Perfetto trace loader doesn't support "FEXTRA" multi-block gzip files. How to reproduce:
- Install
https://github.com/vinlyx/mgzip
- Take any exiting JSON trace
- Code
with open("trace.json") as fin:
with mgzip.open("trace.json.gz", "wt", thread=8, blocksize=1 << 16) as fout:
while buffer := fin.read(1 << 16):
fout.write(buffer)
- For example,
../trace_processor --httpd trace.json.gz
JSON trace file is incomplete
- This will work:
gzip -d trace.json.gz
gzip trace.json
../trace_processor --httpd trace.json.gz
Why does this weird gzip format property matter to me? We, 100B-parameter base LLM trainers in PyTorch, deal with a few hundred megs of profile that require considerable time to compress every few minutes, so compressing them on 192 available CPU cores gives a considerable benefit.
Perfetto trace loader doesn't support "FEXTRA" multi-block gzip files. How to reproduce:
https://github.com/vinlyx/mgzipWhy does this weird gzip format property matter to me? We, 100B-parameter base LLM trainers in PyTorch, deal with a few hundred megs of profile that require considerable time to compress every few minutes, so compressing them on 192 available CPU cores gives a considerable benefit.