I see constantly growing memory with rapidjson.load comparing to standard json.load
I unpacked about 10k gzipped json files like this
for file_name in os.listdir(path):
with open(os.path.join(path, file_name), 'rb') as f:
content = f.read()
with BytesIO(contents) as file_object:
with gzip.GzipFile(fileobj=file_object, filename='objects.json') as archive:
return rapidjson.load(archive)
Memory consumption of rapidjson
With json lib memory is almost constant.
I see constantly growing memory with
rapidjson.loadcomparing to standardjson.loadI unpacked about 10k gzipped json files like this
Memory consumption of rapidjson
With json lib memory is almost constant.