I have a potentially infinite array of objects I want to parse:
I do not really care about the outer array, but am much rather looking for the fastest way to stream the objects contained in the array.
The popular command-line tool jq supports a way to "strip" layers of JSON while streaming the output without requiring the outer array's structure to be fully available, i.e., it continues to work even if the end of the array never arrives.
jq -ercn --stream 'fromstream(1|truncate_stream(inputs))'
I am looking to do something just like this in my own application, which uses simdjson's iterate_many API. My (perhaps naïve) thinking is that the API likely needs an additional parameter that specifies how many layers to strip away.
I have a potentially infinite array of objects I want to parse:
I do not really care about the outer array, but am much rather looking for the fastest way to stream the objects contained in the array.
The popular command-line tool
jqsupports a way to "strip" layers of JSON while streaming the output without requiring the outer array's structure to be fully available, i.e., it continues to work even if the end of the array never arrives.jq -ercn --stream 'fromstream(1|truncate_stream(inputs))'I am looking to do something just like this in my own application, which uses simdjson's
iterate_manyAPI. My (perhaps naïve) thinking is that the API likely needs an additional parameter that specifies how many layers to strip away.