Add experimental multi-threaded ondemand stream parsing [research] - #2788
Open
lemire wants to merge 2 commits into
Open
Add experimental multi-threaded ondemand stream parsing [research]#2788lemire wants to merge 2 commits into
lemire wants to merge 2 commits into
Conversation
lemire
force-pushed
the
parallel-ondemand-stream
branch
from
July 29, 2026 00:49
1aee6e1 to
b73005e
Compare
Introduce simdjson::experimental::parse_many_parallel, which scales the
stage-2 (value materialization) step of iterate_many across many threads.
A single dispatcher thread carves the input into document-aligned slices and
feeds a pool of worker threads through a bounded queue. Each worker owns its
own ondemand::parser and its own output vector, so no locking happens on the
hot path; extracted values are returned as one vector per worker ('shards').
Two stream formats are supported, each sliced at a delimiter byte that cannot
occur inside a JSON value, so a document is never split and boundaries are
found with a single memchr:
- whitespace_delimited (ndjson): cut just after a newline
- json_sequence (RFC 7464): cut just before a record separator (0x1E)
Because the dispatcher stays cheap, throughput scales with the parser threads
(compute bound) or up to memory bandwidth (light workloads).
This targets the case the built-in two-thread iterate_many cannot help with:
when stage 2 dominates, overlapping stage 1 of the next batch only buys a
small constant factor, whereas running many parser threads scales with cores.
Adds benchmark/bench_parallel_stream.cpp comparing serial, built-in
two-thread, and parallel(N) over the amazon_cellphones corpus and a
number-heavy synthetic set, plus a format-coverage check that confirms the
parallel document count and checksum match the serial parse for each format.
Header-only; falls back to single-threaded when SIMDJSON_THREADS_ENABLED is
not defined.
Slicing needs no state from the preceding slice: from any offset, the next boundary is the next delimiter. A dedicated dispatcher thread therefore knows nothing a worker could not compute itself, so each worker now claims a byte range from a shared atomic counter and snaps both ends forward. Worker i ends at snap(raw + slice_bytes) and worker i+1 begins at snap of the same value, so the slices remain contiguous and never split a document; when two claims land inside one long document the earlier one covers it and the later slice is empty. This removes the bounded blocking queue, its mutex and two condition variables, and the dispatcher thread itself, and makes options.threads mean exactly the number of threads spawned rather than one fewer than the number created. The default is correspondingly hardware_concurrency() rather than hardware_concurrency() - 1. Results are unchanged, verified against the previous implementation on newline-delimited corpora including deeply nested and multi-hundred-kilobyte documents.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce simdjson::experimental::parse_many_parallel, which scales the stage-2 (value materialization) step of iterate_many across many threads. A single dispatcher thread carves the input into document-aligned slices and feeds a pool of worker threads through a bounded queue. Each worker owns its own ondemand::parser and its own output vector, so no locking happens on the hot path; extracted values are returned as one vector per worker ('shards').
Two stream formats are supported, each sliced at a delimiter byte that cannot occur inside a JSON value, so a document is never split and boundaries are found with a single memchr:
This is EXPERIMENTAL for now. The goal is to scale to a high number of threads when processing large inputs.
cc @jaja360