Skip to content

Add experimental multi-threaded ondemand stream parsing [research] - #2788

Open
lemire wants to merge 2 commits into
masterfrom
parallel-ondemand-stream
Open

Add experimental multi-threaded ondemand stream parsing [research]#2788
lemire wants to merge 2 commits into
masterfrom
parallel-ondemand-stream

Conversation

@lemire

@lemire lemire commented Jul 25, 2026

Copy link
Copy Markdown
Member

Introduce simdjson::experimental::parse_many_parallel, which scales the stage-2 (value materialization) step of iterate_many across many threads. A single dispatcher thread carves the input into document-aligned slices and feeds a pool of worker threads through a bounded queue. Each worker owns its own ondemand::parser and its own output vector, so no locking happens on the hot path; extracted values are returned as one vector per worker ('shards').

Two stream formats are supported, each sliced at a delimiter byte that cannot occur inside a JSON value, so a document is never split and boundaries are found with a single memchr:

  • whitespace_delimited (ndjson): cut just after a newline
  • json_sequence (RFC 7464): cut just before a record separator (0x1E) Because the dispatcher stays cheap, throughput scales with the parser threads (compute bound) or up to memory bandwidth (light workloads).

This is EXPERIMENTAL for now. The goal is to scale to a high number of threads when processing large inputs.

cc @jaja360

@lemire
lemire force-pushed the parallel-ondemand-stream branch from 1aee6e1 to b73005e Compare July 29, 2026 00:49
lemire added 2 commits July 28, 2026 20:50
Introduce simdjson::experimental::parse_many_parallel, which scales the
stage-2 (value materialization) step of iterate_many across many threads.
A single dispatcher thread carves the input into document-aligned slices and
feeds a pool of worker threads through a bounded queue. Each worker owns its
own ondemand::parser and its own output vector, so no locking happens on the
hot path; extracted values are returned as one vector per worker ('shards').

Two stream formats are supported, each sliced at a delimiter byte that cannot
occur inside a JSON value, so a document is never split and boundaries are
found with a single memchr:
 - whitespace_delimited (ndjson): cut just after a newline
 - json_sequence (RFC 7464): cut just before a record separator (0x1E)
Because the dispatcher stays cheap, throughput scales with the parser threads
(compute bound) or up to memory bandwidth (light workloads).

This targets the case the built-in two-thread iterate_many cannot help with:
when stage 2 dominates, overlapping stage 1 of the next batch only buys a
small constant factor, whereas running many parser threads scales with cores.

Adds benchmark/bench_parallel_stream.cpp comparing serial, built-in
two-thread, and parallel(N) over the amazon_cellphones corpus and a
number-heavy synthetic set, plus a format-coverage check that confirms the
parallel document count and checksum match the serial parse for each format.
Header-only; falls back to single-threaded when SIMDJSON_THREADS_ENABLED is
not defined.
Slicing needs no state from the preceding slice: from any offset, the next
boundary is the next delimiter. A dedicated dispatcher thread therefore knows
nothing a worker could not compute itself, so each worker now claims a byte
range from a shared atomic counter and snaps both ends forward. Worker i ends
at snap(raw + slice_bytes) and worker i+1 begins at snap of the same value, so
the slices remain contiguous and never split a document; when two claims land
inside one long document the earlier one covers it and the later slice is
empty.

This removes the bounded blocking queue, its mutex and two condition
variables, and the dispatcher thread itself, and makes options.threads mean
exactly the number of threads spawned rather than one fewer than the number
created. The default is correspondingly hardware_concurrency() rather than
hardware_concurrency() - 1.

Results are unchanged, verified against the previous implementation on
newline-delimited corpora including deeply nested and multi-hundred-kilobyte
documents.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant