Description

This PR adds support for two additional JSON streaming formats via the existing parse_many and iterate_many APIs:

1. RFC 7464 JSON text sequences (application/json-seq) - Documents prefixed with RS (0x1E) character
2. Comma-delimited documents - Documents separated by commas (e.g., {"a":1},{"b":2})

RFC 7464 was previously entirely unsupported, whereas Comma-delimited documents
were supported but only in single-batch mode without threading (via the
now-deprecated allow_comma_separated parameter). This PR implements both
formats with proper multi-batch processing and threading support.

Related issues:

• RFC 7464 support? #2032 (RFC 7464 support request)
• Enhance iterate_many to enable parsing of comma separated documents #1999 (original comma-separated feature request)
• Support stripping away layers in iterate_many #2252 (this PR doesn't solve this issue, but makes it simpler to solve in the future)

Type of change

• Bug fix
• Optimization
• New feature
• Refactor / cleanup
• Documentation / tests
• Other (please describe):

How to verify / test

New test coverage

• tests/dom/document_stream_tests.cpp: Added rfc7464_tests() and comma_delimited_tests()
• tests/ondemand/ondemand_document_stream_tests.cpp: Equivalent tests for On-Demand API

Running the new tests

cmake -B build -D SIMDJSON_DEVELOPER_MODE=ON
cmake --build build -j$(nproc)

# Run the document stream tests (includes both RFC 7464 and comma-delimited tests)
ctest --test-dir build --output-on-failure -R 'document_stream_tests'

Performance verification

cmake --build build --target bench_stream_formats
./build/benchmark/bench_stream_formats --benchmark_repetitions=10

Benchmark results (AMD Zen 5, Ryzen 9 9900X)

Performance overview:

• NDJSON: No regression observed. Performance variations are within measurement noise.
• RFC 7464: Slightly slower than NDJSON, due to the additional post-stage1
  filtering. Since this format is currently unsupported, I don't think that's
  a concern. We can optimize further by integrating the RS handling into
  stage1 (instead of the small post-processing in this PR) in the future if needed.
• Comma-delimited: Like RFC 7464 format, comma-delimited is slightly slower
  than NDJSON due to the post-stage1 filtering. However, this is outweighted by
  the speedup we get from the threaded support (which we didn't have in the old
  single-batch implementation).

1. NDJSON regression test (baseline vs final)

2. Stream format comparison

3. Comma-delimited: old API vs new API

Implementation notes

API additions

// New enum in simdjson/base.h
enum class stream_format {
  whitespace_delimited,  // NDJSON, whitespace-separated (default)
  json_sequence,         // RFC 7464: RS-delimited
  comma_delimited        // Comma-separated documents
};

// DOM API
simdjson_result<document_stream> parser.parse_many(
    padded_string_view input,
    size_t batch_size,
    stream_format format);

// On-Demand API
simdjson_result<document_stream> parser.iterate_many(
    padded_string_view input,
    size_t batch_size,
    stream_format format);

Deprecation

The allow_comma_separated boolean parameter is deprecated in favor of stream_format::comma_delimited:

// Old (deprecated)
parser.iterate_many(input, batch_size, true);

// New (recommended)
parser.iterate_many(input, batch_size, stream_format::comma_delimited);

Checklist before submitting

• I added/updated tests covering my change (if applicable)
• Code builds locally and passes all tests
• Documentation / README updated if needed
• Commits are atomic and messages are clear
• I linked the related issue (if applicable)
• Benchmark results show no regression for existing functionality