src: use simdutf for UTF-8 ⇄ UTF-16 transcoding in StringDecoder and StringBytes::Write - #65324
Open
codebytere wants to merge 2 commits into
Open
src: use simdutf for UTF-8 ⇄ UTF-16 transcoding in StringDecoder and StringBytes::Write#65324codebytere wants to merge 2 commits into
codebytere wants to merge 2 commits into
Conversation
Collaborator
|
Review requested:
|
StringDecoder used v8::String::NewFromUtf8() for UTF-8, while
Buffer#toString() goes through StringBytes::Encode(), which has
simdutf-backed ASCII, Latin-1 and UTF-16 paths and only falls back
to NewFromUtf8() for input that contains invalid sequences. Route
the decoder through the same function, so streams with
setEncoding('utf8') and readline decode at the same speed as
Buffer#toString(). U+FFFD replacement is unchanged because invalid
input still ends up in NewFromUtf8(), and the ERR_STRING_TOO_LONG
check is kept explicit so over-long input fails as before.
benchmark/string_decoder/string-decoder.js (encoding=utf8) and a
readline-over-pipe workload improve by 2-3x for chunks >= 1 KiB;
64 KiB newline-delimited JSON round trips over child stdio improve
by ~30% on the reading side alone.
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
StringBytes::Write() already used simdutf to encode one-byte strings as UTF-8 but sent every two-byte (UTF-16) string through v8::String::WriteUtf8V2(), which is several times slower. That path is behind Buffer.from(string), buf.write(), fs.write*() with string data and every string written to a libuv stream, and JSON.stringify() output is a two-byte string as soon as any value in the payload is outside Latin-1. Encode two-byte strings with simdutf as well whenever their UTF-8 form is guaranteed to fit in the target: well-formed input is converted directly, and input with unpaired surrogates is converted from a copy passed through simdutf::to_well_formed_utf16(), which replaces each unpaired surrogate with U+FFFD exactly like kReplaceInvalidUtf8 (this mirrors what TextEncoder already does). Writes that have to truncate at a character boundary keep using WriteUtf8V2(), so their output is byte-for-byte unchanged, and so do strings of up to 32 code units, for which V8 is already as fast (the same threshold TextEncoder uses). buf.write() of a 2 KiB two-byte string improves ~5x (astral-heavy and lone-surrogate strings ~3.5x and ~5x), Buffer.from() of a 64 KiB JSON string ~2.7x; one-byte strings are unaffected. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
codebytere
force-pushed
the
perf/src-simdutf-utf8-transcoding
branch
from
August 16, 2026 15:51
ec59ddb to
0c89308
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #65324 +/- ##
==========================================
- Coverage 90.13% 90.11% -0.03%
==========================================
Files 752 752
Lines 251568 251578 +10
Branches 47270 47273 +3
==========================================
- Hits 226759 226706 -53
- Misses 16168 16216 +48
- Partials 8641 8656 +15
🚀 New features to boost your workflow:
|
anonrig
approved these changes
Aug 16, 2026
gurgunday
approved these changes
Aug 16, 2026
mertcanaltin
approved these changes
Aug 16, 2026
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
StringDecodergets 3–12× faster on UTF-8, and writing non-ASCII strings as UTF-8 (Buffer#write,Buffer.from(string),every stream/socket/fs string write) gets 4–5× faster. A 64 KiB JSON-RPC round trip over a child's stdio (stringify → write
→ readline → parse, non-ASCII payload) drops from 1.28 ms to 0.90 ms with only the parent patched, 0.67 ms with both ends.
(Linux x64,
benchmark/compare.js, 30 runs, significance as incompare.R.)Two commits:
string_decoder: decode UTF-8 via StringBytes::Encode- the decoder built strings withString::NewFromUtf8();buffer.toString()already goes throughStringBytes::Encode(), which uses simdutf.MakeString()now calls the samefunction (the
kMaxLength→ERR_STRING_TOO_LONGcheck stays in front), so the decoder returns exactly whattoString()returns for the same bytes. The partial-character bookkeeping is untouched.
src: use simdutf for two-byte strings in UTF-8 writes-StringBytes::Write(UTF8)usedString::WriteUtf8V2()fortwo-byte strings. For strings longer than 32 code units it now validates with
simdutf::validate_utf16()(lone surrogatesare repaired into a scratch buffer with
to_well_formed_utf16(), so the output stays byte-identical to V8'skReplaceInvalidUtf8) and transcodes withconvert_utf16_to_utf8()- but only when the destination is known to fit(
buflen >= 3 × lengthor>= utf8_length_from_utf16()); otherwise it falls back to V8, so partial writes truncate at thesame character boundary as before. Shorter strings and one-byte strings keep the V8 path.
Tests:
test-string-decoder-utf8-large.js(new: large inputs, chunk boundaries inside multi-byte sequences, invalidsequences vs
toString());test-buffer-write-utf8-two-byte.js(new: independent reference encoder; BMP/astral/lonesurrogates at start/middle/end; exact-fit, one-short and 3× destinations; partial writes). Existing string_decoder, buffer,
stream, net, http, child_process and readline suites pass.
Disclosure: the code, test, measurements and this description were written by Claude Code, directed and reviewed by @codebytere.