fs: read small files in one thread pool round trip - #65327
Open
codebytere wants to merge 1 commit into
Open
Conversation
fs.readFile(path) took four libuv thread pool round trips for a typical small file -- open, fstat, read and close, each its own uv_fs request with its own queue wait, completion callback and JS/C++ crossing -- and fs.promises.readFile(path) did the same through a FileHandle. For the small files applications read most, the round trips are the cost, and each occupies a slot in the pool that concurrent dns.lookup(), fs and crypto work is also queueing for. Add ReadFileJob (an AsyncWrap + ThreadPoolWork) that performs open + fstat + read-to-EOF + close as one pool task and reports the whole content, or, when the file turns out to be larger than one chunk (kReadFileBufferLength, 512 KiB), stops after fstat() and hands the fd and size back so that the existing chunked reader continues unchanged (large reads stay interleaved and abortable between chunks, and still save the fstat round trip). fs.readFile() and fs.promises.readFile() use it for path arguments without a user buffer; file descriptors, FileHandles, options.buffer and an active VFS keep their paths. Behavior is otherwise kept: same bytes for every size and encoding; open failures report syscall 'open' with the path, read failures 'read'; permission errors are delivered through the callback/promise as before; an abort that arrives while the read is in flight still wins; the job is an FSREQCALLBACK resource for async_hooks; a handed back fd is tracked exactly like one from a plain open(). Tests that asserted the internal open/fstat/read/close request chain, used readFile() as a proxy for an fstat trace event, or injected faults through FileHandle.prototype for path-based reads are adjusted to keep testing what they test (a file just over one chunk where the chain shape matters, fs.fstat() for the fstat trace, a larger file so the FileHandle path is taken). fs.readFile() of 4 KiB files at concurrency 64 goes from ~51k to ~306k files per second, and a mixed stat/readFile/dns.lookup burst from ~66k to ~312k operations per second. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
codebytere
force-pushed
the
perf/fs-readfile-one-roundtrip
branch
from
August 16, 2026 16:54
0eaf058 to
21db0ab
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fs.readFile()/fs.promises.readFile()of small files get 3–6× faster, and use one libuv thread-pool task instead offour, by doing open + fstat + read + close in a single round trip.
(Linux x64,
--set duration=2, 30 runs.)Today a path-based
readFileissuesopen,fstat,readandcloseas four separateuv_fs_*requests, each withits own queue wait, completion callback and JS↔C++ crossing; the promise API does the same through a
FileHandle. Forsmall files those round trips are the whole cost, and each one takes a pool slot away from concurrent dns/zlib/crypto/fs
work.
ReadFileJob(anAsyncWrap+ThreadPoolWork, providerFSREQCALLBACK) runs open + fstat + read-to-EOF + close as onetask and returns the content. If the file is larger than one chunk (
kReadFileBufferLength, 512 KiB) it stops afterfstatand hands back the fd and size, and the existing chunked reader continues exactly as today (interleaved,abortable between chunks). Both
readFiles use it for path arguments without a user buffer; fds andFileHandles areunchanged.
Preserved on purpose: identical results for every size/encoding;
openerrors reportsyscall: 'open'+path, readerrors
'read'; permission errors arrive through the callback/promise; an abort that lands while the round trip is inflight still wins; the handed-back fd is tracked and closed like any other; size-0 files (procfs) are read to EOF.
One open point: 16–32 MiB reads via
fs.promises.readFile(…, 'utf-8')at concurrency 10 measure −2…3 % (***),reproducibly; the same sizes as Buffers, via the callback API, or at concurrency 1 are flat. They take the hand-back path
with identical syscalls, and direct timing shows ≤2 %, so I haven't pinned it down. If preferred, the promise API can keep
its current path and only the callback API changes.
Tests:
test-fs-readfile-one-roundtrip.js(new; also passes on currentmain): sizes across the 512 KiB threshold,encodings, flags, error shapes, abort before/during, fd/
FileHandleinputs, no fd leak on hand-back, procfs/sysfs,async_hooks lifecycle. Adjusted to keep testing what they test:
test/async-hooks/test-fsreqcallback-readFile.js(acceptsone request instead of exactly four),
test-graph.fsreq-readFile.js(reads a 512 KiB+1 file so the four-requestchain keeps its shape),
test-async-exec-resource-match.js(resource + ≥1 fs request),test-trace-events-fs-async.js(uses
fs.fstat()for the fstat trace instead ofreadFileas a proxy),test-fs-promises-readfile.js(zero-size-liarcase goes through a
FileHandle),test-fs-promises-file-handle-{op,aggregate,close}-errors.js(use a >512 KiB fileso the patched
FileHandlepath is taken). fs, async-hooks, permission, worker, process and child_process suites pass.Disclosure: the code, test, measurements and this description were written by Claude Code, directed and reviewed by @codebytere.