Repository navigation
perf: stat manifest files concurrently (#99) - #129
Conversation
|
A note on the failing Nothing in the diff is involved — Node Tests, UI Test and Release Package are green. Since the bot runs on |
|
🤖 Automated first-pass review TL;DR The change is small, well-scoped, and consistent with the file's existing concurrency helper; correctness looks sound. Only one real review point: the CHANGELOG classifies a performance change under Analysis
Suggested next step(s)
🤖 Generated by DeepSeek as an automated first pass — not a maintainer decision. |
|
Addressing the first-pass review:
|
Carl723000
left a comment
There was a problem hiding this comment.
Thanks @rsyuzyov for this focused optimization and the follow-up clarifications. The concurrency preserves discovery order, map insertion order, fingerprint fields and ENOENT index gaps, including deliberately out-of-order completions with an eight-worker ceiling. It changes Claude's scanner, not the separate Codex compressed-log discovery path.
One error-path regression needs attention before merging: when a stat rejects with a non-ENOENT error, Promise.all rejects the scan but surviving mapWithConcurrency workers keep dequeuing files. A 40-file synthetic fixture with the first stat returning EACCES made 1 call on the sequential baseline versus 40 calls on this head, including 32 started after rejection reached the caller. This is more than merely allowing already-in-flight calls to finish. The rest of a failed history scan can continue in the background.
Please stop scheduling queued work after the first fatal error (already-in-flight I/O may settle), and add a deterministic regression proving that bound. Controlled completion order and an explicit eight-worker maximum would also be stronger than the 500 ms wall-clock assertion, which can still fail on a stalled runner.
The generated merge against released main places the changelog entry inside 2.5.0 (merge snapshot); please move it under Unreleased. This optimization is not in the published 2.5.0 release.
Reviewed head: 0c091c89f6aa6905a2e922cdeb1cee9a51f55815. The ordering/error reproductions and focused manifest suites were independently rerun; this is not a new native Windows performance benchmark. Please keep the changes in this original PR so the contribution and authorship stay intact.
🤖 Generated by OpenAI Codex as an automated first pass — not a maintainer decision.
scanUsageManifest awaited one fs.stat per transcript in sequence. Each stat is a round trip through the file-system filter stack, so on a busy Windows workstation a 976-file history cost 0.5-3.3 s of every refresh tick just to learn that nothing changed; with live transcripts appended between ticks that scan ran dozens of times a day. The stats now fan out through the existing mapWithConcurrency (limit 8, as the timestamp probe already uses), keeping results in discovery order. A quiet bench scan of the same history goes from ~200 ms to ~110 ms; the win grows with contention, where the sequential loop hurt most. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…udeCodeUsage#99) Co-Authored-By: Claude Code <noreply@anthropic.com>
A non-ENOENT stat rejected the scan but the remaining workers kept dequeuing the rest of the history. Already-in-flight calls may settle; nothing further starts. The overlap test now proves the eight-worker ceiling by completion order instead of a wall-clock bound. Co-Authored-By: Claude Code <noreply@anthropic.com>
2.5.0 is published. The generated merge had folded this entry into that section; it is not in the released build. Co-Authored-By: Claude Code <noreply@anthropic.com>
0c091c8 to
d5ea3a9
Compare
|
Addressing the review of
|
Carl723000
left a comment
There was a problem hiding this comment.
The follow-up at d5ea3a95ce850fa727221f4f9617262c84b5caa0 addresses the previously reported fatal-error scheduling issue. Independent synthetic checks confirm that no new stats are dequeued after rejection, already-in-flight calls may settle, the eight-worker ceiling is retained, and results/fingerprints preserve discovery order, including ENOENT gaps. Falsy/non-Error rejection values also stop scheduling.
The required Node Tests, UI Test and Release Package checks are green at this exact head. No remaining blocking finding in the reviewed scope; approving to supersede my earlier changes-requested review. This is not a new native Windows benchmark or a claim that all of #99 is resolved.
🤖 Generated by OpenAI Codex as an automated first pass — not a maintainer decision.
Resolve only the Unreleased CHANGELOG conflict, retaining both the deletion and concurrent-stat entries. Production source and tests remain exactly those from the original PR and the already-merged ClaudeCodeUsage#129 contribution. Validation: full Node/policy suite (2657 passed, 2 skipped); synthetic mixed-event, ownership-order and prior-snapshot checks against full parsing. Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>
Problem
scanUsageManifestawaited onefs.statper transcript in a sequential loop, on every refresh tick. Each stat is a round trip through the file-system filter stack (antivirus included), so on a busy Windows workstation a 976-file history cost 0.5–3.3 s of every tick just to learn whether anything changed — and with live transcripts appended between ticks, that scan runs dozens of times a day. It was the largest remaining per-pass cost once #119–#123 removed the rebuild storms.Change
The stat loop fans out through the existing
mapWithConcurrencyhelper (limit 8, same as the earliest-timestamp probe in this file). Results are reassembled in discovery order, so fingerprints, tie-breaking and diagnostics are unchanged. The disappearance-race handling (ENOENTtolerated, everything else thrown) is kept per file.Measured
976-file history, quiet bench: ~200 ms → ~110 ms per scan. The win grows under contention, where the sequential loop hurt most (live log:
manifest=2850–3265msticks on a loaded machine).Tests
manifest scanning stats files concurrently and keeps discovery order— a delayed-stat fake proves the stats overlap, finish far under the sequential time, and the manifest keeps discovery order.🤖 Generated with Claude Code