Skip to content

perf: stat manifest files concurrently (#99) - #129

Merged
Carl723000 merged 5 commits into
ClaudeCodeUsage:mainfrom
rsyuzyov:perf/parallel-manifest-stat
Oct 16, 2026
Merged

Carl723000 merged 5 commits into
ClaudeCodeUsage:mainfrom
rsyuzyov:perf/parallel-manifest-stat

Conversation

@rsyuzyov

@rsyuzyov rsyuzyov commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Problem

scanUsageManifest awaited one fs.stat per transcript in a sequential loop, on every refresh tick. Each stat is a round trip through the file-system filter stack (antivirus included), so on a busy Windows workstation a 976-file history cost 0.5–3.3 s of every tick just to learn whether anything changed — and with live transcripts appended between ticks, that scan runs dozens of times a day. It was the largest remaining per-pass cost once #119–#123 removed the rebuild storms.

Change

The stat loop fans out through the existing mapWithConcurrency helper (limit 8, same as the earliest-timestamp probe in this file). Results are reassembled in discovery order, so fingerprints, tie-breaking and diagnostics are unchanged. The disappearance-race handling (ENOENT tolerated, everything else thrown) is kept per file.

Measured

976-file history, quiet bench: ~200 ms → ~110 ms per scan. The win grows under contention, where the sequential loop hurt most (live log: manifest=2850–3265ms ticks on a loaded machine).

Tests

  • New: manifest scanning stats files concurrently and keeps discovery order — a delayed-stat fake proves the stats overlap, finish far under the sequential time, and the manifest keeps discovery order.
  • Existing manifest and index suites pass unchanged.

🤖 Generated with Claude Code

@rsyuzyov

rsyuzyov commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

A note on the failing review check: it did not get as far as reviewing. The job log shows First-pass formatting failed: Trusted first-pass attribution invariant failed — the formatter counts generated with|by markers in the final comment and found more than its own footer. The likely trigger: the first-pass reply quoted or mentioned this PR body's own Generated with … footer mid-line, where sanitizeUntrustedBody's line-start anchor does not strip it.

Nothing in the diff is involved — Node Tests, UI Test and Release Package are green. Since the bot runs on opened only, a retry needs a workflow run arranged on your side; happy to reword the PR body if that helps the next pass.

@github-actions

Copy link
Copy Markdown

🤖 Automated first-pass review

TL;DR

The change is small, well-scoped, and consistent with the file's existing concurrency helper; correctness looks sound. Only one real review point: the CHANGELOG classifies a performance change under ### Fixed, and the new test asserts overlap with a timing threshold that may be flaky on a loaded CI machine.

Analysis

  • Correctness / semantics preserved: mapWithConcurrency maps results back in discovery order, and the post-loop for (const fingerprint of fingerprints) if (fingerprint) entries.set(...) preserves the previous insertion order and the per-file ENOENT tolerance (return null) while still rethrowing everything else. The hasStableIdentity logic is unchanged. No issue found.
  • Concurrency limit 8 matches the earliest-timestamp probe mentioned in the description, so no new tuning constant is introduced.
  • Test quality: maxInFlight > 1 is a good structural proof of overlap. The second assertion (< 20 * 50 ms wall clock) is a real-time threshold; on a heavily contended CI runner it can be marginal even though it would pass locally. Consider asserting a much looser bound (e.g. clearly less than the sequential sum with a wide margin) or deriving the bound from the observed overlap so the test proves behavior without a wall-clock race. Also confirm the fake stat wrapper does not hide the real ENOENT path — the existing "fails closed on permission errors but tolerates stat ENOENT" test still covers it, so this is only a note.
  • CHANGELOG: the entry is placed under ### Fixed in [Unreleased]. It is a performance change, not a defect fix; ### Changed (or an explicit performance subsection) is more accurate under Keep a Changelog. This is a convention nit, not a blocker — the entry text itself is fine and correctly notes discovery order is preserved.
  • No src/i18n.ts impact: this is host-side scanning, no user-facing strings were added or changed, so the eight-locale requirement does not apply.
  • No architecture/module ownership change, so no ARCHITECTURE.md or Chinese-companion update is needed for this diff.
  • PR title uses the perf: prefix while the branch/release intent is unclear; since release labeling is maintainer-applied, this is fine.

Suggested next step(s)

  1. Move the CHANGELOG entry to ### Changed, or add a short performance note there, keeping the existing wording and the discovery-order guarantee.
  2. Relax the wall-clock assertion in the new test (or replace it with an overlap-derived bound) so it cannot flake on a loaded CI host; keep maxInFlight > 1 as the primary structural assertion.
  3. Confirm scanUsageManifest's callers do not rely on stats being observed in a strictly serialized order (e.g. logging or cancellation mid-loop); if they do, note it in the PR description.

🤖 Generated by DeepSeek as an automated first pass — not a maintainer decision.

@rsyuzyov

Copy link
Copy Markdown
Contributor Author

Addressing the first-pass review:

  1. CHANGELOG — moved the entry from ### Fixed to ### Changed, wording unchanged. (Earlier entries in this performance series went under Fixed as defect-repairs; happy to move them too if the maintainer prefers the stricter Keep-a-Changelog reading across the board.)
  2. Timing assertion — maxInFlight > 1 stays the structural proof; the wall-clock bound is now half the sequential sum with a ~3× margin over a fanned-out run, so a loaded CI host cannot flake it.
  3. Caller order assumptions — checked: scanUsageManifest's only outputs are the entries map (reassembled by index, so insertion order equals discovery order exactly as before) and the two timing fields. Nothing logs or cancels mid-loop; the only behavioral difference is which stats are in flight when a non-ENOENT error throws, and the observable outcome — the scan fails — is unchanged.

@Carl723000 Carl723000 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rsyuzyov for this focused optimization and the follow-up clarifications. The concurrency preserves discovery order, map insertion order, fingerprint fields and ENOENT index gaps, including deliberately out-of-order completions with an eight-worker ceiling. It changes Claude's scanner, not the separate Codex compressed-log discovery path.

One error-path regression needs attention before merging: when a stat rejects with a non-ENOENT error, Promise.all rejects the scan but surviving mapWithConcurrency workers keep dequeuing files. A 40-file synthetic fixture with the first stat returning EACCES made 1 call on the sequential baseline versus 40 calls on this head, including 32 started after rejection reached the caller. This is more than merely allowing already-in-flight calls to finish. The rest of a failed history scan can continue in the background.

Please stop scheduling queued work after the first fatal error (already-in-flight I/O may settle), and add a deterministic regression proving that bound. Controlled completion order and an explicit eight-worker maximum would also be stronger than the 500 ms wall-clock assertion, which can still fail on a stalled runner.

The generated merge against released main places the changelog entry inside 2.5.0 (merge snapshot); please move it under Unreleased. This optimization is not in the published 2.5.0 release.

Reviewed head: 0c091c89f6aa6905a2e922cdeb1cee9a51f55815. The ordering/error reproductions and focused manifest suites were independently rerun; this is not a new native Windows performance benchmark. Please keep the changes in this original PR so the contribution and authorship stay intact.


🤖 Generated by OpenAI Codex as an automated first pass — not a maintainer decision.

rsyuzyov and others added 5 commits October 15, 2026 22:00
scanUsageManifest awaited one fs.stat per transcript in sequence. Each stat is
a round trip through the file-system filter stack, so on a busy Windows
workstation a 976-file history cost 0.5-3.3 s of every refresh tick just to
learn that nothing changed; with live transcripts appended between ticks that
scan ran dozens of times a day. The stats now fan out through the existing
mapWithConcurrency (limit 8, as the timestamp probe already uses), keeping
results in discovery order. A quiet bench scan of the same history goes from
~200 ms to ~110 ms; the win grows with contention, where the sequential loop
hurt most.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A non-ENOENT stat rejected the scan but the remaining workers kept
dequeuing the rest of the history. Already-in-flight calls may settle;
nothing further starts. The overlap test now proves the eight-worker
ceiling by completion order instead of a wall-clock bound.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2.5.0 is published. The generated merge had folded this entry into that
section; it is not in the released build.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@rsyuzyov
rsyuzyov force-pushed the perf/parallel-manifest-stat branch from 0c091c8 to d5ea3a9 Compare October 15, 2026 17:00
@rsyuzyov

Copy link
Copy Markdown
Contributor Author

Addressing the review of 0c091c8, rebased onto current main:

  1. Fatal stat no longer keeps scanning. mapWithConcurrency stops dequeuing after the first rejection. Calls already in flight may settle; nothing further starts. Test: a fatal stat stops scheduling files that have not started — 40 files, the first stat returns EACCES, at most the eight-worker ceiling starts, and a turn after the rejection starts nothing more.

  2. No wall-clock bound. The overlap test holds every stat, waits until eight are in flight, releases one and asserts exactly one replacement under the same ceiling, then completes in reverse of start order and checks discovery order.

  3. Changelog. The note is under Unreleased, not the published 2.5.0 section.

@Carl723000 Carl723000 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The follow-up at d5ea3a95ce850fa727221f4f9617262c84b5caa0 addresses the previously reported fatal-error scheduling issue. Independent synthetic checks confirm that no new stats are dequeued after rejection, already-in-flight calls may settle, the eight-worker ceiling is retained, and results/fingerprints preserve discovery order, including ENOENT gaps. Falsy/non-Error rejection values also stop scheduling.

The required Node Tests, UI Test and Release Package checks are green at this exact head. No remaining blocking finding in the reviewed scope; approving to supersede my earlier changes-requested review. This is not a new native Windows benchmark or a claim that all of #99 is resolved.


🤖 Generated by OpenAI Codex as an automated first pass — not a maintainer decision.

@Carl723000
Carl723000 merged commit cbf04bf into ClaudeCodeUsage:main Oct 16, 2026
8 checks passed
Carl723000 added a commit to rsyuzyov/ClaudeCodeUsage that referenced this pull request Oct 16, 2026
Resolve only the Unreleased CHANGELOG conflict, retaining both the deletion and concurrent-stat entries. Production source and tests remain exactly those from the original PR and the already-merged ClaudeCodeUsage#129 contribution.

Validation: full Node/policy suite (2657 passed, 2 skipped); synthetic mixed-event, ownership-order and prior-snapshot checks against full parsing.

Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants