Repository navigation
perf: keep multi-file appends on the incremental path (#99) - #111
Merged
Carl723000 merged 3 commits intoSep 22, 2026
Merged
Conversation
…e#99) The content analysis fast path required exactly one changed file (sourcePlans.length === 1 && plans.length === 1). A machine running several agents appends to several transcripts between two refreshes, so that gate never held: every refresh fell back to replaceWithFullAnalysisPlans() and re-read every file. The diagnostics show the shape of it - changed=3 reused=641 alongside bodies=644 and bytes=1.4 GB. The application side already loops over plans, so the gate is what limited it. Allow any number of pure tail appends, and make two ordering guarantees the single-file case never needed: - parse appends in full-scan order, so the first file to carry a UUID owns it exactly as in the full loader (the comparator returned 0 for two appends, leaving discovery order to decide); - attribute each new UUID to the appended file that actually carries it, earliest first, instead of to parsedPlans[0]. The per-file UUID ownership check that forces a rebuild now runs for every appended file rather than only a lone one. Measured on 15 real transcripts (34 MB), three files appended: 15 body reads and 8680 ms became 3 body reads and 586 ms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🤖 Automated first-pass review The configured analysis model did not return a usable analysis for this pull request. The item has still been received and remains available for maintainer review. To help with review, please ensure the change summary, tests, and any user-facing documentation or localization updates are included. 🤖 Posted automatically by CCU Bot — not a maintainer decision. |
Preserve both contributor changelog entries while bringing ClaudeCodeUsage#111 onto the latest tested main. Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>
This was referenced Sep 22, 2026
This was referenced Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second half of the CPU path in #99, on top of #109 (independent — this one stands alone, but the profile in #109 is what pointed here).
Problem
With #109 merged, the extension host on this machine still held a core. The profile moved rather than cleared: 74%
addToBucket, 25%analyzeLine. Not a parsing cost — a volume one. The diagnostics say it plainly:Three files changed; 644 bodies re-read, 1.4 GB, 56–86 s per refresh. With
startup,focus,watchandpollall triggering, refreshes never finish before the next one is due.The cause is the content-analysis gate:
Exactly one changed file. A machine running several agents appends to several transcripts between two refreshes, so the gate never holds and
replaceWithFullAnalysisPlans()turns all 644 files into rebuild plans. One user with one session gets the fast path; anyone running concurrent sessions gets a full rebuild forever.Fix
The application side already loops over
parsedPlans, so only the gate limited this. Allowing N pure tail appends needs two ordering guarantees the single-file case never exercised:0for two appends (if (left.kind === 'append') return 0), leaving discovery order to decide — which is exactly what the existing testconcurrent appends resolve new content UUID owners in full-scan file orderforbids. That test failed with the gate opened and passes once the comparator orders appends by timestamp.parsedPlans[0].fileId— correct while only one plan could ever reach that code, wrong as soon as two can.The per-file UUID ownership check that forces a rebuild now runs for every appended file instead of only a lone one.
Measurement
Real transcripts, 15 files / 34 MB, three of them appended (frozen clock so the analysis cutoff does not move between passes):
Tests
New case
concurrent appends to several files stay incremental instead of forcing a full rebuild: 64 files, three appended, assertsbodyReads === 3, exact appended byte count, anddeepEqualagainstClaudeDataLoader.loadUsageRecords. It fails onmainwithactual: 64, expected: 3.claudeIncrementalIndexsuite: 39/40, the one failure (content analysis is materialized from per-file contributions without a second body scan) fails identically on a cleanmainhere.Full
npm test: 1025 pass on this branch and 1025 onmain. The 8–9 failures are Windows filesystem cases (EPERM: symlink, temp rename/rmdir races) in the Codex lease/manifest suites, and their exact set shifts between runs on the same commit — two runs of this branch failed different members of that group.