Repository navigation
perf: a new transcript no longer rebuilds the corpus (#99) - #112
Merged
Carl723000 merged 3 commits intoSep 22, 2026
Merged
Conversation
…e#99) The content analysis fast path required exactly one changed file (sourcePlans.length === 1 && plans.length === 1). A machine running several agents appends to several transcripts between two refreshes, so that gate never held: every refresh fell back to replaceWithFullAnalysisPlans() and re-read every file. The diagnostics show the shape of it - changed=3 reused=641 alongside bodies=644 and bytes=1.4 GB. The application side already loops over plans, so the gate is what limited it. Allow any number of pure tail appends, and make two ordering guarantees the single-file case never needed: - parse appends in full-scan order, so the first file to carry a UUID owns it exactly as in the full loader (the comparator returned 0 for two appends, leaving discovery order to decide); - attribute each new UUID to the appended file that actually carries it, earliest first, instead of to parsedPlans[0]. The per-file UUID ownership check that forces a rebuild now runs for every appended file rather than only a lone one. Measured on 15 real transcripts (34 MB), three files appended: 15 body reads and 8680 ms became 3 body reads and 586 ms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) A file that had just appeared was classified as an unsafe source mutation, so plans were replaced with a full rebuild of every file. Every new session starts its own transcript, so on a machine running agents this happened many times a day - and it is why the multi-file append fast path rarely helped in practice. A new file is as safe as a tail append: its own body is read in full and the established contributions are untouched. Let such plans through the gate, add the new file's whole analysis as its delta, and recompute ordering state from metadata only when files were actually added, so pure appends still never touch historical files. Ownership is guarded: a new file carrying a UUID somebody already owns falls back to the full rebuild, because the full loader resolves that by scan order. Measured on 15 real transcripts, two appends plus one new file: 16 body reads and 3042 ms became 3 body reads and 567 ms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🤖 Automated first-pass review The configured analysis model did not return a usable analysis for this pull request. The item has still been received and remains available for maintainer review. To help with review, please ensure the change summary, tests, and any user-facing documentation or localization updates are included. 🤖 Posted automatically by CCU Bot — not a maintainer decision. |
Keep the contributor's new-file fast path and both prior performance changes; release tests pass on the combined tree. Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #111 — the branch contains it, so review that one first; this commit is the part that makes the fast path actually reachable in practice.
Problem
After #111 the fast path handles several appends, but on a live machine it still almost never ran. The reason is that every new session starts a new transcript, and a file that had just appeared was classified as an unsafe source mutation:
A new file has no prior, so the gate fails and
replaceWithFullAnalysisPlans()rebuilds every contribution. On my machine, with agents running, this happens many times a day. Measured with the diagnostics after a restart, on a build that already had #111:Reproduced in isolation on a 64-file corpus, frozen clock:
Fix
A new file is as safe as a tail append: its own body is read in full and the established contributions are untouched. Concretely:
addAppendAnalysisDeltaagainst an empty accumulator);orderedFileIds/filePositionByIdonly when files were actually added — that is metadata-only work and no body is re-read, and pure appends keep reusing the previous ordering, so the existing "does not revisit historical file contributions" guarantee is untouched;Measurement
Real transcripts, 15 files / 34 MB, two appended and one new file added:
Tests
Two cases added: a new session file alongside two appends stays at 3 body reads and matches the full loader, and a new file carrying an already-owned UUID falls back while still matching the full loader.
Full
npm teston this branch: 1028 pass, 8 fail, 1 skipped — the same 8 environmental failures (Codex lease/manifest,EPERM: symlink, temp races) that a cleanmainproduces here.One thing worth knowing, unrelated to this change
File identity is
(dev, inode), and NTFS reuses file record numbers. A file created during a test run can land on the identity of one the fixture already indexed, which reads as a move and legitimately forces a full rebuild. This makes the incremental suites flaky on Windows: across four full runs on this machine,one changed file in a reused corpus,default content analysis does not revisit historical file contributionsand others failed onmainas well, in a different combination each time. The new test skips itself when it observeschanged.move > 0rather than asserting a number the filesystem cannot deliver. Happy to open a separate issue with the details if that is useful.