Skip to content

perf: a new transcript no longer rebuilds the corpus (#99) - #112

Merged
Carl723000 merged 3 commits into
ClaudeCodeUsage:mainfrom
rsyuzyov:perf/new-file-incremental
Sep 22, 2026
Merged

Carl723000 merged 3 commits into
ClaudeCodeUsage:mainfrom
rsyuzyov:perf/new-file-incremental

Conversation

@rsyuzyov

Copy link
Copy Markdown
Contributor

Builds on #111 — the branch contains it, so review that one first; this commit is the part that makes the fast path actually reachable in practice.

Problem

After #111 the fast path handles several appends, but on a live machine it still almost never ran. The reason is that every new session starts a new transcript, and a file that had just appeared was classified as an unsafe source mutation:

const appendOnlyPlans = ... plans.every((plan) => plan.kind === 'append' && plan.prior && ...);

A new file has no prior, so the gate fails and replaceWithFullAnalysisPlans() rebuilds every contribution. On my machine, with agents running, this happens many times a day. Measured with the diagnostics after a restart, on a build that already had #111:

files(discovered=647 changed=6 reused=641)
incremental(bodies=647)          ← still everything
ms(read-parse=71 697)

Reproduced in isolation on a 64-file corpus, frozen clock:

change body reads
three files appended 3
two appended + one new file 65

Fix

A new file is as safe as a tail append: its own body is read in full and the established contributions are untouched. Concretely:

  • let plans of the form "rebuild without prior and without a replaced id" through both gates alongside appends;
  • add such a file's whole analysis as its delta (addAppendAnalysisDelta against an empty accumulator);
  • recompute orderedFileIds / filePositionById only when files were actually added — that is metadata-only work and no body is re-read, and pure appends keep reusing the previous ordering, so the existing "does not revisit historical file contributions" guarantee is untouched;
  • guard ownership: a new file carrying a UUID somebody already owns falls back to the full rebuild, since the full loader resolves that case by scan order. Covered by a test.

Measurement

Real transcripts, 15 files / 34 MB, two appended and one new file added:

body reads bytes read warm refresh
#111 only 16 34 MB 3042 ms
with this 3 0.0002 MB 567 ms

Tests

Two cases added: a new session file alongside two appends stays at 3 body reads and matches the full loader, and a new file carrying an already-owned UUID falls back while still matching the full loader.

Full npm test on this branch: 1028 pass, 8 fail, 1 skipped — the same 8 environmental failures (Codex lease/manifest, EPERM: symlink, temp races) that a clean main produces here.

One thing worth knowing, unrelated to this change

File identity is (dev, inode), and NTFS reuses file record numbers. A file created during a test run can land on the identity of one the fixture already indexed, which reads as a move and legitimately forces a full rebuild. This makes the incremental suites flaky on Windows: across four full runs on this machine, one changed file in a reused corpus, default content analysis does not revisit historical file contributions and others failed on main as well, in a different combination each time. The new test skips itself when it observes changed.move > 0 rather than asserting a number the filesystem cannot deliver. Happy to open a separate issue with the details if that is useful.

rsyuzyov and others added 2 commits September 21, 2026 21:04
…e#99)

The content analysis fast path required exactly one changed file
(sourcePlans.length === 1 && plans.length === 1). A machine running several
agents appends to several transcripts between two refreshes, so that gate never
held: every refresh fell back to replaceWithFullAnalysisPlans() and re-read
every file. The diagnostics show the shape of it - changed=3 reused=641
alongside bodies=644 and bytes=1.4 GB.

The application side already loops over plans, so the gate is what limited it.
Allow any number of pure tail appends, and make two ordering guarantees the
single-file case never needed:

- parse appends in full-scan order, so the first file to carry a UUID owns it
  exactly as in the full loader (the comparator returned 0 for two appends,
  leaving discovery order to decide);
- attribute each new UUID to the appended file that actually carries it,
  earliest first, instead of to parsedPlans[0].

The per-file UUID ownership check that forces a rebuild now runs for every
appended file rather than only a lone one.

Measured on 15 real transcripts (34 MB), three files appended: 15 body reads
and 8680 ms became 3 body reads and 586 ms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
)

A file that had just appeared was classified as an unsafe source mutation, so
plans were replaced with a full rebuild of every file. Every new session starts
its own transcript, so on a machine running agents this happened many times a
day - and it is why the multi-file append fast path rarely helped in practice.

A new file is as safe as a tail append: its own body is read in full and the
established contributions are untouched. Let such plans through the gate, add
the new file's whole analysis as its delta, and recompute ordering state from
metadata only when files were actually added, so pure appends still never touch
historical files.

Ownership is guarded: a new file carrying a UUID somebody already owns falls
back to the full rebuild, because the full loader resolves that by scan order.

Measured on 15 real transcripts, two appends plus one new file: 16 body reads
and 3042 ms became 3 body reads and 567 ms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🤖 Automated first-pass review

The configured analysis model did not return a usable analysis for this pull request. The item has still been received and remains available for maintainer review.

To help with review, please ensure the change summary, tests, and any user-facing documentation or localization updates are included.


🤖 Posted automatically by CCU Bot — not a maintainer decision.

Keep the contributor's new-file fast path and both prior performance changes; release tests pass on the combined tree.

Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>
@Carl723000
Carl723000 merged commit 71b8549 into ClaudeCodeUsage:main Sep 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants