Repository navigation
perf: a file already outside the window keeps its empty contribution (#99) - #119
Merged
Carl723000 merged 2 commits intoSep 30, 2026
Conversation
…laudeCodeUsage#99) Measured on a 979-file history with several agents appending: the append fast path never ran. Every refresh reported 53 analysis payload rebases, and any payload rebase disables `fastAppendAnalysis`, so each tick of a few kilobytes paid the slow path — rebuilding the seen-UUID set over the whole history. The 53 files are the ones whose every event is older than the window. On each refresh the cutoff moves, `rebaseFileAnalysisCutoff` takes the "last event is before the cutoff" branch and returns a fresh `newAnalysisAcc`. The payload is empty before and after, but `sameAnalysisPayload` compares collections by identity (by design: a metadata-only rebase keeps them), so every such file reads as changed on every refresh, for good. That branch is now taken only when the last event has just left the window (`last >= prior.analysis.cutoffMs`). A file that was already past the previous cutoff is empty already — `analyzeLine` ignores lines older than the cutoff — and falls through to the metadata-only rebase, keeping its collections. On the same history: `payloadRebases` 53 -> 0 on every refresh. Test: `a file already outside the window keeps its empty contribution as the window drifts` — it fails on the parent commit ("Values have same structure but are not reference-equal") and asserts the analysis equals the full loader's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🤖 Automated first-pass review TL;DR The fix is well-reasoned and the added test looks like a genuine red-green case. Two gaps stand out before merge: no Analysis Correctness of the change:
Test quality:
Gaps / merge-readiness:
Suggested next step(s)
🤖 Generated by DeepSeek as an automated first pass — not a maintainer decision. |
This was referenced Sep 28, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Carl723000
approved these changes
Sep 30, 2026
Carl723000
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed the empty-contribution and cutoff-metadata paths, including expiry, appends and backward cutoff movement. The original regression and the combined Node suite pass; the required Node, UI and package checks are green. No blocking finding.
🤖 Generated with OpenAI Codex
Collaborator
|
Merged. Thank you @rsyuzyov for the profiling, minimal fix and regression test. Keeping already-expired contributions empty without recreating their collections removes a real obstacle to incremental refreshes. This is merged into main; it is not yet a published extension update. 🤖 Generated with OpenAI Codex |
Carl723000
pushed a commit
that referenced
this pull request
Oct 4, 2026
…it (#99) (#123) * perf: a new transcript no longer re-tags every file discovered after it (#99) Discovery order only breaks ties between files with the same first timestamp, but it is stored as the manifest position. A transcript appearing (a new session, and every sub-agent run) shifts the position of every file after it, and each shifted file went through `retagFileOrdering`: all of its records were removed and re-added, re-bucketing every aggregate they touch. Measured on a live history (1,043 files, a new file at position 312): one new transcript cost 20.7 s of CPU and 116,131 aggregate mutations for a single body read. The extension log of one working day showed ~25 such refreshes at 15-40 s each — after #119-#121 the largest remaining cost of a refresh. Files the index already holds now keep their stored rank as long as their relative order is unchanged, and a new file takes a rank between its neighbours. The order is the manifest's, so every tie-break still agrees with the full loader. If known files change their relative order, or the gap between two neighbours runs out, the manifest positions are used as before. Ranks are only compared, never used as positions. Same measurement with the change: 2.2 s of CPU and 19 aggregate mutations. Test: `a new transcript does not re-tag the files discovered after it` inserts a file in the middle of twelve, tying with its right neighbour on first timestamp and message so that only discovery order picks the counted copy. It fails on the parent commit (33 mutations for 1 body read), asserts that the full loader counts the new file's copy, and that records and content analysis equal the full loader's — also after two known files swap order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: changelog entry for stable discovery ranks (#99) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured on a 979-file history with several agents appending: the append fast
path never ran. Every refresh reported 53 analysis payload rebases, and any
payload rebase disables
fastAppendAnalysis, so each tick of a few kilobytespaid the slow path — rebuilding the seen-UUID set over the whole history.
The 53 files are the ones whose every event is older than the window. On each
refresh the cutoff moves,
rebaseFileAnalysisCutofftakes the "last event isbefore the cutoff" branch and returns a fresh
newAnalysisAcc. The payload isempty before and after, but
sameAnalysisPayloadcompares collections byidentity (by design: a metadata-only rebase keeps them), so every such file
reads as changed on every refresh, for good.
That branch is now taken only when the last event has just left the window
(
last >= prior.analysis.cutoffMs). A file that was already past the previouscutoff is empty already —
analyzeLineignores lines older than the cutoff —and falls through to the metadata-only rebase, keeping its collections.
On the same history:
payloadRebases53 -> 0 on every refresh.Test:
a file already outside the window keeps its empty contribution as the window drifts— it fails on the parent commit ("Values have same structure butare not reference-equal") and asserts the analysis equals the full loader's.
Context for #99: this is the first of the remaining per-tick costs found by
profiling a live history (979 files, ~17 concurrent agent sessions). The
largest one left after this is the continuously rolling cutoff re-reading whole
files whose events sit at the 30-day boundary — I'll open that separately, since
it touches the full loader's cutoff contract and deserves its own discussion.
Local note: on Windows
a 645-session corpus reads only seven live tails…anddefault content analysis does not revisit historical…fail onmainas well(NTFS reuses file identities); they pass in CI.
🤖 Generated with Claude Code