Skip to content

perf: deleting a transcript no longer re-reads the whole history (#99) - #127

Merged
Carl723000 merged 5 commits into
ClaudeCodeUsage:mainfrom
rsyuzyov:perf/delete-without-rebuild
Oct 16, 2026
Merged

Carl723000 merged 5 commits into
ClaudeCodeUsage:mainfrom
rsyuzyov:perf/delete-without-rebuild

Conversation

@rsyuzyov

@rsyuzyov rsyuzyov commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Problem

Deleting any .jsonl transcript tripped unsafeSourceMutation (deletions.length > 0) and forced replaceWithFullAnalysisPlans(): every surviving file was re-read from disk. On a workstation where Claude prunes or archives transcripts this fired several times a day — measured 30.09: three full rebuilds, 75–110 s each.

Change

One clause removed. Every piece of the incremental deletion path already existed:

  • removeFile drops the deleted file's records and stored per-file contribution;
  • the ownership-restoration pass already collects removedOwnedUuids from deletions and re-reads only a surviving twin that also carries those UUIDs (analysisReason: 'ownership');
  • the append fast path already requires deletions.length === 0, so a refresh with a deletion falls through to buildAnalysisRuntimeState — an in-memory rebuild from retained per-file contributions, no body reads.

Since #123 a deletion no longer re-tags the surviving files' discovery ranks either, so metadataRetags stays empty and canonicalOrderChanged does not fire.

Measured

976-file, 1.58 GB corpus, one mid-history transcript deleted:

CPU body reads bytes aggregate mutations
before 30.0 s 975 1.58 GB 148 496
after 1.0 s 0 0 6

Tests

  • New: a deleted transcript is dropped without re-reading the remaining history — bodyReads === 0, analysis and records match the full loader.
  • New: deleting the owner of a duplicated uuid re-reads only the surviving twin — bodyReads === 1 (the twin, via an 'ownership' plan), analysis matches the full loader.
  • Updated: cross-file content UUID clones keep the legacy first-owner deduplication previously passed for the wrong reason — the full rebuild re-read the one surviving file. Its timestamps are now pinned inside the analysis window so bodyReads === 1 asserts the ownership transfer itself; on the old code with a live window the same delete would have re-read every file.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

🤖 Automated first-pass review

TL;DR

The change is well-reasoned and the incremental machinery it relies on is described as already existing, but the diff as supplied only shows the guard-clause edit plus tests — it does not show removeFile, the ownership-restoration pass, or the append fast-path condition. I can review the intent and the test strategy, but I cannot verify the three load-bearing claims (records dropped, removedOwnedUuids collected from deletions, append path requires deletions.length === 0) from the diff. Merge-readiness hinges on those being true.

Analysis

  • Correctness framing. Removing deletions.length > 0 from unsafeSourceMutation is only safe if every deletion-dependent invariant is handled elsewhere. The PR body enumerates them; the diff does not demonstrate them. The new tests assert the outcomes (bodyReads === 0, analysis/records match the full loader), which is the right acceptance shape, but they pass only because the described internals hold — if any of the three claims is inaccurate, the tests may be green while a subtler stale-aggregate case regresses.
  • Residual risk class. The dangerous deletion path is not "ordinary delete" but deletion where the deleted file contributed to cross-file response identity (removedOwnedUuids), or where deletion interacts with canonicalOrderChanged/metadataRetags. The two new tests plus the reworked clone test cover the ownership-transfer case at bodyReads === 1; that is a reasonable boundary. What is not evidenced: deletion combined with a move in the same refresh (moves.length > 0 still forces full rebuild — good), and deletion that changes canonical order without a move. The body asserts perf: a new transcript no longer re-tags every file discovered after it (#99) #123 makes metadataRetags empty, but there is no test here pinning canonicalOrderChanged === false for a delete.
  • Test hygiene. Pinning Date.now inside the analysis window is a real improvement and correctly makes the old code fail for the right reason. The manual Date.now save/restore + try/finally is repeated across three tests; that is acceptable but a shared helper would reduce the chance a future test forgets the finally. No mock framework issue apparent.
  • CHANGELOG. An entry is present under [Unreleased]/Fixed and follows Keep a Changelog. Good. Note the PR body and CHANGELOG disagree on the corpus figures: body says "976-file, 1.58 GB"; CHANGELOG says "650-file, 1.4 GB". One of these is stale — pick one and make body and CHANGELOG match.
  • Docs / i18n. This is a pure internal performance fix with no user-facing string and no new setting, so the eight-locale requirement and the nine-README update do not apply. The AGENTS.md invariant "runtime refreshes must not fall back to a full-corpus body read" is exactly what this PR advances, so no doc contradiction.
  • Privacy invariants. No new diagnostics, no path/session leakage introduced by the diff; the CHANGELOG text contains only counts and bytes. Fine.
  • Attribution. The PR footer uses the Claude Code banner; per CONTRIBUTING that is the tool actually used, so truthful. The commit trailer requirement in AGENTS.md applies to maintainer-authored commits, not this external PR — no action here.
  • Merge readiness. Scope is one logical change, on-branch, tests included. The only blocker is verifiability of the three internal claims plus the corpus-number discrepancy.

Suggested next step(s)

  1. Reconcile the corpus figures between PR body (976-file / 1.58 GB) and CHANGELOG (650-file / 1.4 GB); update whichever is stale.
  2. Add one focused assertion that a plain delete does not set canonicalOrderChanged / metadataRetags, pinning the perf: a new transcript no longer re-tags every file discovered after it (#99) #123 dependency this PR leans on.
  3. If verifiable from the base repo, confirm in the PR description (or a comment) the exact lines of removeFile, the ownership-restoration removedOwnedUuids collection, and the deletions.length === 0 condition in the append fast path — so a reviewer can check the three claims without reconstructing them from the full module.

🤖 Generated by DeepSeek as an automated first pass — not a maintainer decision.

@rsyuzyov

Copy link
Copy Markdown
Contributor Author

Addressing the first-pass review:

  1. Corpus figures reconciled — the stale 650-file/1.4 GB numbers came from the 30.09 live observations; the comment and CHANGELOG now carry the bench measurement (976 files, 1.58 GB, 30.0 s CPU / 975 body reads → 1.0 s / 0 body reads).
  2. No-retag contract pinned — the plain-delete test now asserts changed deep-equals { append: 0, rebuild: 0, move: 0, delete: 1 }. Since canonicalOrderChanged escalates through replaceWithFullAnalysisPlans() (which sets changed.rebuild to the full corpus), rebuild === 0 proves no canonical-order escalation happened.
  3. The three load-bearing lines (this branch, src/claudeIncrementalIndex.ts): records and the stored contribution are dropped by removeFile at line 2968, applied to deletions at line 3008; removedOwnedUuids is collected from deletions at line 2753; the append fast path requires deletions.length === 0 at line 2800, which is what routes a delete refresh to the in-memory buildAnalysisRuntimeState rebuild instead of the fast path.

@Carl723000 Carl723000 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rsyuzyov for the measurements and for updating the earlier review points. The bounded deletion optimization is useful, but a fresh check against released main d6d69aa found two mixed-refresh content-analysis regressions before this can merge.

  1. Deleted owner + appended surviving claimant. A owns UUID X; B's existing prefix contains X's suppressed duplicate. Delete A and append unrelated content to B in the same refresh. Removing the deletion guard exposes the ownership-restoration pass's plannedPaths.has(claimant) skip (head line 2772). B receives only a tail read, so X disappears from content analysis and remains missing on the next unchanged refresh. The released baseline restores X.
  2. Unrelated deletion + earlier append preempting a later owner. With B earlier than C in canonical discovery order, let C own X, delete unrelated A and append X to B. Deletion disables fastAppendAnalysis, so the collision probe (head line 2891) does not run. C's content incorrectly remains canonical instead of B's. The released baseline matches the full loader.

Both cases were independently rerun with synthetic files. These are content-analysis mismatches; the fixtures did not demonstrate incorrect token totals, and prior snapshots stayed unchanged. Existing tests, including the contribution's new tests, pass—the mixed-event cases are missing coverage.

Please account for already-planned append claimants that need ownership replay, keep preemption detection active in mixed deletion refreshes, and add full-loader-equivalence tests including an unchanged follow-up refresh and prior-snapshot immutability. Also place the new release note under Unreleased, not the now-published 2.5.0 section. This contribution should continue through this original PR, preserving your authorship.

Reviewed head: b8b63ac3e215ea97cd6da91368c4d70dc9568af5. Combining the proposals with #128 fixes the second fixture but not the first, so merging both is not a substitute for the ownership-restoration fix.


🤖 Generated by OpenAI Codex as an automated first pass — not a maintainer decision.

rsyuzyov and others added 4 commits October 15, 2026 22:00
…udeCodeUsage#99)

A removed .jsonl tripped unsafeSourceMutation and every surviving file was
rebuilt from its body. The deletion path was already incremental everywhere
else: removeFile drops the records and the stored contribution, the ownership
restoration pass re-reads only a surviving twin that shares UUIDs, and the
analysis fast path's deletions.length === 0 condition already routes the
materialized view through an in-memory rebuild from retained per-file
contributions.

Measured on a 976-file, 1.58 GB corpus, deleting one mid-history transcript:
30.0 s CPU, 975 body reads, 148k aggregate mutations before; 1.0 s CPU, zero
body reads, 6 mutations after.

The cross-file UUID clone test now pins the reason a twin is re-read (live
window, ownership transfer) instead of the old full rebuild it happened to
coincide with.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ge#99)

A deleted owner's surviving claimant was skipped when that claimant was
already planned as a tail append, so a prefix duplicate disappeared. An
unrelated deletion also turned the later-owner probe off, so an earlier
append kept the wrong canonical owner. Append claimants are promoted to
a full replay, and mixed deletion refreshes keep the probe.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@rsyuzyov
rsyuzyov force-pushed the perf/delete-without-rebuild branch from b8b63ac to d78f46c Compare October 15, 2026 17:04
@rsyuzyov

Copy link
Copy Markdown
Contributor Author

Addressing the review of b8b63ac, rebased onto current main:

  1. Deleted owner + appended claimant. The ownership-restoration pass skipped any file already in plannedPaths, so a claimant planned as a tail append never replayed the prefix duplicate. An append whose retained analysisAllUuids intersect the removed owner's UUIDs is now promoted to a full replay; a plan that already re-reads its body is left alone. Test: deleting an owner while appending its claimant replays that claimant — one body read (a bystander is not read), full-loader content-analysis equivalence, an unchanged follow-up with zero body reads, and the prior index/analysis frozen before the refresh.

  2. Unrelated deletion + earlier append preempting a later owner. Deletion kept the refresh off the append fast path, which was the only place the later-owner probe ran. Mixed deletion refreshes now run that probe and still fall back to one ordered rebuild on collision. Test: an unrelated deletion does not hide an earlier append preempting a later owner, with the same equivalence, follow-up and snapshot checks.

  3. Changelog. The note is under Unreleased, not the published 2.5.0 section.

Resolve only the Unreleased CHANGELOG conflict, retaining both the deletion and concurrent-stat entries. Production source and tests remain exactly those from the original PR and the already-merged ClaudeCodeUsage#129 contribution.

Validation: full Node/policy suite (2657 passed, 2 skipped); synthetic mixed-event, ownership-order and prior-snapshot checks against full parsing.

Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>

@Carl723000 Carl723000 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The follow-up deletion fixes at d78f46c4cb10f148ff5aa3209166defd81175c23 address both previously reported mixed-event ownership regressions. The original PR branch has now been integrated with main after #129, resolving only the shared Unreleased CHANGELOG insertion; neither contribution's production source or tests was rewritten.

On that integrated state, the full local Node/policy suite passed (2,657 passed, 2 skipped), and independent synthetic deletion/append, earlier-owner preemption, rewrite/append and first-timestamp/order fixtures agree with the full loader, preserve previous snapshots, and retain correct unchanged follow-ups. The separate #128 ordering finding does not reproduce on this PR's retained correctness-first guard.

Reviewed integrated head: 8173a7d30c502c3733673f62e0aa049f101705ef. Required CI for this updated head must pass before merge. No remaining blocking finding in this reviewed scope; approving to supersede my earlier changes-requested review. This does not establish that all of #99 is resolved or provide a new native performance benchmark.


🤖 Generated by OpenAI Codex as an automated first pass — not a maintainer decision.

@Carl723000
Carl723000 merged commit e0917c1 into ClaudeCodeUsage:main Oct 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants