Repository navigation
perf: deleting a transcript no longer re-reads the whole history (#99) - #127
Conversation
|
🤖 Automated first-pass review TL;DR The change is well-reasoned and the incremental machinery it relies on is described as already existing, but the diff as supplied only shows the guard-clause edit plus tests — it does not show Analysis
Suggested next step(s)
🤖 Generated by DeepSeek as an automated first pass — not a maintainer decision. |
|
Addressing the first-pass review:
|
Carl723000
left a comment
There was a problem hiding this comment.
Thanks @rsyuzyov for the measurements and for updating the earlier review points. The bounded deletion optimization is useful, but a fresh check against released main d6d69aa found two mixed-refresh content-analysis regressions before this can merge.
- Deleted owner + appended surviving claimant. A owns UUID X; B's existing prefix contains X's suppressed duplicate. Delete A and append unrelated content to B in the same refresh. Removing the deletion guard exposes the ownership-restoration pass's
plannedPaths.has(claimant)skip (head line 2772). B receives only a tail read, so X disappears from content analysis and remains missing on the next unchanged refresh. The released baseline restores X. - Unrelated deletion + earlier append preempting a later owner. With B earlier than C in canonical discovery order, let C own X, delete unrelated A and append X to B. Deletion disables
fastAppendAnalysis, so the collision probe (head line 2891) does not run. C's content incorrectly remains canonical instead of B's. The released baseline matches the full loader.
Both cases were independently rerun with synthetic files. These are content-analysis mismatches; the fixtures did not demonstrate incorrect token totals, and prior snapshots stayed unchanged. Existing tests, including the contribution's new tests, pass—the mixed-event cases are missing coverage.
Please account for already-planned append claimants that need ownership replay, keep preemption detection active in mixed deletion refreshes, and add full-loader-equivalence tests including an unchanged follow-up refresh and prior-snapshot immutability. Also place the new release note under Unreleased, not the now-published 2.5.0 section. This contribution should continue through this original PR, preserving your authorship.
Reviewed head: b8b63ac3e215ea97cd6da91368c4d70dc9568af5. Combining the proposals with #128 fixes the second fixture but not the first, so merging both is not a substitute for the ownership-restoration fix.
🤖 Generated by OpenAI Codex as an automated first pass — not a maintainer decision.
…udeCodeUsage#99) A removed .jsonl tripped unsafeSourceMutation and every surviving file was rebuilt from its body. The deletion path was already incremental everywhere else: removeFile drops the records and the stored contribution, the ownership restoration pass re-reads only a surviving twin that shares UUIDs, and the analysis fast path's deletions.length === 0 condition already routes the materialized view through an in-memory rebuild from retained per-file contributions. Measured on a 976-file, 1.58 GB corpus, deleting one mid-history transcript: 30.0 s CPU, 975 body reads, 148k aggregate mutations before; 1.0 s CPU, zero body reads, 6 mutations after. The cross-file UUID clone test now pins the reason a twin is re-read (live window, ownership transfer) instead of the old full rebuild it happened to coincide with. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…udeCodeUsage#99) Co-Authored-By: Claude Code <noreply@anthropic.com>
…ge#99) A deleted owner's surviving claimant was skipped when that claimant was already planned as a tail append, so a prefix duplicate disappeared. An unrelated deletion also turned the later-owner probe off, so an earlier append kept the wrong canonical owner. Append claimants are promoted to a full replay, and mixed deletion refreshes keep the probe. Co-Authored-By: Claude Code <noreply@anthropic.com>
b8b63ac to
d78f46c
Compare
|
Addressing the review of
|
Resolve only the Unreleased CHANGELOG conflict, retaining both the deletion and concurrent-stat entries. Production source and tests remain exactly those from the original PR and the already-merged ClaudeCodeUsage#129 contribution. Validation: full Node/policy suite (2657 passed, 2 skipped); synthetic mixed-event, ownership-order and prior-snapshot checks against full parsing. Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>
Carl723000
left a comment
There was a problem hiding this comment.
The follow-up deletion fixes at d78f46c4cb10f148ff5aa3209166defd81175c23 address both previously reported mixed-event ownership regressions. The original PR branch has now been integrated with main after #129, resolving only the shared Unreleased CHANGELOG insertion; neither contribution's production source or tests was rewritten.
On that integrated state, the full local Node/policy suite passed (2,657 passed, 2 skipped), and independent synthetic deletion/append, earlier-owner preemption, rewrite/append and first-timestamp/order fixtures agree with the full loader, preserve previous snapshots, and retain correct unchanged follow-ups. The separate #128 ordering finding does not reproduce on this PR's retained correctness-first guard.
Reviewed integrated head: 8173a7d30c502c3733673f62e0aa049f101705ef. Required CI for this updated head must pass before merge. No remaining blocking finding in this reviewed scope; approving to supersede my earlier changes-requested review. This does not establish that all of #99 is resolved or provide a new native performance benchmark.
🤖 Generated by OpenAI Codex as an automated first pass — not a maintainer decision.
Problem
Deleting any
.jsonltranscript trippedunsafeSourceMutation(deletions.length > 0) and forcedreplaceWithFullAnalysisPlans(): every surviving file was re-read from disk. On a workstation where Claude prunes or archives transcripts this fired several times a day — measured 30.09: three full rebuilds, 75–110 s each.Change
One clause removed. Every piece of the incremental deletion path already existed:
removeFiledrops the deleted file's records and stored per-file contribution;removedOwnedUuidsfrom deletions and re-reads only a surviving twin that also carries those UUIDs (analysisReason: 'ownership');deletions.length === 0, so a refresh with a deletion falls through tobuildAnalysisRuntimeState— an in-memory rebuild from retained per-file contributions, no body reads.Since #123 a deletion no longer re-tags the surviving files' discovery ranks either, so
metadataRetagsstays empty andcanonicalOrderChangeddoes not fire.Measured
976-file, 1.58 GB corpus, one mid-history transcript deleted:
Tests
a deleted transcript is dropped without re-reading the remaining history—bodyReads === 0, analysis and records match the full loader.deleting the owner of a duplicated uuid re-reads only the surviving twin—bodyReads === 1(the twin, via an'ownership'plan), analysis matches the full loader.cross-file content UUID clones keep the legacy first-owner deduplicationpreviously passed for the wrong reason — the full rebuild re-read the one surviving file. Its timestamps are now pinned inside the analysis window sobodyReads === 1asserts the ownership transfer itself; on the old code with a live window the same delete would have re-read every file.🤖 Generated with Claude Code