Repository navigation
fix: share one exact content-analysis cutoff (#99) - #121
Conversation
…eUsage#99) The analysis window was `now - windowDays` to the millisecond, so it moved on every refresh. Each event it passes forces the incremental index to re-read that file in full ('cutoff' / 'window' plans): the stored aggregate still counts the expired event and a tail read cannot subtract it. A transcript that was busy 30 days ago at this time of day loses an event almost every refresh, so it is re-read almost every refresh. Measured on a live history (979 files, several agents appending): with the millisecond cutoff most refreshes re-read 3-5 such files, 5-48 MB and 4-12 s of parse per tick, and the append fast path stayed off (`calibrationCutoffIsStable` is false whenever the cutoff crosses the calibration frontier). With an hourly cutoff the same refreshes read only the appended tails, 3-100 KB, on the fast path. `analysisWindowCutoffMs(now, windowDays)` rounds the cutoff down to a whole hour and is shared by the full loader (analysis and its calibration, which now also use one cutoff instead of two `Date.now()` reads) and the incremental index, so the two still produce the same analysis. The window is at most one hour wider than `windowDays`, and expired events leave in hourly batches — the file is re-read once per step instead of once per event. Tests: `the window drifting within one cutoff step re-reads nothing` asserts no body reads across three refreshes inside one step, equality with the full loader, and a single expiry at the next step. Four tests that expired an event by moving the clock one minute past it now move it one step (12:00 -> 13:01), which keeps what they check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🤖 Automated first-pass review TL;DR The change is coherent and well-motivated: sharing one stepped cutoff between Analysis
Suggested next step(s)
🤖 Generated by DeepSeek as an automated first pass — not a maintainer decision. |
|
Thanks for the first pass. Addressed:
🤖 Generated with Claude Code |
Carl723000
left a comment
There was a problem hiding this comment.
Thank you @rsyuzyov for the profiling and for separating this change from #119. The shared-cutoff extraction and reduced boundary-file reads have real value.
I am holding this change out of the current crash/performance patch because it changes the established exact rolling-expiry contract:
analysisWindowCutoffMsinsrc/dataLoader.tsfloors the exact cutoff to an epoch hour. This can retain expired content, prompt samples and calibration contributions for up to one additional hour on a fresh analysis pass. All-time usage totals are unchanged, but content-analysis results are not.- A synthetic boundary example illustrates that the numerical difference is not necessarily small: at September 10, 12:01 UTC, a response from August 11, 12:00:30 UTC with 4,000 output tokens is excluded by the current 30-day cutoff, but retained by the proposed cutoff. With a recent response of 2 output tokens, calibration is 2 versus 4,002.
- Agreement between the two modified loaders establishes consistency with the new policy, not compatibility with the old one. Applying the change while retaining the current cutoff tests fails four cases: calibration expiry, configured-zone midnight expiry, same-day expiry, and release of the global Skill-retention cap. The PR adjusts those test clocks to accept the new hourly policy.
Please preserve exact rolling expiry for this patch while exploring cheaper expiry processing from retained contributions. The extraction of one shared exact cutoff can be kept independently. An hourly approximation can be considered separately, but needs an explicit product-contract decision, matching architecture/user documentation, and boundary tests rather than being treated as a transparent optimization.
This PR remains open; the profiling and implementation should not be discarded or replaced without preserving your contribution.
🤖 Generated with OpenAI Codex
Retain the contributor shared-cutoff extraction and single captured full-loader window. Keep millisecond-precise expiry, restore the original boundary clocks, and add exact frontier and clock-drift regression coverage. Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>
Carl723000
left a comment
There was a problem hiding this comment.
The earlier expiry concern is resolved by the maintainer amendment on this original PR. The shared helper and single full-loader cutoff are retained, but both loaders keep millisecond precision. The original calibration, same-day, configured-zone midnight and Skill-cap boundary clocks are restored; new regressions cover equality at a frontier, immediate expiry one millisecond later and clock drift during parsing. The combined Node suite passes, and the latest-head Node, UI and package CI checks are all green. Approved.
🤖 Generated with OpenAI Codex
|
Merged after the amendment on this original branch. Thank you @rsyuzyov for profiling the boundary-file costs and extracting the shared cutoff. We retained that useful part, aligned full-loader analysis and calibration to one captured cutoff, and preserved exact millisecond expiry. The original boundary clocks and new one-millisecond/frontier regressions pass; the latest Node, UI and package CI checks are green. This is merged into main, not yet a published extension update. 🤖 Generated with OpenAI Codex |
…it (#99) (#123) * perf: a new transcript no longer re-tags every file discovered after it (#99) Discovery order only breaks ties between files with the same first timestamp, but it is stored as the manifest position. A transcript appearing (a new session, and every sub-agent run) shifts the position of every file after it, and each shifted file went through `retagFileOrdering`: all of its records were removed and re-added, re-bucketing every aggregate they touch. Measured on a live history (1,043 files, a new file at position 312): one new transcript cost 20.7 s of CPU and 116,131 aggregate mutations for a single body read. The extension log of one working day showed ~25 such refreshes at 15-40 s each — after #119-#121 the largest remaining cost of a refresh. Files the index already holds now keep their stored rank as long as their relative order is unchanged, and a new file takes a rank between its neighbours. The order is the manifest's, so every tie-break still agrees with the full loader. If known files change their relative order, or the gap between two neighbours runs out, the manifest positions are used as before. Ranks are only compared, never used as positions. Same measurement with the change: 2.2 s of CPU and 19 aggregate mutations. Test: `a new transcript does not re-tag the files discovered after it` inserts a file in the middle of twelve, tying with its right neighbour on first timestamp and message so that only discovery order picks the counted copy. It fails on the parent commit (33 mutations for 1 body read), asserts that the full loader counts the new file's copy, and that records and content analysis equal the full loader's — also after two known files swap order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: changelog entry for stable discovery ranks (#99) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This retains @rsyuzyov's useful shared-cutoff extraction from the boundary-file profiling for #99, with a maintainer amendment to preserve the existing exact rolling-expiry contract.
Current changes
analysisWindowCutoffMsbetween the full loader and incremental index, retaining millisecond precision.The hourly approximation and its benchmark gains are not part of the amended implementation. The existing frontier optimization still avoids body reads when the exact cutoff moves without expiring an admitted event. This does not claim to eliminate all boundary-file rereads or resolve every source of high CPU.
Original profiling and whole-hour prototype (historical context)
The analysis window was
now - windowDaysto the millisecond, so it moved onevery refresh. Each event it passes forces the incremental index to re-read
that file in full ('cutoff' / 'window' plans): the stored aggregate still counts
the expired event and a tail read cannot subtract it. A transcript that was busy
30 days ago at this time of day loses an event almost every refresh, so it is
re-read almost every refresh.
Measured on a live history (979 files, several agents appending): with the
millisecond cutoff most refreshes re-read 3-5 such files, 5-48 MB and 4-12 s of
parse per tick, and the append fast path stayed off (
calibrationCutoffIsStableis false whenever the cutoff crosses the calibration frontier). With an hourly
cutoff the same refreshes read only the appended tails, 3-100 KB, on the fast
path.
analysisWindowCutoffMs(now, windowDays)rounds the cutoff down to a whole hourand is shared by the full loader (analysis and its calibration, which now also
use one cutoff instead of two
Date.now()reads) and the incremental index, sothe two still produce the same analysis. The window is at most one hour wider
than
windowDays, and expired events leave in hourly batches — the file isre-read once per step instead of once per event.
Tests:
the window drifting within one cutoff step re-reads nothingasserts nobody reads across three refreshes inside one step, equality with the full
loader, and a single expiry at the next step. Four tests that expired an event
by moving the clock one minute past it now move it one step (12:00 -> 13:01),
which keeps what they check.
This changes the full loader's contract from a continuously rolling cutoff to an
hourly one, which is why it is a separate PR from #119. The measured fast-path
numbers above are with #119 applied as well: without it the 53 files already
outside the window still keep the fast path off, so the two together are what
make an ordinary refresh cost only its appended tails. If a finer step is
preferred,
ANALYSIS_CUTOFF_STEP_MSis the one knob — the re-read cost scaleswith how many events each step expires, not with the step itself.
Local note: on Windows
a 645-session corpus reads only seven live tails…anddefault content analysis does not revisit historical…fail onmainas well(NTFS reuses file identities); they pass in CI.
🤖 Generated with Claude Code
🤖 Generated with OpenAI Codex