Skip to content

fix: share one exact content-analysis cutoff (#99) - #121

Merged
Carl723000 merged 4 commits into
ClaudeCodeUsage:mainfrom
rsyuzyov:perf/stepped-analysis-cutoff
Sep 30, 2026
Merged

Carl723000 merged 4 commits into
ClaudeCodeUsage:mainfrom
rsyuzyov:perf/stepped-analysis-cutoff

Conversation

@rsyuzyov

@rsyuzyov rsyuzyov commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

This retains @rsyuzyov's useful shared-cutoff extraction from the boundary-file profiling for #99, with a maintainer amendment to preserve the existing exact rolling-expiry contract.

Current changes

  • Share analysisWindowCutoffMs between the full loader and incremental index, retaining millisecond precision.
  • Capture one cutoff for full-loader content analysis and calibration, so a clock advance during parsing cannot give them different windows.
  • Keep the original same-day, configured-zone midnight, calibration-expiry and global Skill-cap test clocks.
  • Cover equality at an event frontier, immediate expiry one millisecond later, zero body reads between frontiers, and clock drift during a full load.
  • Document the same contract in both architecture editions.

The hourly approximation and its benchmark gains are not part of the amended implementation. The existing frontier optimization still avoids body reads when the exact cutoff moves without expiring an admitted event. This does not claim to eliminate all boundary-file rereads or resolve every source of high CPU.

Original profiling and whole-hour prototype (historical context)

The analysis window was now - windowDays to the millisecond, so it moved on
every refresh. Each event it passes forces the incremental index to re-read
that file in full ('cutoff' / 'window' plans): the stored aggregate still counts
the expired event and a tail read cannot subtract it. A transcript that was busy
30 days ago at this time of day loses an event almost every refresh, so it is
re-read almost every refresh.

Measured on a live history (979 files, several agents appending): with the
millisecond cutoff most refreshes re-read 3-5 such files, 5-48 MB and 4-12 s of
parse per tick, and the append fast path stayed off (calibrationCutoffIsStable
is false whenever the cutoff crosses the calibration frontier). With an hourly
cutoff the same refreshes read only the appended tails, 3-100 KB, on the fast
path.

analysisWindowCutoffMs(now, windowDays) rounds the cutoff down to a whole hour
and is shared by the full loader (analysis and its calibration, which now also
use one cutoff instead of two Date.now() reads) and the incremental index, so
the two still produce the same analysis. The window is at most one hour wider
than windowDays, and expired events leave in hourly batches — the file is
re-read once per step instead of once per event.

Tests: the window drifting within one cutoff step re-reads nothing asserts no
body reads across three refreshes inside one step, equality with the full
loader, and a single expiry at the next step. Four tests that expired an event
by moving the clock one minute past it now move it one step (12:00 -> 13:01),
which keeps what they check.

This changes the full loader's contract from a continuously rolling cutoff to an
hourly one, which is why it is a separate PR from #119. The measured fast-path
numbers above are with #119 applied as well: without it the 53 files already
outside the window still keep the fast path off, so the two together are what
make an ordinary refresh cost only its appended tails. If a finer step is
preferred, ANALYSIS_CUTOFF_STEP_MS is the one knob — the re-read cost scales
with how many events each step expires, not with the step itself.

Local note: on Windows a 645-session corpus reads only seven live tails… and
default content analysis does not revisit historical… fail on main as well
(NTFS reuses file identities); they pass in CI.

🤖 Generated with Claude Code


🤖 Generated with OpenAI Codex

…eUsage#99)

The analysis window was `now - windowDays` to the millisecond, so it moved on
every refresh. Each event it passes forces the incremental index to re-read
that file in full ('cutoff' / 'window' plans): the stored aggregate still counts
the expired event and a tail read cannot subtract it. A transcript that was busy
30 days ago at this time of day loses an event almost every refresh, so it is
re-read almost every refresh.

Measured on a live history (979 files, several agents appending): with the
millisecond cutoff most refreshes re-read 3-5 such files, 5-48 MB and 4-12 s of
parse per tick, and the append fast path stayed off (`calibrationCutoffIsStable`
is false whenever the cutoff crosses the calibration frontier). With an hourly
cutoff the same refreshes read only the appended tails, 3-100 KB, on the fast
path.

`analysisWindowCutoffMs(now, windowDays)` rounds the cutoff down to a whole hour
and is shared by the full loader (analysis and its calibration, which now also
use one cutoff instead of two `Date.now()` reads) and the incremental index, so
the two still produce the same analysis. The window is at most one hour wider
than `windowDays`, and expired events leave in hourly batches — the file is
re-read once per step instead of once per event.

Tests: `the window drifting within one cutoff step re-reads nothing` asserts no
body reads across three refreshes inside one step, equality with the full
loader, and a single expiry at the next step. Four tests that expired an event
by moving the clock one minute past it now move it one step (12:00 -> 13:01),
which keeps what they check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🤖 Automated first-pass review

TL;DR

The change is coherent and well-motivated: sharing one stepped cutoff between ClaudeDataLoader and updateClaudeUsageIndex keeps the two producers aligned, and the cost analysis (per-step re-read instead of per-event) is sound. Two merge-readiness gaps stand out: the full loader's comparison test in the new case uses a real Date.now() while the incremental path uses the stubbed clock, and there is no CHANGELOG.md entry for a user-visible behavior change. Neither is fatal, but both should be resolved before merge.

Analysis

  1. Test clock asymmetry in the new test (src/test/claudeIncrementalIndex.test.ts). updateClaudeUsageIndex is called after Date.now = () => now, but ClaudeDataLoader.loadUsageRecords is called in the same block. If the loader reads Date.now() internally (which the diff shows it does, via analysisWindowCutoffMs(Date.now(), windowDays)), it will use the stubbed value only because the stub is global — worth confirming explicitly that the assert.deepEqual(warm.contentAnalysis, full.contentAnalysis) comparison actually exercises the stepped cutoff and not two real-clock readings. If the stub is restored or Date.now is read before the assignment in any code path, this comparison silently becomes a no-op. Prefer a deterministic assertion (or freeze the clock for the whole body) so the equality check cannot pass trivially.

  2. Contract change needs a CHANGELOG.md entry. AGENTS.md says user-facing changes require a CHANGELOG.md entry and matching documentation. The PR description explicitly states the full loader's contract changes from a continuous to an hourly cutoff, and the analysis window widens by up to one hour. That is user-visible (an event up to ~1 h older than windowDays can now be included) and should be called out. No CHANGELOG.md or README change appears in the diff.

  3. Independence from perf: a file already outside the window keeps its empty contribution (#99) #119. The PR body is candid that the fast-path numbers depend on perf: a file already outside the window keeps its empty contribution (#99) #119 also being applied, and that without it the two changes are not sufficient. This is fine as a stacked PR but should be stated clearly in the merge checklist so a reviewer does not apply fix: share one exact content-analysis cutoff (#99) #121 alone and conclude the fast path is broken. Confirm the intended merge order (likely perf: a file already outside the window keeps its empty contribution (#99) #119 first, or both together) and that CI on this branch reflects that combination.

  4. Four existing tests were adjusted (12:00 → 13:01, 16:00:30 → 17:00:30). The tweaks preserve what each test checks, but they shift the boundary by one step; if any of those tests was specifically verifying "an event exactly at the cutoff boundary", the new time may no longer exercise the exact boundary. Spot-check that the boundary semantics (inclusive vs. exclusive at the floored step) are still covered somewhere, ideally by a focused assertion.

  5. analysisWindowCutoffMs is exported and used in two modules. That is the right shape. Consider a brief note in ARCHITECTURE.md near the claudeIncrementalIndex.ts row, since the module map says that module owns "content-analysis contributions" and the cutoff step is now a shared contract between two modules. Small, optional.

  6. Local Windows note. The PR body flags two test names failing on main on NTFS due to file-identity reuse and passing in CI. That is a pre-existing condition, not introduced here, but worth confirming the new test does not join that set on Windows — the new test writes two fixed filenames and asserts bodyReads === 0, which is exactly the pattern the noted failures involve. Verify on Windows or note the expected behavior.

  7. Step constant. ANALYSIS_CUTOFF_STEP_MS = 60 * 60 * 1000 is exported as "the one knob". Good. Make sure nothing else in the codebase computes an analysis window independently (a grep for windowDays * 24 * 60 * 60 * 1000 or similar) so the single-knob claim holds.

Suggested next step(s)

  • Add a CHANGELOG.md entry describing the stepped analysis cutoff and the up-to-one-hour window widening.
  • Harden the new test's clock handling so ClaudeDataLoader is deterministically evaluated under the same stubbed time, or explicitly assert the cutoff value on both sides.
  • Confirm and document merge order with perf: a file already outside the window keeps its empty contribution (#99) #119 (stacked or combined).
  • Grep for other window computations to validate the single-knob claim, and spot-check that the floor boundary semantics remain covered after the four test time shifts.
  • Confirm the new test passes on Windows, or document that it is expected to pass only in CI alongside the two known NTFS-sensitive tests.

🤖 Generated by DeepSeek as an automated first pass — not a maintainer decision.

)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rsyuzyov

Copy link
Copy Markdown
Contributor Author

Thanks for the first pass. Addressed:

  • CHANGELOG — added under Changed (d33b107): the window may now include up to one extra hour, and expired events leave in hourly batches.
  • Clock in the new test — the stub is global and set before both calls, so ClaudeDataLoader.loadUsageRecords reads the same frozen clock as updateClaudeUsageIndex; nothing in the body restores it before the finally. The equality check also can't pass on two real-clock readings: the test first asserts the cutoff itself (2026-08-11T12:00:00Z for the whole step, 13:00:00Z after it), which only the stubbed clock produces.
  • Merge order — perf: a file already outside the window keeps its empty contribution (#99) #119 first, then this one (or both together). Each is correct on its own and CI is green on both; the fast-path numbers in the description need both, because without perf: a file already outside the window keeps its empty contribution (#99) #119 the files already outside the window still turn the fast path off.

🤖 Generated with Claude Code

@Carl723000 Carl723000 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @rsyuzyov for the profiling and for separating this change from #119. The shared-cutoff extraction and reduced boundary-file reads have real value.

I am holding this change out of the current crash/performance patch because it changes the established exact rolling-expiry contract:

  • analysisWindowCutoffMs in src/dataLoader.ts floors the exact cutoff to an epoch hour. This can retain expired content, prompt samples and calibration contributions for up to one additional hour on a fresh analysis pass. All-time usage totals are unchanged, but content-analysis results are not.
  • A synthetic boundary example illustrates that the numerical difference is not necessarily small: at September 10, 12:01 UTC, a response from August 11, 12:00:30 UTC with 4,000 output tokens is excluded by the current 30-day cutoff, but retained by the proposed cutoff. With a recent response of 2 output tokens, calibration is 2 versus 4,002.
  • Agreement between the two modified loaders establishes consistency with the new policy, not compatibility with the old one. Applying the change while retaining the current cutoff tests fails four cases: calibration expiry, configured-zone midnight expiry, same-day expiry, and release of the global Skill-retention cap. The PR adjusts those test clocks to accept the new hourly policy.

Please preserve exact rolling expiry for this patch while exploring cheaper expiry processing from retained contributions. The extraction of one shared exact cutoff can be kept independently. An hourly approximation can be considered separately, but needs an explicit product-contract decision, matching architecture/user documentation, and boundary tests rather than being treated as a transparent optimization.

This PR remains open; the profiling and implementation should not be discarded or replaced without preserving your contribution.


🤖 Generated with OpenAI Codex

Retain the contributor shared-cutoff extraction and single captured full-loader window. Keep millisecond-precise expiry, restore the original boundary clocks, and add exact frontier and clock-drift regression coverage.

Co-authored-by: OpenAI Codex <215057067+openai-codex[bot]@users.noreply.github.com>
@Carl723000 Carl723000 changed the title perf: move the content-analysis cutoff in whole-hour steps (#99) fix: share one exact content-analysis cutoff (#99) Sep 30, 2026
@github-actions github-actions Bot added the fix Patch — bug fix label Sep 30, 2026

@Carl723000 Carl723000 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The earlier expiry concern is resolved by the maintainer amendment on this original PR. The shared helper and single full-loader cutoff are retained, but both loaders keep millisecond precision. The original calibration, same-day, configured-zone midnight and Skill-cap boundary clocks are restored; new regressions cover equality at a frontier, immediate expiry one millisecond later and clock drift during parsing. The combined Node suite passes, and the latest-head Node, UI and package CI checks are all green. Approved.


🤖 Generated with OpenAI Codex

@Carl723000
Carl723000 merged commit fbd9713 into ClaudeCodeUsage:main Sep 30, 2026
5 checks passed
@Carl723000

Copy link
Copy Markdown
Collaborator

Merged after the amendment on this original branch. Thank you @rsyuzyov for profiling the boundary-file costs and extracting the shared cutoff. We retained that useful part, aligned full-loader analysis and calibration to one captured cutoff, and preserved exact millisecond expiry. The original boundary clocks and new one-millisecond/frontier regressions pass; the latest Node, UI and package CI checks are green. This is merged into main, not yet a published extension update.


🤖 Generated with OpenAI Codex

Carl723000 pushed a commit that referenced this pull request Oct 4, 2026
…it (#99) (#123)

* perf: a new transcript no longer re-tags every file discovered after it (#99)

Discovery order only breaks ties between files with the same first timestamp,
but it is stored as the manifest position. A transcript appearing (a new
session, and every sub-agent run) shifts the position of every file after it,
and each shifted file went through `retagFileOrdering`: all of its records were
removed and re-added, re-bucketing every aggregate they touch.

Measured on a live history (1,043 files, a new file at position 312): one new
transcript cost 20.7 s of CPU and 116,131 aggregate mutations for a single
body read. The extension log of one working day showed ~25 such refreshes at
15-40 s each — after #119-#121 the largest remaining cost of a refresh.

Files the index already holds now keep their stored rank as long as their
relative order is unchanged, and a new file takes a rank between its
neighbours. The order is the manifest's, so every tie-break still agrees with
the full loader. If known files change their relative order, or the gap
between two neighbours runs out, the manifest positions are used as before.
Ranks are only compared, never used as positions.

Same measurement with the change: 2.2 s of CPU and 19 aggregate mutations.

Test: `a new transcript does not re-tag the files discovered after it` inserts
a file in the middle of twelve, tying with its right neighbour on first
timestamp and message so that only discovery order picks the counted copy. It
fails on the parent commit (33 mutations for 1 body read), asserts that the
full loader counts the new file's copy, and that records and content analysis
equal the full loader's — also after two known files swap order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: changelog entry for stable discovery ranks (#99)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Patch — bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants