Repository navigation
[Bug]: v2.2.2 — one changed file re-reads the entire 1.1 GB corpus; Extension Host pegged at 100% CPU / 3.9 GB RSS #87
Description
Environment
| Extension | 2.2.2 |
| VS Code | 1.132.0 |
| OS | macOS (Darwin 25.6.0), arm64 |
~/.claude/projects |
1.3 GB across 2261 .jsonl files |
Summary
Related to #70, but I believe this is a distinct root cause, which is why I'm opening it separately rather than commenting there.
The fixes tracked in #70 (removing the 8s polling override, quiet delays, background suspension in #83) all target how often a refresh runs. Those appear to work. The remaining problem is what one refresh costs: any non-zero changed count re-reads and re-parses the entire corpus, not just the changed files.
On a 1.3 GB corpus that is 12–19 seconds of Extension Host main-thread time per refresh. During an active Claude Code session the transcript is appended every few seconds, so refreshes retrigger faster than they complete and the Extension Host never returns to idle.
Evidence — anonymous refresh: lines
Per the guidance in #70, these are refresh: lines only — counts and timings, no paths, prompts, session IDs or raw JSONL.
refresh: trigger=focus files(discovered=2264 changed=0 reused=2264 removed=0 failed=0) io(bytes=0 lines=0) events(watcher=0 coalesced=0) ms(manifest=157.5 read-parse=0.0 aggregate-render=0.0 total=174.7)
refresh: trigger=focus files(discovered=2263 changed=0 reused=2263 removed=0 failed=0) io(bytes=0 lines=0) events(watcher=0 coalesced=1) ms(manifest=178.9 read-parse=0.0 aggregate-render=0.0 total=193.4)
refresh: trigger=watch files(discovered=2264 changed=1 reused=2263 removed=0 failed=0) io(bytes=1124628493 lines=300015) events(watcher=1 coalesced=1) ms(manifest=136.8 read-parse=6344.4 aggregate-render=7917.1 total=14399.0)
refresh: trigger=focus files(discovered=2264 changed=2 reused=2262 removed=0 failed=0) io(bytes=1124637408 lines=300023) events(watcher=0 coalesced=0) ms(manifest=182.1 read-parse=11298.0 aggregate-render=8092.1 total=19618.3)
refresh: trigger=focus files(discovered=2264 changed=2 reused=2262 removed=0 failed=0) io(bytes=1124716652 lines=300058) events(watcher=0 coalesced=0) ms(manifest=244.2 read-parse=8658.1 aggregate-render=10566.0 total=19470.3)
refresh: trigger=watch files(discovered=2263 changed=1 reused=2262 removed=0 failed=0) io(bytes=1123831864 lines=299802) events(watcher=3 coalesced=1) ms(manifest=275.0 read-parse=5963.7 aggregate-render=7120.8 total=13361.0)
refresh: trigger=watch files(discovered=2263 changed=1 reused=2262 removed=0 failed=0) io(bytes=1123857735 lines=299818) events(watcher=2 coalesced=1) ms(manifest=175.9 read-parse=5607.5 aggregate-render=6321.4 total=12105.4)
refresh: trigger=watch files(discovered=2263 changed=1 reused=2262 removed=0 failed=0) io(bytes=1123877234 lines=299834) events(watcher=3 coalesced=1) ms(manifest=377.5 read-parse=7537.5 aggregate-render=9882.1 total=17798.3)
Read the two groups side by side:
changed |
io(bytes) |
total |
|---|---|---|
0 |
0 |
~175–193 ms |
1 |
1_124_628_493 |
~14 400 ms |
2 |
1_124_637_408 |
~19 618 ms |
The changed=0 short-circuit works correctly. The moment one file changes, the full 1.12 GB / 300k lines is read and parsed again.
Note that reused=2263 is reported on the very same line that reads 1.12 GB. Whatever reused is counting (mtime comparison in the manifest stage, judging by manifest=136.8 ms), it isn't gating the read/parse stage — read-parse and aggregate-render both scale with the whole corpus, not with changed.
Extension Host impact
Measured on an otherwise idle window with only this extension active:
- 100% CPU sustained, never returning to idle while Claude Code is running
- 3.9 GB RSS peak (2.9 GB within 60 seconds of Extension Host start)
A sample(1) of the Extension Host shows the main thread almost entirely in fs read completions:
4079 Thread_... DispatchQueue_1: com.apple.main-thread
3453 uv_run
3452 uv__io_poll
3452 uv__async_io
3405 uv__work_done
3403 node::MakeLibuvRequestCallback<uv_fs_s, ...>::Wrapper(uv_fs_s*)
3101 node::fs::FileHandle::CloseReq::Resolve()
... → JIT frames
Because the Extension Host is single-threaded and shared, this stalls every other extension in the window — that's how it surfaced for me.
Expected behaviour
For an append-only JSONL corpus, a refresh triggered by changed=1 should read only the appended bytes of that one file (offset + length from the manifest) and merge the delta into the existing aggregate — cost proportional to changed, not to discovered.
Suggested checks
- Confirm whether the
reusedset is actually skipped in the read/parse stage, or only computed for the manifest and then discarded. aggregate-renderis ~7–10 s independently ofread-parse, so the aggregation step also looks like it rebuilds from the full record set rather than merging a delta. Both stages likely need the same fix.- A corpus-size guard would help as a stopgap — at 1.3 GB this is unusable, and transcript corpora only grow.
Happy to run any build or capture further profiles — I have a reliably reproducing setup.
🤖 Generated with Claude Code
Activity
Carl723000 commentedon Aug 7, 2026
Thanks for the exceptionally clear diagnostics. They make the failure mode reproducible, and we have confirmed the underlying cause.
Root cause
The current manifest comparison is only a reload gate. It can identify unchanged files, but once any file changes, the loader still reads and parses the complete JSONL corpus and rebuilds the aggregates from all records. As a result,
changed=1still causes roughly 1.12 GB of I/O. During an active session, new appends can arrive faster than a full refresh completes, keeping the Extension Host busy and allowing memory use to grow.Planned solution
How we will verify it
Regression coverage will include append, deletion, truncation, atomic replacement, incomplete final lines, and read failures. With a large reused corpus, anonymous diagnostics should show bytes read, lines parsed, and aggregation work scaling with the changed or appended data while exact token totals remain unchanged.
Longer refresh intervals remain a useful temporary mitigation, but the actual fix is to stop repeated full-corpus reads and rebuilds.
🤖 Generated with OpenAI Codex
Carl723000 commentedon Aug 20, 2026
Thanks again for the exceptionally clear diagnostics. They made the failure mode directly measurable.
We have confirmed that the root-cause analysis is accurate: the manifest comparison currently acts only as a reload gate. Once
changedis non-zero, the extension still re-reads and re-parses the complete JSONL corpus and then rebuilds the aggregate views from the completerecords[]collection.Work on the fix has started and is the current highest-priority item. The implementation direction is to retain contributions per file, read only appended bytes during normal JSONL growth, rebuild only a file that is truncated or replaced, remove only the contribution of a deleted file, and publish the new state atomically after a successful update. The aggregation layer must also become incremental; reducing I/O alone would leave the main-thread aggregation stall in place.
We are measuring the prototype against the same full-rebuild results and large-corpus scenarios before assigning it to a release. The target version will be confirmed only after those measurements show that I/O, parsing, and aggregation work scale with the changed data while every reported total remains exactly equal to a clean full rebuild.
Thank you for providing diagnostics detailed enough to define both the root cause and the acceptance criteria.
🤖 Generated with OpenAI Codex
Carl723000 commentedon Aug 21, 2026
Thanks for the detailed report. We reproduced the performance problem and isolated the main cause.
The current refresh path can cheaply detect that nothing changed, but once even a small log append is detected it rereads and reparses the complete Claude history, reruns content analysis, and rebuilds the aggregates. On a 616-file, 338 MB corpus, a representative changed refresh took about 6.41 seconds, while an unchanged check took about 24 ms.
We have validated an incremental in-memory indexing prototype:
The intended production design keeps per-file cursors, parses only appended data, updates global deduplication ownership and aggregates by delta, and falls back to a safe rebuild when a file is truncated or replaced. This makes normal refresh work scale with the amount of changed data instead of total history size.
We plan to make this the core performance fix in the next patch release. The production implementation will retain manual refresh and correctness fallbacks rather than relying only on a longer polling interval.
🤖 Generated with OpenAI Codex
Carl723000 commentedon Aug 28, 2026
Implemented in #93 for v2.3.0. Claude refreshes now use an exact in-memory per-file index: unchanged refreshes read no JSONL bodies, normal appends read only the verified tail, and truncate, replacement, move, and delete events rebuild only the affected contributions. Regression coverage also verifies full-loader equivalence, failure atomicity, and bounded work in a 100-file corpus.\n\nThank you again for the unusually clear diagnostics. If the problem remains after installing v2.3.0, please reopen this issue with new anonymous refresh diagnostic lines.