diff --git a/.claude/handoffs/codegraph-tool-surface-rethink-2026-05-27.md b/.claude/handoffs/codegraph-tool-surface-rethink-2026-05-27.md new file mode 100644 index 000000000..398e783d5 --- /dev/null +++ b/.claude/handoffs/codegraph-tool-surface-rethink-2026-05-27.md @@ -0,0 +1,114 @@ +--- +name: codegraph-tool-surface-rethink-2026-05-27 +date: 2026-05-27 15:11 +project: codegraph +branch: feat/go-multi-module-trace-quality +summary: PR #494 multi-language audit revealed structural ~$0.04-$0.08 tiny-repo cost overhead from MCP tool-defs; user pivoted to questioning whether codegraph_context / 5+ tools are even necessary — suggested `explore` + `trace` only. +--- + +# Handoff: Should codegraph cut to just `explore` + `trace`? + +## Resume here — read this first +**Current state:** PR #494 (`feat/go-multi-module-trace-quality`, 13 commits, all 1076 tests pass) ships every safe optimization for the cosmos/etcd Go work AND the cross-language extensions (generated-detection, IFACE_OVERRIDE_LANGS, sibling-inlining, path-proximity, tool gating at <150 files to 5 core tools). Empirically PROVED that cutting below 5 tools regresses every tiny repo (3-tool gate: cobra 17→48% loss; 1-tool gate: express -43% WIN flipped to +107% LOSS). User just asked the right question: **"Why do we need codegraph_context, or any of these massive amounts of tools? All it really needs is explore, and trace if you ask me."** + +**Immediate next step:** Open the next session by treating the user's question as a design pivot, not a continuation of the cost-gap whack-a-mole. The right reply is a focused honest analysis: what does each of the 10 tools actually do that explore + trace alone can't, where does codegraph_context's value-add hold up (or not), and what would removing context/search/node from the default surface ACTUALLY cost in measured loss-of-flow-coverage. Don't start cutting tools yet — present the analysis first. + +> Suggested next message: "Walk me through what each codegraph_* tool actually does on a real flow question that explore + trace alone can't, and which ones agents are picking in our recent audits. If context/search/node aren't earning their seat, propose cutting them and measure on cosmos-Q1 + etcd-Q1 + prometheus + cobra n=2 each." + +## Goal +Decide whether codegraph's 10-tool MCP surface should be cut down to ~2 core tools (explore + trace) as the user proposed. The empirical iteration in this session showed that the 5 omitted "auxiliary" tools (callers, callees, impact, status, files) only add cost on tiny repos and aren't earning their seat. The real question now: **does the same logic apply to context + search + node?** If yes, codegraph becomes 2 tools + a smaller MCP surface = lower fixed prompt overhead = closes the tiny-repo cost gap structurally instead of patching it. If no, name the specific flows where they do unique work. + +## Key findings (this session) + +- **PR #494 status**: 13 commits, all 1076 tests pass, https://github.com/colbymchenry/codegraph/pull/494. Already pushed: + - Generated-file detection: `src/extraction/generated-detection.ts` (multi-language patterns, applied in `findSymbol`/`findAllSymbols`/`handleSearch`/`handleExplore` file ranking/`context/formatter.ts`) + - Go gRPC bridge: `goGrpcStubImplEdges` in `src/resolution/callback-synthesizer.ts:341` (467 bridge edges on cosmos-sdk) + - Trace failure inlining + path-proximity pairing + less-canonical-path penalty + sibling-from-TO-file inlining: all in `src/mcp/tools.ts` `handleTrace` + - `IFACE_OVERRIDE_LANGS` extended from `{java,kotlin}` to `{java,kotlin,csharp,typescript,javascript,swift,scala}`; loop iterates `class` AND `struct` kinds + - Tool-def trims (~7KB → 5KB) in `src/mcp/tools.ts` + - Tiny-repo tool gating: `ToolHandler.getTools()` filters to 5 core tools when `fileCount < 150` + - Tiny-tier explore budget in `getExploreOutputBudget(fileCount < 150)`: 13K total / 4 files / `includeRelationships: true` + - `handleContext` default `maxNodes` drops from 20 → 8 when `fileCount < 150` +- **Cosmos Q1 flipped**: WIN ($0.257 vs $0.449, n=1; n=2 avg $0.341 vs $0.350 tied). The breakthrough was `inlineEndpoint`'s "Other functions in TO's file" siblings — `msgServer.Send`'s real callee `k.Keeper.SendCoins` is an embedded-interface call tree-sitter can't statically resolve, so static `getCallees` returns only utility funcs; the *actual* flow lives in `x/bank/keeper/send.go`'s file-mates. See `handleTrace` line ~1430. +- **Empirical lower bounds on tool gating** (n=2-3 audits): + - 5 tools (search+context+node+explore+trace) = current setting, works + - 3 tools (search+context+trace) = cobra 17→48% loss, sinatra 18→96% loss; agent falls back to Reads when node/explore unavailable + - 1 tool (search only) = catastrophic, express -43% WIN → +107% LOSS +- **n=3 measurements confirm structural floor:** cobra WITH consistently $0.28 (variance <5%), WITHOUT consistently $0.24. The $0.04 gap is structural, not noise. +- **The user's pivot question challenges this:** their hypothesis is that context+search+node may also be earning less than they cost. The audits we have can't directly answer that — every test had all 10 (or 5) tools available. To test, expose ONLY explore+trace on a controlled batch and re-measure. +- **Cross-language status (single-run each):** WINS = Go (multi-mod), Rust, Java, C#, Kotlin, Swift, Svelte, prometheus, ky (post-gating), express (JS). TIES = cobra (n=2 tied $0.27/$0.27), excalidraw, django, redis, json, Masonry, flutter, vapor, spring. LOSSES = sinatra, slim, flask, scala-play, Fusion, vue-core (variance), Drupal, NestJS, FastAPI, Laravel, ASP.NET, axum, actix, Rocket, gorilla/mux, SvelteKit, Charts bridge (slight), RN segmented-control (slight). +- **Loss pattern is structural, not language-specific.** All losses are tiny example/starter repos where the without-arm grep+read path costs ~$0.20-0.30 and codegraph's MCP overhead can't be amortized. + +## Gotchas + +- **PR-494 is a Go-multi-module PR by title but the body is now cross-cutting** — generated-detection, IFACE_OVERRIDE_LANGS, tool gating, all language-agnostic. Don't let the title narrow what's in it. +- **The variance on the WITHOUT arm is enormous** — same-repo single-run cost can swing $0.04 to $0.80 depending on whether the agent goes grep-heavy or read-heavy that turn. **Never conclude WIN/LOSS from n=1.** The session has many single-run results that need confirming. +- **Cobra (~50 files) is the canary** — every aggressive cut that helps ky or sinatra has regressed cobra at least once. It's the most-tested tiny repo because of that. +- **Don't try the 1-tool or 3-tool gate again** — both are explicitly documented as regressions in `getTools()` comments (`src/mcp/tools.ts` around line 660). Cutting below 5 forces the agent to Read. +- **Kong's first audit was a 0-byte index** — parallel `audit.sh` runs against the same .codegraph dir can corrupt each other. If kong/any-repo's audit shows wildly wrong numbers, check `stat /tmp/codegraph-corpus//.codegraph/codegraph.db` before iterating on the result. +- **48-parallel audit launches FAIL silently** — system resource limits. Stay at 6-8 parallel max. Use `wait` between waves. +- **The MCP daemon caches the tool list** at process start — when iterating on `getTools()` you MUST `pkill -f "codegraph.js serve --mcp"` between rebuilds or you'll be testing stale code. +- **`maxCharsPerFile` monotonic invariant** is pinned by `__tests__/explore-output-budget.test.ts` (the spec is `a larger tier must NEVER get a smaller maxCharsPerFile than a smaller tier`). Honor it. + +## How to test & validate + +- `npm test` → "Tests 1076 passed | 2 skipped". Must stay green. +- `npm run build 2>&1 | tail -3` → check dist rebuilt cleanly. +- `pkill -f "codegraph.js serve --mcp" ; sleep 2` → ALWAYS run before agent-eval after a build, otherwise the daemon serves stale code. +- Single-question audit: `AGENT_EVAL_OUT=/tmp/cg-NAME /Users/colby/Development/Personal/codegraph/scripts/agent-eval/run-all.sh "" headless`. Outputs `run-headless-with.jsonl` and `run-headless-without.jsonl`. +- Parse: `node scripts/agent-eval/parse-run.mjs /tmp/cg-NAME/run-headless-{with,without}.jsonl` → cost, duration, turns, tool sequence. +- **For real conclusions, always n=2 minimum.** n=3 is the right bar to separate variance from signal — last session's data on cobra showed WITH had <5% variance but WITHOUT swung 95%. +- **The explore + trace experiment** the user wants: modify `getTools()` to filter visible tools to `new Set(['codegraph_explore', 'codegraph_trace'])` for ALL repos (or just the tiny tier first), re-run cosmos-Q1, etcd-Q1, prometheus, cobra n=2 each, and compare. + +## Repo state + +- branch `feat/go-multi-module-trace-quality`, last commit `ae5364c docs(mcp): pin empirical lower bound on tool gating after n=2 micro test` +- uncommitted: clean +- PR: https://github.com/colbymchenry/codegraph/pull/494 (13 commits, ready for review unless we land the tool-surface redesign) + +## Open threads / TODO + +- [ ] **The user's pivot**: prove or disprove that explore + trace alone is sufficient. Set up a 4-repo × n=2 batch (cosmos-Q1, etcd-Q1, prometheus, cobra) with ONLY explore+trace exposed, compare to current 5-tool / 10-tool baselines. +- [ ] If explore+trace alone wins → cut the tool surface across the board. **This is a breaking API change** — callers/callees/impact/status/files/node would disappear from default exposure. Need a clean way to retain them for users who script against the MCP directly (env var? `--full-tools` flag?). +- [ ] If explore+trace alone loses → identify which of context/search/node is doing the structural work, and propose cutting only the others. +- [ ] **README update either way**: the current "~35% cheaper" claim averages 7 medium/large repos. Either commit to that scope ("real codebases (~200+ files)") or re-measure after the tool surface change. +- [ ] Liquid, Pascal/Delphi, React Router, TurboModules, Expo Modules, Paper view managers — still untested categories from the README. Bridges Swift↔ObjC/RN-legacy/RN-events/Fabric were tested in wave 3 — 1 win, 2 tied, 1 slight loss. The rest are still gaps. +- [ ] If we ship the PR as-is, write a CHANGELOG entry under `[Unreleased]` summarizing the 13 commits — currently the CHANGELOG entry covers commits 1-2 (generated-detection + gRPC bridge + trace UX); commits 3-13 need their own bullets. + +## Recent transcript (last ~10 turns, oldest → newest) + +### Turn N-9 — User: "Have we tested every language/framework per the playbook?" +- **Claude:** Audited coverage honestly — ~70% of README's claims (22 langs + 14 frameworks + 8 bridges) had NOT been validated. Offered three scopes (smoke / medium / full playbook). +- **Outcome:** User chose "Full playbook"; later refined to "Smoke-test only the completely-untested categories." + +### Turn N-8 — Wave 1 smoke-test (8 parallel: JS/Dart/Drupal/NestJS/FastAPI/Vapor/Spring/Rails) +- **Claude:** Launched 8 audits via `run-all.sh`. Results: express WIN (-43%), 3 tied, 4 LOSS (Drupal, NestJS, FastAPI, Rails). NestJS 113% loss, FastAPI 194% loss were the standouts. +- **Outcome:** Documented the loss pattern correlates with repo size, not language. + +### Turn N-7 — Wave 2 (7 frameworks: Laravel, ASP.NET, axum, actix, Rocket, gorilla/mux, SvelteKit) + Wave 3 (4 bridges: Charts, async-storage, RN-geolocation, RN-segmented-control) +- **Claude:** All wave-2 frameworks lost on cost (small example/starter repos). All wave-3 bridges won/tied on cost AND won on reads. +- **Outcome:** Confirmed loss pattern is tiny-repo-structural, not framework-specific. Bridges (codegraph's marquee feature) work. + +### Turn N-6 — User: "We have a lot of losses. Work on those until they are all clear wins. You cannot stop or give up or push back." +- **Claude:** Ran n=3 on the 8 worst losses to separate variance from signal. First attempt with 48 parallel processes silently failed (system limits). Re-ran in waves of 6. +- **Outcome:** n=3 cobra: WITH consistent $0.28 (variance <5%), WITHOUT consistent $0.24. **Confirmed losses are structural, not variance.** + +### Turn N-5 — Tested 3-tool gate (search + context + trace only) +- **Claude:** Edited `getTools()` to expose only 3 tools on `<150` files. Re-ran cobra/ky/sinatra. Results: cobra 17→48% LOSS, ky 47→42% loss, sinatra 18→96% LOSS. Reverted. +- **Outcome:** Cutting below 5 tools regresses because the agent falls back to extra Reads. + +### Turn N-4 — Tested 1-tool gate (search only) on micro (<50 files) +- **Claude:** Edited gate. cobra 126% LOSS, ky 44% loss, express FLIPPED from -43% WIN to +107% LOSS. Catastrophic. Reverted. +- **Outcome:** Documented the empirical lower bound: 5 tools. Committed `ae5364c` with comments warning future tuners not to relitigate. + +### Turn N-3 — Claude proposed three paths forward (auto-disable / README honesty / context-rewrite) +- **Claude:** Used AskUserQuestion to offer auto-disable on <50, honest README, or last-ditch context shrink. +- **Outcome:** User pivoted instead of picking any of those. + +### Turn N-2 — User: "Why do we need codegraph_context, or any of these massive amounts of tools? All it really needs is explore, and trace if you ask me." +- **Claude:** Recognized this is a design pivot, not a continuation of the cost-gap iteration. The right next move is to actually test the user's hypothesis, not to write another response defending the status quo. +- **Outcome:** This handoff captures the pivot for a fresh session to answer properly. + +### Turn N-1 — User: `/handoff save` +- **Claude:** Wrote this file. +- **Outcome:** Handoff persisted. Next session reads it and engages the explore+trace-only design question with measurement, not opinion. diff --git a/.claude/handoffs/explore-flow-tool-adoption.md b/.claude/handoffs/explore-flow-tool-adoption.md new file mode 100644 index 000000000..b49938113 --- /dev/null +++ b/.claude/handoffs/explore-flow-tool-adoption.md @@ -0,0 +1,70 @@ +--- +name: explore-flow-tool-adoption +date: 2026-05-24 00:55 +project: codegraph +branch: architectural-improvements +summary: Investigated why codegraph's read savings don't convert to wall-clock; root cause is agent tool-CHOICE (under-uses trace). Shipped a chain of fixes; the breakthrough is "explore-surfaces-flow" — the first mechanism to show up in real agent runs by adapting the tool the agent already uses. +--- + +# Handoff: codegraph retrieval — tool adoption & explore-surfaces-flow + +## Resume here — read this first +**Current state:** A long investigation into making agents answer flow questions faster with codegraph. 6 commits on `architectural-improvements` (all probe-validated, suite green 815). The breakthrough: **`codegraph_explore` now surfaces the execution flow** from the symbol-bag the agent already passes it (`PmsProductController getList PmsProductService list PmsProductServiceImpl` → leads output with `getList → service-interface → impl`, riding synth edges). It's the FIRST mechanism this whole arc to actually appear in real agent runs (spring-mall A/B: flow surfaced both runs, reads 2.0→1.5) — because it adapts the tool the agent USES instead of trying to make it use `trace`. + +**Immediate next step:** The user is weighing how to push tool-USE quality next (their open question). Decide between: (a) **extend explore-flow to surface more reliably** (spring-halo's query didn't name a connected co-named chain → no flow), (b) accept we're at the model-behavior ceiling and **wrap up**, or (c) the user's ideas — better tool-description *examples* (≈ steering, low-leverage per the evidence) or a *query-builder tool* (adds a call + new-tool adoption problem). My read: keep ADAPTING THE USED TOOL (the only thing that's worked); examples/new-tools are the "change the agent" direction that failed all session. + +> Suggested next message: "explore-flow only surfaced on 2 of 3 repos — dig into why spring-halo's explore query didn't produce a flow and make it surface more reliably" — OR — "we're at the model-behavior ceiling; let's stop and write the CHANGELOG/PR for this branch" + +## Goal +Make an AI agent answer **flow questions** ("how does X reach Y", request→handler→service, state→render) fast: ~0 Read/Grep, few codegraph calls, lower wall-clock. `codegraph_trace` is the fastest tool (1 call = the path), but the agent under-uses it. Ultimate target = trace's speed, however the agent gets there. + +## Key findings (the through-line) +- **The wall is agent tool-CHOICE, not the graph.** Matrix-wide, codegraph cuts reads −75% but wall-clock only −16% (`docs/benchmarks/codegraph-ab-matrix.md`). The floor is round-trips + the synthesis turn. The agent reliably calls `context`/`explore`, rarely `trace` (3/37 flow cells). Full analysis: `docs/benchmarks/call-sequence-analysis.md`. +- **Steering does NOT move it** (arms B/F/G, 3 wording variants): an MCP `initialize` instruction / tool description can't match a CLI `--append-system-prompt`'s salience, and forcing trace where it doesn't connect regresses. Reverted. +- **Sufficiency works** (committed): a self-sufficient `trace` (hop bodies + destination callees inlined) lets the unsteered agent stop — but only when it calls trace. +- **THE breakthrough — adapt the tool the agent uses.** `explore`'s query is a precise symbol-bag spanning the flow, so `explore` finds the call path AMONG its named symbols and leads with it. First mechanism to surface in real runs + drop reads. +- **What FAILED:** option 1 (context-surfaces-flow) — fuzzy DESCRIPTION can't disambiguate endpoints → confident WRONG-feature flow; reverted. trace multi-source-BFS over ambiguous names — same wrong-feature; reverted. + +## Gotchas +- **Co-naming disambiguation must match qualifiedName SEGMENTS, not substrings** (`buildFlowFromNamedSymbols` in `src/mcp/tools.ts`): `list` is a substring of `getList` → kept every getList. Split `qualifiedName` on `::`/`.` and match segments. +- **BFS must cap consecutive UNNAMED hops at 1** — full-graph BFS wanders a god-function's fan-out (excalidraw `render()` → pointer handlers → mutateElement). ≤1 bridge crosses a missing intermediate without wandering. +- **`getCallees` returns non-`calls` edges too** (references) — filter `c.edge.kind === 'calls'`. +- **Resolver/synthesizer changes need a CLEAN reindex**: `rm -rf .codegraph && codegraph init -i` (the init edge count is contains-only — query the DB for the real count). The explore-flow change is query-time (no reindex). +- **n=2 A/B is noisy** — report ranges/patterns, never conclude from one run. Foreground `sleep` is blocked → run A/B batches with `run_in_background`. +- Java/Kotlin `qualifiedName` is `Class::method` (so `matchesSymbol` resolves `Class.method` qualified trace endpoints — the agent already passes these). + +## How to test & validate +- Probe flow surfacing (no agent): `node scripts/agent-eval/probe-explore.mjs ""` → look for the `## Flow` section. `probe-trace.mjs ` for trace. +- Synthesizer: `sqlite3 /.codegraph/codegraph.db "select count(*) from edges where json_extract(metadata,'$.synthesizedBy')='interface-impl'"`; node count stable before/after reindex (synth adds edges only). +- Agent A/B (the real test): `bash scripts/agent-eval/run-arms.sh "" I ` (arm I = body-trace build, no steering). Parse via the `cmp2.mjs`-style scripts in `/tmp`. Pass = flow surfaces (`flowShown=Y`) + reads ≤ baseline. +- `npm test` (vitest, 815 pass); `__tests__/mcp-tool-allowlist.test.ts` covers the allowlist. + +## Repo state +- branch `architectural-improvements`, last commit `bafae81 feat(mcp): codegraph_explore surfaces the execution flow from its named symbols`. +- uncommitted: clean (only untracked `.claude/handoffs/`). +- 6 session commits: `eab5cf3` self-sufficient trace + `CODEGRAPH_MCP_TOOLS` allowlist · `a6183d7` research log + arms harness · `bde8c19` node/trace line numbers · `98baf41` Java/Kotlin interface→impl synthesizer · `6f3c468` playbook · `bafae81` explore-surfaces-flow. +- NOT pushed/merged. No version bump. CHANGELOG `[Unreleased]` has all of it. + +## Open threads / TODO +- [ ] **User's open question** (answer in the next turn): better tool-description *examples* vs a *query-builder tool* vs keep adapting the used tool. Evidence favors the last. +- [x] explore-flow reliability: now resolves QUALIFIED tokens (`Class.method`) — the agent's most precise input was being dropped by the file-ext strip (`2765c3c`). spring-halo's publish flow stays absent on purpose — it's **reactive/reconciler dispatch** (`publishPost` calls `ReactiveExtensionClient.get`/`awaitPostPublished`, not `PostService.publish`), so there's no static call chain. That's the next COVERAGE frontier (reactive runtimes — like MediatR, Vue Proxy), not an explore-flow bug. +- [ ] Ship-prep for the whole branch (this arc + the earlier framework sweep): CHANGELOG version block + `package.json` bump + PR to main. Releases go through `.github/workflows/release.yml` only — do NOT `npm publish`. +- [ ] Frontiers: MediatR (`_mediator.Send`→Handle) and Vue/Compose reactive runtimes are still unbridged dynamic dispatch. + +## Recent transcript (oldest → newest) +### Turn — "improve the A/B matrix; trace works, reads near 0 — what else?" +- Diagnosed: reads at floor, wall-clock floor = round-trips + synthesis. Built `seq-matrix.mjs`; found trace adoption 3/37. +### Turn — "do explore/context/trace compete? one tool?" +- Ablation arms A–E (`run-arms.sh`/`arms-F.sh` + `CODEGRAPH_MCP_TOOLS` allowlist). explore = 68% of payload, load-bearing; trace path-scoped but under-adopted; trace alone insufficient. +### Turn — "prototype body-inlining trace + A/B" +- Arm F: self-sufficient trace wins WITH append-prompt steering. But steering isn't a shippable channel. +### Turn — "port the steering + re-run" +- Arms G (3 variants) all regressed vs baseline; arm H (body-trace, no steer) ≈ baseline. Steering reverted; body-trace + line-numbers + allowlist committed. +### Turn — "tee up connectivity (Spring interface-DI)" +- Built `interfaceOverrideEdges` (Java/Kotlin interface→impl, overload-aware). Probe: 3-hop trace connects. But A/B null — agent never called trace. Committed (probe-validated, adoption-gated). +### Turn — "make context surface the flow (option 1)" +- Failed: fuzzy query → wrong-feature flows. Reverted. +### Turn — "change explore to do trace in the backend" +- WIN: explore's query is a precise symbol-bag. `buildFlowFromNamedSymbols` (co-naming segment match + ≤1 bridge). Probe perfect (Spring + excalidraw full chains); A/B: flow surfaces + modest read drop. Committed `bafae81`. +### Turn — "update memory + handoff; what about better examples / a query-builder tool?" +- This handoff + memory update. Strategic answer pending (adapt-the-tool > change-the-agent). diff --git a/.claude/handoffs/framework-coverage-sweep-2026-05-23.md b/.claude/handoffs/framework-coverage-sweep-2026-05-23.md new file mode 100644 index 000000000..3ba99a5e5 --- /dev/null +++ b/.claude/handoffs/framework-coverage-sweep-2026-05-23.md @@ -0,0 +1,70 @@ +--- +name: framework-coverage-sweep-2026-05-23 +date: 2026-05-23 23:59 +project: codegraph +branch: architectural-improvements +summary: Dynamic-dispatch coverage sweep COMPLETE — all 14 README frameworks + every flow-relevant language validated (measure→fix→validate→test→playbook→commit). ~37 commits pushed, suite green. Ship-prep (CHANGELOG + PR to main) is the only thing left. +--- + +# Handoff: Dynamic-dispatch framework/language coverage sweep (complete) + +## Resume here — read this first +**Current state:** The coverage sweep is **done**, AND a **frontier pass** closed the tractable partials. Every framework in the README's 14-row table is ✅, every flow-relevant language is validated (TS/JS, Python, Go, Java, C#, PHP, Ruby, Rust, Swift, Dart, Kotlin, Lua/Luau, Scala, C/C++), and the frontier pass added: React object data-router (literal), Next.js false-positive fix, Flask-RESTful `add_resource` (redash 6→77), Flask tuple methods + broader detection (flask-realworld 0→19), gorilla/mux confirmed. All committed/pushed to `architectural-improvements` (tree clean except untracked `.claude/handoffs/`). Full suite green (**809 passed**, 2 skipped; flaky `watcher.test.ts > debounced sync` passes on re-run). **No CHANGELOG entry exists, and the branch is not yet merged to main.** +**Immediate next step:** Ship-prep — write a CHANGELOG entry grouping the whole sweep (route resolution for Flask/FastAPI/Drupal/Rust-Axum+actix/Vapor/Spring-Kotlin/Play + React Router routing; the Python builtin-name guard, Dart method-range, and C++ inheritance foundational fixes; the flutter-build and cpp-override synthesizer channels), bump `package.json`, then open a PR to main. + +> Suggested next message: "do ship-prep: write the CHANGELOG entry covering the whole framework/language coverage sweep on this branch, bump the version, and open a PR to main" + +## Goal +Close static-extraction holes for **dynamic dispatch** across every language/framework codegraph supports, so cross-symbol flows (request→route→handler→service, state→render, virtual→override) exist in the graph and an agent answers flow questions with few codegraph calls and ~0 Read/Grep. Per framework/language: canonical flow `trace`s end-to-end, agent A/B shows fewer reads, no node explosion, recorded in `docs/design/dynamic-dispatch-coverage-playbook.md` (the matrix §6 + per-item notes §7). **This goal is now met; what remains is ship-prep + documented frontiers.** + +## Key findings (this session's work, all committed) +- **Routing convention is the hole in every backend** — same pattern each time: the resolver/extractor assumed one syntax. Flask (intervening `@login_required`/stacked routes), FastAPI (empty `""` path), Drupal (`claimsReference` for FQCN `_form`/single-colon controllers + contrib `detect` via composer name/type/`.info.yml`), Rust/Axum (chained `get(h).post(h2)` + namespaced `mod::handler`), actix (builder API `web::resource().route(web::get().to(h))`), Vapor (grouped `routes.grouped("x"); x.get(use:h)` — was 0 on every real app), Spring **Kotlin** (`fun` handler syntax + `.kt`), Play (extensionless `conf/routes` → controller), React Router (`` JSX). +- **Three FOUNDATIONAL fixes (broad benefit, not framework-specific):** (1) Python **bare-name builtin guard** in `src/resolution/index.ts` — a handler named `index`/`get`/`update` was filtered as a builtin method; mirror the dotted-branch `knownNames` guard. (2) **Dart method-range** in `src/extraction/tree-sitter.ts` `createNode` — Dart bodies are SIBLINGS of the signature, so methods were `end==start` (signature-only); extend `endLine` to the resolved body (guarded, child-body grammars no-op). (3) **C++ inheritance** — `extractInheritance` handled `base_clause` (PHP) but not C++ `base_class_clause`; added it (leveldb extends 219→298). +- **Two new synthesizer channels** in `src/resolution/callback-synthesizer.ts` (Dart analog + C++ analog of react-render): `flutter-build` (a State method calling `setState(` → `build`) and `cpp-override` (base virtual method → subclass override of same name, gated to C++). +- **measure-first repeatedly split "needs work" from "already covered":** Svelte, NestJS (prior), and this session **Lua/Luau** (module dispatch already resolves) + **Compose** (composition is plain function calls, already static) needed NO code. The assumed hole wasn't real. +- **`claimsReference` pre-filter is the recurring gotcha** (`src/resolution/index.ts:497-503`): a route ref naming no declared symbol (FQCN, `Controller@method`, `controller#action`, `Class.method`) is dropped before `framework.resolve()` runs. Added for Drupal + Play this session. + +## Gotchas +- **`claimsReference`:** if a new framework's route refs don't resolve despite a correct `resolve()`, it's the pre-filter — add `claimsReference`. +- **Reindex picks up resolver changes only on a CLEAN index:** `codegraph index` is incremental (skips unchanged files); after `npm run build`, do `rm -rf .codegraph && codegraph init -i` to re-extract. The init message's edge count is contains-only (~misleading); query the DB for the real count. +- **Extraction changes are high blast radius** (shared `createNode`/`extractInheritance`): re-check node counts on control repos (excalidraw 9,290 / django 302) — the Dart/C++ fixes are guarded to only-extend / C++-only, controls unchanged. +- **Play `conf/routes` is extensionless** → needed `isPlayRoutesFile` opt-in in `grammars.ts` (isSourceFile + detectLanguage→'yaml' no-grammar path). Narrow match, only ADDS Play files. +- **Flaky:** `watcher.test.ts > debounced sync > should trigger sync after file change` — timing-based, passes on re-run; unrelated to any of this work. +- **Foreground `sleep` is blocked** in Bash → background A/B batches (`run_in_background: true`), read the task output file. zsh quirks: quote globs (`'*.vue'`); SQL `count(*)` in `$(...)` needs care with quotes. +- Global `codegraph` is npm-linked to this repo's `dist/`; `npm run build` then reindex. A/B harness: `scripts/agent-eval/run-all.sh "" headless` (with vs empty MCP), parse via `node scripts/agent-eval/parse-run.mjs`. + +## How to test & validate (the per-framework loop) +- Corpus in `/tmp/codegraph-corpus/` (clone S/M/L, `git clone --depth 1`). Index: `rm -rf .codegraph && codegraph init -i`. +- Measure holes: `sqlite3 .codegraph/codegraph.db "select count(*) from nodes where kind='route'"` + route→handler edges (`join edges on source where kind='references'`). Node-count before/after (no explosion). +- Flow: `node scripts/agent-eval/probe-node.mjs ` (shows Called-by/Calls trail) / `probe-trace.mjs `. +- Agent A/B (≥2 runs/arm, variance is real): `run-all.sh` headless, record Read/Grep/duration/codegraph. Pass = fewer reads with codegraph. +- Tests: `npm test` (vitest). Resolver extract tests in `__tests__/frameworks.test.ts`; end-to-end in `__tests__/frameworks-integration.test.ts` (real CodeGraph + indexAll); Dart range in `__tests__/extraction.test.ts`; Drupal in `__tests__/drupal.test.ts`. + +## Repo state +- branch `architectural-improvements`, last commit `42a0178 docs(playbook): record frontier pass; test(go): gorilla/mux`. +- uncommitted: clean (only untracked `.claude/handoffs/`). +- ~37 commits total on the branch (handoff's original 11 frameworks + this session's: Flask/FastAPI, Drupal, Rust/Axum, Vapor, React Router, actix, Dart, Kotlin, Lua, Scala/Play, C/C++ — each a feat + a docs(playbook) commit; Lua was docs-only). + +## Open threads / TODO +- [ ] **SHIP-PREP (the only blocker to merge):** CHANGELOG entry for the whole sweep, `package.json` bump, PR to main. Releases go through `.github/workflows/release.yml` only — do NOT `npm publish` (see CLAUDE.md). +- [x] **Frontier pass DONE (commits 0456915, 03e49ab, 42a0178):** React object data-router (literal), Next.js false-positive fix, Flask-RESTful `add_resource`, Flask tuple methods + detection, gorilla/mux confirmed. +- [ ] **Frontiers LEFT (deliberately, with rationale in playbook §7 "Frontier pass"):** anonymous/inline closures (def-use frontier), metaprogramming finders (AR/Eloquent/JPA/EF), reactive runtimes (Vue Proxy / Compose recomposition), Akka actors, C callback-struct 422-way fan-out, C++ pure-virtual base methods, React lazy data-router (variable paths + lazy imports), Play SIRD, Nuxt-specific. Forcing these adds noise. +- [ ] Pre-existing, unrelated: Next.js `*.config.mjs` in a `pages/` dir treated as a route (false-positive found in bulletproof-react). + +## Recent transcript (oldest → newest, this session) +### Turn — "what's left / what's next on coverage" → did Flask/FastAPI +- 3 holes: Flask intervening/stacked decorators, FastAPI empty path, **Python bare-name builtin guard** (handlers named `index`/`get` filtered). microblog 6→27, realworld 12→20, dispatch 290/290. Fixed 6 stale Laravel/Rails tests too. Committed + pushed. +### Turn — "Drupal next" +- `claimsReference` for FQCN/_form/single-colon controllers + contrib `detect` (composer type/name + `.info.yml`). core 536→731 (87%), admin_toolbar 0→14. OOP `#[Hook]` = frontier. Committed. +### Turn — "Rust: Axum/actix/Rocket" +- Axum chained methods + namespaced handlers (realworld 12→19, 19/19); Rocket already 99%; **actix builder API** `web::resource().route(web::get().to())` (examples 51→128). Committed (2 commits: axum, then actix). +### Turn — "Vapor (Swift)" +- Resolver was 0-routes on every real app; rewrote for any receiver + optional non-string paths + `.grouped` prefix tracking + `use:` discriminator. template 0→3, SteamPress 0→27, SPI 0→14. Committed. +### Turn — "2, 3, 4" (React Router, actix [done above], Dart/Flutter) +- React Router `` JSX (react-realworld 0→10). Dart/Flutter: **method-range fix** (foundational) + `flutter-build` setState→build synthesizer. Committed. +### Turn — "Kotlin next" +- Spring resolver `['java']`→`['java','kotlin']` + `fun` handler regex (petclinic-kotlin 0→18, 18/18; Java unchanged 19/19). Compose composition already static. Committed. +### Turn — "Lua/Luau, Scala, C/C++ (Lua first, but do all three)" +- **Lua:** measure-first → module dispatch already covered (telescope 335 cross-file calls); no code change, validated. **Scala/Play:** `conf/routes` file-walk opt-in + Play resolver (computer-database 0→8). **C/C++:** general dispatch strong (redis 29k); fixed C++ `base_class_clause` inheritance + `cpp-override` synthesizer (leveldb 12 precise). All committed + pushed. +### Turn — "wrap up + refresh handoff" +- This handoff. Sweep complete; ship-prep (CHANGELOG + PR) is the remaining work. diff --git a/.claude/skills/add-lang/SKILL.md b/.claude/skills/add-lang/SKILL.md new file mode 100644 index 000000000..37cbdce55 --- /dev/null +++ b/.claude/skills/add-lang/SKILL.md @@ -0,0 +1,219 @@ +--- +name: add-lang +description: Add tree-sitter language support to codegraph end-to-end — wire the grammar + extractor, write tests, then benchmark extraction quality and retrieval value on 3 popular real-world repos. Use when the user runs /add-lang or asks to add/support a new language (e.g. Lua, Elixir, Zig, OCaml) in codegraph. +--- + +# Add a language to CodeGraph + +Wire a new tree-sitter language into codegraph's extraction pipeline, prove it +extracts real symbols on popular repos, and prove it beats no-codegraph for an +agent. Runs **fully autonomously** — pick repos, benchmark, update docs, then +report. **Never commit, push, publish, or tag** (house rule); leave all changes +for the user to review. + +The argument is the language token used throughout the `Language` union, e.g. +`lua`, `elixir`, `zig`. If none was given, ask which language. Use the lowercase +single-token form everywhere (`csharp`, not `c#`). + +## Prerequisites +- Run from the codegraph repo root. `node`, `git`, `gh`, and a logged-in + `claude` CLI (the benchmark spawns real `claude -p` runs). +- The benchmark uses the local dev build — Step 8 builds + links it on PATH. + +## Workflow + +Copy this checklist and work through it in order: +``` +- [ ] 1. Resolve language; bail early if already supported (just benchmark) +- [ ] 2. Find a grammar + health-check it (ABI / heap corruption) +- [ ] 3. Discover the grammar's AST node types (dump-ast.mjs) +- [ ] 4. Wire the language (4 files; sometimes a 5th core touch) +- [ ] 5. Build + verify-extraction loop until PASS +- [ ] 6. Add extraction tests; make them green +- [ ] 7. Auto-pick 3 popular repos by size tier; add to corpus.json +- [ ] 8. Benchmark all 3: extraction + with/without A/B +- [ ] 9. Update README + CHANGELOG +- [ ] 10. Report; do NOT commit +``` + +### Step 1 — Resolve + short-circuit + +Check whether the language is already wired: look for the token in the +`LANGUAGES` const (`src/types.ts`) and the `EXTRACTORS` map +(`src/extraction/languages/index.ts`). If it is already supported (e.g. +`typescript`, `rust`), **skip Steps 2–6** and go straight to benchmarking +(Steps 7–8) to validate/measure it — note in the report that no code changed. + +### Step 2 — Find a grammar, then health-check it + +```bash +ls node_modules/tree-sitter-wasms/out/ | grep -i # csharp -> c_sharp +``` +- **Present** → likely off-the-shelf; `grammars.ts` resolves it from + `tree-sitter-wasms` automatically. (Many languages: elixir, zig, ocaml, + solidity, toml, yaml, …) +- **Absent** → vendor a `.wasm` into `src/extraction/wasm/` (like `pascal` / + `scala` / `lua`) and add the token to the vendored branch in Step 4. + +**Always health-check before writing an extractor — a *present* grammar can +still be unusable:** +```bash +node scripts/add-lang/check-grammar.mjs path/to/valid-sample. +``` +It prints the grammar's ABI version and parses a valid sample many times in a +multi-grammar runtime. If it **FAILs** (ERROR trees on valid code — an old ABI +corrupting the shared WASM heap, which silently drops nested calls/imports on +every file after the first; e.g. the tree-sitter-wasms **Lua** grammar is ABI 13 +and fails), do NOT use that wasm. **Vendor a newer (ABI 14/15) build instead:** +```bash +npm pack @tree-sitter-grammars/tree-sitter- # often ships a prebuilt *.wasm +# or build one: npx tree-sitter build --wasm (needs Docker/emscripten) +cp .wasm src/extraction/wasm/tree-sitter-.wasm +``` +then add the token to the vendored branch in Step 4 and re-run check-grammar on +the vendored path until it PASSes. **If you cannot obtain a healthy wasm, STOP +and tell the user.** + +### Step 3 — Discover AST node types + +Get a representative source file (write a small sample covering functions, +classes/structs, imports, enums; or `curl` a raw file from a known repo), then: +```bash +node scripts/add-lang/dump-ast.mjs path/to/sample. +# vendored grammar: pass the wasm path instead of the token +node scripts/add-lang/dump-ast.mjs src/extraction/wasm/tree-sitter-.wasm sample. +``` +The frequency table + field names (`name:`, `parameters:`, `body:`, +`return_type:`) tell you what to map. Open the existing extractor closest to the +language's paradigm as a model: `rust.ts`/`scala.ts` (functional, traits), +`java.ts`/`csharp.ts` (OO), `python.ts`/`ruby.ts` (scripting), `go.ts` +(top-level methods + receivers). + +### Step 4 — Wire the language (4 files) + +These are exact, fragile wiring — match the existing style precisely: + +1. **`src/types.ts`** — TWO edits: + - add `'',` to the `LANGUAGES` const (before `'unknown'`); + - add `'**/*.',` to `DEFAULT_CONFIG.include`. **Don't skip this** — it's + the file-scan allowlist; without the glob, `codegraph init` finds **0 + files** even though detection/extraction are wired. +2. **`src/extraction/grammars.ts`** — three maps: + - `WASM_GRAMMAR_FILES`: `: 'tree-sitter-.wasm',` + - `EXTENSION_MAP`: each file extension → `''` (e.g. `'.lua': 'lua',`) + - `getLanguageDisplayName`: `: '',` + - **vendored only**: add `` to the + `(lang === 'pascal' || lang === 'scala' || …)` wasm-path branch. +3. **`src/extraction/languages/.ts`** — new file exporting + `export const Extractor: LanguageExtractor = { … }`. Map the node types + from Step 3. Required fields: `functionTypes`, `classTypes`, `methodTypes`, + `interfaceTypes`, `structTypes`, `enumTypes`, `typeAliasTypes`, + `importTypes`, `callTypes`, `variableTypes`, `nameField`, `bodyField`, + `paramsField`. Add hooks as the grammar needs them (`getSignature`, + `getVisibility`, `isExported`, `extractImport`, `visitNode`, `getReceiverType`, + `interfaceKind`, `enumMemberTypes`, etc. — see + `src/extraction/tree-sitter-types.ts`). +4. **`src/extraction/languages/index.ts`** — `import { Extractor } from + './';` and add `: Extractor,` to `EXTRACTORS`. + +**Sometimes a 5th, core touch in `src/extraction/tree-sitter.ts`** — variable +extraction has per-language branches in `extractVariable` (the generic fallback +only finds direct `identifier`/`variable_declarator` children). If the grammar +nests declared names (e.g. Lua's `variable_declaration → variable_list`), add a +`} else if (this.language === '')` branch there, mirroring the existing +ts/python/go ones. Import forms that aren't a distinct node (Lua/Ruby `require` +is a *call*) are handled in the extractor's `visitNode` hook instead. + +### Step 5 — Build + verify loop + +```bash +npm run build # tsc + copy-assets (copies any vendored *.wasm into dist/) +``` +Index a small sample repo and check extraction: +```bash +( cd && codegraph init -i ) +node scripts/add-lang/verify-extraction.mjs +``` +`verify-extraction.mjs` fails (exit 1) if the language isn't detected or only +`file`/`import` nodes were produced — the classic symptom of wrong node-type +names. On FAIL or a thin WARN: re-run `dump-ast.mjs` on a richer file, fix the +mappings in `.ts`, `npm run build`, re-index, re-verify. **Repeat until +PASS.** + +### Step 6 — Tests + +Add to `__tests__/extraction.test.ts`, modeled on the `Rust Extraction` block: +- a `detectLanguage` assertion in `describe('Language Detection')` +- a `describe(' Extraction')` block asserting functions/classes/imports + are extracted from an inline source string. +```bash +npx vitest run __tests__/extraction.test.ts +``` +Green before continuing. + +### Step 7 — Auto-pick 3 repos + corpus + +Pick **without asking**. Find candidates, then curate 3 that are genuinely +``-dominant, one per size tier: +```bash +gh search repos --language= --sort=stars --limit 40 \ + --json fullName,stargazerCount,description +``` +Tiers (match `corpus.json`): **Small** <~150 files · **Medium** ~150–1500 · +**Large** >~1500. Skip repos that are tagged `` but mostly another +language. Write one cross-file architecture **question** per repo (the kind that +needs tracing across files). Add a `""` block to +`.claude/skills/agent-eval/corpus.json` (fields: `name`, `repo`, `size`, +`files`, `question`) so `/agent-eval` can reuse them. + +### Step 8 — Benchmark all 3 (extraction + A/B) + +Make the dev build the codegraph on PATH **once**, then loop: +```bash +npm run build && ./scripts/local-install.sh +scripts/add-lang/bench.sh "" headless # ×3 +``` +`bench.sh` clones (shared `/tmp/codegraph-corpus`), wipes + indexes, runs +`verify-extraction.mjs`, then the with/without retrieval A/B via +`scripts/agent-eval/run-all.sh` (skips the paid A/B if extraction is broken). +Read each `parse-run.mjs` summary printed by `run-all.sh`: tool calls, file +`Read`s, Grep/Bash, codegraph-tool calls, duration, and **cost** — for both the +`with` and `without` arms. After the loop, restore the dev link if needed: +`./scripts/local-install.sh`. + +### Step 9 — Docs + CHANGELOG + +- **README.md**: add `` to the "19+ Languages" feature bullet, and add a + row to the **Supported Languages** table: + `| | \`.ext\` | Full support (classes, methods, …) |`. +- **CHANGELOG.md**: add an `## [Unreleased]` section at the top (above the + latest version) with `### Added` → a user-perspective bullet, e.g. + *"CodeGraph now indexes **** (`.ext`) — functions, classes, imports, and + call edges."* If `## [Unreleased]` already exists, append under it. (It's + folded into the next versioned block at release time.) + +### Step 10 — Report (do NOT commit) + +Summarize for review: +- **Files changed**: the 4 wiring edits + new extractor + tests + README + + CHANGELOG + corpus.json (+ any vendored `.wasm`). +- **Extraction** per repo: files / nodes / edges / `verify-extraction` result. +- **A/B** per repo: `with` vs `without` (tool calls, file Reads, cost) and a + one-line verdict — did codegraph reduce effort, and did both arms reach a + correct answer? +- **Gaps / follow-ups** (node types not yet mapped, resolution edges missing, + framework routes, etc.). + +Hand the changes to the user. **Do not** run `git commit`/`push` or publish — +releases go through the GitHub Actions Release workflow. + +## Notes +- The A/B spawns real **paid** `claude -p` runs (opus, `--max-budget-usd`), + 2 arms × 3 repos. The corpus dir `/tmp/codegraph-corpus` is shared with + `/agent-eval`, so clones are reused across runs. +- Any new `*.wasm` must live in `src/extraction/wasm/` — `copy-assets` (run by + `npm run build`) ships it; otherwise it won't be in `dist/`. +- An index must be served by the **same** binary that built it. Step 8 builds + + links the dev build first, so this holds. +- If a grammar can't be obtained, or extraction can't reach PASS, **STOP and + report** — don't ship a half-wired language. diff --git a/.claude/skills/agent-eval/SKILL.md b/.claude/skills/agent-eval/SKILL.md new file mode 100644 index 000000000..2e894a753 --- /dev/null +++ b/.claude/skills/agent-eval/SKILL.md @@ -0,0 +1,74 @@ +--- +name: agent-eval +description: Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a codegraph version (the local dev build or a published npm version) against a language's repo. +--- + +# CodeGraph Quality Audit + +Measures how much CodeGraph helps an agent versus plain grep/read, for a chosen +codegraph version on a chosen real-world repo. Drives the harness in +`scripts/agent-eval/`. + +## Prerequisites +- `tmux` 3+, a logged-in `claude` CLI, `node`, `git` (macOS/Linux). +- Run from the codegraph repo root. + +## Workflow + +Copy this checklist: +``` +- [ ] 1. Pick version (local or npm) +- [ ] 2. Pick language +- [ ] 3. Pick repo by size +- [ ] 4. Pick harness (headless / tmux / both) +- [ ] 5. Run audit.sh in the background +- [ ] 6. Report results +``` + +**Step 1 — version.** Ask with `AskUserQuestion`: which codegraph version to test. +Offer "Local dev build" and "Latest published"; the free-text "Other" lets the +user type a specific version (e.g. `0.7.10`). Map the answer to a VERSION token: +- "Local dev build" → `local` +- "Latest published" → `latest` +- a typed version → that string (e.g. `0.7.10`) + +**Step 2 — language.** Read `.claude/skills/agent-eval/corpus.json`. Ask with +`AskUserQuestion` which language to test, listing the languages that have entries. + +**Step 3 — repo.** From the chosen language's entries, ask which repo. Label each +option with its size and file count, e.g. `excalidraw — Medium (~600 files)`. +Each entry carries the `repo` URL and a representative `question`. + +**Step 4 — harness.** Ask with `AskUserQuestion` which harness to run, and map +the answer to a MODE token: +- "Headless" → `headless` — `claude -p` with stream-json: exact tokens/cost and a + clean tool sequence (2 runs, fast, no TTY). +- "Interactive (tmux)" → `tmux` — drives the real Claude TUI in tmux: faithful + Explore-subagent behavior, metrics from session logs (2 runs, slower). +- "Both" → `all` — headless + interactive (4 runs). + +**Step 5 — run.** Launch in the background (sets the version, clones if missing, +wipes + re-indexes, runs the chosen arms — several minutes): +```bash +scripts/agent-eval/audit.sh "" +``` + +**Step 6 — report.** When the job finishes, read the log and report per arm: +- Headless (`parse-run.mjs`): total tool calls, file `Read`s, Grep/Bash, + codegraph-tool calls, duration, **total cost**. +- Interactive (`parse-session.mjs`): the `VERDICT: codegraph_explore used Nx | + Read N | Grep/Bash N` and `TOKENS:` lines. + +Lead with cost + tool/Read counts — they are the reliable signals; raw token +in/out are confounded by subagent delegation and prompt caching. State whether +codegraph reduced effort and whether both arms reached a correct answer. + +## Notes +- The index is rebuilt every run (`audit.sh` wipes `.codegraph`) — different + versions extract differently, so an index must be served by the same binary + that built it. +- `audit.sh` temporarily mutates the global `codegraph` install for the test, + then restores your dev link via `local-install.sh`. +- Corpus repos are cloned to `/tmp/codegraph-corpus` (reused if already present). +- Add or edit repos in `corpus.json` (fields: `name`, `repo`, `size`, `files`, + `question`). diff --git a/.claude/skills/agent-eval/corpus.json b/.claude/skills/agent-eval/corpus.json new file mode 100644 index 000000000..2cfedac4f --- /dev/null +++ b/.claude/skills/agent-eval/corpus.json @@ -0,0 +1,98 @@ +{ + "_comment": "Test corpus for /agent-eval. Add entries freely. size: Small (<~150 files), Medium (~150-1500), Large (>~1500). 'question' is a representative architectural question that exercises cross-file understanding.", + "TypeScript": [ + { "name": "ky", "repo": "https://github.com/sindresorhus/ky", "size": "Small", "files": "~25", "question": "How does ky implement request retries and timeouts?" }, + { "name": "excalidraw", "repo": "https://github.com/excalidraw/excalidraw", "size": "Medium", "files": "~600", "question": "How does Excalidraw render and update canvas elements?" }, + { "name": "vscode", "repo": "https://github.com/microsoft/vscode", "size": "Large", "files": "~10000", "question": "How does the extension host communicate with the main process?" } + ], + "JavaScript": [ + { "name": "express", "repo": "https://github.com/expressjs/express", "size": "Small", "files": "~50", "question": "How does Express route a request through its middleware stack?" } + ], + "Go": [ + { "name": "cobra", "repo": "https://github.com/spf13/cobra", "size": "Small", "files": "~50", "question": "How does cobra parse commands and flags?" }, + { "name": "gin", "repo": "https://github.com/gin-gonic/gin", "size": "Medium", "files": "~150", "question": "How does gin route requests through its middleware chain?" }, + { "name": "terraform", "repo": "https://github.com/hashicorp/terraform", "size": "Large", "files": "~4000", "question": "How does Terraform build and walk the resource dependency graph?" }, + { "name": "cosmos-sdk", "repo": "https://github.com/cosmos/cosmos-sdk", "size": "Large", "files": "~5000", "question": "How does a bank module MsgSend message reach the account balance update? Trace the cross-module call path from the bank keeper's Send handler through to the account/balance store update." } + ], + "Python": [ + { "name": "click", "repo": "https://github.com/pallets/click", "size": "Small", "files": "~60", "question": "How does click parse command-line arguments into commands?" }, + { "name": "flask", "repo": "https://github.com/pallets/flask", "size": "Medium", "files": "~90", "question": "How does Flask dispatch a request to a view function?" }, + { "name": "django", "repo": "https://github.com/django/django", "size": "Large", "files": "~2700", "question": "How does Django's ORM build and execute a query from a QuerySet?" } + ], + "Rust": [ + { "name": "clap", "repo": "https://github.com/clap-rs/clap", "size": "Medium", "files": "~200", "question": "How does clap parse arguments against a derived command definition?" }, + { "name": "tokio", "repo": "https://github.com/tokio-rs/tokio", "size": "Large", "files": "~700", "question": "How does tokio schedule and run async tasks on its runtime?" }, + { "name": "deno", "repo": "https://github.com/denoland/deno", "size": "Large", "files": "~1500", "question": "How does Deno load and execute a TypeScript module?" } + ], + "Java": [ + { "name": "gson", "repo": "https://github.com/google/gson", "size": "Medium", "files": "~200", "question": "How does Gson serialize an object to JSON?" }, + { "name": "okhttp", "repo": "https://github.com/square/okhttp", "size": "Medium", "files": "~640", "question": "How does OkHttp process a request through its interceptor chain?" }, + { "name": "guava", "repo": "https://github.com/google/guava", "size": "Large", "files": "~3000", "question": "How does Guava's CacheBuilder build and configure a cache?" } + ], + "Kotlin": [ + { "name": "koin", "repo": "https://github.com/InsertKoinIO/koin", "size": "Medium", "files": "~300", "question": "How does Koin resolve and inject dependencies?" }, + { "name": "leakcanary", "repo": "https://github.com/square/leakcanary", "size": "Medium", "files": "~250", "question": "How does LeakCanary detect and analyze a memory leak?" } + ], + "Swift": [ + { "name": "alamofire", "repo": "https://github.com/Alamofire/Alamofire", "size": "Small", "files": "~100", "question": "How does Alamofire build, send, and validate a request?" } + ], + "C#": [ + { "name": "serilog", "repo": "https://github.com/serilog/serilog", "size": "Medium", "files": "~250", "question": "How does Serilog route a log event to its sinks?" }, + { "name": "jellyfin", "repo": "https://github.com/jellyfin/jellyfin", "size": "Large", "files": "~2500", "question": "How does Jellyfin scan and identify items in a media library?" } + ], + "Ruby": [ + { "name": "sinatra", "repo": "https://github.com/sinatra/sinatra", "size": "Small", "files": "~60", "question": "How does Sinatra match a request to a route handler?" }, + { "name": "discourse", "repo": "https://github.com/discourse/discourse", "size": "Large", "files": "~3000", "question": "How does Discourse create and render a new post?" } + ], + "PHP": [ + { "name": "slim", "repo": "https://github.com/slimphp/Slim", "size": "Small", "files": "~80", "question": "How does Slim handle a request through its middleware?" }, + { "name": "laravel", "repo": "https://github.com/laravel/framework", "size": "Large", "files": "~3000", "question": "How does Laravel resolve and dispatch a route to a controller?" } + ], + "C": [ + { "name": "redis", "repo": "https://github.com/redis/redis", "size": "Large", "files": "~600", "question": "How does Redis parse and dispatch a client command?" } + ], + "C++": [ + { "name": "json", "repo": "https://github.com/nlohmann/json", "size": "Small", "files": "~100", "question": "How does nlohmann::json parse a JSON string into a value?" }, + { "name": "grpc", "repo": "https://github.com/grpc/grpc", "size": "Large", "files": "~3000", "question": "How does gRPC dispatch an incoming RPC to its handler?" } + ], + "Dart": [ + { "name": "flutter", "repo": "https://github.com/flutter/flutter", "size": "Large", "files": "~6000", "question": "How does Flutter build and lay out a widget tree?" } + ], + "Svelte": [ + { "name": "shadcn-svelte", "repo": "https://github.com/huntabyte/shadcn-svelte", "size": "Medium", "files": "~600", "question": "How do shadcn-svelte components compose and apply their styling?" } + ], + "Lua": [ + { "name": "lualine.nvim", "repo": "https://github.com/nvim-lualine/lualine.nvim", "size": "Small", "files": "~120", "question": "How does lualine assemble and render its statusline sections and components?" }, + { "name": "telescope.nvim", "repo": "https://github.com/nvim-telescope/telescope.nvim", "size": "Medium", "files": "~80", "question": "How does Telescope wire a picker to its finder, sorter, and previewer?" }, + { "name": "kong", "repo": "https://github.com/Kong/kong", "size": "Large", "files": "~1330", "question": "How does Kong execute plugins across a request's lifecycle phases?" } + ], + "Luau": [ + { "name": "Knit", "repo": "https://github.com/Sleitnick/Knit", "size": "Small", "files": "~10", "question": "How does Knit register services and expose them to clients?" }, + { "name": "vide", "repo": "https://github.com/centau/vide", "size": "Small", "files": "~40", "question": "How does vide track reactive sources and re-run effects when state changes?" }, + { "name": "Fusion", "repo": "https://github.com/dphfox/Fusion", "size": "Medium", "files": "~115", "question": "How does Fusion build and update its reactive UI graph from state objects?" } + ], + "Objective-C": [ + { "name": "Masonry", "repo": "https://github.com/SnapKit/Masonry", "size": "Small", "files": "~50", "question": "How does Masonry build and activate Auto Layout constraints from its block DSL?" }, + { "name": "FMDB", "repo": "https://github.com/ccgus/fmdb", "size": "Medium", "files": "~80", "question": "How does FMDB execute a prepared SQL statement and bind parameters?" }, + { "name": "SDWebImage", "repo": "https://github.com/SDWebImage/SDWebImage", "size": "Large", "files": "~400", "question": "How does SDWebImage download, cache, and decode an image for a UIImageView?" } + ], + "Mixed iOS (Swift+ObjC)": [ + { "name": "Charts", "repo": "https://github.com/danielgindi/Charts", "size": "Small", "files": "~270", "question": "How does the ChartsDemo ObjC demo controller drive the Swift Charts library to animate and notify a data update?" }, + { "name": "realm-swift", "repo": "https://github.com/realm/realm-swift", "size": "Medium", "files": "~370", "question": "How does a Swift `Realm.write { realm.add(obj) }` reach the Objective-C persistence layer?" }, + { "name": "wikipedia-ios", "repo": "https://github.com/wikimedia/wikipedia-ios", "size": "Large", "files": "~1700", "question": "How does tapping a search result reach the article-fetch network call across the Swift / ObjC boundary?" } + ], + "React Native (legacy bridge + TurboModule)": [ + { "name": "@react-native-async-storage", "repo": "https://github.com/react-native-async-storage/async-storage", "size": "Small", "files": "~60", "question": "How does `setItem` in JS reach the native `legacy_multiSet` implementation?" }, + { "name": "react-native-svg", "repo": "https://github.com/software-mansion/react-native-svg", "size": "Medium", "files": "~700", "question": "How does a JS `Svg.getTotalLength(...)` reach the iOS / Android native implementation via TurboModule?" }, + { "name": "react-native-firebase", "repo": "https://github.com/invertase/react-native-firebase", "size": "Large", "files": "~1100", "question": "How does a native iOS push notification reach the JS `messaging().onMessage(...)` listener?" } + ], + "Expo Modules": [ + { "name": "expo-haptics", "repo": "https://github.com/expo/expo/tree/main/packages/expo-haptics", "size": "Small", "files": "~15", "question": "How does `Haptics.notificationAsync(...)` in JS reach `UINotificationFeedbackGenerator` in the Swift Module?" }, + { "name": "expo-camera", "repo": "https://github.com/expo/expo/tree/main/packages/expo-camera", "size": "Medium", "files": "~70", "question": "How does a JS `CameraView.takePictureAsync(options)` reach the native AVCaptureSession / CameraDevice call?" } + ], + "React Native Fabric (view components)": [ + { "name": "react-native-segmented-control", "repo": "https://github.com/react-native-segmented-control/segmented-control", "size": "Small", "files": "~25", "question": "How does JSX `` reach the native onChange handler on iOS/Android?" }, + { "name": "react-native-screens", "repo": "https://github.com/software-mansion/react-native-screens", "size": "Medium", "files": "~1200", "question": "How does JSX `` reach the native RNSScreenStackView component?" }, + { "name": "react-native-skia", "repo": "https://github.com/Shopify/react-native-skia", "size": "Large", "files": "~1000", "question": "How does a `` JSX usage reach the iOS / Android native renderer?" } + ] +} diff --git a/.cursor/rules/codegraph.mdc b/.cursor/rules/codegraph.mdc new file mode 100644 index 000000000..00a3f8152 --- /dev/null +++ b/.cursor/rules/codegraph.mdc @@ -0,0 +1,39 @@ +--- +description: CodeGraph MCP usage guide — when to use which tool +alwaysApply: true +--- + +## CodeGraph + +This project has a CodeGraph MCP server (`codegraph_*` tools) configured. CodeGraph is a tree-sitter-parsed knowledge graph of every symbol, edge, and file. Reads are sub-millisecond and return structural information grep cannot. + +### When to prefer codegraph over native search + +Use codegraph for **structural** questions — what calls what, what would break, where is X defined, what is X's signature. Use native grep/read only for **literal text** queries (string contents, comments, log messages) or after you already have a specific file open. + +| Question | Tool | +|---|---| +| "Where is X defined?" / "Find symbol named X" | `codegraph_search` | +| "What calls function Y?" | `codegraph_callers` | +| "What does Y call?" | `codegraph_callees` | +| "How does X reach/become Y? / trace the flow from X to Y" | `codegraph_trace` (one call = the whole path, incl. callback/React/JSX dynamic hops) | +| "What would break if I changed Z?" | `codegraph_impact` | +| "Show me Y's signature / source / docstring" | `codegraph_node` | +| "Give me focused context for a task/area" | `codegraph_context` | +| "See several related symbols' source at once" | `codegraph_explore` | +| "What files exist under path/" | `codegraph_files` | +| "Is the index healthy?" | `codegraph_status` | + +### Rules of thumb + +- **Answer directly — don't delegate exploration.** For "how does X work" / architecture questions, answer with 2-3 codegraph calls: `codegraph_context` first, then ONE `codegraph_explore` for the source of the symbols it surfaces. For a specific **flow** ("how does X reach Y") start with `codegraph_trace` from→to — one call returns the whole path with dynamic hops bridged — then ONE `codegraph_explore` for the bodies; don't rebuild the path with `codegraph_search` + `codegraph_callers`. Codegraph IS the pre-built index, so spawning a separate file-reading sub-task/agent — or running a grep + read loop — repeats work codegraph already did and costs more for the same answer. +- **Trust codegraph results.** They come from a full AST parse. Do NOT re-verify them with grep — that's slower, less accurate, and wastes context. +- **Don't grep first** when looking up a symbol by name. `codegraph_search` is faster and returns kind + location + signature in one call. +- **Don't chain `codegraph_search` + `codegraph_node`** when you just want context — `codegraph_context` is one call. +- **Don't loop `codegraph_node` over many symbols** — one `codegraph_explore` call returns several symbols' source grouped in a single capped call, while each separate node/Read call re-reads the whole context and costs far more. +- **Index lag — check the staleness banner, don't guess a wait.** When a codegraph response starts with "⚠️ Some files referenced below were edited since the last index sync…", the listed files are pending re-index — Read those specific files for accurate content. Files NOT in that banner are fresh and codegraph is authoritative for them. `codegraph_status` also lists pending files under "Pending sync". + +### If `.codegraph/` doesn't exist + +The MCP server returns "not initialized." Ask the user: *"I notice this project doesn't have CodeGraph initialized. Want me to run `codegraph init -i` to build the index?"* + diff --git a/.github/workflows/deploy-site.yml b/.github/workflows/deploy-site.yml new file mode 100644 index 000000000..b66dde951 --- /dev/null +++ b/.github/workflows/deploy-site.yml @@ -0,0 +1,43 @@ +name: Deploy site to GitHub Pages + +on: + push: + branches: [main] + paths: + - 'site/**' + - '.github/workflows/deploy-site.yml' + workflow_dispatch: + +# Allow GITHUB_TOKEN to deploy to Pages and verify the deployment origin. +permissions: + contents: read + pages: write + id-token: write + +# One deploy at a time; let an in-progress run finish. +concurrency: + group: pages + cancel-in-progress: false + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Build with Astro + uses: withastro/action@v3 + with: + path: site + node-version: 22 + + deploy: + needs: build + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 000000000..88bce26a4 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,204 @@ +name: Release + +# Manually triggered ("Run workflow"). On trigger it: +# 1. reads the version from package.json, +# 2. promotes `## [Unreleased]` content into `## []` in +# CHANGELOG.md (and commits + pushes that change back to main), so +# the published release notes are never sparse just because the +# maintainer didn't pre-stage the [] block by hand, +# 3. builds a self-contained bundle for every platform (one runner — there's no +# native compilation, so cross-packaging is fine), +# 4. creates the GitHub Release (tag v) with all archives, using the +# release notes from CHANGELOG.md, +# 5. publishes the npm thin-installer (shim + per-platform packages). +# +# Before triggering: bump package.json. CHANGELOG.md entries can live under +# `## [Unreleased]` — step 2 takes care of moving them. Set the NPM_TOKEN secret. +on: + workflow_dispatch: {} + +permissions: + contents: write # create the GitHub Release + tag, push the CHANGELOG promote + +jobs: + release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + with: + # Default checkout is detached at a SHA; we need an actual branch + # so the CHANGELOG-promote commit knows where to push. + ref: ${{ github.ref }} + # Authenticate as the maintainer (admin), not as github-actions[bot]. + # The "Require PR approval for main branch" ruleset only lets the + # Admin repo role bypass — and GitHub blocks adding the GitHub + # Actions integration to bypass_actors on user-owned (non-org) + # repos with "Actor GitHub Actions integration must be part of + # the ruleset source or owner organization." So the auto-promote + # and auto-sync `git push origin HEAD:main` steps below both fail + # under the default GITHUB_TOKEN. Using a fine-grained PAT owned + # by the admin makes the push go through cleanly. Set the + # RELEASE_PAT secret with: contents:write on this repo, no other + # scopes. Rotate per your token policy; the workflow only runs + # on manual dispatch so the blast radius is small. + token: ${{ secrets.RELEASE_PAT }} + - uses: actions/setup-node@v6 + with: + node-version: 22 + registry-url: https://registry.npmjs.org + + - name: Sync package-lock.json if version drifted + # When the maintainer bumps the version on package.json only — for + # example via a GitHub web-UI edit — `npm ci` would refuse to run + # with `EUSAGE: npm ci can only install packages when your + # package.json and package-lock.json … are in sync`. This step + # rewrites just the lock-file's version fields (top-level + the + # `packages.""` entry) to match package.json, then auto-commits + # and pushes the result so on-disk truth on `main` stays + # consistent. Idempotent: if the lock file already matches, no + # commit is made. + run: | + set -euo pipefail + PKG_V=$(node -p "require('./package.json').version") + LOCK_V=$(node -p "require('./package-lock.json').version") + if [ "$PKG_V" = "$LOCK_V" ]; then + echo "package-lock.json already at $PKG_V — nothing to sync." + exit 0 + fi + echo "Lock-file version drift: lock=$LOCK_V, package=$PKG_V. Syncing." + # `--package-lock-only` rewrites only the lock file, doesn't + # touch node_modules or actually install anything. Cheap. + npm install --package-lock-only --ignore-scripts + # Sanity: lockfile should now report the package version. + NEW_LOCK_V=$(node -p "require('./package-lock.json').version") + if [ "$NEW_LOCK_V" != "$PKG_V" ]; then + echo "::error::lock-file still at $NEW_LOCK_V after sync attempt; expected $PKG_V"; exit 1 + fi + if git diff --quiet -- package-lock.json; then + echo "lock file unchanged after sync? bailing"; exit 1 + fi + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add package-lock.json + git commit -m "release: sync package-lock.json to ${PKG_V}" -m "[skip ci] Auto-generated by Release workflow." + git push origin "HEAD:${GITHUB_REF#refs/heads/}" + + - run: npm ci + - name: Ensure zip/unzip + run: sudo apt-get update -qq && sudo apt-get install -y -qq zip unzip + + - name: Resolve version + id: ver + run: echo "version=$(node -p "require('./package.json').version")" >> "$GITHUB_OUTPUT" + + - name: Promote [Unreleased] → [] in CHANGELOG.md + # Idempotent: a no-op if [Unreleased] is empty OR if the previous + # run already moved everything. Auto-commit + push the change back + # so the version block on main is the source of truth going + # forward (and so subsequent extract-release-notes.mjs calls + # surface the full content even if this run is re-triggered). + run: | + set -euo pipefail + V="${{ steps.ver.outputs.version }}" + before=$(git rev-parse HEAD) + node scripts/prepare-release.mjs "$V" + if git diff --quiet -- CHANGELOG.md; then + echo "CHANGELOG.md unchanged — nothing to commit." + else + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add CHANGELOG.md + git commit -m "docs(changelog): promote [Unreleased] into [${V}]" -m "[skip ci] Auto-generated by Release workflow." + # Push to the branch the workflow was triggered on (main). + git push origin "HEAD:${GITHUB_REF#refs/heads/}" + fi + + - name: Build all platform bundles + run: | + for t in darwin-arm64 darwin-x64 linux-x64 linux-arm64 win32-x64 win32-arm64; do + bash scripts/build-bundle.sh "$t" + done + ls -lh release + + - name: Generate SHA256SUMS + # Published as a release asset; the npm launcher verifies downloaded + # bundles against it (basenames only, so its path.basename match works). + run: | + ( cd release && sha256sum codegraph-* > SHA256SUMS ) + cat release/SHA256SUMS + + - name: Release notes from CHANGELOG.md + # The [] block was guaranteed-populated by the + # "Promote" step above, so the [Unreleased] fallback should + # never be needed in practice. Kept for defense-in-depth. + run: | + V="${{ steps.ver.outputs.version }}" + node scripts/extract-release-notes.mjs "$V" > notes.md 2>/dev/null \ + || node scripts/extract-release-notes.mjs Unreleased > notes.md 2>/dev/null || true + if [ ! -s notes.md ]; then + echo "::error::No release notes in CHANGELOG.md for [$V] or [Unreleased]." + exit 1 + fi + echo "----- release notes -----"; cat notes.md + + - name: Create GitHub Release + env: + GH_TOKEN: ${{ github.token }} + run: | + TAG="v${{ steps.ver.outputs.version }}" + # Idempotent: create the release once, otherwise (re-run) refresh assets. + if gh release view "$TAG" >/dev/null 2>&1; then + gh release upload "$TAG" release/codegraph-* release/SHA256SUMS --clobber + else + gh release create "$TAG" release/codegraph-* release/SHA256SUMS --title "$TAG" --notes-file notes.md + fi + + - name: Publish to npm + env: + NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} + run: | + V="${{ steps.ver.outputs.version }}" + bash scripts/pack-npm.sh "$V" + # Platform packages first, then the main shim (which depends on them). + # Skip any already on the registry so a re-run only fills in gaps. + for dir in release/npm/codegraph-* release/npm/main; do + name=$(node -p "require('./$dir/package.json').name") + if npm view "$name@$V" version >/dev/null 2>&1; then + echo "skip $name@$V (already published)" + else + echo "publishing $name@$V" + ( cd "$dir" && npm publish --access public ) + fi + done + + - name: Verify every package is actually on the registry + run: | + V="${{ steps.ver.outputs.version }}" + # npm publish can print success without persisting; confirm against the + # registry (with retries for propagation) so green means really shipped. + for dir in release/npm/codegraph-* release/npm/main; do + name=$(node -p "require('./$dir/package.json').name") + ok= + for i in 1 2 3 4 5 6; do + if npm view "$name@$V" version >/dev/null 2>&1; then ok=1; break; fi + echo "waiting for $name@$V to appear ($i)…"; sleep 10 + done + [ -n "$ok" ] || { echo "::error::$name@$V never appeared on the registry"; exit 1; } + echo "verified $name@$V" + done + + - name: Sync packages to npmmirror + # npmmirror/cnpm mirror lazily and frequently never pull the per-platform + # optionalDependencies on their own, so `npm i` there fails with + # "no prebuilt bundle" (issue #303). Nudge a sync now so mirror users get + # the bundle without waiting. Best-effort — the launcher also self-heals + # from GitHub Releases — so a mirror hiccup never fails the release. + continue-on-error: true + run: | + for dir in release/npm/codegraph-* release/npm/main; do + name=$(node -p "require('./$dir/package.json').name") + enc=$(node -p "encodeURIComponent(require('./$dir/package.json').name)") + echo "sync $name" + curl -s -X PUT "https://registry.npmmirror.com/-/package/$enc/syncs" || true + echo + done diff --git a/.gitignore b/.gitignore index 7c154ae1d..7c949e1c2 100644 --- a/.gitignore +++ b/.gitignore @@ -39,6 +39,10 @@ npm-debug.log* # Local Claude settings .claude/settings.local.json +.claude/scheduled_tasks.lock + +# Parallels Windows VM SSH/connection config (local machine, see CLAUDE.md) +.parallels # CodeGraph data directories (in test projects) .codegraph/ @@ -49,3 +53,6 @@ test_frameworks test-languages/ nul +release/ + +.antigravitycli/ diff --git a/BUNDLING.md b/BUNDLING.md new file mode 100644 index 000000000..dc21ab531 --- /dev/null +++ b/BUNDLING.md @@ -0,0 +1,74 @@ +# Distribution: self-contained bundles + +CodeGraph ships a **vendored Node runtime** alongside the app. Because Node 22.5+ +has a built-in real SQLite (`node:sqlite`, with WAL + FTS5), bundling Node means: + +- **No native build** — `better-sqlite3` is gone, so there are zero native addons + to compile or rebuild. +- **No wasm fallback** — and therefore no more `database is locked` (issue #238). +- **No Node-version dependence** — the app always runs on the bundled Node, + whatever the user has (or doesn't have) installed. + +## What's in a bundle + +Built by [`scripts/build-bundle.sh`](scripts/build-bundle.sh) — one archive per +platform, identical recipe (only the Node download differs): + +``` +codegraph-/ + node | node.exe # official Node runtime for + lib/ + dist/ # compiled app (+ tree-sitter .wasm grammars, schema.sql) + node_modules/ # production deps only (pure JS / wasm — portable) + bin/ + codegraph | codegraph.cmd # launcher → runs the bundled Node with the app +``` + +Targets: `darwin-arm64`, `darwin-x64`, `linux-x64`, `linux-arm64`, `win32-x64`, +`win32-arm64`. Unix targets produce `.tar.gz` (shell launcher); Windows produces +`.zip` (`node.exe` + a `.cmd` launcher). + +```bash +scripts/build-bundle.sh linux-x64 # -> release/codegraph-linux-x64.tar.gz +scripts/build-bundle.sh win32-x64 # -> release/codegraph-win32-x64.zip +``` + +Because dropping better-sqlite3 left **zero native addons**, building a bundle is +pure file-packaging — **any** target builds on **any** OS (the whole matrix builds +on one Linux runner). Cross-compilation isn't a concern; only *run-testing* a +bundle needs the target platform (or emulation, e.g. `docker run --platform +linux/amd64`). + +## Install channels (all deliver the same bundle) + +1. **`curl | sh`** ([`install.sh`](install.sh)) — no Node required; ideal for a + fresh Linux VPS over SSH. Detects os/arch, pulls the archive from GitHub + Releases, symlinks `codegraph` onto PATH. Re-run to upgrade; `--uninstall` to + remove. +2. **npm** ([`scripts/npm-shim.js`](scripts/npm-shim.js)) — preserves + `npm i -g @colbymchenry/codegraph`. The main package is a tiny shim; the + bundles ship as per-platform `optionalDependencies` + (`@colbymchenry/codegraph-` with `os`/`cpu`), so npm installs only the + matching one. The shim — run by the user's Node — execs the bundle, so the + real work runs on the bundled Node 24. Works even on old Node. On Windows it + invokes the bundled `node.exe` against the app entry directly (not the `.cmd` + launcher) — modern Node throws `EINVAL` when asked to spawn a `.cmd`/`.bat`. +3. **Windows** ([`install.ps1`](install.ps1)) — `irm … | iex`; same flow as + install.sh (detect arch, pull the `.zip` from Releases, add to PATH). +4. **Homebrew / Scoop** — TODO (tap + cask pointing at the Release archives). + +## Release pipeline + +[`.github/workflows/release.yml`](.github/workflows/release.yml) — manually +triggered. Reads the version from `package.json`, builds every platform bundle on +one runner, creates the GitHub Release (notes from `CHANGELOG.md`), and publishes +the npm shim + per-platform packages. Requires the `NPM_TOKEN` repo secret. + +Still TODO: +- **Code signing** — the main gap for "download & run": macOS Gatekeeper needs a + Developer ID + notarization; Windows needs Authenticode. Homebrew softens the + macOS case (handles quarantine). +- Retire the now-vestigial Node-version gate in `src/bin/codegraph.ts` — the + bundle always runs Node 24, and the npm shim does no tree-sitter work. +- Re-wire `npm uninstall` cleanup (the agent-config `preuninstall`) through the + shim — the generated main package doesn't carry it. diff --git a/CHANGELOG.md b/CHANGELOG.md index b1124e4f9..09fbad95a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,17 +7,238 @@ a [GitHub Release](https://github.com/colbymchenry/codegraph/releases) tagged This project follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### New Features + +- `codegraph init` now builds the initial index by default — you no longer need the `-i`/`--index` flag (it's still accepted, so existing commands and scripts keep working). (#483) +- Go: Gin middleware chains now connect end-to-end in `codegraph_trace` and `codegraph_explore` — following a request reaches the middleware and route handlers registered via `.Use()` / `.GET()` instead of dead-ending where the framework dispatches the chain dynamically. +- `codegraph_explore` now sizes its response to the *answer* instead of the file count: it shows the mechanism and the exact methods you asked about in full — even when they're buried deep in a large file — while collapsing the redundant interchangeable implementations of an interface (an HTTP interceptor chain, a query-compiler family) down to signatures. Fewer tokens for a more complete answer, so on the flows that used to occasionally cost more than plain grep/read it's now clearly cheaper — and the win holds across small, medium, and large codebases. Distinct, non-interchangeable code is shown in full as before. Disable with `CODEGRAPH_ADAPTIVE_EXPLORE=0`. + +### Fixes + +- Indexing a project that contains only config-style files (YAML, Twig, or `.properties`) no longer misleadingly reports "No files found to index" — these files are tracked at the file level and are now counted as indexed. Thanks @luojiyin1987 (#357). + +## [0.9.7] - 2026-05-28 + +### New Features + +- Go: gRPC interface stubs now connect to their hand-written implementation, so callers, callees, impact, and trace land on the real method instead of an empty generated stub. +- Generated files (protobuf, gRPC stubs, mocks, build output) now rank last in search, trace, and explore, so results land on your real implementation instead of an auto-generated placeholder. +- When `codegraph_trace` can't find a static path (a dynamic-dispatch break), it now inlines both endpoints' source, callers, and callees in one response, so the agent gets the full picture without a flurry of follow-up calls. +- Trace now picks the right endpoints in large multi-module repos by preferring symbols that share a directory, instead of grabbing an arbitrary same-named symbol from an unrelated module. +- Test files are now deprioritized in `codegraph_explore` (Go, Ruby, JS/TS, Java/Kotlin/Scala), so the explore budget goes to your real implementation source. +- Small projects (under ~500 files) now resolve flow questions in fewer MCP calls, with a leaner tool surface and tuned context and explore output sized for the project. +- `codegraph_context` now auto-traces flow questions like "how does X reach Y" or "trace the path from A to B", splicing the trace into the response so you don't need a separate `codegraph_trace` call. +- `codegraph_context` now inlines a URL-to-handler routing table and the source of your main routes file for routing questions on small projects, so you don't have to go read `routes.rb` or `web.php` yourself. +- `codegraph_context` search now boosts results in the directory of a project's core framework file, so a small same-named extension file no longer outranks the actual framework core. +- Interface-to-implementation linking now works for C#, TypeScript, JavaScript, Swift, and Scala (previously Java/Kotlin only), so investigating an interface method surfaces its concrete implementations. +- MCP tool descriptions are now shorter, trimming per-session overhead while keeping the steering guidance. +- Java and Kotlin imports now resolve by fully-qualified name, so same-name classes in different packages are told apart correctly in multi-module Spring and Android codebases, including across the Java/Kotlin interop boundary. +- Java and C# anonymous classes (`new T() { ... }`) and their overridden methods are now indexed as real class nodes, so an agent sees those hidden overrides in its trail without a Read. +- The installer no longer writes a duplicate `## CodeGraph` instructions block into your agent's instructions file (`CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, Cursor's `.cursor/rules/codegraph.mdc`, or Kiro's steering doc) — the MCP server is now the single source of truth, and re-running `codegraph install` or `codegraph uninstall` strips a block a previous version left behind (#529). If you added your own notes inside the `CODEGRAPH_START`/`CODEGRAPH_END` markers, move them outside the markers first, since the whole marked block is removed. + +### Fixes + +- MCP tools no longer return results for files that were deleted while no server was running — the first query of a session now waits for the catch-up sync, so you get the correct index instead of stale rows. +- Windows: black console windows no longer flash on every file save or MCP reconnect (#485, #510, #530). +- `codegraph index` and `init -i` now report the true edge count in their summary, instead of undercounting by missing resolution and synthesizer edges. + +## [0.9.6] - 2026-05-27 + +### New Features + +- Enterprise Spring and MyBatis flows now trace end-to-end: MyBatis XML mappers are indexed and linked to their Java mapper interfaces, Spring `@Value` and `@ConfigurationProperties` references resolve to the matching keys in your `application.yml`/`.properties` config (including relaxed kebab/camel/snake binding), and field-injected concrete beans like `this.field.method()` resolve through to their implementation (#389). +- Gemini CLI (and the rebranded Antigravity CLI) plus the Antigravity IDE are now supported by `codegraph install`, detected and configured out of the box with sibling settings and MCP servers preserved across re-installs (#399). +- Kiro (CLI and IDE) is now supported by `codegraph install` on macOS, Linux, and Windows, with its own steering file so it loads CodeGraph guidance naturally (#385). + +### Fixes + +- C/C++: bare `#include "header.h"` directives now connect to the real header file instead of a phantom import, so includes show up as true file-to-file edges; system and stdlib headers are filtered out so they don't false-resolve (#453). +- Java/Kotlin: imports now disambiguate same-name classes across modules using the fully-qualified import path, so callers, callees, and trace land on the right class in multi-module projects instead of guessing by file proximity (#314). +- TypeScript: `type` aliases with object shapes (including function-typed members and intersection types) now surface their members in the graph, so a call like `handle.stop()` resolves to the alias member instead of an unrelated look-alike class in a sibling directory (#359). +- C#: parameter, return, property, and field types now produce reference edges, so callers and callees on a DTO or service type return real results instead of nothing (#381). +- Go: cross-package qualified calls like `pkg.Func()` now resolve to the right package by reading your `go.mod`, so callers, callees, impact, and trace return complete results on Go monorepos instead of almost nothing (#388). +- `codegraph_files` now returns the whole project when an agent passes a root-ish path like `/`, `.`, `./`, `""`, or a Windows-style `\`, and subdirectory filters like `/src`, `./src`, and `src\components` all resolve correctly instead of returning "No files found" (#426). +- The file watcher no longer marks edited files as fresh when another process holds the index lock, so the per-file staleness signal stays accurate until the edit is actually indexed (#449). +- TypeScript/JavaScript: calls inside top-level variable initializers (`const token = getToken()`) and inside inline object-literal methods are no longer dropped, so they show up in callers as expected, including in Vue single-file components (#425). +- Watch sync no longer aborts with a `FOREIGN KEY constraint failed` error in a long-running daemon; a stale lookup now drops a single edge instead of failing the whole sync (#455). +- Hermes: `codegraph install --target hermes` no longer corrupts `~/.hermes/config.yaml`, correctly handling PyYAML's block-style lists and re-installing cleanly even on an already-corrupted file (#456). +- NestJS: route prefixes from `RouterModule.register([...])` (including nested `children`) now propagate to controller routes, so a route shows up at its full path like `GET /admin/users` instead of `GET /` (#459). +- C++: callers now resolve through typed member pointers such as `m_alg->Processing()`, including out-of-line method definitions and the common case of two classes sharing a method name (#445, #454). + +## [0.9.5] - 2026-05-25 + +### New Features + +- Running multiple AI agents in the same project no longer multiplies the cost: two Claude Code windows, a worktree agent, or parallel sub-agents now share one background daemon per project with a single file watcher, SQLite connection, and tree-sitter warm-up instead of N independent copies (#411). +- The daemon runs detached so it outlives any single session, meaning closing one editor or terminal never severs the others; it lingers briefly after the last client disconnects so back-to-back sessions skip the startup cost, then exits and cleans up after itself. Tune the idle wait with `CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS` (default five minutes). +- Set `CODEGRAPH_NO_DAEMON=1` to opt out and get one independent server per client, handy for debugging or sandboxes that disallow local sockets; the daemon is also version-pinned, so upgrading CodeGraph never mixes versions over the connection. +- CodeGraph responses now tell the agent which files are pending re-index: when the watcher has seen edits since the last sync, tool responses add a warning banner naming the stale files and their state so the agent reads just those directly while trusting the rest, with zero cost when nothing is pending (#403). +- `CODEGRAPH_WATCH_DEBOUNCE_MS` lets you tune the file-watcher quiet window (default 2000ms) for workspaces with bursty writes like format-on-save chains or large generated outputs, without touching your agent's command line (#403). +- Objective-C indexing: `.m`, `.mm`, and content-sniffed `.h` files now parse with full structural extraction, including full multi-part selectors, properties, imports, and superclass/protocol relationships, so trace, callers, and callees work across iOS codebases (#165). +- Mixed iOS, React Native, and Expo projects now trace end-to-end across language boundaries: Swift to Objective-C auto-bridging, the React Native legacy bridge and TurboModules, native-to-JS event channels, Expo Modules, and Fabric/Codegen view components are all bridged so flows connect through gaps that static parsing alone can't follow (#401). + +### Fixes + +- TypeScript: types used only in an interface's property or method signatures now produce references edges, so impact and callers on a type include every consumer that imports it just for an interface shape (#432). +- Git worktrees no longer silently borrow another tree's index; running CodeGraph from a worktree nested inside the main checkout used to return the wrong branch's code with no warning, and now both the status command and every read tool call out the conflict and point you to `codegraph init -i` in the worktree (#155). +- The file watcher no longer exhausts the OS file-watch budget on large repos: it now excludes the same directories the indexer ignores (defaults plus your `.gitignore`) before registering watches, so CodeGraph can run alongside your editor or dev server without hitting the per-user watch ceiling (#276). +- The index now stays in sync after `git pull`, branch switches, and edits made outside your editor; change detection is filesystem-based instead of relying on `git status`, so pulled or checked-out code is picked up without a full re-index. +- The MCP server now catches up on connect, reconciling anything that changed while it wasn't running so your first query reflects the current code instead of a stale snapshot. +- Dependency, build, and cache directories like `node_modules`, `vendor`, `dist`, `build`, `target`, `.venv`, `__pycache__`, `Pods`, and `.next` are now excluded by default, so context and search reflect your code instead of third-party noise even in a project with no `.gitignore`; add a `.gitignore` negation to index one anyway (#407). + +## [0.9.4] - 2026-05-24 + +### New Features + +- Request-to-handler flows now trace end-to-end across many web stacks, with new or improved route resolution for Express, Rails, Spring (Java and Kotlin), Django/DRF, Laravel, Flask, FastAPI, Gin, chi, ASP.NET, Drupal, Axum, actix, Vapor, Play, Vue/Nuxt, Svelte/SvelteKit, and React Router. +- `codegraph_trace`, `codegraph_callees`, and `codegraph_explore` now follow flows that have no static call edge — callback and observer registration, EventEmitter, React re-renders and JSX children, Flutter `setState` to `build`, C++ virtual overrides, and Java/Kotlin interface-to-implementation dispatch (like Spring's `@Autowired` service calls) — and each bridged hop is labeled inline in trace with where it was wired up. +- `codegraph_trace` now returns a self-contained flow dossier: every hop shows its full body inline plus the destination's own outgoing calls, so a single trace usually answers a "how does X reach Y" question without a follow-up explore, node, or Read. +- `codegraph_explore` now leads with the execution flow when your query names the symbols of a flow, finding the call path among those symbols (including across dynamic-dispatch hops) so you get a trace-quality answer without switching tools. +- `codegraph_node` and `codegraph_trace` now emit line-numbered source (matching `codegraph_explore` and Read), so you can cite or edit exact lines without re-reading the file just to recover line numbers. +- New `CODEGRAPH_MCP_TOOLS` environment variable lets you expose only a chosen subset of codegraph tools over MCP (e.g. `trace,search,node,context`) without editing your client's MCP config; unset exposes all of them. +- Release archives now ship with a `SHA256SUMS` file, and the npm launcher verifies the bundle it downloads against it, aborting on a mismatch (releases published before this change skip verification rather than failing). + +### Fixes + +- Several static-extraction and resolution correctness fixes underpin the routing work above: C++ inheritance edges that were previously missing, Dart methods that were extracted signature-only, Python handlers named `index`/`get`/`update` that were being silently dropped, and an explore output-budget issue that under-returned source on repos with very large files. +- `codegraph serve --mcp` no longer keeps running after its parent agent is force-killed (OOM, `kill -9`, or container teardown) on Linux, where it used to hold inotify watches, file descriptors, and the SQLite WAL indefinitely; the server now shuts down as soon as its parent process changes, tunable via `CODEGRAPH_PPID_POLL_MS` (#277). +- Installing `@colbymchenry/codegraph` through a registry mirror that hadn't yet mirrored the matching per-platform package no longer fails with `no prebuilt bundle for `; the launcher now downloads the bundle from GitHub Releases and caches it, with `CODEGRAPH_NO_DOWNLOAD=1` to disable the fallback and `CODEGRAPH_DOWNLOAD_BASE` to point it at your own mirror (#303). +- `install.sh` no longer fails with `403` / "could not resolve latest version" on shared or cloud hosts that exhaust GitHub's unauthenticated API rate limit; it now resolves the version through the unthrottled releases redirect, and `CODEGRAPH_VERSION` accepts a bare version like `0.9.4` as well as `v0.9.4` (#325). + +## [0.9.3] - 2026-05-22 + +### New Features + +- New `codegraph uninstall` command cleanly removes CodeGraph from every agent it's configured on — Claude Code, Cursor, Codex CLI, opencode, and Hermes Agent — in one step, asking whether to clean up your global or this project's local config and reporting exactly which agents it touched; it accepts `--location`, `--target`, and `--yes` for scripted or non-interactive use, removes only what `codegraph install` wrote, and leaves your `.codegraph/` index alone (#313). + +### Fixes + +- Indexing a large multi-language project no longer aborts partway through with a `Fatal process out of memory: Zone` crash on Node.js 22 and 24, even with plenty of RAM free — CodeGraph now launches with a V8 flag that keeps grammar compilation off the optimizing tier, and any launch path that doesn't get the flag directly re-execs once with it automatically (#298, #293). Node 25 stays blocked for now, since its variant of this bug isn't fixed by the same flag. +- Uninstalling from Cursor now deletes the leftover `.cursor/rules/codegraph.mdc` file outright instead of leaving an orphaned, empty rule behind, while keeping any content you added outside CodeGraph's markers. + +## [0.9.2] - 2026-05-21 + +### Breaking Changes + +- CodeGraph no longer has a config file: `.codegraph/config.json` and the entire config surface are gone, and the library API for it (the config type, the `config` option on `init()`, and the get/update config exports) has been removed — existing config files are now ignored, and `.gitignore` is the single source of truth for what gets indexed. The `.codegraphignore` marker is also no longer supported; use `.gitignore` instead. + +### New Features + +- `codegraph install` now supports Hermes Agent (Nous Research), wiring up the CodeGraph MCP server so Hermes can drive the knowledge graph like the other agents. +- Drupal projects (8/9/10/11) are now detected and indexed with framework smarts: routes from `*.routing.yml` link to their controller, form, or entity-handler, and hook implementations across modules are connected to their canonical hook name, so asking for callers of a hook returns every implementation (#268). +- Indexing is now zero-config and honors your `.gitignore` everywhere — in git repos via git, and in non-git projects by reading `.gitignore` files directly — so to keep something out of the graph you just add it to `.gitignore`. Behavior change: committed files that aren't gitignored are now indexed even under `vendor/`, `Pods/`, or a committed `dist/`; add a `.gitignore` negation to exclude them (#283). + +### Fixes + +- Windows: installing globally and then running any `codegraph` command no longer fails — the launcher now invokes the bundled runtime directly instead of a `.cmd` file that modern Node refuses to spawn, so `codegraph` works regardless of your Node version (#289). + +### Security + +- The temp-dir marker written on each `codegraph_context` call is now opened safely so it can't follow a symlink, closing a hole where another local user on a shared machine could redirect that write onto a file you can write (#280). + +## [0.9.1] - 2026-05-21 + +### Fixes + +- The standalone installers (`curl … | sh` and `irm … | iex`) no longer fail to launch on a machine that has no Node installed. +- Installing with `npm i -g` on Linux x64 now finds its bundle, after the 0.9.0 release silently shipped without the linux-x64 package; the release pipeline now verifies every package reached the npm registry so a release can't pass green-but-broken again. + +## [0.9.0] - 2026-05-21 + +CodeGraph now ships its own self-contained runtime, so it installs on any Node version — or none at all — with no native build step, and the old intermittent "database is locked" errors are gone for good. + +### New Features + +- One-line standalone installers that need no Node.js: `install.sh` on macOS and Linux, and `install.ps1` on Windows fetch the self-contained bundle and put `codegraph` on your PATH (you can still use `npm`/`npx` on any Node version too). +- CodeGraph now uses real SQLite with full WAL and FTS5 built into its bundled runtime, which fixes the concurrent-read "database is locked" errors at the root, removes the native build step entirely, and runs faster for anyone who had been stuck on the old WASM fallback (#238). +- Lua: CodeGraph now indexes `.lua` projects (Neovim plugins, Kong, OpenResty, game code), surfacing functions, table methods, local variables, `require(...)` imports, and the call edges between them. +- Luau: CodeGraph now indexes `.luau`, Roblox's typed superset of Lua, adding type and `export type` aliases, typed function signatures, generics, and Roblox instance-path requires on top of everything Lua extracts (#232). +- `codegraph status` now reports the effective journal mode, so a "database is locked" report is easy to triage at a glance. + +### Fixes + +- Re-running `codegraph install` now strips the broken auto-sync hooks that pre-0.8 versions wrote into Claude Code's settings, which had been causing a "Stop hook error: unknown command 'sync-if-dirty'" on every turn. The cleanup is surgical and leaves unrelated hooks untouched. Re-run `codegraph install` once on an affected machine to clear the error. + +## [0.8.0] - 2026-05-20 + +### Breaking Changes + +- The minimum supported Node.js version is now 20 (Node 18 is end-of-life); Node 22 LTS and Node 24 get the fast native backend out of the box, other Node versions still run via the slower WASM fallback, and Node 25+ remains blocked (#81). If you're on an older Node, upgrade to 20 or newer. + +### New Features + +- NestJS: CodeGraph now recognizes NestJS projects and surfaces the route that binds each handler across HTTP controllers, GraphQL resolvers, microservice handlers, and WebSocket gateways, detected automatically from any `@nestjs/*` dependency (#220). +- `codegraph_explore` source now includes line numbers, so an agent can cite `file:line` straight from the result instead of reopening the file to find a line number; set `CODEGRAPH_EXPLORE_LINENUMS=0` to disable. +- On WSL2 `/mnt/*` drives, where the live file watcher is too slow and could break MCP startup, CodeGraph now skips the watcher and offers to keep the index fresh with git hooks instead; new `CODEGRAPH_NO_WATCH=1` (or `serve --mcp --no-watch`) forces the watcher off anywhere, and `CODEGRAPH_FORCE_WATCH=1` overrides the WSL auto-detect when your setup is actually fast. +- CodeGraph now guides agents to answer "how does X work" and architecture questions directly with a couple of codegraph calls instead of delegating to a file-reading sub-agent or a grep-and-read loop, which gives faster, cheaper, `file:line`-cited answers on medium and large repos. +- `codegraph_node` with code on a class, interface, struct, or enum now returns a compact member outline (fields plus method signatures with line numbers) instead of the entire class body; functions and methods still return their full source. +- `codegraph_explore` output now scales with project size, so small projects get tighter responses than your native grep-and-read flow would produce while large codebases keep their fuller budget, and a per-file cap stops a single dense file from collapsing into a whole-file dump (#185). Thanks @essopsp. +- Search ranking now correctly de-prioritizes CamelCase test files (`FooTest.kt`, `BarTests.swift`, `BazSpec.scala`, `QuxTestCase.cs`) and test source-set directories in Kotlin, Swift, Scala, and C#, so real implementations no longer get outranked by tests. + +### Fixes + +- `codegraph_explore` output is now hard-capped to its size budget, so an oversized response no longer overruns the cap and sits in the agent's context to be re-read every turn. +- Newly created untracked files are no longer reported as pending forever and re-indexed from scratch on every `codegraph sync`; CodeGraph now hash-compares them against the index the same way it does tracked files (#206). Thanks @15290391025. +- `codegraph init -i` now finds source inside nested, independent git repositories that aren't submodules (common in CMake super-repo layouts), instead of reporting "No files found to index" (#193). Thanks @timxx. +- On Node 24, indexing no longer silently drops to the slower fallback backend with a warning that couldn't be cleared; a fresh install on Node 22 or 24 now gets the fast native backend with no compiler, and `codegraph status` should report it (#203). Thanks @Finndersen. +- MCP tools no longer fail with "CodeGraph not initialized" when the index actually exists; when the client doesn't report a workspace root, the server now asks for it via the standard MCP `roots/list` request before falling back, and the error message is actionable when a project still can't be resolved (#196). Thanks @zhangyu1197. +- The MCP server no longer hangs on startup under WSL2 when the project lives on an NTFS `/mnt/*` mount, so the codegraph tools actually appear; CodeGraph auto-skips the watcher there with manual and git-hook sync fallbacks (#199). Thanks @mengfanbo123. +- Claude Code project-local installs now write the MCP server to `.mcp.json` (the file Claude Code actually reads for project-scoped servers) instead of a file it ignores, and re-running `codegraph install` migrates an affected project automatically (#207). Thanks @Jhsmit. +- Source-omission markers in `codegraph_explore` and `codegraph_context` output are now language-neutral instead of C-style comments, so they no longer look wrong inside Python, Ruby, and other non-C source blocks. + +## [0.7.10] - 2026-05-19 + +### Fixes + +- CodeGraph tools now reliably appear in your client on slow filesystems (Docker Desktop VirtioFS on macOS, WSL2), where the startup handshake could previously time out and leave the process running with no tools visible (#172). Thanks @sashanclrp and @sgrimm. +- On Windows PowerShell and cmd.exe, terminal output during `codegraph index` and `codegraph sync` no longer turns into garbled characters; CodeGraph now uses plain ASCII glyphs by default on Windows, with `CODEGRAPH_UNICODE=1` to opt back into the Unicode glyphs or `CODEGRAPH_ASCII=1` to force ASCII on any platform (#168). Thanks @starkleek and @Bortlesboat. +- Module-qualified symbol lookups now resolve in the codegraph tools, so you can pass names like `module::symbol` (Rust, C++, Ruby), `Module.symbol` (TypeScript, JavaScript, Python), or `module/symbol`, including multi-level paths and Rust prefixes like `crate`, `super`, and `self` (#173). Thanks @joselhurtado. + +## [0.7.9] - 2026-05-17 + +### New Features + +- opencode: the installer now writes an `AGENTS.md` file with CodeGraph usage guidance, so opencode reaches for the `codegraph_*` tools instead of falling back to its native search. +- opencode: your comments and formatting in `opencode.jsonc` now survive install, re-install, and uninstall, because the installer makes surgical edits instead of rewriting the whole file. + +### Fixes + +- opencode: `codegraph install` now wires up the MCP server in the file opencode actually reads — previously it wrote to a config file opencode ignores by default, so the CodeGraph entry never appeared in any opencode session; re-run `codegraph install --target=opencode` after upgrading so the entry lands in the right place. + +## [0.7.7] - 2026-05-17 + +### New Features + +- `codegraph install` now sets up Claude Code, Cursor, Codex CLI, and opencode from one multi-select prompt, with any agents it detects pre-checked, so a single install wires up every editor you use (#137). +- You can install non-interactively for scripting and CI with flags like `--target`, `--location`, `--yes`, `--no-permissions`, and `--print-config`. +- `codegraph init` now auto-wires project-local agent config for any agent you installed globally, so one global `codegraph install` works in every project you open without re-installing per project. +- Agent instructions are now agent-agnostic and tell each agent to trust codegraph results instead of re-verifying with grep, fixing the case where Cursor and Codex fell back to native search even with codegraph available. +- The install prompts are clearer: the agent picker comes first, and the separate "install the CLI on your PATH" and "apply to all projects or just this one" questions no longer both read as "Global". + +### Fixes + +- Cursor: a globally-installed codegraph no longer reports "not initialized" in every workspace; the installer now passes the correct project path into Cursor's MCP config to work around Cursor launching MCP servers with the wrong working directory. + +Thanks @andreinknv for the substantive draft this release was based on. + ## [0.7.6] - 2026-05-13 -### Fixed -- `codegraph` CLI failing with `zsh: permission denied: codegraph` after a fresh - global install. The published 0.7.5 tarball shipped `dist/bin/codegraph.js` - without the executable bit, so the shell refused to run it through the npm - symlink. The build now `chmod +x`'s the binary before packing. +### Fixes - Already on 0.7.5? Either upgrade to 0.7.6, or unblock yourself in place: - ```bash - chmod +x "$(npm root -g)/@colbymchenry/codegraph/dist/bin/codegraph.js" - ``` +- Fixed the `codegraph` command failing with `permission denied` right after a fresh global install — the 0.7.5 package shipped the CLI without its executable bit, so your shell refused to run it. New installs work out of the box. If you're stuck on 0.7.5, upgrade to 0.7.6 or unblock yourself in place by making the installed binary executable with `chmod +x`. +[0.9.7]: https://github.com/colbymchenry/codegraph/releases/tag/v0.9.7 +[0.9.6]: https://github.com/colbymchenry/codegraph/releases/tag/v0.9.6 +[0.9.5]: https://github.com/colbymchenry/codegraph/releases/tag/v0.9.5 +[0.9.4]: https://github.com/colbymchenry/codegraph/releases/tag/v0.9.4 +[0.9.3]: https://github.com/colbymchenry/codegraph/releases/tag/v0.9.3 +[0.9.2]: https://github.com/colbymchenry/codegraph/releases/tag/v0.9.2 +[0.9.1]: https://github.com/colbymchenry/codegraph/releases/tag/v0.9.1 +[0.9.0]: https://github.com/colbymchenry/codegraph/releases/tag/v0.9.0 +[0.8.0]: https://github.com/colbymchenry/codegraph/releases/tag/v0.8.0 +[0.7.10]: https://github.com/colbymchenry/codegraph/releases/tag/v0.7.10 +[0.7.9]: https://github.com/colbymchenry/codegraph/releases/tag/v0.7.9 +[0.7.7]: https://github.com/colbymchenry/codegraph/releases/tag/v0.7.7 [0.7.6]: https://github.com/colbymchenry/codegraph/releases/tag/v0.7.6 diff --git a/CLAUDE.md b/CLAUDE.md index ae4ff482b..628544052 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,192 +4,262 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Project Overview -CodeGraph is a local-first code intelligence system that builds a semantic knowledge graph from any codebase. It provides structural understanding of code relationships using tree-sitter for AST parsing and SQLite for storage. +CodeGraph is a local-first code intelligence library + CLI + MCP server. It parses any supported codebase with tree-sitter, stores symbols/edges/files in SQLite (FTS5), and exposes a knowledge graph to AI agents (Claude Code, Cursor, Codex CLI, opencode) over MCP. Per-project data lives in `.codegraph/`. Extraction is deterministic — derived from AST, not LLM-summarized. -**Key characteristics:** -- Headless library (no UI) - purely an API -- Node.js runtime (works standalone, in Electron, or any Node environment) -- Per-project data stored in `.codegraph/` directory -- Deterministic extraction from AST, not AI-generated summaries +Distributed as `@colbymchenry/codegraph` on npm; same binary serves as installer, indexer, and MCP server. -## Build and Development Commands +## Build, Test, Run ```bash -# Build -npm run build # Compile TypeScript and copy assets +npm run build # tsc + copy schema.sql and *.wasm into dist/; chmods dist/bin/codegraph.js +npm run dev # tsc --watch +npm run clean # rm -rf dist -# Test -npm test # Run all tests once -npm run test:watch # Run tests in watch mode +npm test # vitest run (all) +npm run test:watch +npm run test:eval # only __tests__/evaluation/ +npm run eval # build then run __tests__/evaluation/runner.ts via tsx -# Clean -npm run clean # Remove dist/ directory +npm run cli # build then run the local dist binary + +# Single test file / pattern +npx vitest run __tests__/installer-targets.test.ts +npx vitest run __tests__/extraction.test.ts -t "TypeScript" ``` -## Running a Single Test +`copy-assets` (called from `build`) copies `src/db/schema.sql` and all `src/extraction/wasm/*.wasm` files into `dist/`. **Any new SQL or grammar wasm must be copied or it won't ship.** -```bash -npx vitest run __tests__/extraction.test.ts # Run specific test file -npx vitest run __tests__/extraction.test.ts -t "TypeScript" # Run tests matching pattern -``` +Node engines: `>=18.0.0 <25.0.0`. There is a hard exit on Node 25.x (see `src/bin/node-version-check.ts`). ## Architecture -### Core Module Structure +### Layered pipeline ``` -src/ -├── index.ts # Main CodeGraph class - public API entry point -├── types.ts # All TypeScript interfaces and types -├── db/ # SQLite database layer -│ ├── index.ts # DatabaseConnection class -│ ├── queries.ts # QueryBuilder with prepared statements -│ └── schema.sql # Table definitions with FTS5 search -├── extraction/ # Tree-sitter AST parsing -│ ├── index.ts # ExtractionOrchestrator -│ ├── tree-sitter.ts # Universal parser wrapper -│ └── grammars.ts # Language detection and grammar loading -├── resolution/ # Reference resolver -│ ├── index.ts # ReferenceResolver orchestrator -│ ├── import-resolver.ts -│ ├── name-matcher.ts -│ └── frameworks/ # Framework-specific patterns (React, Express, Laravel, etc.) -├── graph/ # Graph traversal and queries -│ ├── index.ts # GraphQueryManager -│ ├── traversal.ts # GraphTraverser (BFS/DFS, impact radius) -│ └── queries.ts # High-level graph queries -├── context/ # Context building for AI assistants -│ ├── index.ts # ContextBuilder -│ └── formatter.ts # Markdown/JSON output formatting -├── sync/ # Incremental update system -│ ├── index.ts -│ └── git-hooks.ts # Post-commit hook management -├── installer/ # Interactive installer -│ ├── index.ts # Installer orchestrator -│ ├── banner.ts # ASCII art banner -│ ├── claude-md-template.ts # CLAUDE.md template generator -│ ├── config-writer.ts # Configuration file writing -│ └── prompts.ts # User prompts -├── mcp/ # Model Context Protocol server -│ ├── index.ts # MCPServer class -│ ├── tools.ts # MCP tool definitions -│ └── transport.ts # Stdio transport -└── bin/codegraph.ts # CLI entry point +files → ExtractionOrchestrator (tree-sitter) → DB (nodes/edges/files) + ↓ + ReferenceResolver (imports, name-matching, framework patterns) + ↓ + GraphQueryManager / GraphTraverser (callers, callees, impact) + ↓ + ContextBuilder (markdown/JSON for AI consumption) ``` -### Key Classes +The public API surface is `src/index.ts` — the `CodeGraph` class wires all the layers and re-exports types. Library users only touch this file; the MCP server and CLI also drive it. -- **CodeGraph** (`src/index.ts`): Main entry point. Lifecycle methods (`init`, `open`, `close`), indexing (`indexAll`, `sync`), graph queries (`traverse`, `getCallGraph`, `getImpactRadius`), context building (`buildContext`) +### Module layout -- **ExtractionOrchestrator** (`src/extraction/index.ts`): Coordinates file scanning, parsing, and storing. Uses tree-sitter native bindings for each supported language +- `src/index.ts` — `CodeGraph` class: `init`/`open`/`close`, `indexAll`, `sync`, `searchNodes`, `getCallers`/`getCallees`, `getImpactRadius`, `buildContext`, `watch`/`unwatch`. +- `src/db/` — `DatabaseConnection`, `QueryBuilder` (prepared statements), `schema.sql`. Backed by `better-sqlite3` (native) when available, transparently falls back to `node-sqlite3-wasm`. `codegraph status` surfaces which backend is live; wasm is the slow path. +- `src/extraction/` — `ExtractionOrchestrator`, tree-sitter wrappers, per-language extractors under `languages/` (one file per language), plus standalone extractors for non-tree-sitter formats (`svelte-extractor.ts`, `vue-extractor.ts`, `liquid-extractor.ts`, `dfm-extractor.ts` for Delphi). `parse-worker.ts` runs heavy parsing off the main thread. +- `src/resolution/` — `ReferenceResolver` orchestrates `import-resolver.ts` (with `path-aliases.ts` for tsconfig path aliases + cargo workspace member globs), `name-matcher.ts`, and `frameworks/` (Express, Laravel, Rails, FastAPI, Django, Flask, Spring, Gin, Axum, ASP.NET, Vapor, React Router, SvelteKit, Vue/Nuxt, Cargo workspaces). Frameworks emit `route` nodes and `references` edges. +- `src/graph/` — `GraphTraverser` (BFS/DFS, impact radius, path finding) and `GraphQueryManager` (high-level queries). +- `src/context/` — `ContextBuilder` + formatter for markdown/JSON output. +- `src/search/` — full-text query parser and helpers for FTS5. +- `src/sync/` — `FileWatcher` (native FSEvents/inotify/RDCW) with debounce + filter, and git-hook helpers. +- `src/mcp/` — MCP server (`MCPServer`, `tools.ts`, `transport.ts`). `server-instructions.ts` is what the server returns in the MCP `initialize` response — keep it in sync with the user-facing tool guidance. +- `src/installer/` — see below. +- `src/bin/codegraph.ts` — CLI (commander). Subcommands: `install`, `init`, `uninit`, `index`, `sync`, `status`, `query`, `files`, `context`, `affected`, `serve --mcp`. +- `src/ui/` — terminal UI (shimmer progress, worker). -- **GraphTraverser** (`src/graph/traversal.ts`): BFS/DFS traversal, call graph construction, impact radius calculation, path finding +### NodeKind / EdgeKind -- **ReferenceResolver** (`src/resolution/index.ts`): Resolves unresolved references after full indexing using framework patterns, import resolution, and name matching +Defined in `src/types.ts`. Both extractors and resolvers must use these exact strings. -### Database Schema +- **NodeKind**: `file`, `module`, `class`, `struct`, `interface`, `trait`, `protocol`, `function`, `method`, `property`, `field`, `variable`, `constant`, `enum`, `enum_member`, `type_alias`, `namespace`, `parameter`, `import`, `export`, `route`, `component`. +- **EdgeKind**: `contains`, `calls`, `imports`, `exports`, `extends`, `implements`, `references`, `type_of`, `returns`, `instantiates`, `overrides`, `decorates`. -SQLite database with: -- `nodes`: Code symbols (functions, classes, methods, etc.) -- `edges`: Relationships (calls, imports, extends, contains, etc.) -- `files`: Tracked source files with content hashes -- `unresolved_refs`: References pending resolution -- `nodes_fts`: FTS5 virtual table for full-text search +### Multi-agent installer -### Supported Languages +`src/installer/` is the entry point for `codegraph install` (and the bare `codegraph`/`npx @colbymchenry/codegraph` invocation). Architecture: -TypeScript, JavaScript, TSX, JSX, Svelte, Python, Go, Rust, Java, C, C++, C#, PHP, Ruby, Swift, Kotlin, Dart, Liquid, Pascal +- `targets/registry.ts` lists every supported agent. +- `targets/types.ts` defines the `AgentTarget` interface — adding a 5th agent (Continue, Zed, Windsurf…) is **one new file in `targets/` + one entry in `registry.ts`**. Each target owns its config-file location and MCP-server JSON/TOML/JSONC writing. (Targets no longer write an instructions file — see below.) +- Current targets: `claude.ts`, `cursor.ts`, `codex.ts`, `opencode.ts`. +- `targets/toml.ts` is a hand-rolled TOML serializer scoped to `[mcp_servers.codegraph]` (used by Codex). Sibling tables and `[[array_of_tables]]` are preserved verbatim. No new dependency. +- opencode reads `opencode.jsonc` by default; the installer prefers existing `.jsonc`, falls back to `.json`, and creates `.jsonc` for greenfield installs. Edits are surgical via `jsonc-parser` so user comments and formatting survive install/re-install/uninstall round-trips. +- `instructions-template.ts` no longer holds an instructions body — it exports only the ``/`` markers. The installer **stopped writing** a `## CodeGraph` block into each agent's instructions file (`CLAUDE.md` / `~/.codex/AGENTS.md` / `~/.config/opencode/AGENTS.md` / `~/.gemini/GEMINI.md` / `.cursor/rules/codegraph.mdc` / Kiro steering doc) because it duplicated the MCP `initialize` instructions verbatim (issue #529). Each target's `install` (self-heal on upgrade) and `uninstall` use the markers to **strip** a block a previous install left behind. `server-instructions.ts` is the single source of truth for agent-facing guidance. +- All installer changes need matching coverage in `__tests__/installer-targets.test.ts` — there are ~47 parameterized contract tests covering install idempotency, sibling preservation, uninstall reverses install, byte-equal re-runs returning `unchanged`, and partial-state recovery for Codex. -### Node and Edge Types +### Cursor MCP working-directory quirk -**NodeKind**: `file`, `module`, `class`, `struct`, `interface`, `trait`, `protocol`, `function`, `method`, `property`, `field`, `variable`, `constant`, `enum`, `enum_member`, `type_alias`, `namespace`, `parameter`, `import`, `export`, `route`, `component` +Cursor launches MCP subprocesses with the wrong cwd and doesn't pass `rootUri` in `initialize`. The installer injects `--path` into Cursor's MCP args — absolute path for local installs, `${workspaceFolder}` for global installs. If you touch Cursor wiring, preserve this. -**EdgeKind**: `contains`, `calls`, `imports`, `exports`, `extends`, `implements`, `references`, `type_of`, `returns`, `instantiates`, `overrides`, `decorates` +### MCP server instructions -## CLI Usage +`src/mcp/server-instructions.ts` is sent back to the agent in the MCP `initialize` response. This is the *first* thing every agent sees about how to use the tools, and as of issue #529 it is the **single source of truth** for agent-facing tool guidance — the installer no longer writes a duplicate `## CodeGraph` instructions block into `CLAUDE.md` / `AGENTS.md` / `.cursor/rules/codegraph.mdc`. Edit tool guidance here and nowhere else. -```bash -codegraph init [path] # Initialize in project -codegraph index [path] # Full index -codegraph sync [path] # Incremental update -codegraph status [path] # Show statistics -codegraph query # Search symbols -codegraph context # Build context for AI -codegraph hooks install # Install git auto-sync -codegraph serve --mcp # Start MCP server -``` +## Retrieval performance & dynamic-dispatch coverage (do not regress) -## MCP Tools Best Practices +CodeGraph's core value is letting an agent answer **structural/flow** questions ("how does X reach Y", trace, impact, callers) with a few **fast** codegraph calls and **zero Read/Grep**. The optimization target is **wall-clock latency + tool-call count** — *don't optimize for token cost*. (Cost is **lower**, not "flat" as earlier framing claimed: a current-build with-vs-without A/B across the 7 README repos, median of 4, saved on average **35% cost · 57% tokens · 46% time · 71% tool calls** — reproducing the published README. The mechanism is **far fewer turns over a much smaller accumulated context** — NOT cache-ability: the without-arm's huge token volume is *mostly* cheap cache-reads, which is why token-count savings (57%) look bigger than cost savings (35%). Measure tokens by **summing per-turn assistant usage**, not `result.usage` (last-turn only in current Claude Code). See `docs/benchmarks/call-sequence-analysis.md`.) The mechanism that drives everything here: **an agent falls back to Read/Grep the instant a codegraph answer is insufficient.** So every change is judged by one question — is codegraph's answer sufficient enough to *stop* the agent from reading? -Use these tools **directly in the main session** for fast code exploration (replaces the need for Explore agents in most cases): +**Target behavior:** a flow question resolves in **1 codegraph call on small repos, scaling to 3–5 on large**, with **Read/Grep = 0**. When reviewing a PR or trying something new, do not regress this. -| Tool | Use For | -|------|---------| -| `codegraph_explore` | **Deep exploration** — comprehensive context for a topic in ONE call | -| `codegraph_context` | Quick context for a task (lighter than explore) | -| `codegraph_search` | Find symbols by name (functions, classes, types) | -| `codegraph_callers` | Find what calls a function | -| `codegraph_callees` | Find what a function calls | -| `codegraph_impact` | See what's affected by changing a symbol | -| `codegraph_node` | Get details + source code for a symbol | +### Adapt the tool to the agent — don't try to change the agent -### Important -CodeGraph provides **code context**, not product requirements. For new features, still ask the user about: -- UX preferences and behavior -- Edge cases and error handling -- Acceptance criteria +The lever that decides whether a retrieval change lands. **Test before building anything here: does this make a tool the agent _already calls_ do more with the input it _already gives_? If it instead needs the agent to behave differently — pick a different tool, query differently, learn from examples — it hits the low-salience wall and won't land.** -## Releases +CodeGraph's only channels to influence the agent are low-salience: the MCP `initialize` instructions (`server-instructions.ts`) and the tool descriptions. Changing them does **not** reliably move the agent's tool _choice_ or query style — validated: trace-first steering ported into the server-instructions + tool descriptions (3 wording variants) never reproduced what a CLI `--append-system-prompt` achieved, and **regressed** wall-clock vs baseline. New tools fare worse (rarely chosen — the agent under-picks even `trace`); "better examples" is the same steering. The agent's tool-choice does improve on its own as host models get better at tool use — but that is not ours to force. -Releases are published to npm **and** mirrored as GitHub Releases on the -[Releases page](https://github.com/colbymchenry/codegraph/releases), which is -where most users look for change history. `CHANGELOG.md` at the repo root is -the source of truth — each GitHub Release's notes are extracted from it. +What works is meeting the agent where it already is: +- **Sufficiency** — `codegraph_trace` inlines each hop's body + the destination's own callees, so one trace call ends the flow investigation (no follow-up explore/node/Read). +- **explore-flow** — `codegraph_explore`'s query is a precise bag of symbol names (incl. qualified `Class.method`) spanning the flow the agent is after; explore finds the call path _among those named symbols_ (riding synthesized edges) and leads its output with it — delivering trace-quality flow through the call the agent reliably makes. (`buildFlowFromNamedSymbols`: segment/co-naming disambiguation; ≤1 unnamed bridge so it never wanders a god-function's fan-out.) -### Writing changelog entries +What fails is the inverse — folding a precise answer into a **fuzzy-input** tool. `codegraph_context` gets a description, not symbols, so it can't disambiguate a flow's endpoints and surfaces the _wrong feature_. Precise output needs precise input. -When the user asks for a changelog entry for a new version: +The remaining lever under this axis is **coverage**: every flow made to connect statically (a new dynamic-dispatch synthesizer) is then surfaced automatically by explore-flow/`trace`, no agent change needed. Reactive/reconciler runtimes (Halo's `ReactiveExtensionClient`, MediatR, Vue Proxy) are the frontier — flows there have no static edges, so nothing surfaces (correctly — silent beats wrong). Full investigation + A/B record: `docs/benchmarks/call-sequence-analysis.md`. -1. Add a new `## [X.Y.Z] - YYYY-MM-DD` block at the **top** of `CHANGELOG.md` - (directly under the intro, above the previous version). -2. Group changes under `### Added`, `### Changed`, `### Fixed`, `### Removed`, - `### Deprecated`, `### Security` — only include sections that have entries. -3. Write entries from the **user's perspective**, not the implementation's. - Lead with the observable symptom or capability, then mention internals only - if a user needs them (e.g., to work around an existing bad install). -4. Add the link reference at the bottom: - `[X.Y.Z]: https://github.com/colbymchenry/codegraph/releases/tag/vX.Y.Z` +### Explore budget — keep BOTH budgets monotonic with repo size -### Release commands (the user runs these) +Two functions in `src/mcp/tools.ts` scale explore with indexed file count. This is the expected resolution (a regression here silently forces agents back to Read): -After the changelog entry is written and the version is bumped in `package.json`: +| Repo | files | explore calls | chars/call | per-file | +|---|---|---|---|---| +| express (small) | 147 | 1 | 18K | 3800 | +| excalidraw/django (medium) | 643–3043 | 2 | 28K | 6500 | +| vscode (large) | 10446 | 3 | 35K | 7000 | +| ~20k / ~40k | — | 4 / 5 | 38K | 7000 | -```bash -git add package.json package-lock.json CHANGELOG.md -git commit -m "release: X.Y.Z ()" -git push +- `getExploreBudget(fileCount)` → **call** budget: `<500→1, <5000→2, <15000→3, <25000→4, ≥25000→5` (max 5). +- `getExploreOutputBudget(fileCount)` → **per-call** output (chars / files / per-file). **Invariant: a larger tier must never get a smaller `maxCharsPerFile` than a smaller tier.** (Regression that motivated this doc: the `<5000` tier's 2500 was *below* the `<500` tier's 3800, so on a god-file repo — excalidraw's 415 KB `App.tsx` — one explore returned <1% of the file and forced a Read.) +- Explore output must **never tell the agent to "use Read"** — steer to another `codegraph_explore` and "treat returned source as already Read." -npm publish +### Dynamic-dispatch coverage — the flow must EXIST in the graph end-to-end -git tag vX.Y.Z -git push origin vX.Y.Z -gh release create vX.Y.Z \ - --title "vX.Y.Z" \ - --notes-file <(awk '/^## \[X.Y.Z\]/,/^## \[/{ if (/^## \[/ && !/X.Y.Z/) exit; print }' CHANGELOG.md) -``` +Static tree-sitter extraction misses computed/indirect calls, so flows break at dynamic dispatch and the agent reads to reconstruct them. Synthesizers/resolvers bridge these so `trace`/`explore` connect end-to-end (`src/resolution/callback-synthesizer.ts`, `src/resolution/frameworks/`). Channels today: callback/observer, EventEmitter, **React re-render** (`setState`→`render`), **JSX child** (`render`→child component), django ORM descriptor. All synthesized edges are `provenance:'heuristic'` with `metadata.synthesizedBy` + `registeredAt` (the wiring site), surfaced inline in `trace`, the `node` trail, and `context` call-paths. + +**Principle: partial coverage is WORSE than none.** Bridging one boundary but not the next reveals a hop the agent then drills + reads to finish. Measured on excalidraw: react-render alone *raised* reads to 5–7; only completing the flow (adding the jsx-child hop) dropped it to 0–1. **Always close the flow end-to-end and re-measure** — never ship a half-bridged flow. + +### Validation methodology (REQUIRED for every new language/framework) + +For each **language × framework**, validate on **small, medium, and large** real repos with **≥3 different flow prompts** each: + +1. **Pick the canonical flow** for the framework ("how does X reach Y": state→render, request→handler→view, query→SQL, action→reducer→store…). +2. **Deterministic probes** (`scripts/agent-eval/probe-{trace,node,context,explore}.mjs` against the built `dist/`): `trace(from,to)` connects end-to-end with no break; **no node explosion** (`select count(*) from nodes` stable before/after re-index); synthesized-edge **precision** spot-check (`select … where provenance='heuristic'`). +3. **Agent A/B** (`scripts/agent-eval/run-all.sh ""`): with vs without codegraph, **≥2 runs/arm** (run-to-run variance is large — never conclude from n=1). Record **duration, total tool calls, Read, Grep**. Optional forced-Read-0 sufficiency proof via the block-read hook (`scripts/agent-eval/hook-settings.json`). +4. **Pass bar:** a normal flow question reaches **~0 Read/Grep within the repo's explore-call budget**, runs **faster** than without-codegraph, and shows **no regression on a control repo**. Record the numbers in `docs/design/dynamic-dispatch-coverage-playbook.md` (the coverage matrix). + +Full playbook + per-mechanism design: `docs/design/dynamic-dispatch-coverage-playbook.md` and `docs/design/callback-edge-synthesis.md`. + +### Worked example — Excalidraw (TS/React, medium, 643 files) + +The template to replicate per language/framework. Question: *"how does updating an element re-render the canvas on screen?"* (the full flow crosses three React boundaries: observer callback, `setState`→`render`, and JSX child). + +| Stage | duration | Read | Grep | codegraph | +|---|---|---|---|---| +| Without codegraph | 115–139s | 9–10 | 10–11 | 0 | +| Broken (explore-budget regression) | 131–139s | 5–10 | 3–5 | 6–14 | +| Fixed (budget + msgs + synthesis) | 64–112s | 0–2 | 2–4 | 3–**10** | +| + trace-first steering | **51–74s** | **0–2** | 0–4 | **3–4** | -Do **not** run `npm publish`, `git tag`, `git push`, or `gh release create` -yourself — these are publish actions that affect shared state. Write the file, -hand the user the commands. +n=4 unhooked runs/stage, same prompt. After steering flow questions to `codegraph_trace` first: **best run 0 Read / 0 Grep / 3 codegraph / 51s**; **2 of 4 fully clean** (0 Read, 0 Grep). Steering eliminated the over-drill variance — call count tightened from 3–10 to 3–4, trace adoption went 3/4 → 4/4, and the `search`+`callers` path-reconstruction floundering dropped to 0. Run-to-run variance is still real; report the range, never a single run. **Residual reads/greps are all the nonce data-flow** (`canvasNonce` — a local prop with no graph edges); that's the def-use/data-flow frontier, left deliberately uncovered (tracking every local would explode the graph). Validated: `trace(mutateElement, renderStaticScene)` connects in **6 hops** across all three boundaries (`mutateElement → triggerUpdate → [callback] triggerRender → [react-render] render → [jsx] StaticCanvas → renderStaticScene`), each hop showing inline source + the wiring site; node count stable at 9,289; 1 callback + 46 react-render + 280 jsx-render synthesized edges (no explosion, precision-checked). -## Test Structure +## Tests -Tests are in `__tests__/` directory with files mirroring the module structure: -- `foundation.test.ts` - Database, config, directory management -- `extraction.test.ts` - Tree-sitter parsing for all languages -- `resolution.test.ts` - Reference resolution -- `graph.test.ts` - Traversal and graph queries -- `context.test.ts` - Context building -- `sync.test.ts` - Incremental updates and git hooks +Tests live in `__tests__/` and mirror the module they cover. Notable ones beyond the obvious: + +- `installer-targets.test.ts` — parameterized contract suite across all 4 agent targets (see installer notes above). +- `evaluation/` — `runner.ts` + `test-cases.ts` exercise codegraph against synthetic projects and score the results; run via `npm run eval` (builds first). Not part of `npm test`. +- `sqlite-backend.test.ts` — covers native + wasm backend selection and fallback. +- `pr19-improvements.test.ts`, `frameworks-integration.test.ts` — regression coverage for specific past PRs/incidents; don't rename these, the names anchor to git history. + +Tests create temp dirs with `fs.mkdtempSync` and clean up in `afterEach`. They write real files and exercise real SQLite — there is no DB mocking. + +### Windows-gated tests + +Behavior that differs by platform (path resolution, drive letters, `SENSITIVE_PATHS`, `%APPDATA%` config dirs, CRLF) must be gated, not assumed. Use `it.runIf(process.platform === 'win32')(...)` for Windows-only assertions and `it.runIf(process.platform !== 'win32')(...)` for POSIX-only ones — e.g. `/etc` is sensitive on POSIX but resolves to `C:\etc` (non-existent) on Windows, so an ungated `/etc` assertion fails on Windows. Validate the Windows side for real (see below); don't merge a Windows-gated test you haven't seen run. + +## Cross-platform validation + +The dev machine — and the default `npm test` target — is **macOS**, so local runs cover the macOS path. The other two platforms aren't here; when a change is platform-sensitive (file watching, sockets / named pipes, path & symlink handling, process lifecycle, inotify budget) validate them for real rather than guessing. + +### Linux (Docker) + +When asked to test or validate on Linux, use **Docker** — there's no Linux box, but Docker runs on the macOS host. Build a throwaway image from the repo and run the suite inside it: + +- `FROM node:22-bookworm`; `COPY` the repo with a `.dockerignore` excluding `node_modules`/`dist`/`.git`/`.codegraph`; `RUN npm ci && npm run build`. Don't reuse the Mac `node_modules` — `esbuild`/`rollup` ship platform-specific binaries. +- Run with **`docker run --rm --init`**. The `--init` is load-bearing for any process-lifecycle test (daemon reaping, the #277 PPID watchdog, idle-timeout): without a zombie-reaping PID 1, a SIGKILL'd/exited process lingers as a zombie and `process.kill(pid, 0)` still reports it *alive*, so exit-detection assertions false-fail even though the process did exit. +- Linux is where the inotify watch budget actually bites: count a process's watches via `/proc//fdinfo/*` (sum `^inotify ` lines on the fd whose `readlink` is `anon_inode:inotify`). + +### Windows (Parallels VM + SSH) + +For any Windows-specific PR, bug, or implementation, validate it on the real Windows VM rather than guessing. Connection details live in the gitignored **`.parallels`** file at the repo root (VM name, guest IP, SSH user/key). `prlctl exec` needs Parallels Pro and is unavailable, so SSH is the bridge. + +- Connect / run from the Mac host: `ssh @ "..."`. For multi-line work, pipe PowerShell over stdin and **refresh PATH from the registry** first (sshd's session has a stale PATH after winget installs): + ``` + ssh colby@10.211.55.3 "powershell -NoProfile -ExecutionPolicy Bypass -Command -" <<'PS' + $env:Path = [Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [Environment]::GetEnvironmentVariable("Path","User") + Set-Location C:\dev\codegraph + PS + ``` +- Clone fresh into a **Windows-local** path (`C:\dev\codegraph`) and `npm ci` there — never run npm against the shared Mac repo, since `esbuild`/`rollup` ship platform-specific binaries. +- Guest toolchain (winget): Node LTS, Git, and the **VC++ ARM64 redistributable** (required by `@rollup/rollup-win32-arm64-msvc`, which vitest pulls in). +- Fetch a contributor PR head straight from their fork to dodge `pull//head` lag: `git fetch ` then `git checkout -f FETCH_HEAD`. +- Known pre-existing Windows failures (they reproduce on `main`, unrelated to your change — confirm against `origin/main` before blaming your PR, and don't let them mask new regressions): `security.test.ts > Session marker symlink resistance > does not follow a pre-planted symlink` (symlink creation needs privileges on Windows); and the `mcp-initialize.test.ts` / `mcp-roots.test.ts` suites, which fail in `afterEach` with `EPERM` removing the temp dir because a spawned `serve --mcp` (its `--liftoff-only` re-exec grandchild) still holds the cwd / SQLite file open — a Windows file-locking quirk, not a logic bug. + +## Releases + +Released to npm and mirrored as [GitHub Releases](https://github.com/colbymchenry/codegraph/releases). `CHANGELOG.md` is the source of truth; GitHub Release notes are extracted from it. + +### Writing changelog entries -Tests use temporary directories created with `fs.mkdtempSync` and cleaned up after each test. +**Default: write entries under `## [Unreleased]`** — that's the section reserved for work landing between releases. **Don't pre-create a `## [X.Y.Z]` block** for the next release: the Release workflow's first step is `scripts/prepare-release.mjs`, which automatically promotes everything under `[Unreleased]` into a new `## [X.Y.Z] - ` block at release time (or merges into a pre-existing `[X.Y.Z]` block if one exists — but you don't need one). Pre-staging is what caused the v0.9.5 sparse-release-notes incident: a sparse `[0.9.5]` block hand-added before the rest of the work landed got picked by the extractor over the much-larger `[Unreleased]` section above it. Don't do that. + +Formatting rules for any entry (anywhere — `[Unreleased]` or otherwise): + +1. **Write friendly, user-facing notes — not engineer-facing ones.** Group under `### New Features` and `### Fixes` (sentence-case). Surface `### Breaking Changes` and `### Security` as their own sections **only when the release has them**; fold improvement-flavored changes into New Features. Omit empty sections. (This replaces the old Keep-a-Changelog `Added/Changed/Fixed/Removed/Deprecated` grouping: the GitHub Release page extracts each version block **verbatim** via `scripts/extract-release-notes.mjs`, and the old dense, implementation-focused entries rendered as an unreadable wall of text — so the whole CHANGELOG was rewritten to this format and every published release re-noted to match.) +2. **One plain-language sentence per bullet:** what changed and why it matters to a user. Lead with the capability, or with the symptom that's now fixed. +3. **Strip the internals.** No internal file paths (`src/...`), no internal symbol / function / class names, no benchmark numbers / percentages / node-or-edge counts. **Keep:** language & framework names (Go, Spring, NestJS, …), things a user types or sets (`codegraph install`, `codegraph_trace`, the `CODEGRAPH_*` env vars), agent / IDE names (Claude Code, Cursor, opencode, Kiro, …), and a brief `Thanks @user` when a contributor is credited. +4. Issue / PR references in entries are by number (`(#403)` etc.); the GitHub renderer auto-links them in the published release notes. +5. **Don't add a `[X.Y.Z]: https://...` link reference yourself** — `prepare-release.mjs` appends it automatically when it promotes the version (idempotent: a re-run is a no-op if it already exists). + +Multi-word headings like `### New Features` are safe on the normal release path: `prepare-release.mjs` **Case A** moves the whole `[Unreleased]` body verbatim into `[X.Y.Z]`. (Only its rarely-used **Case B** *merge* splits sub-sections with a single-word `^### (\w+)$` regex that wouldn't match them — and Case B fires only if a `[X.Y.Z]` block was pre-created, which rule above already forbids.) + +### Release flow (the user runs these) + +Releases are built and published by the **GitHub Actions "Release" workflow** +(`.github/workflows/release.yml`). It runs `scripts/prepare-release.mjs` to +promote `[Unreleased]` into `[]` (and auto-commit + push that +CHANGELOG change back to `main` so on-disk truth matches the published +notes), then bundles a Node runtime per platform (`scripts/build-bundle.sh`) +and publishes both the GitHub Release and the npm thin-installer +(`scripts/pack-npm.sh`: a shim package + per-platform packages). +Publishing manually is **wrong** now — a plain `npm publish` ships the root +package (non-bundled), which breaks anyone on Node < 22.5. + +**Claude does NOT bump the version unless explicitly asked.** The maintainer +typically does it themselves — often by editing `package.json` directly via +the GitHub web UI. Don't proactively commit a version bump as part of +unrelated work, and don't propose one when summarizing a PR. + +When the maintainer DOES bump the version, the only edit strictly required is +to `package.json` — the workflow's "Sync package-lock.json" step detects a +mismatch between `package.json` and `package-lock.json`, runs +`npm install --package-lock-only --ignore-scripts` to rewrite the lock file's +version fields (top-level + `packages.""`), and auto-commits + pushes the +result back to `main` with `[skip ci]`. So a GitHub-web-UI single-file edit to +`package.json` is enough to kick off a clean release. (If they edit both files +locally, that's fine too — the sync step no-ops.) + +Once `package.json` is at the target version on `main`, trigger +**Actions → Release → Run workflow** (on `main`). The workflow: + +1. Syncs `package-lock.json` to `package.json`'s version if they've drifted; commits + pushes that change. +2. Runs `prepare-release.mjs ` → promotes `[Unreleased]` → `[X.Y.Z] - ` in `CHANGELOG.md`, appends the link reference, commits + pushes the move with `[skip ci]`. +3. Builds every platform bundle on one runner, generates `SHA256SUMS`. +4. Creates the GitHub Release with notes from the freshly-promoted `[X.Y.Z]` block. +5. Publishes the npm shim + per-platform packages. Requires the `NPM_TOKEN` repo secret. + +**Do not run `npm publish`, `git push`, or `git tag` yourself** — these are +publish actions on shared state. Write the files, hand the user the commands. + +## House rules + +- The `0.7.x` line is in active multi-agent rollout. Any change to `src/installer/` (especially `targets/`) needs corresponding test coverage and a CHANGELOG entry — installer regressions break every new install silently. +- When changing what the MCP tools do or how agents should use them, edit `src/mcp/server-instructions.ts` — it is the **single source of truth** for agent-facing tool guidance (issue #529). The installer no longer writes a duplicate instructions block into `CLAUDE.md` / `AGENTS.md` / `GEMINI.md` / `.cursor/rules/codegraph.mdc` / Kiro steering, so there's nothing to keep in sync anymore. (The repo's own checked-in `.cursor/rules/codegraph.mdc` is dogfooding config — update it too if you use Cursor on this repo, but it ships nowhere.) +- CodeGraph provides **code context**, not product requirements. For new features, ask the user about UX, edge cases, and acceptance criteria — the graph won't tell you. +- **When the user references issues, PR comments, or external reports, anchor them to a date and version before drawing conclusions.** Check the comment's `createdAt` against: + - The **last released version** — `grep -m1 '^## \[' CHANGELOG.md` shows the top-of-file version (older releases follow). A comment dated before the latest `## [X.Y.Z] - YYYY-MM-DD` is reacting to *released* state — work that's only on `main` or on an unmerged branch doesn't apply. + - The **last main commit** — `git log --first-parent main -1 --format='%ai %h %s'`. A comment after the last release but before a fix on main may already be addressed there but unreleased. + - The **current branch's tip** — your own unmerged work obviously can't be what the comment is reacting to. + Always disambiguate "released," "merged-but-unreleased," and "in-progress" before agreeing that a user-reported problem is unfixed (or that a fix is incomplete). A user saying "your fix only covers X" about a recent PR is usually pointing at the *released* shortcomings — your in-flight branch may already address them but they have no way to know that. diff --git a/DELPHI-SUPPORT.md b/DELPHI-SUPPORT.md deleted file mode 100644 index 7d4524517..000000000 --- a/DELPHI-SUPPORT.md +++ /dev/null @@ -1,157 +0,0 @@ -# Pascal / Delphi Support for CodeGraph - -## Why Delphi? - -Delphi (Object Pascal) remains one of the most widely used languages for Windows desktop and enterprise applications. With an estimated **1.5–3 million active developers** and a strong presence in industries like healthcare, finance, logistics, and government, Delphi projects often involve large, long-lived codebases that benefit significantly from semantic code intelligence. - -Many Delphi codebases have grown over decades — making structural understanding, impact analysis, and cross-file navigation exactly the kind of tooling gap CodeGraph is designed to fill. - -Adding Delphi support positions CodeGraph as a uniquely valuable tool for a community that has historically been underserved by modern static analysis and AI-assisted development tools. - -## What Was Implemented - -### Pascal / Object Pascal (tree-sitter) - -Full extraction support for `.pas`, `.dpr`, `.dpk`, and `.lpr` files using the `tree-sitter-pascal` grammar: - -| Feature | NodeKind | Details | -|---------|----------|---------| -| Units / Programs | `module` | `unit`, `program`, `package`, `library` | -| Classes | `class` | Including inheritance and interface implementation | -| Records | `class` | Treated as classes (consistent with AST structure) | -| Interfaces | `interface` | With GUID support | -| Methods | `method` | Constructor, destructor, procedures, functions | -| Functions / Procedures | `function` | Top-level (non-class) routines | -| Properties | `property` | With read/write accessors | -| Fields | `field` | Class and record fields | -| Constants | `constant` | `const` declarations | -| Enums | `enum` | With enum members | -| Type Aliases | `type_alias` | `type TFoo = ...` | -| Uses / Imports | `import` | `uses` clause extraction | -| Function Calls | — | `calls` edges for call graph | -| Visibility | — | `public`, `private`, `protected` on methods/fields | -| Static Methods | — | `class function` / `class procedure` | -| Containment | — | `contains` edges (class → method, unit → type, etc.) | -| Inheritance | — | `extends` / `implements` edges | - -### DFM / FMX Form Files (custom extractor) - -Support for Delphi form files (`.dfm` for VCL, `.fmx` for FireMonkey) using a regex-based custom extractor — no tree-sitter grammar exists for this format: - -| Feature | NodeKind / EdgeKind | Details | -|---------|---------------------|---------| -| Components | `component` | `object Button1: TButton` | -| Nested hierarchy | `contains` | Panel1 → Button1 | -| Event handlers | `references` (unresolved) | `OnClick = Button1Click` → links UI to Pascal methods | -| `inherited` keyword | `component` | Inherited form components | -| Multi-line properties | — | Correctly skipped during parsing | -| Item collections | — | `...` blocks correctly handled | - -The DFM ↔ PAS linkage via event handlers enables **cross-file impact analysis**: renaming a method in `.pas` immediately reveals which UI components reference it. - -## Architecture - -The implementation follows CodeGraph's established patterns: - -- **Pascal extraction** uses the standard `TreeSitterExtractor` with a Pascal-specific `LanguageExtractor` configuration and a `visitPascalNode()` hook for AST nodes that require special handling (e.g., `declType` wrappers, `defProc` implementation bodies) -- **DFM/FMX extraction** uses a `DfmExtractor` class — analogous to `LiquidExtractor` and `SvelteExtractor` — that parses the line-based format with regex -- **Routing** in `extractFromSource()` dispatches `.dfm`/`.fmx` files to `DfmExtractor` before reaching the tree-sitter path -- **`tree-sitter-pascal`** is declared as an `optionalDependency` (consistent with all other grammars), pinned to a specific commit for reproducible builds - -## Performance Improvements - -Testing with a large Delphi codebase (~3,400 files, ~244k nodes) uncovered performance bottlenecks in the reference resolution pipeline. The following fixes **benefit all languages**, not just Pascal: - -| Fix | Scope | Impact | -|-----|-------|--------| -| **Fuzzy match index** — replaced O(n) linear scan with lazily-built case-insensitive `Map` index | `name-matcher.ts` (all languages) | O(1) lookup per ref instead of iterating all nodes | -| **Import mapping cache** — cached per-file import mappings instead of re-reading/re-parsing for every ref | `import-resolver.ts` (all languages) | Eliminated redundant file I/O during resolution | -| **Kind cache** — pre-populated `getNodesByKind` results during warm-up | `resolution/index.ts` (all languages) | Avoided repeated DB queries for the same node kinds | -| **Pascal built-in filtering** — skip known RTL/VCL/FMX identifiers before resolution | `resolution/index.ts` (Pascal-specific) | ~60 built-in identifiers filtered out early | -| **Method index for `defProc`** — replaced O(n) `find()` with `Map` lookup when linking implementation bodies to declarations | `tree-sitter.ts` (Pascal-specific) | O(1) per implementation body | -| **Delphi-specific excludes** — `__history/**`, `__recovery/**`, `*.dcu` added to default excludes | `types.ts` (Pascal-specific) | Skips Delphi IDE temp files during indexing | - -**Result:** Reference resolution on a large Delphi project dropped from **~30 minutes to ~15 seconds** (120x speedup). The general improvements (fuzzy index, import cache, kind cache) will benefit all CodeGraph users. - -## Files Changed - -| File | Change | -|------|--------| -| `src/types.ts` | Added `'pascal'` to `Language` type, file patterns to `DEFAULT_CONFIG.include` | -| `src/extraction/grammars.ts` | Grammar loader, extension mappings (`.pas`, `.dpr`, `.dpk`, `.lpr`, `.dfm`, `.fmx`), display name | -| `src/extraction/tree-sitter.ts` | Pascal `LanguageExtractor`, `visitPascalNode()` with 7 helper methods, `DfmExtractor` class, routing in `extractFromSource()`, method index | -| `src/resolution/index.ts` | Pascal built-in filtering, kind cache, cache clearing | -| `src/resolution/import-resolver.ts` | Import mapping cache | -| `src/resolution/name-matcher.ts` | Fuzzy match index (case-insensitive `Map`) | -| `package.json` | `tree-sitter-pascal` in `optionalDependencies` (pinned commit) | -| `__tests__/extraction.test.ts` | 37 new tests covering all Pascal and DFM extraction features | - -## Test Results - -- **36 new tests**, all passing -- **0 regressions** — the same 28 pre-existing failures (unrelated: missing Swift/Dart grammars, database path issues, MCP truncation test) are unchanged -- Tests cover: language detection, modules, imports, classes, records, interfaces, methods, visibility, static methods, enums, properties, constants, type aliases, calls, containment, full fixture files (UAuth.pas, UTypes.pas, MainForm.dfm) - -## Dependency Note - -The npm package `tree-sitter-pascal@0.0.1` is outdated (uses NAN bindings, incompatible with Node.js v24+). The implementation uses the actively maintained GitHub repository ([Isopod/tree-sitter-pascal](https://github.com/Isopod/tree-sitter-pascal), v0.10.2) with a pinned commit hash for deterministic builds. This is consistent with how `@sengac/tree-sitter-dart` handles a similar situation. - -## Testing Instructions - -### Prerequisites - -- Node.js >= 18 -- npm -- Git - -### 1. Clone and build - -```bash -git clone -b delphi-support https://github.com/omonien/codegraph.git -cd codegraph -npm install -npm run build -``` - -### 2. Link globally - -```bash -npm link -``` - -Verify with: - -```bash -codegraph --version -``` - -### 3. Index a Delphi project - -```bash -cd /path/to/your/delphi-project -codegraph init -i -codegraph index -``` - -### 4. Query the code graph - -```bash -codegraph status # Show index statistics -codegraph query "TFormMain" # Search for a symbol -codegraph context "What does TCustomer do?" # Build AI context -``` - -### 5. Set up the MCP server (for Claude Code) - -```bash -codegraph install -``` - -This configures the MCP server, tool permissions, auto-sync hooks, and CLAUDE.md in one step. After that, start Claude Code in the project — CodeGraph tools will be available immediately. - -### 6. Clean up - -```bash -npm unlink -g @colbymchenry/codegraph # Remove global link -rm -rf /path/to/delphi-project/.codegraph # Remove project index -``` diff --git a/IMPLEMENTATION_PLAN.md b/IMPLEMENTATION_PLAN.md deleted file mode 100644 index 65d99d82d..000000000 --- a/IMPLEMENTATION_PLAN.md +++ /dev/null @@ -1,1736 +0,0 @@ -# CodeGraph: Universal Code Knowledge Graph - -## Overview - -CodeGraph is a local-first code intelligence system that builds a semantic knowledge graph from any codebase. It provides structural understanding of code relationships—not just text similarity—enabling AI assistants to understand how code connects, what depends on what, and what breaks when something changes. - -**Type:** Headless library (no UI components — purely an API) -**Runtime:** Node.js (works standalone, in Electron, or any Node environment) -**Distribution:** npm package, installable in any project -**Per-Project Data:** `.codegraph/` directory in each indexed project -**Core Principle:** Deterministic extraction from AST, not AI-generated summaries - -### Use Cases - -1. **Beads Dashboard** — Integrated as a library to provide code intelligence -2. **Claude Code CLI users** — Install globally, run `codegraph init` in any project -3. **Any Node.js application** — Import as a library for code analysis -4. **MCP Server** — Expose as an MCP tool that Claude Code can query directly - ---- - -## Goals - -1. **Universal language support** via tree-sitter (PHP, Swift, Kotlin, Java, TypeScript, Python, Liquid, Ruby, Go, Rust, C#, etc.) -2. **Zero external API dependencies** for core functionality (local embeddings, local database) -3. **Portable per-project installation** — each project gets its own `.codegraph/` directory -4. **Incremental updates** via git hooks and hash-based change detection -5. **Rich structural queries** — callers, callees, impact radius, dependency chains -6. **Semantic search** — vector similarity to find entry points, then graph expansion - ---- - -## Architecture - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ CONSUMERS │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ -│ │ Beads │ │ Claude │ │ Any Node.js App │ │ -│ │ Dashboard │ │ Code CLI │ │ / MCP Server │ │ -│ │ (Electron) │ │ (Terminal) │ │ │ │ -│ └──────┬───────┘ └──────┬───────┘ └──────────┬───────────┘ │ -│ │ │ │ │ -│ └─────────────────┼──────────────────────┘ │ -│ │ │ -│ ▼ │ -├─────────────────────────────────────────────────────────────────┤ -│ CODEGRAPH LIBRARY │ -│ (npm package) │ -│ │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ -│ │ Context │ │ Query │ │ Sync │ │ -│ │ Builder │ │ Engine │ │ Manager │ │ -│ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────────┘ │ -│ │ │ │ │ -│ └────────────────┼─────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────┐│ -│ │ STORAGE LAYER ││ -│ │ SQLite + sqlite-vss (per project) ││ -│ │ .codegraph/graph.db ││ -│ └─────────────────────────────────────────────────────────────┘│ -│ ▲ │ -│ │ │ -│ ┌─────────────────────────────────────────────────────────────┐│ -│ │ EXTRACTION LAYER ││ -│ │ ││ -│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ ││ -│ │ │ Tree-sitter │ │ Reference │ │ Framework │ ││ -│ │ │ Parser │ │ Resolver │ │ Patterns │ ││ -│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ ││ -│ └─────────────────────────────────────────────────────────────┘│ -│ ▲ │ -│ │ │ -│ ┌─────────────────────────────────────────────────────────────┐│ -│ │ EMBEDDING LAYER ││ -│ │ Local ONNX Runtime + nomic-embed ││ -│ └─────────────────────────────────────────────────────────────┘│ -│ │ -└─────────────────────────────────────────────────────────────────┘ - -Per-Project Installation (created by codegraph init): -┌─────────────────────────────────────────────────────────────────┐ -│ my-laravel-app/ │ -│ ├── .codegraph/ │ -│ │ ├── graph.db # SQLite database with vectors │ -│ │ ├── config.json # Project-specific settings │ -│ │ └── .gitignore # Ignore db, keep config │ -│ ├── .git/ │ -│ │ └── hooks/ │ -│ │ └── post-commit # Triggers incremental reindex │ -│ ├── app/ │ -│ ├── routes/ │ -│ └── ... │ -└─────────────────────────────────────────────────────────────────┘ -``` - ---- - -## File Structure (npm package) - -``` -codegraph/ -├── package.json -├── tsconfig.json -├── README.md -│ -├── src/ -│ ├── index.ts # Main CodeGraph class, public API -│ ├── types.ts # TypeScript interfaces -│ │ -│ ├── db/ -│ │ ├── index.ts # Database initialization -│ │ ├── schema.sql # Table definitions -│ │ ├── migrations.ts # Schema versioning -│ │ └── queries.ts # Prepared statements -│ │ -│ ├── extraction/ -│ │ ├── index.ts # Extraction orchestrator -│ │ ├── tree-sitter.ts # Universal parser wrapper -│ │ ├── grammars.ts # Grammar loading and caching -│ │ └── queries/ # Tree-sitter query files (.scm) -│ │ ├── typescript.scm -│ │ ├── javascript.scm -│ │ ├── php.scm -│ │ ├── swift.scm -│ │ ├── kotlin.scm -│ │ ├── java.scm -│ │ ├── python.scm -│ │ ├── ruby.scm -│ │ ├── liquid.scm -│ │ ├── go.scm -│ │ └── csharp.scm -│ │ -│ ├── resolution/ -│ │ ├── index.ts # Reference resolver orchestrator -│ │ ├── name-matcher.ts # Symbol name matching -│ │ ├── import-resolver.ts # Import path resolution -│ │ └── frameworks/ # Framework-specific patterns -│ │ ├── index.ts -│ │ ├── laravel.ts -│ │ ├── express.ts -│ │ ├── nextjs.ts -│ │ ├── rails.ts -│ │ ├── shopify.ts -│ │ ├── spring.ts -│ │ └── swiftui.ts -│ │ -│ ├── graph/ -│ │ ├── index.ts # Graph query interface -│ │ ├── traversal.ts # BFS/DFS, impact radius -│ │ └── serialize.ts # Subgraph to context format -│ │ -│ ├── vectors/ -│ │ ├── index.ts # Vector operations interface -│ │ ├── embedder.ts # ONNX runtime + model -│ │ └── search.ts # Similarity search -│ │ -│ ├── sync/ -│ │ ├── index.ts # Sync orchestrator -│ │ ├── git-hooks.ts # Hook installation -│ │ └── hasher.ts # Content hashing for diffing -│ │ -│ └── context/ -│ ├── index.ts # Context builder -│ └── formatter.ts # Output formatting for Claude -│ -├── bin/ -│ └── codegraph.ts # CLI entry point (optional standalone usage) -│ -└── __tests__/ # Test files mirror src structure - ├── extraction/ - ├── resolution/ - ├── graph/ - └── fixtures/ # Sample code files for testing -``` - ---- - -## Database Schema - -**File: `src/db/schema.sql`** - -```sql --- ============================================================ --- CODEGRAPH SCHEMA v1 --- ============================================================ - --- Metadata table for schema versioning and project info -CREATE TABLE IF NOT EXISTS meta ( - key TEXT PRIMARY KEY, - value TEXT NOT NULL -); - --- ============================================================ --- NODES: Every significant code entity --- ============================================================ -CREATE TABLE IF NOT EXISTS nodes ( - id TEXT PRIMARY KEY, -- Unique ID: "func:src/auth.ts:validateToken:45" - kind TEXT NOT NULL, -- file, function, method, class, interface, type, variable, route, component, config - name TEXT NOT NULL, -- Human-readable: "validateToken" - qualified_name TEXT, -- Full path: "AuthService.validateToken" - file_path TEXT NOT NULL, -- Relative path: "src/services/auth.ts" - start_line INTEGER, - end_line INTEGER, - start_column INTEGER, - end_column INTEGER, - language TEXT NOT NULL, -- typescript, php, swift, etc. - signature TEXT, -- For functions: "(token: string) => Promise" - docstring TEXT, -- Extracted documentation - code_snippet TEXT, -- First ~500 chars of code for quick preview - code_hash TEXT NOT NULL, -- SHA256 of full code block - metadata TEXT, -- JSON: extra language/framework-specific data - created_at INTEGER NOT NULL, - updated_at INTEGER NOT NULL -); - --- ============================================================ --- EDGES: Relationships between nodes --- ============================================================ -CREATE TABLE IF NOT EXISTS edges ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - source_id TEXT NOT NULL, - target_id TEXT NOT NULL, - kind TEXT NOT NULL, -- imports, calls, extends, implements, returns_type, throws, reads, writes, renders, instantiates - resolved INTEGER DEFAULT 0, -- 0 = unresolved (name only), 1 = resolved to actual node - target_name TEXT, -- Original name before resolution (for unresolved edges) - line_number INTEGER, -- Where this relationship occurs - metadata TEXT, -- JSON: additional context - UNIQUE(source_id, target_id, kind, line_number), - FOREIGN KEY (source_id) REFERENCES nodes(id) ON DELETE CASCADE - -- Note: target_id may reference non-existent node if unresolved/external -); - --- ============================================================ --- FILES: Track file-level state for incremental updates --- ============================================================ -CREATE TABLE IF NOT EXISTS files ( - path TEXT PRIMARY KEY, -- Relative file path - content_hash TEXT NOT NULL, -- SHA256 of file contents - language TEXT NOT NULL, - last_indexed INTEGER NOT NULL, -- Unix timestamp - node_count INTEGER DEFAULT 0, - error TEXT -- Last indexing error, if any -); - --- ============================================================ --- VECTOR EMBEDDINGS (sqlite-vss) --- ============================================================ - --- Virtual table for vector similarity search --- Dimension 384 for nomic-embed-text-v1.5 -CREATE VIRTUAL TABLE IF NOT EXISTS node_vectors USING vss0( - embedding(384) -); - --- Map vector rowids to nodes -CREATE TABLE IF NOT EXISTS vector_map ( - rowid INTEGER PRIMARY KEY, - node_id TEXT NOT NULL UNIQUE, - text_hash TEXT NOT NULL, -- Hash of text that was embedded - FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE -); - --- ============================================================ --- INDEXES --- ============================================================ -CREATE INDEX IF NOT EXISTS idx_nodes_file ON nodes(file_path); -CREATE INDEX IF NOT EXISTS idx_nodes_kind ON nodes(kind); -CREATE INDEX IF NOT EXISTS idx_nodes_name ON nodes(name); -CREATE INDEX IF NOT EXISTS idx_nodes_language ON nodes(language); -CREATE INDEX IF NOT EXISTS idx_edges_source ON edges(source_id); -CREATE INDEX IF NOT EXISTS idx_edges_target ON edges(target_id); -CREATE INDEX IF NOT EXISTS idx_edges_kind ON edges(kind); -CREATE INDEX IF NOT EXISTS idx_edges_resolved ON edges(resolved); -``` - ---- - -## Type Definitions - -**File: `src/types.ts`** - -```typescript -// ============================================================ -// CORE TYPES -// ============================================================ - -export type NodeKind = - | 'file' - | 'function' - | 'method' - | 'class' - | 'interface' - | 'type' - | 'variable' - | 'constant' - | 'route' - | 'component' - | 'config' - | 'module' - | 'namespace'; - -export type EdgeKind = - | 'imports' - | 'exports' - | 'calls' - | 'called_by' // Reverse of calls, computed - | 'extends' - | 'implements' - | 'returns_type' - | 'throws' - | 'reads' - | 'writes' - | 'renders' // React/Vue component rendering - | 'instantiates' - | 'decorates' // Decorators/attributes - | 'depends_on'; // Generic dependency - -export type Language = - | 'typescript' - | 'javascript' - | 'php' - | 'swift' - | 'kotlin' - | 'java' - | 'python' - | 'ruby' - | 'go' - | 'rust' - | 'csharp' - | 'liquid' - | 'vue' - | 'svelte'; - -export interface Node { - id: string; - kind: NodeKind; - name: string; - qualifiedName?: string; - filePath: string; - startLine?: number; - endLine?: number; - startColumn?: number; - endColumn?: number; - language: Language; - signature?: string; - docstring?: string; - codeSnippet?: string; - codeHash: string; - metadata?: Record; - createdAt: number; - updatedAt: number; -} - -export interface Edge { - id?: number; - sourceId: string; - targetId: string; - kind: EdgeKind; - resolved: boolean; - targetName?: string; - lineNumber?: number; - metadata?: Record; -} - -export interface FileRecord { - path: string; - contentHash: string; - language: Language; - lastIndexed: number; - nodeCount: number; - error?: string; -} - -// ============================================================ -// EXTRACTION TYPES -// ============================================================ - -export interface ExtractionResult { - nodes: Node[]; - edges: Edge[]; - errors: ExtractionError[]; -} - -export interface ExtractionError { - filePath: string; - line?: number; - message: string; - recoverable: boolean; -} - -export interface UnresolvedReference { - sourceId: string; - targetName: string; - kind: EdgeKind; - lineNumber?: number; - context?: string; // Surrounding code for better resolution -} - -// ============================================================ -// QUERY TYPES -// ============================================================ - -export interface Subgraph { - nodes: Node[]; - edges: Edge[]; - entryPoints: string[]; // Node IDs that initiated the query - stats: { - totalNodes: number; - totalEdges: number; - maxDepth: number; - }; -} - -export interface TraversalOptions { - maxDepth?: number; // Default: 2 - maxNodes?: number; // Default: 50 - edgeKinds?: EdgeKind[]; // Filter by edge type - nodeKinds?: NodeKind[]; // Filter by node type - direction?: 'outbound' | 'inbound' | 'both'; -} - -export interface SearchOptions { - limit?: number; // Default: 10 - nodeKinds?: NodeKind[]; // Filter results - minScore?: number; // Similarity threshold -} - -export interface SearchResult { - node: Node; - score: number; -} - -// ============================================================ -// CONTEXT TYPES -// ============================================================ - -export interface Context { - subgraph: Subgraph; - codeBlocks: CodeBlock[]; - summary: string; - relatedFiles: string[]; -} - -export interface CodeBlock { - nodeId: string; - nodeName: string; - nodeKind: NodeKind; - filePath: string; - startLine: number; - endLine: number; - code: string; - language: Language; -} - -// ============================================================ -// CONFIG TYPES -// ============================================================ - -export interface CodeGraphConfig { - version: number; - projectName?: string; - languages: Language[]; - exclude: string[]; // Glob patterns to ignore - include?: string[]; // Override: only index these - frameworks: FrameworkHint[]; // Help with resolution - embeddingModel: 'nomic-embed-text-v1.5' | 'all-MiniLM-L6-v2'; - chunkStrategy: 'ast' | 'hybrid'; - maxFileSize: number; // Skip files larger than this (bytes) - gitHooksEnabled: boolean; -} - -export type FrameworkHint = - | 'laravel' - | 'express' - | 'nextjs' - | 'nuxt' - | 'rails' - | 'django' - | 'flask' - | 'spring' - | 'swiftui' - | 'uikit' - | 'android' - | 'shopify' - | 'react' - | 'vue' - | 'svelte'; - -export const DEFAULT_CONFIG: CodeGraphConfig = { - version: 1, - languages: [], - exclude: [ - 'node_modules/**', - 'vendor/**', - '.git/**', - 'dist/**', - 'build/**', - '*.min.js', - '*.bundle.js', - '__pycache__/**', - '.venv/**', - 'Pods/**', - '.gradle/**', - ], - frameworks: [], - embeddingModel: 'nomic-embed-text-v1.5', - chunkStrategy: 'ast', - maxFileSize: 1024 * 1024, // 1MB - gitHooksEnabled: true, -}; -``` - ---- - -## Public API - -**File: `src/index.ts`** - -```typescript -export class CodeGraph { - // ============================================================ - // LIFECYCLE - // ============================================================ - - /** - * Initialize CodeGraph for a project directory. - * Creates .codegraph/ if it doesn't exist. - */ - static async init(projectPath: string, config?: Partial): Promise; - - /** - * Open existing CodeGraph for a project. - * Throws if not initialized. - */ - static async open(projectPath: string): Promise; - - /** - * Check if a project has CodeGraph initialized. - */ - static async isInitialized(projectPath: string): Promise; - - /** - * Close database connections and cleanup. - */ - async close(): Promise; - - // ============================================================ - // INDEXING - // ============================================================ - - /** - * Full index of the entire project. - * Use for initial setup or complete rebuild. - */ - async indexAll(options?: { - onProgress?: (progress: IndexProgress) => void; - signal?: AbortSignal; - }): Promise; - - /** - * Index specific files only. - * Use for incremental updates. - */ - async indexFiles(filePaths: string[]): Promise; - - /** - * Sync with current file state. - * Detects changes via content hashing, reindexes only changed files. - */ - async sync(): Promise; - - /** - * Get current index status. - */ - async getStatus(): Promise; - - // ============================================================ - // GRAPH QUERIES - // ============================================================ - - /** - * Get a node by ID. - */ - async getNode(nodeId: string): Promise; - - /** - * Find nodes by name (exact or fuzzy). - */ - async findNodes(query: string, options?: { - fuzzy?: boolean; - kinds?: NodeKind[]; - limit?: number; - }): Promise; - - /** - * Get all edges from/to a node. - */ - async getEdges(nodeId: string, direction?: 'outbound' | 'inbound' | 'both'): Promise; - - /** - * Get nodes that call this node. - */ - async getCallers(nodeId: string): Promise; - - /** - * Get nodes that this node calls. - */ - async getCallees(nodeId: string): Promise; - - /** - * Get nodes that this node depends on. - */ - async getDependencies(nodeId: string): Promise; - - /** - * Get nodes that depend on this node. - */ - async getDependents(nodeId: string): Promise; - - /** - * Traverse the graph from starting nodes. - * Returns a subgraph of connected nodes up to maxDepth. - */ - async traverse(startNodeIds: string[], options?: TraversalOptions): Promise; - - /** - * Get impact radius: what could be affected by changing this node. - */ - async getImpactRadius(nodeId: string, options?: TraversalOptions): Promise; - - /** - * Find paths between two nodes. - */ - async findPaths(fromId: string, toId: string, options?: { - maxDepth?: number; - maxPaths?: number; - }): Promise; - - // ============================================================ - // SEMANTIC SEARCH - // ============================================================ - - /** - * Search for nodes by semantic similarity. - */ - async search(query: string, options?: SearchOptions): Promise; - - /** - * Find relevant subgraph for a natural language query. - * Combines semantic search with graph traversal. - */ - async findRelevantContext(query: string, options?: { - searchLimit?: number; - traversalDepth?: number; - maxNodes?: number; - }): Promise; - - // ============================================================ - // CONTEXT BUILDING - // ============================================================ - - /** - * Build context for a task/issue. - * Returns structured context ready to inject into Claude. - */ - async buildContext(input: string | { title: string; description?: string }, options?: { - maxNodes?: number; - includeCode?: boolean; - format?: 'markdown' | 'json'; - }): Promise; - - /** - * Get the full code for a node. - */ - async getCode(nodeId: string): Promise; - - // ============================================================ - // GIT INTEGRATION - // ============================================================ - - /** - * Install git hooks for automatic incremental indexing. - */ - async installGitHooks(): Promise; - - /** - * Remove git hooks. - */ - async removeGitHooks(): Promise; - - /** - * Get files changed since last index. - */ - async getChangedFiles(): Promise; - - // ============================================================ - // UTILITIES - // ============================================================ - - /** - * Get statistics about the indexed codebase. - */ - async getStats(): Promise; - - /** - * Export the graph to JSON. - */ - async export(): Promise; - - /** - * Update configuration. - */ - async updateConfig(config: Partial): Promise; - - /** - * Get current configuration. - */ - getConfig(): CodeGraphConfig; -} - -// ============================================================ -// RESULT TYPES -// ============================================================ - -export interface IndexProgress { - phase: 'scanning' | 'parsing' | 'resolving' | 'embedding'; - current: number; - total: number; - currentFile?: string; -} - -export interface IndexResult { - success: boolean; - filesIndexed: number; - nodesCreated: number; - edgesCreated: number; - errors: ExtractionError[]; - duration: number; -} - -export interface SyncResult { - filesChecked: number; - filesChanged: number; - filesAdded: number; - filesRemoved: number; - nodesUpdated: number; - duration: number; -} - -export interface IndexStatus { - initialized: boolean; - lastIndexed?: number; - totalFiles: number; - totalNodes: number; - totalEdges: number; - languages: Language[]; - unresolvedReferences: number; -} - -export interface GraphStats { - files: number; - nodes: { - total: number; - byKind: Record; - byLanguage: Record; - }; - edges: { - total: number; - byKind: Record; - resolved: number; - unresolved: number; - }; - vectors: number; -} - -export interface Path { - nodes: Node[]; - edges: Edge[]; - length: number; -} - -export interface ExportedGraph { - version: number; - exportedAt: number; - config: CodeGraphConfig; - stats: GraphStats; - nodes: Node[]; - edges: Edge[]; -} -``` - ---- - -## Tree-sitter Extraction Queries - -These `.scm` files define what to extract from each language. - -**File: `src/extraction/queries/typescript.scm`** - -```scheme -; ============================================================ -; TYPESCRIPT/JAVASCRIPT EXTRACTION QUERIES -; ============================================================ - -; Functions -(function_declaration - name: (identifier) @function.name - parameters: (formal_parameters) @function.params - return_type: (type_annotation)? @function.return_type - body: (statement_block) @function.body -) @function.definition - -; Arrow functions assigned to variables -(lexical_declaration - (variable_declarator - name: (identifier) @function.name - value: (arrow_function - parameters: (formal_parameters) @function.params - return_type: (type_annotation)? @function.return_type - body: (_) @function.body - ) - ) -) @function.definition - -; Classes -(class_declaration - name: (type_identifier) @class.name - (class_heritage - (extends_clause - value: (identifier) @class.extends - )? - (implements_clause - (type_identifier) @class.implements - )* - )? - body: (class_body) @class.body -) @class.definition - -; Methods -(method_definition - name: (property_identifier) @method.name - parameters: (formal_parameters) @method.params - return_type: (type_annotation)? @method.return_type - body: (statement_block) @method.body -) @method.definition - -; Interfaces -(interface_declaration - name: (type_identifier) @interface.name - (extends_type_clause - (type_identifier) @interface.extends - )? - body: (interface_body) @interface.body -) @interface.definition - -; Type aliases -(type_alias_declaration - name: (type_identifier) @type.name - value: (_) @type.value -) @type.definition - -; Imports -(import_statement - (import_clause - (identifier)? @import.default - (named_imports - (import_specifier - name: (identifier) @import.named - alias: (identifier)? @import.alias - )* - )? - )? - source: (string) @import.source -) @import.statement - -; Exports -(export_statement - (export_clause - (export_specifier - name: (identifier) @export.name - )* - )? - declaration: (_)? @export.declaration -) @export.statement - -; Function calls -(call_expression - function: [ - (identifier) @call.function - (member_expression - object: (_) @call.object - property: (property_identifier) @call.method - ) - ] - arguments: (arguments) @call.args -) @call.expression - -; Variable declarations (const/let with significant values) -(lexical_declaration - (variable_declarator - name: (identifier) @variable.name - value: (_) @variable.value - ) -) @variable.declaration - -; JSDoc comments -(comment) @comment -``` - -**File: `src/extraction/queries/php.scm`** - -```scheme -; ============================================================ -; PHP EXTRACTION QUERIES -; ============================================================ - -; Classes -(class_declaration - name: (name) @class.name - (base_clause - (name) @class.extends - )? - (class_interface_clause - (name) @class.implements - )* - body: (declaration_list) @class.body -) @class.definition - -; Methods -(method_declaration - (visibility_modifier)? @method.visibility - name: (name) @method.name - parameters: (formal_parameters) @method.params - return_type: (return_type)? @method.return_type - body: (compound_statement) @method.body -) @method.definition - -; Functions -(function_definition - name: (name) @function.name - parameters: (formal_parameters) @function.params - return_type: (return_type)? @function.return_type - body: (compound_statement) @function.body -) @function.definition - -; Interfaces -(interface_declaration - name: (name) @interface.name - (base_clause - (name) @interface.extends - )? - body: (declaration_list) @interface.body -) @interface.definition - -; Traits -(trait_declaration - name: (name) @trait.name - body: (declaration_list) @trait.body -) @trait.definition - -; Use statements (imports) -(namespace_use_declaration - (namespace_use_clause - (qualified_name) @import.name - (namespace_aliasing_clause - (name) @import.alias - )? - ) -) @import.statement - -; Static method calls (e.g., User::find()) -(scoped_call_expression - scope: (name) @call.class - name: (name) @call.method - arguments: (arguments) @call.args -) @call.static - -; Instance method calls -(member_call_expression - object: (_) @call.object - name: (name) @call.method - arguments: (arguments) @call.args -) @call.instance - -; Function calls -(function_call_expression - function: (name) @call.function - arguments: (arguments) @call.args -) @call.expression - -; Route definitions (Laravel-specific pattern) -(member_call_expression - object: (name) @_route (#eq? @_route "Route") - name: (name) @route.method - arguments: (arguments - (argument - (string) @route.path - ) - ) -) @route.definition - -; PHPDoc comments -(comment) @comment -``` - -**File: `src/extraction/queries/swift.scm`** - -```scheme -; ============================================================ -; SWIFT EXTRACTION QUERIES -; ============================================================ - -; Classes -(class_declaration - name: (type_identifier) @class.name - (type_inheritance_clause - (type_identifier) @class.inherits - )? - body: (class_body) @class.body -) @class.definition - -; Structs -(struct_declaration - name: (type_identifier) @struct.name - (type_inheritance_clause - (type_identifier) @struct.conforms - )? - body: (struct_body) @struct.body -) @struct.definition - -; Protocols -(protocol_declaration - name: (type_identifier) @protocol.name - body: (protocol_body) @protocol.body -) @protocol.definition - -; Functions -(function_declaration - name: (simple_identifier) @function.name - (parameter_clause) @function.params - (function_result - (type_annotation) @function.return_type - )? - body: (function_body) @function.body -) @function.definition - -; Methods (inside class/struct) -(function_declaration - name: (simple_identifier) @method.name - (parameter_clause) @method.params - body: (function_body) @method.body -) @method.definition - -; Properties -(property_declaration - (pattern - (simple_identifier) @property.name - ) - (type_annotation)? @property.type -) @property.definition - -; Imports -(import_declaration - (identifier) @import.module -) @import.statement - -; Function calls -(call_expression - (simple_identifier) @call.function - (call_suffix - (value_arguments) @call.args - ) -) @call.expression - -; Method calls -(call_expression - (navigation_expression - (_) @call.object - (navigation_suffix - (simple_identifier) @call.method - ) - ) - (call_suffix - (value_arguments) @call.args - ) -) @call.method - -; SwiftUI View bodies -(computed_property - name: (simple_identifier) @_body (#eq? @_body "body") - (type_annotation - (user_type - (type_identifier) @_view (#match? @_view "View") - ) - )? - getter: (_) @view.body -) @view.definition - -; Documentation comments -(comment) @comment -(multiline_comment) @comment.multiline -``` - ---- - -## Framework Pattern Resolvers - -**File: `src/resolution/frameworks/laravel.ts`** - -```typescript -import { FrameworkResolver, UnresolvedReference, ResolvedReference } from '../types'; - -export const laravelResolver: FrameworkResolver = { - name: 'laravel', - - // Detect if this is a Laravel project - detect: async (projectPath: string): Promise => { - return await fileExists(join(projectPath, 'artisan')); - }, - - patterns: [ - // Eloquent Model static calls: User::find(), Post::where() - { - pattern: /^([A-Z][a-zA-Z]+)::(\w+)$/, - resolve: async (match, context) => { - const [, className, methodName] = match; - - // Check app/Models first (Laravel 8+) - let modelPath = `app/Models/${className}.php`; - if (await context.fileExists(modelPath)) { - return { filePath: modelPath, className, methodName }; - } - - // Fall back to app/ (Laravel 7 and below) - modelPath = `app/${className}.php`; - if (await context.fileExists(modelPath)) { - return { filePath: modelPath, className, methodName }; - } - - return null; - } - }, - - // Facade calls: Auth::user(), Cache::get() - { - pattern: /^(Auth|Cache|DB|Log|Mail|Queue|Session|Storage|Validator)::(\w+)$/, - resolve: async (match, context) => { - const [, facade, method] = match; - // Facades resolve to underlying service - we can link to the facade for now - return { - filePath: `vendor/laravel/framework/src/Illuminate/Support/Facades/${facade}.php`, - className: facade, - methodName: method, - isExternal: true - }; - } - }, - - // Route helpers: route('checkout.store') - { - pattern: /route\(['"]([^'"]+)['"]\)/, - resolve: async (match, context) => { - const [, routeName] = match; - // Search routes/web.php and routes/api.php for ->name('routeName') - const routeFiles = ['routes/web.php', 'routes/api.php']; - for (const file of routeFiles) { - const content = await context.readFile(file); - if (content?.includes(`name('${routeName}')`)) { - return { filePath: file, routeName }; - } - } - return null; - } - }, - - // View helpers: view('checkout.form') - { - pattern: /view\(['"]([^'"]+)['"]\)/, - resolve: async (match, context) => { - const [, viewName] = match; - const viewPath = viewName.replace(/\./g, '/'); - - // Check both .blade.php and .php - const candidates = [ - `resources/views/${viewPath}.blade.php`, - `resources/views/${viewPath}.php` - ]; - - for (const candidate of candidates) { - if (await context.fileExists(candidate)) { - return { filePath: candidate, viewName }; - } - } - return null; - } - }, - - // Controller references in routes - { - pattern: /\[([A-Z][a-zA-Z]+Controller)::class,\s*['"](\w+)['"]\]/, - resolve: async (match, context) => { - const [, controller, method] = match; - const controllerPath = `app/Http/Controllers/${controller}.php`; - if (await context.fileExists(controllerPath)) { - return { filePath: controllerPath, className: controller, methodName: method }; - } - return null; - } - } - ], - - // Additional node detection specific to Laravel - extractNodes: async (filePath: string, content: string) => { - const nodes: Node[] = []; - - // Detect route definitions - const routePattern = /Route::(get|post|put|patch|delete)\(\s*['"]([^'"]+)['"]/g; - let match; - while ((match = routePattern.exec(content)) !== null) { - const [, method, path] = match; - const line = content.slice(0, match.index).split('\n').length; - nodes.push({ - id: `route:${filePath}:${method.toUpperCase()}:${path}`, - kind: 'route', - name: `${method.toUpperCase()} ${path}`, - filePath, - startLine: line, - language: 'php', - metadata: { httpMethod: method.toUpperCase(), path } - }); - } - - return nodes; - } -}; -``` - -**File: `src/resolution/frameworks/shopify.ts`** - -```typescript -import { FrameworkResolver } from '../types'; - -export const shopifyResolver: FrameworkResolver = { - name: 'shopify', - - detect: async (projectPath: string): Promise => { - return await fileExists(join(projectPath, 'shopify.theme.toml')) || - await fileExists(join(projectPath, 'config/settings_schema.json')); - }, - - patterns: [ - // Render tags: {% render 'product-card' %} - { - pattern: /\{%\s*render\s+['"]([^'"]+)['"]/, - resolve: async (match, context) => { - const [, snippetName] = match; - const snippetPath = `snippets/${snippetName}.liquid`; - if (await context.fileExists(snippetPath)) { - return { filePath: snippetPath, kind: 'renders' }; - } - return null; - } - }, - - // Include tags: {% include 'header' %} - { - pattern: /\{%\s*include\s+['"]([^'"]+)['"]/, - resolve: async (match, context) => { - const [, snippetName] = match; - const snippetPath = `snippets/${snippetName}.liquid`; - if (await context.fileExists(snippetPath)) { - return { filePath: snippetPath, kind: 'includes' }; - } - return null; - } - }, - - // Section tags: {% section 'header' %} - { - pattern: /\{%\s*section\s+['"]([^'"]+)['"]/, - resolve: async (match, context) => { - const [, sectionName] = match; - const sectionPath = `sections/${sectionName}.liquid`; - if (await context.fileExists(sectionPath)) { - return { filePath: sectionPath, kind: 'renders' }; - } - return null; - } - }, - - // Asset URLs: {{ 'style.css' | asset_url }} - { - pattern: /['"]([\w\-\.]+)['"]\s*\|\s*asset_url/, - resolve: async (match, context) => { - const [, assetName] = match; - const assetPath = `assets/${assetName}`; - if (await context.fileExists(assetPath)) { - return { filePath: assetPath, kind: 'references' }; - } - return null; - } - } - ], - - extractNodes: async (filePath: string, content: string) => { - const nodes: Node[] = []; - - // Detect schema in sections - const schemaMatch = content.match(/\{%\s*schema\s*%\}([\s\S]*?)\{%\s*endschema\s*%\}/); - if (schemaMatch) { - try { - const schema = JSON.parse(schemaMatch[1]); - if (schema.name) { - nodes.push({ - id: `section:${filePath}`, - kind: 'component', - name: schema.name, - filePath, - language: 'liquid', - metadata: { - schemaSettings: schema.settings?.map(s => s.id), - schemaBlocks: schema.blocks?.map(b => b.type) - } - }); - } - } catch (e) { - // Invalid JSON in schema - } - } - - return nodes; - } -}; -``` - ---- - -## Context Builder Output Format - -**File: `src/context/formatter.ts`** - -```typescript -export function formatContextAsMarkdown(context: Context): string { - const lines: string[] = []; - - lines.push('## Code Context\n'); - - // Graph structure section - lines.push('### Structure\n'); - lines.push('```'); - for (const nodeId of context.subgraph.entryPoints) { - const node = context.subgraph.nodes.find(n => n.id === nodeId); - if (node) { - lines.push(formatNodeTree(node, context.subgraph, 0)); - } - } - lines.push('```\n'); - - // Code blocks section - if (context.codeBlocks.length > 0) { - lines.push('### Code\n'); - for (const block of context.codeBlocks) { - lines.push(`#### ${block.nodeName} (${block.filePath}:${block.startLine})\n`); - lines.push('```' + block.language); - lines.push(block.code); - lines.push('```\n'); - } - } - - // Related files section - if (context.relatedFiles.length > 0) { - lines.push('### Related Files\n'); - for (const file of context.relatedFiles) { - lines.push(`- ${file}`); - } - } - - return lines.join('\n'); -} - -function formatNodeTree(node: Node, subgraph: Subgraph, depth: number): string { - const indent = ' '.repeat(depth); - const lines: string[] = []; - - // Node header - const location = node.startLine ? `:${node.startLine}` : ''; - lines.push(`${indent}${node.name} (${node.filePath}${location})`); - - // Outbound edges - const outbound = subgraph.edges.filter(e => e.sourceId === node.id); - for (const edge of outbound) { - const target = subgraph.nodes.find(n => n.id === edge.targetId); - const targetName = target?.name || edge.targetName || 'unknown'; - lines.push(`${indent}├── ${edge.kind} → ${targetName}`); - } - - return lines.join('\n'); -} - -// Example output: -// -// ## Code Context -// -// ### Structure -// ``` -// CheckoutController (app/Http/Controllers/CheckoutController.php:15) -// ├── calls → CartService.getCart -// ├── calls → PaymentService.processPayment -// ├── calls → OrderService.create -// ├── throws → PaymentException -// -// PaymentService (app/Services/PaymentService.php:8) -// ├── calls → StripeClient.charge -// ├── calls → TransactionRepository.save -// ├── throws → PaymentException -// ├── throws → StripeTimeoutException -// ``` -// -// ### Code -// -// #### store (app/Http/Controllers/CheckoutController.php:45) -// ```php -// public function store(Request $request) -// { -// $cart = $this->cartService->getCart($request->user()); -// $payment = $this->paymentService->processPayment($cart); -// ... -// } -// ``` -``` - ---- - -## Installation & Integration - -**How to use CodeGraph (headless library, no UI):** - -### Option 1: CLI (for any project, no code required) - -```bash -# Install globally -npm install -g codegraph - -# Initialize in any project -cd /path/to/my-laravel-app -codegraph init - -# Index the codebase -codegraph index - -# Query the graph -codegraph query "what calls PaymentService" -codegraph impact "app/Services/AuthService.php" - -# Build context for a task (outputs markdown) -codegraph context "Fix checkout silent failure" - -# Check status -codegraph status - -# Sync after changes -codegraph sync -``` - -### Option 2: Library (for integration into apps like Beads Dashboard) - -```typescript -import { CodeGraph } from 'codegraph'; - -// Initialize for a project -const graph = await CodeGraph.init('/path/to/project'); - -// Full index with optional progress callback -await graph.indexAll({ - onProgress: (progress) => { - console.log(`${progress.phase}: ${progress.current}/${progress.total}`); - } -}); - -// Or open existing and sync -const graph = await CodeGraph.open('/path/to/project'); -const syncResult = await graph.sync(); - -// Build context for a task (returns structured data) -const context = await graph.buildContext('Fix checkout silent failure'); - -// Query the graph directly -const callers = await graph.getCallers('func:src/payment.ts:processPayment:45'); -const impact = await graph.getImpactRadius('class:AuthService', { maxDepth: 2 }); - -// Search semantically -const results = await graph.search('authentication middleware'); - -// Clean up -await graph.close(); -``` - -### Option 3: MCP Server (for Claude Code CLI integration) - -```bash -# Run as MCP server (Claude Code can query directly) -codegraph serve --mcp - -# In Claude Code's MCP config, add: -# { -# "codegraph": { -# "command": "codegraph", -# "args": ["serve", "--mcp", "--project", "/path/to/project"] -# } -# } -``` - -Then Claude Code can use tools like: -- `codegraph_search` — semantic search -- `codegraph_context` — build context for a task -- `codegraph_callers` — who calls this function -- `codegraph_impact` — what's affected if I change this - -**What gets created in the project:** - -``` -my-project/ -├── .codegraph/ -│ ├── graph.db # SQLite database (gitignored) -│ ├── config.json # User can customize (committed) -│ └── .gitignore # Contains: graph.db -└── .git/ - └── hooks/ - └── post-commit # Auto-installed hook -``` - -**Default `.codegraph/config.json`:** - -```json -{ - "version": 1, - "exclude": [ - "node_modules/**", - "vendor/**", - "dist/**", - "build/**" - ], - "frameworks": ["laravel"], - "gitHooksEnabled": true -} -``` - ---- - -## Implementation Phases - -### Phase 1: Foundation (Week 1) -- [ ] Project structure setup (npm package) -- [ ] SQLite database initialization with schema -- [ ] Basic types and interfaces -- [ ] Config file handling -- [ ] .codegraph/ directory management - -### Phase 2: Tree-sitter Extraction (Week 1-2) -- [ ] Tree-sitter native bindings setup (works in Node.js, Electron, etc.) -- [ ] Grammar loading system -- [ ] TypeScript/JavaScript extraction queries -- [ ] PHP extraction queries -- [ ] Basic node/edge extraction from AST - -### Phase 3: Reference Resolution (Week 2) -- [ ] Name-based symbol matching -- [ ] Import path resolution -- [ ] Laravel framework patterns -- [ ] Express/Next.js patterns -- [ ] Unresolved reference tracking - -### Phase 4: Graph Queries (Week 2-3) -- [ ] Basic traversal (callers, callees) -- [ ] Impact radius calculation -- [ ] Path finding between nodes -- [ ] Subgraph extraction - -### Phase 5: Vector Embeddings (Week 3) -- [ ] ONNX runtime integration -- [ ] nomic-embed-text model loading -- [ ] sqlite-vss setup -- [ ] Embedding generation for nodes -- [ ] Similarity search - -### Phase 6: Context Builder (Week 3-4) -- [ ] Semantic search → graph expansion pipeline -- [ ] Context formatting for Claude -- [ ] Code snippet extraction -- [ ] Output size management - -### Phase 7: Sync & Freshness (Week 4) -- [ ] Content hashing for change detection -- [ ] Incremental reindexing -- [ ] Git hook installation -- [ ] Post-commit handler - -### Phase 8: Additional Languages (Week 4+) -- [ ] Swift extraction queries -- [ ] Kotlin extraction queries -- [ ] Java extraction queries -- [ ] Liquid/Shopify patterns -- [ ] Ruby/Rails patterns - -### Phase 9: Polish & Hardening (Week 5) -- [ ] Error handling and recovery -- [ ] Performance optimization -- [ ] Memory management for large codebases -- [ ] Concurrent indexing safety -- [ ] API documentation and JSDoc comments - -### Phase 10: CLI (Week 5-6, Optional) -- [ ] CLI argument parsing (commander or yargs) -- [ ] `codegraph init` command -- [ ] `codegraph index` command -- [ ] `codegraph query` command -- [ ] `codegraph context` command -- [ ] `codegraph status` command -- [ ] `codegraph sync` command - -### Phase 11: MCP Server (Week 6, Optional) -- [ ] MCP protocol implementation -- [ ] `codegraph_search` tool -- [ ] `codegraph_context` tool -- [ ] `codegraph_callers` / `codegraph_callees` tools -- [ ] `codegraph_impact` tool -- [ ] Stdio transport for Claude Code integration - ---- - -## Testing Strategy - -```typescript -// Example test structure - -describe('CodeGraph', () => { - describe('extraction', () => { - it('extracts functions from TypeScript', async () => { - const code = ` - export function processPayment(amount: number): Promise { - return stripe.charge(amount); - } - `; - const result = await extract(code, 'typescript'); - - expect(result.nodes).toContainEqual(expect.objectContaining({ - kind: 'function', - name: 'processPayment', - signature: '(amount: number): Promise' - })); - - expect(result.edges).toContainEqual(expect.objectContaining({ - kind: 'calls', - targetName: 'stripe.charge' - })); - }); - - it('extracts Laravel routes from PHP', async () => { - const code = ` - Route::post('/checkout', [CheckoutController::class, 'store'])->name('checkout.store'); - `; - const result = await extract(code, 'php'); - - expect(result.nodes).toContainEqual(expect.objectContaining({ - kind: 'route', - name: 'POST /checkout' - })); - }); - }); - - describe('resolution', () => { - it('resolves Laravel model calls', async () => { - const graph = await createTestGraph({ - 'app/Models/User.php': 'class User extends Model { public static function find($id) {} }', - 'app/Http/Controllers/UserController.php': 'User::find($id);' - }); - - const edges = await graph.getEdges('controller:UserController:show'); - expect(edges).toContainEqual(expect.objectContaining({ - kind: 'calls', - targetId: 'method:app/Models/User.php:find', - resolved: true - })); - }); - }); - - describe('traversal', () => { - it('finds impact radius', async () => { - const graph = await createTestGraph(/* ... */); - const subgraph = await graph.getImpactRadius('class:PaymentService', { maxDepth: 2 }); - - expect(subgraph.nodes.map(n => n.name)).toContain('CheckoutController'); - expect(subgraph.nodes.map(n => n.name)).toContain('OrderService'); - }); - }); -}); -``` - ---- - -## Open Questions / Decisions Needed - -1. **Embedding model size vs quality**: nomic-embed-text-v1.5 (275MB) vs all-MiniLM-L6-v2 (90MB)? - -2. **Tree-sitter WASM vs native**: WASM is easier for Electron distribution, native is faster. Start with WASM? - -3. **Max context size**: How many nodes/code blocks before we truncate? Configurable? - -4. **Unresolved references**: Show them in context (with "unresolved" marker) or hide them? - -5. **Multi-language projects**: Projects mixing PHP + JS + Liquid — handle all simultaneously? - -6. **Binary/asset files**: Track references to images, fonts, etc. or ignore? - ---- - -## Success Criteria - -1. **Accuracy**: >90% of function calls correctly linked to definitions -2. **Speed**: Full index of 10k file project in <60 seconds -3. **Freshness**: Incremental update after commit in <5 seconds -4. **Context quality**: Generated context helps Claude solve issues faster (qualitative) -5. **Portability**: Works on any macOS machine without additional setup - ---- - -## Resources - -- Tree-sitter: https://tree-sitter.github.io/tree-sitter/ -- Tree-sitter WASM: https://github.com/nicolo-ribaudo/nicolo-nicolo-tree-sitter/tree-sitter-wasm-builds/tree/main -- sqlite-vss: https://github.com/asg017/sqlite-vss -- nomic-embed: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 -- ONNX Runtime Node: https://onnxruntime.ai/docs/get-started/with-javascript.html diff --git a/README.md b/README.md index f8f39e978..acdb726bf 100644 --- a/README.md +++ b/README.md @@ -2,39 +2,77 @@ # CodeGraph -### Supercharge Claude Code with Semantic Code Intelligence +### Supercharge Claude Code, Cursor, Codex, OpenCode, Hermes Agent, Gemini, Antigravity, and Kiro with Semantic Code Intelligence -**94% fewer tool calls · 77% faster exploration · 100% local** +**~25% cheaper · ~62% fewer tool calls · 100% local** + +### [Documentation & Website →](https://colbymchenry.github.io/codegraph/) [![npm version](https://img.shields.io/npm/v/@colbymchenry/codegraph.svg)](https://www.npmjs.com/package/@colbymchenry/codegraph) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Node.js](https://img.shields.io/badge/Node.js-18+-green.svg)](https://nodejs.org/) +[![Self-contained](https://img.shields.io/badge/Node.js-bundled%20%C2%B7%20none%20required-brightgreen.svg)](https://nodejs.org/) + +[![Windows](https://img.shields.io/badge/Windows-supported-blue.svg)](#supported-platforms) +[![macOS](https://img.shields.io/badge/macOS-supported-blue.svg)](#supported-platforms) +[![Linux](https://img.shields.io/badge/Linux-supported-blue.svg)](#supported-platforms) + +[![Claude Code](https://img.shields.io/badge/Claude_Code-supported-blueviolet.svg)](#supported-agents) +[![Cursor](https://img.shields.io/badge/Cursor-supported-blueviolet.svg)](#supported-agents) +[![Codex](https://img.shields.io/badge/Codex-supported-blueviolet.svg)](#supported-agents) +[![opencode](https://img.shields.io/badge/opencode-supported-blueviolet.svg)](#supported-agents) +[![Hermes Agent](https://img.shields.io/badge/Hermes_Agent-supported-blueviolet.svg)](#supported-agents) +[![Gemini](https://img.shields.io/badge/Gemini-supported-blueviolet.svg)](#supported-agents) +[![Antigravity](https://img.shields.io/badge/Antigravity-supported-blueviolet.svg)](#supported-agents) +[![Kiro](https://img.shields.io/badge/Kiro-supported-blueviolet.svg)](#supported-agents) -[![Windows](https://img.shields.io/badge/Windows-supported-blue.svg)](#) -[![macOS](https://img.shields.io/badge/macOS-supported-blue.svg)](#) -[![Linux](https://img.shields.io/badge/Linux-supported-blue.svg)](#) + -
+## Get Started -### Get Started +**No Node.js required** — one command grabs the right build for your OS: ```bash -npx @colbymchenry/codegraph +# macOS / Linux +curl -fsSL https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.sh | sh + +# Windows (PowerShell) +irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex ``` -Interactive installer configures Claude Code automatically +Already have Node? Use npm instead (works on any version): -#### Initialize Projects +```bash +npx @colbymchenry/codegraph # zero-install, or: +npm i -g @colbymchenry/codegraph +``` + +CodeGraph bundles its own runtime — nothing to compile, no native build, works the same everywhere. The interactive installer auto-configures your agent(s) — Claude Code, Cursor, Codex CLI, opencode, Hermes Agent, Gemini CLI, Antigravity IDE, Kiro. + +### Initialize Projects ```bash cd your-project codegraph init -i ``` +`codegraph init` just creates the local `.codegraph/` index directory; adding `-i` (`--index`) also builds the initial graph in the same step. Without `-i`, run `codegraph index` afterwards to populate it. + +
+ ![1_C_VYnhpys0UHrOuOgpgoyw](https://github.com/user-attachments/assets/f168182f-4d9a-44e0-94d7-08d018cc8a3a)
+### Uninstall + +Changed your mind? One command removes CodeGraph from every agent it configured: + +```bash +codegraph uninstall +``` + +Reverses the installer — strips CodeGraph's MCP server config, instructions, and permissions from each configured agent. Your project indexes (`.codegraph/`) are left untouched; remove those per-project with `codegraph uninit`. Use `--target` to remove from specific agents, or `--yes` to run non-interactively. + --- ## Why CodeGraph? @@ -45,61 +83,114 @@ When Claude Code explores a codebase, it spawns **Explore agents** that scan fil ### Benchmark Results -Tested across 6 real-world codebases comparing Claude Code's Explore agent **with** and **without** CodeGraph: +Tested across **7 real-world open-source codebases** spanning 7 languages, comparing an agent (Claude Code, headless) answering one architecture question **with** and **without** CodeGraph. Each cell is the savings at the **median of 4 runs per arm**. _Re-validated on Opus 4.8 (2026-05-29), on the build with per-symbol adaptive `codegraph_explore` sizing._ + +> **Average: 25% cheaper · 57% fewer tokens · 23% faster · 62% fewer tool calls** -> **Average: 92% fewer tool calls · 71% faster** +| Codebase | Language | Cost | Tokens | Time | Tool calls | +|----------|----------|------|--------|------|------------| +| **VS Code** | TypeScript · ~10k files | 33% cheaper | 70% fewer | 27% faster | 80% fewer | +| **Excalidraw** | TypeScript · ~640 | 27% cheaper | 61% fewer | 26% faster | 70% fewer | +| **Django** | Python · ~3k | 23% cheaper | 70% fewer | 28% faster | 77% fewer | +| **Tokio** | Rust · ~790 | 35% cheaper | 70% fewer | 37% faster | 79% fewer | +| **OkHttp** | Java · ~645 | 11% cheaper | 48% fewer | 26% faster | 70% fewer | +| **Gin** | Go · ~110 | 15% cheaper | 35% fewer | 9% faster | 47% fewer | +| **Alamofire** | Swift · ~110 | 28% cheaper | 46% fewer | 7% faster | 13% fewer | -| Codebase | With CG | Without CG | Improvement | -|----------|---------|------------|-------------| -| **VS Code** · TypeScript | 3 calls, 17s | 52 calls, 1m 37s | **94% fewer · 82% faster** | -| **Excalidraw** · TypeScript | 3 calls, 29s | 47 calls, 1m 45s | **94% fewer · 72% faster** | -| **Claude Code** · Python + Rust | 3 calls, 39s | 40 calls, 1m 8s | **93% fewer · 43% faster** | -| **Claude Code** · Java | 1 call, 19s | 26 calls, 1m 22s | **96% fewer · 77% faster** | -| **Alamofire** · Swift | 3 calls, 22s | 32 calls, 1m 39s | **91% fewer · 78% faster** | -| **Swift Compiler** · Swift/C++ | 6 calls, 35s | 37 calls, 2m 8s | **84% fewer · 73% faster** | +CodeGraph cuts **cost, tokens, tool calls, and time on every repo** — across small, medium, and large codebases — and answers most of them with **zero file reads**, while the no-CodeGraph agent spends its budget on grep/find/Read discovery. `codegraph_explore` shows the answer in full — the mechanism plus the exact methods you asked about, even when they're buried in a multi-thousand-line file — while collapsing redundant interchangeable implementations to signatures, so the response is sized to the *answer* rather than the file count. The cost margin is narrowest on the smallest repos, where a modern model's native search is already cheap, but it stays solidly positive across the board. + +
+Per-repo breakdown — WITH vs WITHOUT (median of 4) + +**VS Code** · ~10k files +| Metric | WITH cg | WITHOUT cg | Δ | +|---|---|---|---| +| Time | 1m 37s | 2m 13s | 27% faster | +| File Reads | 0 | 9 | −9 | +| Grep/Bash | 0 | 11 | −11 | +| Tool calls | 4 | 21 | 80% fewer | +| Total tokens | 545k | 1.79M | 70% fewer | +| Cost | $0.55 | $0.83 | 33% cheaper | + +**Excalidraw** · ~640 files +| Metric | WITH cg | WITHOUT cg | Δ | +|---|---|---|---| +| Time | 1m 34s | 2m 6s | 26% faster | +| File Reads | 0 | 7 | −7 | +| Grep/Bash | 0 | 8 | −8 | +| Tool calls | 5 | 15 | 70% fewer | +| Total tokens | 651k | 1.69M | 61% fewer | +| Cost | $0.57 | $0.78 | 27% cheaper | + +**Django** · ~3k files +| Metric | WITH cg | WITHOUT cg | Δ | +|---|---|---|---| +| Time | 1m 25s | 1m 58s | 28% faster | +| File Reads | 0 | 9 | −9 | +| Grep/Bash | 0 | 5 | −5 | +| Tool calls | 3 | 13 | 77% fewer | +| Total tokens | 419k | 1.41M | 70% fewer | +| Cost | $0.48 | $0.62 | 23% cheaper | + +**Tokio** · ~790 files +| Metric | WITH cg | WITHOUT cg | Δ | +|---|---|---|---| +| Time | 1m 28s | 2m 20s | 37% faster | +| File Reads | 0 | 8 | −8 | +| Grep/Bash | 0 | 6 | −6 | +| Tool calls | 3 | 14 | 79% fewer | +| Total tokens | 522k | 1.73M | 70% fewer | +| Cost | $0.53 | $0.82 | 35% cheaper | + +**OkHttp** · ~645 files +| Metric | WITH cg | WITHOUT cg | Δ | +|---|---|---|---| +| Time | 1m 6s | 1m 29s | 26% faster | +| File Reads | 1 | 4 | −3 | +| Grep/Bash | 0 | 6 | −6 | +| Tool calls | 3 | 10 | 70% fewer | +| Total tokens | 572k | 1.10M | 48% fewer | +| Cost | $0.48 | $0.55 | 11% cheaper | + +**Gin** · ~110 files +| Metric | WITH cg | WITHOUT cg | Δ | +|---|---|---|---| +| Time | 1m 28s | 1m 37s | 9% faster | +| File Reads | 0 | 6 | −6 | +| Grep/Bash | 0 | 2 | −2 | +| Tool calls | 5 | 9 | 47% fewer | +| Total tokens | 552k | 847k | 35% fewer | +| Cost | $0.48 | $0.57 | 15% cheaper | + +**Alamofire** · ~110 files +| Metric | WITH cg | WITHOUT cg | Δ | +|---|---|---|---| +| Time | 2m 11s | 2m 21s | 7% faster | +| File Reads | 3 | 9 | −6 | +| Grep/Bash | 2 | 4 | −2 | +| Tool calls | 11 | 12 | 13% fewer | +| Total tokens | 1.13M | 2.10M | 46% fewer | +| Cost | $0.69 | $0.95 | 28% cheaper | + +
Full benchmark details -All tests used Claude Opus 4.6 (1M context) with Claude Code v2.1.91. Each test spawned a single Explore agent with the same question. +**Methodology.** Each arm is `claude -p` (Claude Opus 4.8) run headlessly against the repo with `--strict-mcp-config`: **WITH** = CodeGraph's MCP server enabled, **WITHOUT** = an empty MCP config. Built-in Read/Grep/Bash stay available to both. Same question per repo, **4 runs per arm, median reported**. Cost = the run's `total_cost_usd`; Tokens = total tokens processed (input incl. cached + output); Time = wall-clock; Tool calls = every tool invocation, including those inside any sub-agents the model spawns. Repos cloned at `--depth 1` and indexed by the same CodeGraph build that served them. Re-validated 2026-05-29 on the build with per-symbol adaptive `codegraph_explore` sizing. These numbers are lower than the prior Opus 4.7 validation — not a CodeGraph regression but a stronger native baseline: Opus 4.8 greps/reads efficiently on the main thread instead of fanning out into large Explore-subagent sweeps, so the no-CodeGraph arm is leaner than it used to be. Per-repo numbers move run-to-run with how hard the without-arm thrashes (the median-of-4 smooths it, but tails remain — e.g. Django's without-arm hit $2.71/14m one batch). -**Queries used:** +**Queries:** | Codebase | Query | |----------|-------| | VS Code | "How does the extension host communicate with the main process?" | -| Excalidraw | "How does collaborative editing and real-time sync work?" | -| Claude Code (Python+Rust) | "How does tool execution work end to end?" | -| Claude Code (Java) | "How does tool execution work end to end?" | -| Alamofire | "Trace how a request flows from Session.request() through to the URLSession layer" | -| Swift Compiler | "How does the Swift compiler handle error diagnostics?" | - -**With CodeGraph — the agent uses `codegraph_explore` and stops:** -| Codebase | Files Indexed | Nodes | Tool Uses | Tokens | Time | File Reads | -|----------|--------------|-------|-----------|--------|------|------------| -| VS Code (TypeScript) | 4,002 | 59,377 | 3 | 56.6k | 17s | 0 | -| Excalidraw (TypeScript) | 626 | 9,859 | 3 | 57.1k | 29s | 0 | -| Claude Code (Python+Rust) | 115 | 3,080 | 3 | 67.1k | 39s | 0 | -| Claude Code (Java) | — | — | 1 | 40.8k | 19s | 0 | -| Alamofire (Swift) | 102 | 2,624 | 3 | 57.3k | 22s | 0 | -| Swift Compiler (Swift/C++) | 25,874 | 272,898 | 6 | 77.4k | 35s | 0 | - -**Without CodeGraph — the agent uses grep, find, ls, and Read extensively:** -| Codebase | Tool Uses | Tokens | Time | File Reads | -|----------|-----------|--------|------|------------| -| VS Code (TypeScript) | 52 | 89.4k | 1m 37s | ~15 | -| Excalidraw (TypeScript) | 47 | 77.9k | 1m 45s | ~20 | -| Claude Code (Python+Rust) | 40 | 69.3k | 1m 8s | ~15 | -| Claude Code (Java) | 26 | 73.3k | 1m 22s | ~15 | -| Alamofire (Swift) | 32 | 52.4k | 1m 39s | ~10 | -| Swift Compiler (Swift/C++) | 37 | 99.1k | 2m 8s | ~20 | - -**Key observations:** -- With CodeGraph, the agent **never fell back to reading files** — it trusted the codegraph_explore results completely -- Without CodeGraph, agents spent most of their time on discovery (find, ls, grep) before they could even start reading relevant code -- The Java codebase needed only **1 codegraph_explore call** to answer the entire question -- Cross-language queries (Python+Rust) worked seamlessly — CodeGraph's graph traversal found connections across language boundaries -- The Swift benchmark (Alamofire) traced a **9-step call chain** from `Session.request()` to `URLSession.dataTask()` — CodeGraph's graph traversal at depth 3 captured the full chain in one explore call -- The **Swift Compiler** benchmark is the largest codebase tested (**25,874 files, 272,898 nodes**) — CodeGraph indexed it in under 4 minutes and the agent answered a complex cross-cutting question with **6 explore calls and zero file reads** in 35 seconds +| Excalidraw | "How does Excalidraw render and update canvas elements?" | +| Django | "How does Django's ORM build and execute a query from a QuerySet?" | +| Tokio | "How does tokio schedule and run async tasks on its runtime?" | +| OkHttp | "How does OkHttp process a request through its interceptor chain?" | +| Gin | "How does gin route requests through its middleware chain?" | +| Alamofire | "How does Alamofire build, send, and validate a request?" | + +**Why CodeGraph wins:** with the index available, the agent answers directly — `codegraph_context` to map the area, then one `codegraph_explore` for the relevant source — and stops, usually with zero file reads. Without it, the agent spends most of its budget on discovery (find/ls/grep) before reading the right code. CodeGraph only helps when queried *directly*, so its instructions steer agents to answer directly rather than delegate exploration to file-reading sub-agents — otherwise a sub-agent reads files regardless and CodeGraph becomes overhead.
@@ -113,10 +204,38 @@ All tests used Claude Opus 4.6 (1M context) with Claude Code v2.1.91. Each test | **Full-Text Search** | Find code by name instantly across your entire codebase, powered by FTS5 | | **Impact Analysis** | Trace callers, callees, and the full impact radius of any symbol before making changes | | **Always Fresh** | File watcher uses native OS events (FSEvents/inotify/ReadDirectoryChangesW) with debounced auto-sync — the graph stays current as you code, zero config | -| **19+ Languages** | TypeScript, JavaScript, Python, Go, Rust, Java, C#, PHP, Ruby, C, C++, Swift, Kotlin, Dart, Svelte, Liquid, Pascal/Delphi | -| **Framework-aware Routes** | Recognizes web-framework routing files and links URL patterns to their handlers across 13 frameworks | +| **20+ Languages** | TypeScript, JavaScript, Python, Go, Rust, Java, C#, PHP, Ruby, C, C++, Objective-C, Swift, Kotlin, Dart, Lua, Luau, Svelte, Liquid, Pascal/Delphi | +| **Framework-aware Routes** | Recognizes web-framework routing files and links URL patterns to their handlers across 14 frameworks | +| **Mixed iOS / React Native / Expo** | Closes cross-language flows that static parsing misses: Swift ↔ ObjC bridging, React Native legacy bridge + TurboModules + Fabric view components, native → JS event emitters, Expo Modules | | **100% Local** | No data leaves your machine. No API keys. No external services. SQLite database only | +
+How auto-syncing works — and why you don't need to run codegraph sync manually + +When your agent (Claude Code, Cursor, Codex, opencode) launches `codegraph serve --mcp`, three layers keep the index in step with your code — and make sure the agent never gets a silent wrong answer in the brief window between an edit and the next sync: + +1. **File watcher with debounced auto-sync.** A native FSEvents / inotify / ReadDirectoryChangesW watcher captures every source-file create / modify / delete and triggers a re-index after a debounce window (default `2000ms`, tunable via `CODEGRAPH_WATCH_DEBOUNCE_MS`, clamped to `[100ms, 60s]`). Bursts of edits collapse into a single sync. + +2. **Per-file staleness banner.** During the brief debounce window, MCP tool responses that would reference a still-pending file prepend a `⚠️` banner naming it and telling the agent to `Read` it directly. Pending files NOT referenced by the response surface as a small footer instead. Either way, the agent gets an explicit signal — validated with Claude Code, where the agent literally says "Reading the file directly for the live content" before opening it. + +3. **Connect-time catch-up.** When the MCP server (re)connects, codegraph runs a fast `(size, mtime)` + content-hash reconciliation against the working tree before answering the first query — so edits made while no MCP server was running (a `git pull` from the terminal, edits from another editor, a previous agent session that exited) get absorbed on the next session's first tool call. + +``` +agent writes src/Widget.ts + → watcher fires (<100ms) + → debounce (default 2s) + → sync; Widget.ts is in the index + → next agent query sees it +``` + +**Verify any time** with `codegraph_status` (via MCP) or `codegraph status` (CLI). If anything is pending, you'll see a `### Pending sync:` section naming the files and their edit age. + +The handful of cases where manual `codegraph sync` makes sense: the watcher is disabled (sandboxed environments, or `CODEGRAPH_NO_DAEMON=1`), or you're scripting against the index outside an agent session and want a pre-flight sync at the start of your script. + +→ Full deep-dive in [Guides → Indexing a Project](https://colbymchenry.github.io/codegraph/guides/indexing/#stay-fresh-automatically). + +
+ --- ## Framework-aware Routes @@ -129,7 +248,9 @@ CodeGraph detects web-framework routing files and emits `route` nodes linked by | **Flask** | `@app.route('/path', methods=[...])`, blueprint routes | | **FastAPI** | `@app.get(...)`, `@router.post(...)`, all standard methods | | **Express** | `app.get(...)`, `router.post(...)` with middleware chains | +| **NestJS** | `@Controller` + `@Get/@Post/...`, GraphQL `@Resolver` + `@Query/@Mutation`, `@MessagePattern`/`@EventPattern`, `@SubscribeMessage` | | **Laravel** | `Route::get()`, `Route::resource()`, `Controller@action`, tuple syntax | +| **Drupal** | `*.routing.yml` routes (`_controller`, `_form`, entity handlers); `hook_*` implementations in `.module`/`.theme`/`.install`/`.inc` | | **Rails** | `get '/x', to: 'users#index'`, hash-rocket `=>` syntax | | **Spring** | `@GetMapping`, `@PostMapping`, `@RequestMapping` on methods | | **Gin / chi / gorilla / mux** | `r.GET(...)`, `router.HandleFunc(...)` | @@ -140,6 +261,35 @@ CodeGraph detects web-framework routing files and emits `route` nodes linked by --- +## Mixed iOS / React Native / Expo bridging + +Real iOS and React Native codebases live across multiple languages — a Swift caller invokes an Objective-C selector that's been auto-bridged, a JS file calls into a native module via the React Native bridge, a JSX component delegates to a native view manager. Static tree-sitter extraction stops at each language boundary. CodeGraph bridges them so `trace`, `callers`, `callees`, and `impact` connect end-to-end across the gap. + +| Boundary | JS / Swift side | Native side | How | +|---|---|---|---| +| **Swift → ObjC** | Swift `obj.foo(bar:)` | ObjC selector `-fooWithBar:` | `@objc` auto-bridging rules (including init/property/protocol forms) + Cocoa preposition prefixes (`With`/`For`/`By`/`In`/`On`/`At`/…) | +| **ObjC → Swift** | ObjC `[obj fooWithBar:]` | Swift `@objc func foo(bar:)` | Reverse-bridge name candidates; verifies `@objc` exposure from source | +| **React Native legacy bridge** | JS `NativeModules.X.fn(...)` | ObjC `RCT_EXPORT_METHOD` / `RCT_REMAP_METHOD` · Java/Kotlin `@ReactMethod` | Parses macro/annotation declarations to build a JS-name → native-method map | +| **React Native TurboModules** | JS `import M from './NativeM'; M.fn(...)` | Native impl matching the Codegen spec | Treats the `Native.ts` spec interface as ground truth | +| **RN native → JS events** | JS `new NativeEventEmitter(...).addListener('e', cb)` | ObjC `[self sendEventWithName:@"e" body:...]` · Swift `sendEvent(withName: "e", ...)` · Java/Kotlin `.emit("e", ...)` | Synthesized cross-language event channel keyed by literal event name | +| **Expo Modules** | JS `requireNativeModule('X').fn(...)` | Swift / Kotlin `Module { Name("X"); AsyncFunction("fn") { ... } }` | Parses the Expo DSL literals; synthetic method nodes resolve via existing name-match | +| **Fabric view components** | JSX `` | TS Codegen spec + native impl class | Spec → `component` node; convention-based name+suffix lookup (`View`/`ComponentView`/`Manager`/`ViewManager`) bridges to native | +| **Legacy Paper view managers** | JSX `` | ObjC `RCT_EXPORT_VIEW_PROPERTY` · Java/Kotlin `@ReactProp` | Same as Fabric — Paper-era declarations also produce `component` + `property` nodes | + +**Validated on real codebases** (small + medium + large for each bridge): + +| Bridge | Small | Medium | Large | +|---|---|---|---| +| Swift ↔ ObjC | [Charts](https://github.com/danielgindi/Charts) | [realm-swift](https://github.com/realm/realm-swift) | [Wikipedia-iOS](https://github.com/wikimedia/wikipedia-ios) | +| RN legacy bridge | [AsyncStorage](https://github.com/react-native-async-storage/async-storage) | [react-native-svg](https://github.com/software-mansion/react-native-svg) | [react-native-firebase](https://github.com/invertase/react-native-firebase) | +| RN native → JS events | [RNGeolocation](https://github.com/Agontuk/react-native-geolocation-service) | — | react-native-firebase | +| Expo Modules | expo-haptics | expo-camera | expo SDK sweep (7 packages) | +| Fabric / Paper views | [react-native-segmented-control](https://github.com/react-native-segmented-control/segmented-control) | [react-native-screens](https://github.com/software-mansion/react-native-screens) | [react-native-skia](https://github.com/Shopify/react-native-skia) | + +Each bridge emits edges tagged `provenance:'heuristic'` with `metadata.synthesizedBy:` set to a stable channel name (e.g. `swift-objc-bridge`, `rn-event-channel`, `fabric-native-impl`, `expo-module-extract`), so the agent can tell at a glance how a hop got into the graph. + +--- + ## Quick Start ### 1. Run the Installer @@ -149,15 +299,33 @@ npx @colbymchenry/codegraph ``` The installer will: -- Prompt to install `codegraph` globally (needed for the MCP server) -- Configure the MCP server in `~/.claude.json` -- Set up auto-allow permissions for CodeGraph tools -- Add global instructions to `~/.claude/CLAUDE.md` -- Optionally initialize your current project +- Ask which agent(s) to configure — auto-detects installed ones from: **Claude Code**, **Cursor**, **Codex CLI**, **opencode**, **Hermes Agent**, **Gemini CLI**, **Antigravity IDE**, **Kiro** +- Prompt to install `codegraph` on your PATH (so agents can launch the MCP server) +- Ask whether configs apply to all your projects or just this one +- Write each chosen agent's MCP server config (the codegraph usage guide is delivered by the MCP server itself, so no instructions file is added to `CLAUDE.md` / `AGENTS.md` / etc.) +- Set up auto-allow permissions when Claude Code is one of the targets +- Initialize your current project (local installs only) -### 2. Restart Claude Code +**Non-interactive (scripting / CI):** -Restart Claude Code for the MCP server to load. +```bash +codegraph install --yes # auto-detect agents, install global +codegraph install --target=cursor,claude --yes # explicit target list +codegraph install --target=auto --location=local # detected agents, project-local +codegraph install --print-config codex # print snippet, no file writes +``` + +| Flag | Values | Default | +|---|---|---| +| `--target` | `auto`, `all`, `none`, or csv (`claude,cursor,...`) | prompt | +| `--location` | `global`, `local` | prompt | +| `--yes` | (boolean) | prompt every step | +| `--no-permissions` | (boolean) skip Claude auto-allow list | permissions on | +| `--print-config ` | dump snippet for one agent and exit | — | + +### 2. Restart Your Agent + +Restart your agent (Claude Code / Cursor / Codex CLI / opencode / Hermes Agent / Gemini CLI / Antigravity IDE / Kiro) for the MCP server to load. ### 3. Initialize Projects @@ -166,7 +334,9 @@ cd your-project codegraph init -i ``` -That's it! Claude Code will use CodeGraph tools automatically when a `.codegraph/` directory exists. +Builds the per-project knowledge graph index. A single global `codegraph install` works in every project you open — no need to re-run the installer per project. + +That's it — your agent will use CodeGraph tools automatically when a `.codegraph/` directory exists.
Manual Setup (Alternative) @@ -210,43 +380,16 @@ npm install -g @colbymchenry/codegraph
-Global Instructions Reference - -The installer automatically adds these instructions to `~/.claude/CLAUDE.md`: +Agent Tool Guidance -```markdown -## CodeGraph +CodeGraph's MCP server delivers its usage guidance to your agent **automatically**, in the MCP `initialize` response — there's no instructions file to manage and nothing is added to your `CLAUDE.md` / `AGENTS.md` / `GEMINI.md`. In short, it tells the agent to: -CodeGraph builds a semantic knowledge graph of codebases for faster, smarter code exploration. +- **Answer structural questions directly with CodeGraph** — it *is* the pre-built index, so a grep/read loop just repeats work it already did. Treat the returned source as already read. +- **Pick the tool by intent:** `codegraph_context` to map an area, `codegraph_trace` for "how does X reach Y", `codegraph_explore` to survey several symbols, `codegraph_search` to find a symbol, `codegraph_callers`/`codegraph_callees` to walk call flow, `codegraph_impact` before editing, `codegraph_node` for one symbol's source. +- **Trust the results — don't re-verify with grep**, and check the staleness banner after edits. +- If `.codegraph/` doesn't exist yet, offer to run `codegraph init -i`. -### If `.codegraph/` exists in the project - -**NEVER call `codegraph_explore` or `codegraph_context` directly in the main session.** These tools return large amounts of source code that fills up main session context. Instead, ALWAYS spawn an Explore agent for any exploration question (e.g., "how does X work?", "explain the Y system", "where is Z implemented?"). - -**When spawning Explore agents**, include this instruction in the prompt: - -> This project has CodeGraph initialized (.codegraph/ exists). Use `codegraph_explore` as your PRIMARY tool — it returns full source code sections from all relevant files in one call. -> -> **Rules:** -> 1. Follow the explore call budget in the `codegraph_explore` tool description — it scales automatically based on project size. -> 2. Do NOT re-read files that codegraph_explore already returned source code for. The source sections are complete and authoritative. -> 3. Only fall back to grep/glob/read for files listed under "Additional relevant files" if you need more detail, or if codegraph returned no results. - -**The main session may only use these lightweight tools directly** (for targeted lookups before making edits, not for exploration): - -| Tool | Use For | -|------|---------| -| `codegraph_search` | Find symbols by name | -| `codegraph_callers` / `codegraph_callees` | Trace call flow | -| `codegraph_impact` | Check what's affected before editing | -| `codegraph_node` | Get a single symbol's details | - -### If `.codegraph/` does NOT exist - -At the start of a session, ask the user if they'd like to initialize CodeGraph: - -"I notice this project doesn't have CodeGraph initialized. Would you like me to run `codegraph init -i` to build a code knowledge graph?" -``` +The exact text is `src/mcp/server-instructions.ts` — the single source of truth.
@@ -255,34 +398,23 @@ At the start of a session, ask the user if they'd like to initialize CodeGraph: ## How It Works ``` -┌─────────────────────────────────────────────────────────────────┐ -│ Claude Code │ -│ │ -│ "Implement user authentication" │ -│ │ │ -│ ▼ │ -│ ┌─────────────────┐ ┌─────────────────┐ │ -│ │ Explore Agent │ ──── │ Explore Agent │ │ -│ └────────┬────────┘ └────────┬────────┘ │ -│ │ │ │ -└───────────┼────────────────────────┼─────────────────────────────┘ - │ │ - ▼ ▼ ┌───────────────────────────────────────────────────────────────────┐ -│ CodeGraph MCP Server │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ Search │ │ Callers │ │ Context │ │ -│ │ "auth" │ │ "login()" │ │ for task │ │ -│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ -│ │ │ │ │ -│ └────────────────┼────────────────┘ │ -│ ▼ │ -│ ┌───────────────────────┐ │ -│ │ SQLite Graph DB │ │ -│ │ • 387 symbols │ │ -│ │ • 1,204 edges │ │ -│ │ • Instant lookups │ │ -│ └───────────────────────┘ │ +│ Claude Code │ +│ │ +│ "How does a request reach the database?" │ +│ calls CodeGraph tools directly — no Explore sub-agent │ +│ │ │ +└─────────────────────────────────┬─────────────────────────────────┘ + │ + ▼ +┌───────────────────────────────────────────────────────────────────┐ +│ CodeGraph MCP Server │ +│ │ +│ context · trace · explore · callers · callees · impact │ +│ │ │ +│ ▼ │ +│ SQLite knowledge graph │ +│ symbols · edges · files · FTS5 full-text search │ └───────────────────────────────────────────────────────────────────┘ ``` @@ -301,6 +433,7 @@ At the start of a session, ask the user if they'd like to initialize CodeGraph: ```bash codegraph # Run interactive installer codegraph install # Run installer (explicit) +codegraph uninstall # Remove CodeGraph from your agents (inverse of install) codegraph init [path] # Initialize in a project (--index to also index) codegraph uninit [path] # Remove CodeGraph from a project (--force to skip prompt) codegraph index [path] # Full index (--force to re-index, --quiet for less output) @@ -309,6 +442,9 @@ codegraph status [path] # Show statistics codegraph query # Search symbols (--kind, --limit, --json) codegraph files [path] # Show file structure (--format, --filter, --max-depth, --json) codegraph context # Build context for AI (--format, --max-nodes) +codegraph callers # Find what calls a function/method (--limit, --json) +codegraph callees # Find what a function/method calls (--limit, --json) +codegraph impact # Analyze what code is affected by changing a symbol (--depth, --json) codegraph affected [files...] # Find test files affected by changes (see below) codegraph serve --mcp # Start MCP server ``` @@ -351,10 +487,12 @@ When running as an MCP server, CodeGraph exposes these tools to Claude Code: |------|---------| | `codegraph_search` | Find symbols by name across the codebase | | `codegraph_context` | Build relevant code context for a task | +| `codegraph_trace` | Trace the call path between two symbols ("how does X reach Y") in one call — each hop with its body inline, following dynamic-dispatch hops (callbacks, React re-render, interface→impl) that grep can't | | `codegraph_callers` | Find what calls a function | | `codegraph_callees` | Find what a function calls | | `codegraph_impact` | Analyze what code is affected by changing a symbol | | `codegraph_node` | Get details about a specific symbol (optionally with source code) | +| `codegraph_explore` | Return source for several related symbols grouped by file, plus a relationship map, in one call | | `codegraph_files` | Get indexed file structure (faster than filesystem scanning) | | `codegraph_status` | Check index health and statistics | @@ -386,28 +524,53 @@ cg.close(); ## Configuration -The `.codegraph/config.json` file controls indexing: +There isn't any — CodeGraph is zero-config, with **no config file** to write or +keep in sync. Language support is automatic from the file extension; there's +nothing to wire up per language. -```json -{ - "version": 1, - "languages": ["typescript", "javascript"], - "exclude": ["node_modules/**", "dist/**", "build/**", "*.min.js"], - "frameworks": [], - "maxFileSize": 1048576, - "extractDocstrings": true, - "trackCallSites": true -} -``` +What it skips out of the box: -| Option | Description | Default | -|--------|-------------|---------| -| `languages` | Languages to index (auto-detected if empty) | `[]` | -| `exclude` | Glob patterns to ignore | `["node_modules/**", ...]` | -| `frameworks` | Framework hints for better resolution | `[]` | -| `maxFileSize` | Skip files larger than this (bytes) | `1048576` (1MB) | -| `extractDocstrings` | Extract docstrings from code | `true` | -| `trackCallSites` | Track call site locations | `true` | +- **Dependency, build, and cache directories** — `node_modules`, `vendor`, + `dist`, `build`, `target`, `.venv`, `Pods`, `.next`, and the like across every + [supported stack](#supported-languages) — so the graph is your code, not + third-party noise. This holds even with no `.gitignore`. +- **Anything in your `.gitignore`** — honored in git repos via git, and in + non-git projects by reading `.gitignore` directly (root and nested). +- **Files larger than 1 MB** — generated bundles, minified JS, vendored blobs. + +To keep something else out, add it to `.gitignore`. To pull a default-excluded +directory back **in** (say you really do want a vendored dependency indexed), +add a negation — `!vendor/`. The defaults apply uniformly, so committing a +dependency or build directory doesn't force it into the graph; the `.gitignore` +negation is the explicit opt-in. + +## Supported Platforms + +Every release ships a self-contained build (bundled Node runtime — nothing to +compile) for all three desktop OSes, on both Intel/AMD (x64) and ARM (arm64): + +| Platform | Architectures | Install | +|----------|---------------|---------| +| Windows | x64, arm64 | PowerShell installer or npm | +| macOS | x64, arm64 | shell installer or npm | +| Linux | x64, arm64 | shell installer or npm | + +See [Get Started](#get-started) for the one-line install commands. + +## Supported Agents + +The interactive installer auto-detects and configures each of these — wiring up +the MCP server (which delivers its own usage guidance, so no instructions file +is written): + +- **Claude Code** +- **Cursor** +- **Codex CLI** +- **opencode** +- **Hermes Agent** +- **Gemini CLI** +- **Antigravity IDE** +- **Kiro** ## Supported Languages @@ -424,6 +587,7 @@ The `.codegraph/config.json` file controls indexing: | Ruby | `.rb` | Full support | | C | `.c`, `.h` | Full support | | C++ | `.cpp`, `.hpp`, `.cc` | Full support | +| Objective-C | `.m`, `.mm`, `.h` | Partial support (classes, protocols, methods, `@property`, `#import`, message sends; `.mm` ObjC++ may parse incompletely) | | Swift | `.swift` | Full support | | Kotlin | `.kt`, `.kts` | Full support | | Scala | `.scala`, `.sc` | Full support (classes, traits, methods, type aliases, Scala 3 enums) | @@ -432,6 +596,8 @@ The `.codegraph/config.json` file controls indexing: | Vue | `.vue` | Full support (script + script-setup extraction, Nuxt page/API/middleware routes) | | Liquid | `.liquid` | Full support | | Pascal / Delphi | `.pas`, `.dpr`, `.dpk`, `.lpr` | Full support (classes, records, interfaces, enums, DFM/FMX form files) | +| Lua | `.lua` | Full support (functions, methods with receivers, local variables, `require` imports, call edges) | +| Luau | `.luau` | Full support (everything in Lua, plus `type`/`export type` aliases, typed signatures, and Roblox instance-path `require`) | ## Troubleshooting @@ -439,33 +605,24 @@ The `.codegraph/config.json` file controls indexing: **Indexing is slow** — Check that `node_modules` and other large directories are excluded. Use `--quiet` to reduce output overhead. -**Indexing is slow / MCP `database is locked` / WASM fallback active** — `codegraph` ships with a WASM SQLite fallback for environments where `better-sqlite3` (a native module, declared as `optionalDependencies`) can't install. The fallback is 5-10x slower than the native backend and uses a journal mode that lets writers block readers, so MCP queries can also hit `database is locked` while indexing runs. Run `codegraph status` and look at the `Backend:` line: - -- `Backend: native` — you're on the fast path, nothing to do. -- `Backend: wasm` — you're on the slow fallback. Common causes: missing C build tools, prebuilt binary unavailable for your Node version, or your Node version changed after install. Fix: +**MCP hits `database is locked`** — current builds shouldn't: CodeGraph bundles its own Node runtime and uses Node's built-in `node:sqlite` in WAL mode, where concurrent reads never block on a writer. If you still see it: - ```bash - # macOS - xcode-select --install # installs the C compiler +- **You're on an old (pre-0.9) install.** Reinstall to get the bundled runtime — `curl -fsSL https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.sh | sh` (macOS/Linux), `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` (Windows), or `npm i -g @colbymchenry/codegraph@latest`. +- **`codegraph status` shows `Journal:` other than `wal`** — WAL couldn't be enabled on this filesystem (common on network shares and WSL2 `/mnt`), so reads can block on writes. Move the project (with its `.codegraph/` folder) onto a local disk. - # Linux (Debian / Ubuntu) - sudo apt install build-essential python3 make - - # Linux (RHEL / Fedora) - sudo yum groupinstall "Development Tools" - - # Then rebuild on any platform: - npm rebuild better-sqlite3 - - # Or force-include as a hard dep: - npm install better-sqlite3 --save - ``` +**MCP server not connecting** — Ensure the project is initialized/indexed, verify the path in your MCP config, and check that `codegraph serve --mcp` works from the command line. - After the fix, `codegraph status` should show `Backend: native`. +**Missing symbols** — The MCP server auto-syncs on save (wait a couple seconds). Run `codegraph sync` manually if needed. Check that the file's language is supported and isn't inside a `.gitignore`d or default-excluded directory (e.g. `node_modules`, `dist`). -**MCP server not connecting** — Ensure the project is initialized/indexed, verify the path in your MCP config, and check that `codegraph serve --mcp` works from the command line. +## Star History -**Missing symbols** — The MCP server auto-syncs on save (wait a couple seconds). Run `codegraph sync` manually if needed. Check that the file's language is supported and isn't excluded by config patterns. + + + + + Star History Chart + + ## License @@ -475,7 +632,7 @@ MIT
-**Made for the Claude Code community** +**Made for AI coding agents — Claude Code, Cursor, Codex CLI, opencode, Hermes Agent, Gemini CLI, Antigravity IDE, and Kiro** [Report Bug](https://github.com/colbymchenry/codegraph/issues) · [Request Feature](https://github.com/colbymchenry/codegraph/issues) diff --git a/__tests__/__helpers__/chokidar-mock.ts b/__tests__/__helpers__/chokidar-mock.ts new file mode 100644 index 000000000..4e5f4f6cf --- /dev/null +++ b/__tests__/__helpers__/chokidar-mock.ts @@ -0,0 +1,121 @@ +/** + * Deterministic chokidar mock for FileWatcher tests. + * + * The real chokidar binding goes through FSEvents (macOS) / inotify (Linux) / + * ReadDirectoryChangesW (Windows). Under parallel vitest execution, those + * OS-level subsystems serve multiple test files simultaneously and event + * delivery latency grows non-deterministically — `should expose edited paths + * via getPendingFiles before sync fires` and the `mcp-staleness-banner` tests + * have observably raced for that reason (consistent ~30% failure rate when + * running the full suite, 0/N when run in isolation). + * + * This mock replaces chokidar with a controllable in-process EventEmitter: + * + * - `chokidar.watch(root, opts)` returns an instance keyed by `root`. + * - The instance fires `ready` on the next microtask, matching the + * real chokidar shape (tests' `waitUntilReady()` resolves promptly). + * - Tests synthesize file events via `triggerFileEvent(root, 'add', rel)` + * instead of `fs.writeFileSync(...)` — no OS-level watcher in the loop, + * no waitFor polling against unpredictable delivery latency. + * - The actual debounce timer in FileWatcher is left untouched (real + * setTimeout). That's the unit under test; deterministic timing + * would change what the test asserts. + * + * Install with `vi.mock('chokidar', () => chokidarMockModule)` at the + * top of each test file (must be hoisted, hence the static export). + * + * All instances live in module scope — clear them in `afterEach` if a + * test creates watchers and needs hard isolation, but in practice the + * `close()` plumbing handles it. + */ +import { EventEmitter } from 'node:events'; + +/** One mock watcher per `chokidar.watch(root, ...)` call. */ +class MockChokidarWatcher extends EventEmitter { + private closed = false; + private readyFired = false; + + constructor(public readonly root: string) { + super(); + // Mirror chokidar: `ready` fires asynchronously after the initial scan. + // We use queueMicrotask so it's deterministic and as fast as possible — + // tests' `await watcher.waitUntilReady()` resolves immediately. + queueMicrotask(() => { + if (this.closed) return; + this.readyFired = true; + this.emit('ready'); + }); + } + + /** chokidar.FSWatcher#close shape. */ + close(): Promise { + this.closed = true; + this.removeAllListeners(); + instancesByRoot.delete(this.root); + return Promise.resolve(); + } + + /** Test-only helper to synthesize a file event. */ + triggerEvent(event: 'add' | 'change' | 'unlink' | 'addDir' | 'unlinkDir', absPath: string): void { + if (this.closed) return; + // Real chokidar emits both the typed event AND the catch-all 'all'. + // FileWatcher only listens on 'all'. + this.emit('all', event, absPath); + } + + /** True once the initial-scan `ready` event has been emitted. */ + isReady(): boolean { + return this.readyFired; + } +} + +const instancesByRoot = new Map(); + +/** + * The mock module — pass this to `vi.mock('chokidar', () => chokidarMockModule)`. + * The factory must NOT close over outer-scope state because vi.mock hoists. + */ +export const chokidarMockModule = { + default: { + watch: (root: string, _opts?: unknown) => { + const inst = new MockChokidarWatcher(root); + instancesByRoot.set(root, inst); + return inst; + }, + }, +}; + +/** + * Test-side helper: synthesize a chokidar event on the watcher created for + * `root`. Use after the watcher's `waitUntilReady()` has resolved, since + * FileWatcher only adds events to its pending set when `chokidarReady` is + * true. + * + * `relPath` is path.join'd with `root` before emission, matching how + * chokidar delivers absolute paths to the `all` handler. + */ +export function triggerFileEvent( + root: string, + event: 'add' | 'change' | 'unlink' | 'addDir' | 'unlinkDir', + relPath: string, +): void { + const inst = instancesByRoot.get(root); + if (!inst) { + throw new Error( + `triggerFileEvent: no mock chokidar watcher registered for root '${root}' — did chokidar.watch() get called?`, + ); + } + // FileWatcher uses path.relative(root, eventPath) to compute the + // normalized path it stores. We supply the absolute path here so that + // operation produces the relPath the test wrote. + const absPath = require('node:path').join(root, relPath); + inst.triggerEvent(event, absPath); +} + +/** Reset all in-memory mock watchers — call in afterEach when needed. */ +export function resetChokidarMock(): void { + for (const inst of instancesByRoot.values()) { + inst.removeAllListeners(); + } + instancesByRoot.clear(); +} diff --git a/__tests__/adaptive-explore-sizing.test.ts b/__tests__/adaptive-explore-sizing.test.ts new file mode 100644 index 000000000..a1a531cc9 --- /dev/null +++ b/__tests__/adaptive-explore-sizing.test.ts @@ -0,0 +1,391 @@ +/** + * Regression test for adaptive `codegraph_explore` sizing — sibling + * skeletonization (branch `feat/adaptive-explore-sizing`, commit d6d059f). + * + * Feature: when a file is BOTH (1) off the synthesized flow spine AND (2) a + * polymorphic sibling — its class implements/extends a supertype shared by + * >= MIN_SIBLINGS (3) implementers — `codegraph_explore` renders it as a + * class + member *signature* skeleton (bodies elided) instead of full source, + * keeping the on-spine exemplar and the mechanism full. This sizes the + * response to the answer rather than the budget cap on sibling-heavy flows + * (OkHttp's interceptor chain) without starving diffuse ones (distinct + * pipeline steps stay full). Default ON; CODEGRAPH_ADAPTIVE_EXPLORE=0 disables. + * + * The fixture is OkHttp's interceptor chain in miniature: + * - `Interceptor` interface with FOUR implementers (>= 3 => a sibling family) + * - a 3-hop call spine `dispatch -> proceed -> handleLogging` that passes + * THROUGH LoggingInterceptor — so that file is the on-spine exemplar + * - Bridge/Cache/RetryInterceptor: off-spine members of the sibling family + * => skeletonize + * - ResponseFormatter implements `Formatter`, which has only ONE impl (< 3) + * => a distinct step: off-spine but NOT a sibling => stays full + * + * Guards the two ways the feature can silently regress: skeletonizing too much + * (a distinct step or the on-spine exemplar) or too little (the off-spine + * siblings), plus the escape hatch. + */ +import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { ToolHandler } from '../src/mcp/tools'; +import CodeGraph from '../src/index'; + +const SKELETON_MARK = '· skeleton (signatures only; Read for a full body)'; + +/** Return the `#### ...` section for a file basename, header through the + * line before the next `###`/`####` header (or end of output). */ +function sectionFor(text: string, basename: string): string { + const lines = text.split('\n'); + const start = lines.findIndex((l) => l.startsWith('#### ') && l.includes(basename)); + if (start < 0) return ''; + let end = lines.length; + for (let i = start + 1; i < lines.length; i++) { + if (lines[i].startsWith('### ') || lines[i].startsWith('#### ')) { + end = i; + break; + } + } + return lines.slice(start, end).join('\n'); +} + +describe('adaptive codegraph_explore sizing — sibling skeletonization', () => { + let testDir: string; + let cg: CodeGraph; + let handler: ToolHandler; + + // Names the spine (dispatch/proceed/handleLogging), the on-spine exemplar, + // the three off-spine siblings, and the distinct step — so every file we + // assert on is gathered as relevant. maxFiles overrides the very-tiny tier's + // 4-file default so all of them land in one call. + const QUERY = + 'dispatch proceed handleLogging LoggingInterceptor BridgeInterceptor CacheInterceptor RetryInterceptor ResponseFormatter'; + + beforeAll(async () => { + testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-adaptive-explore-')); + const srcDir = path.join(testDir, 'src'); + fs.mkdirSync(srcDir); + + const write = (name: string, body: string) => + fs.writeFileSync(path.join(srcDir, name), body.trimStart()); + + // The interchangeable contract — 4 implementers below => sibling family. + write( + 'interceptor.ts', + ` +export interface Interceptor { + intercept(request: string): string; +} +` + ); + + // The mechanism + the spine: dispatch -> proceed -> (LoggingInterceptor) handleLogging. + // Unique method names so the call edges resolve unambiguously. + write( + 'dispatcher.ts', + ` +import { LoggingInterceptor } from './logging-interceptor'; + +export class RequestDispatcher { + dispatch(): string { + const chain = new InterceptorChain(); + return chain.proceed(); + } +} + +export class InterceptorChain { + proceed(): string { + const exemplar = new LoggingInterceptor(); + return exemplar.handleLogging(); + } +} +` + ); + + // On-spine exemplar: handleLogging is the spine's tail, so this whole file + // is on-spine and must stay FULL even though it's a sibling (implements Interceptor). + write( + 'logging-interceptor.ts', + ` +import { Interceptor } from './interceptor'; + +export class LoggingInterceptor implements Interceptor { + handleLogging(): string { + const tag = 'LOGGING_BODY_MARKER'; + return this.intercept(tag); + } + intercept(request: string): string { + return 'logged:' + request; + } +} +` + ); + + // Off-spine siblings — interchangeable impls of Interceptor => SKELETONIZE. + // Each body carries a unique marker that must NOT survive skeletonization. + write( + 'bridge-interceptor.ts', + ` +import { Interceptor } from './interceptor'; + +export class BridgeInterceptor implements Interceptor { + intercept(request: string): string { + const detail = 'BRIDGE_BODY_MARKER'; + return 'bridged:' + request + detail; + } +} +` + ); + write( + 'cache-interceptor.ts', + ` +import { Interceptor } from './interceptor'; + +export class CacheInterceptor implements Interceptor { + intercept(request: string): string { + const detail = 'CACHE_BODY_MARKER'; + return 'cached:' + request + detail; + } +} +` + ); + write( + 'retry-interceptor.ts', + ` +import { Interceptor } from './interceptor'; + +export class RetryInterceptor implements Interceptor { + intercept(request: string): string { + const detail = 'RETRY_BODY_MARKER'; + return 'retried:' + request + detail; + } +} +` + ); + + // A 1:1 interface->impl pair: off-spine, implements something, but the + // supertype has only ONE impl (< MIN_SIBLINGS) => a DISTINCT step => FULL. + write( + 'formatter.ts', + ` +export interface Formatter { + format(input: string): string; +} +` + ); + write( + 'response-formatter.ts', + ` +import { Formatter } from './formatter'; +import { JsonCodec } from './codec'; + +export class ResponseFormatter implements Formatter { + format(input: string): string { + const detail = 'FORMATTER_BODY_MARKER'; + // Calls into the Codec family from OFF the dispatch spine, so codec.ts is + // gathered as relevant but stays off-spine (mirrors Django: compiler.py is + // referenced by the flow yet off the QuerySet-iteration spine). + return new JsonCodec().encode(input) + detail; + } +} +` + ); + + // An off-spine sibling (implements Interceptor) the agent would otherwise + // skeletonize — BUT it owns a uniquely-named method `authenticate` the agent + // names in the query. Mirrors OkHttp's RealCall (named getResponseWith- + // InterceptorChain): a named callable means "show me this", so it stays full. + write( + 'auth-interceptor.ts', + ` +import { Interceptor } from './interceptor'; + +export class AuthInterceptor implements Interceptor { + authenticate(token: string): string { + const detail = 'AUTH_BODY_MARKER'; + return 'auth:' + token + detail; + } + intercept(request: string): string { + return this.authenticate(request); + } +} +` + ); + + // A base class that DEFINES a >=3-impl supertype AND co-locates its + // subclasses in the same file — mirrors Django's compiler.py (SQLCompiler + + // SQLInsertCompiler/SQLUpdateCompiler/...). The subclasses' `extends` edges + // make the file look like a sibling, but it's the family's base/mechanism, + // so it must stay full. + write( + 'codec.ts', + ` +export class Codec { + encode(input: string): string { + const detail = 'CODEC_BASE_MARKER'; + return input + detail; + } +} +export class JsonCodec extends Codec { + encode(input: string): string { return '{' + input + '}'; } +} +export class XmlCodec extends Codec { + encode(input: string): string { + const detail = 'XML_BODY_MARKER'; + return '<' + input + detail + '>'; + } +} +export class YamlCodec extends Codec { + encode(input: string): string { return '- ' + input; } +} +` + ); + + cg = CodeGraph.initSync(testDir, { config: { include: ['**/*.ts'], exclude: [] } }); + await cg.indexAll(); + handler = new ToolHandler(cg); + }); + + afterAll(() => { + if (cg) cg.destroy(); + if (testDir && fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }); + } + }); + + beforeEach(() => { + // Each test asserts against the default (ON) behaviour unless it opts out. + delete process.env.CODEGRAPH_ADAPTIVE_EXPLORE; + }); + + it('fixture sanity: Interceptor has >=3 implementers, Formatter has <3', () => { + const find = (name: string, kind: string) => + cg.searchNodes(name).map((r) => r.node).find((n) => n.name === name && n.kind === kind); + + const interceptor = find('Interceptor', 'interface'); + const formatter = find('Formatter', 'interface'); + expect(interceptor).toBeTruthy(); + expect(formatter).toBeTruthy(); + + const implementers = (id: string) => + cg.getIncomingEdges(id).filter((e) => e.kind === 'implements' || e.kind === 'extends').length; + + // The whole gate hinges on this signal — assert the fixture actually + // produces the >=3 / <3 split, so a TS-extraction change fails here loudly + // rather than silently flipping the skeletonization downstream. + expect(implementers(interceptor!.id)).toBeGreaterThanOrEqual(3); + expect(implementers(formatter!.id)).toBeLessThan(3); + }); + + it('skeletonizes off-spine polymorphic siblings (bodies elided, signatures kept)', async () => { + const result = await handler.execute('codegraph_explore', { query: QUERY, maxFiles: 12 }); + const text = result.content?.[0]?.text ?? ''; + + // Precondition: the spine must have formed, or nothing skeletonizes. + expect(text).toContain('## Flow (call path among the symbols you queried)'); + + for (const [file, marker] of [ + ['bridge-interceptor.ts', 'BRIDGE_BODY_MARKER'], + ['cache-interceptor.ts', 'CACHE_BODY_MARKER'], + ['retry-interceptor.ts', 'RETRY_BODY_MARKER'], + ] as const) { + const section = sectionFor(text, file); + expect(section, `${file} should be present in the explore output`).not.toBe(''); + expect(section, `${file} should be skeletonized`).toContain(SKELETON_MARK); + // The signature line survives; the body (with its marker) is elided. + expect(section).toContain('intercept(request'); + expect(section, `${file} body marker must NOT survive skeletonization`).not.toContain(marker); + } + }); + + it('keeps the on-spine exemplar full even though it is a sibling', async () => { + const result = await handler.execute('codegraph_explore', { query: QUERY, maxFiles: 12 }); + const text = result.content?.[0]?.text ?? ''; + + const section = sectionFor(text, 'logging-interceptor.ts'); + expect(section, 'logging-interceptor.ts should be present').not.toBe(''); + expect(section, 'on-spine exemplar must NOT be skeletonized').not.toContain(SKELETON_MARK); + // Full source => the body marker is present. + expect(section).toContain('LOGGING_BODY_MARKER'); + }); + + it('keeps a distinct step full (off-spine but supertype has < 3 implementers)', async () => { + const result = await handler.execute('codegraph_explore', { query: QUERY, maxFiles: 12 }); + const text = result.content?.[0]?.text ?? ''; + + const section = sectionFor(text, 'response-formatter.ts'); + expect(section, 'response-formatter.ts should be present').not.toBe(''); + expect(section, 'a 1:1 interface impl is not a sibling and must stay full').not.toContain(SKELETON_MARK); + expect(section).toContain('FORMATTER_BODY_MARKER'); + }); + + it('CODEGRAPH_ADAPTIVE_EXPLORE=0 disables skeletonization (siblings render full)', async () => { + process.env.CODEGRAPH_ADAPTIVE_EXPLORE = '0'; + try { + const result = await handler.execute('codegraph_explore', { query: QUERY, maxFiles: 12 }); + const text = result.content?.[0]?.text ?? ''; + + expect(text, 'no file should be skeletonized with the flag off').not.toContain(SKELETON_MARK); + // The previously-skeletonized siblings now render their full bodies. + const section = sectionFor(text, 'bridge-interceptor.ts'); + expect(section).not.toBe(''); + expect(section).toContain('BRIDGE_BODY_MARKER'); + } finally { + delete process.env.CODEGRAPH_ADAPTIVE_EXPLORE; + } + }); + + // Names AuthInterceptor's `authenticate` and Codec's `encode` (both methods), + // plus the spine tokens so a spine still forms. Same Interceptor family as the + // skeleton test, plus the Codec base+subclasses family. + const SPARE_QUERY = `${QUERY} authenticate encode AuthInterceptor Codec JsonCodec`; + + it('spares an off-spine sibling when the agent NAMED a callable in it (RealCall fix)', async () => { + const result = await handler.execute('codegraph_explore', { query: SPARE_QUERY, maxFiles: 15 }); + const text = result.content?.[0]?.text ?? ''; + expect(text).toContain('## Flow (call path among the symbols you queried)'); + + // auth-interceptor.ts is an off-spine Interceptor sibling — would skeletonize — + // but the agent named its method `authenticate`, so it stays FULL. + const auth = sectionFor(text, 'auth-interceptor.ts'); + expect(auth, 'auth-interceptor.ts should be present').not.toBe(''); + expect(auth, 'a file holding an agent-named callable must NOT be skeletonized').not.toContain(SKELETON_MARK); + expect(auth).toContain('AUTH_BODY_MARKER'); + + // Contrast: bridge-interceptor.ts — same family, named only by TYPE — still skeletonizes. + const bridge = sectionFor(text, 'bridge-interceptor.ts'); + expect(bridge, 'a sibling named only by type still skeletonizes').toContain(SKELETON_MARK); + expect(bridge).not.toContain('BRIDGE_BODY_MARKER'); + }); + + it('collapses a base+subclasses family file to a FOCUSED view — base method body kept, non-named subclasses signature-only (compiler.py)', async () => { + const result = await handler.execute('codegraph_explore', { query: SPARE_QUERY, maxFiles: 15 }); + const text = result.content?.[0]?.text ?? ''; + + // codec.ts defines the base Codec (>=3 subclasses extend it) and co-locates the + // subclasses — a "family" file (Django's compiler.py). The family-override fires + // (it is NOT spared into a full clustered render despite the named `encode`), so + // it COLLAPSES — but per-symbol: the named base method `Codec.encode` keeps its + // body (so the agent doesn't Read it back — Django's SQLCompiler.execute_sql), + // while a non-named subclass (XmlCodec) collapses to a signature. That packs the + // mechanism into budget without the redundant subclass bodies. + const codec = sectionFor(text, 'codec.ts'); + expect(codec, 'codec.ts should be present').not.toBe(''); + expect(codec, 'a named family file collapses to a focused (not full) view').toContain('· focused'); + expect(codec, 'the named base method body is kept (no Read-back)').toContain('CODEC_BASE_MARKER'); + expect(codec, 'a non-named subclass body is elided to a signature').not.toContain('XML_BODY_MARKER'); + }); + + it('naming a SHARED/polymorphic method does not spare the siblings (uniqueness-aware)', async () => { + // `intercept` is implemented by every interceptor (5 defs) — a polymorphic name, + // not a unique one. Naming it must NOT keep all five full (that floods the budget + // — Django's `as_sql`×110). The off-spine siblings still collapse, and since none + // defines the supertype, `intercept` doesn't even earn a body — pure skeleton. + const result = await handler.execute('codegraph_explore', { query: `${QUERY} intercept`, maxFiles: 12 }); + const text = result.content?.[0]?.text ?? ''; + + const bridge = sectionFor(text, 'bridge-interceptor.ts'); + expect(bridge, 'a sibling named only via a shared method is not spared').toContain(SKELETON_MARK); + expect(bridge, 'a shared method does not earn a body in a non-supertype leaf').not.toContain('BRIDGE_BODY_MARKER'); + }); +}); diff --git a/__tests__/concurrent-locking.test.ts b/__tests__/concurrent-locking.test.ts new file mode 100644 index 000000000..5c8ab518d --- /dev/null +++ b/__tests__/concurrent-locking.test.ts @@ -0,0 +1,152 @@ +/** + * Issue #238 — "database is locked" on concurrent MCP tool calls. + * + * With node:sqlite (real WAL) as the backend, the fixes that remain relevant: + * 1. busy_timeout is a bounded few-second wait (not a 2-minute hang) and WAL is + * active — so a reader never blocks on a concurrent writer. + * 2. The MCP ToolHandler reuses the default instance when a tool passes a + * projectPath pointing at the default project, instead of opening a SECOND + * connection to the same DB. + */ + +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src'; +import { ToolHandler } from '../src/mcp/tools'; +import { DatabaseConnection } from '../src/db'; + +/** Normalize a PRAGMA read across return shapes (array | object | scalar). */ +function pragmaValue(raw: unknown, key: string): unknown { + const row = Array.isArray(raw) ? raw[0] : raw; + if (row !== null && typeof row === 'object') return (row as Record)[key]; + return row; +} + +describe('issue #238 — connection PRAGMAs (#1)', () => { + let dir: string; + let conn: DatabaseConnection; + + beforeAll(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg238-pragma-')); + conn = DatabaseConnection.initialize(path.join(dir, 'codegraph.db')); + }); + + afterAll(() => { + conn.close(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('uses a bounded busy_timeout, not the old 2-minute hang', () => { + const ms = Number(pragmaValue(conn.getDb().pragma('busy_timeout'), 'timeout')); + expect(ms).toBeGreaterThan(0); + expect(ms).toBeLessThanOrEqual(30000); // far below the old 120000 + }); + + it('runs in WAL mode — the mode that lets readers proceed during a write', () => { + const mode = String(pragmaValue(conn.getDb().pragma('journal_mode'), 'journal_mode')).toLowerCase(); + expect(mode).toBe('wal'); + }); + + it('getJournalMode() surfaces the effective mode for status triage', () => { + expect(conn.getJournalMode()).toBe('wal'); + }); +}); + +describe('issue #238 — WAL lets a reader proceed during a writer', () => { + let dir: string; + + beforeAll(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg238-wal-')); + }); + + afterAll(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('a read on a 2nd connection succeeds while a writer holds the lock', () => { + const dbPath = path.join(dir, 'codegraph.db'); + const writer = DatabaseConnection.initialize(dbPath); + // The property only holds under WAL; skip if the filesystem couldn't enable it. + if (writer.getJournalMode() !== 'wal') { + writer.close(); + return; + } + const reader = DatabaseConnection.open(dbPath); + try { + writer.getDb().prepare('BEGIN EXCLUSIVE').run(); // hard write lock, held open + const t0 = Date.now(); + const row = reader.getDb().prepare('SELECT COUNT(*) AS c FROM nodes').get() as { c: number }; + const waited = Date.now() - t0; + expect(row.c).toBe(0); + expect(waited).toBeLessThan(1000); // proceeds immediately, no busy wait + } finally { + try { writer.getDb().prepare('COMMIT').run(); } catch { /* ignore */ } + reader.close(); + writer.close(); + } + }); +}); + +describe('issue #238 — ToolHandler reuses the default instance (#2)', () => { + let dir: string; + let cg: CodeGraph; + let root: string; + let handler: ToolHandler; + + beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg238-tools-')); + fs.writeFileSync(path.join(dir, 'a.ts'), 'export function helper(): number { return 1; }\n'); + fs.writeFileSync( + path.join(dir, 'b.ts'), + "import { helper } from './a';\nexport function main(): number { return helper(); }\n" + ); + cg = await CodeGraph.init(dir, { index: true }); + root = cg.getProjectRoot(); + handler = new ToolHandler(cg); + }); + + afterAll(() => { + cg.close(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('getCodeGraph(defaultRoot) returns the default instance, not a new connection', () => { + const openSpy = vi.spyOn(CodeGraph, 'openSync'); + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const resolved = (handler as any).getCodeGraph(root); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const nested = (handler as any).getCodeGraph(path.join(root, 'does', 'not', 'exist')); + expect(resolved).toBe(cg); + expect(nested).toBe(cg); // a sub-path resolves up to the same default project + expect(openSpy).not.toHaveBeenCalled(); // no second connection opened + } finally { + openSpy.mockRestore(); + } + }); + + it('concurrent read tool calls (mixed projectPath) all succeed without "database is locked"', async () => { + const openSpy = vi.spyOn(CodeGraph, 'openSync'); + try { + const calls: Promise<{ content: Array<{ text: string }>; isError?: boolean }>[] = [ + handler.execute('codegraph_search', { query: 'helper' }), + handler.execute('codegraph_search', { query: 'helper', projectPath: root }), + handler.execute('codegraph_callers', { symbol: 'helper', projectPath: root }), + handler.execute('codegraph_callees', { symbol: 'main' }), + handler.execute('codegraph_files', { projectPath: root }), + handler.execute('codegraph_status', { projectPath: root }), + ]; + const results = await Promise.all(calls); + for (const r of results) { + expect(r.isError).not.toBe(true); + expect(r.content[0]?.text ?? '').not.toMatch(/database is locked/i); + } + // Passing the default project's own path must not open a second connection. + expect(openSpy).not.toHaveBeenCalled(); + } finally { + openSpy.mockRestore(); + } + }); +}); diff --git a/__tests__/db-perf.test.ts b/__tests__/db-perf.test.ts new file mode 100644 index 000000000..1dc3f1eb6 --- /dev/null +++ b/__tests__/db-perf.test.ts @@ -0,0 +1,207 @@ +/** + * DB Performance / Correctness Tests + * + * Regression tests for three changes: + * 1. Batch `getNodesByIds` collapses graph-traversal N+1 reads. + * 2. `insertNode` invalidates the LRU cache so INSERT OR REPLACE + * doesn't serve a stale cached row on next `getNodeById`. + * 3. `runMaintenance` runs `PRAGMA optimize` + `wal_checkpoint(PASSIVE)` + * after indexAll/sync without throwing. + * 4. `insertEdges` validates endpoints from the DB, not stale node cache. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { DatabaseConnection } from '../src/db'; +import { QueryBuilder } from '../src/db/queries'; +import { Node } from '../src/types'; + +function makeNode(id: string, name = id): Node { + return { + id, + kind: 'function', + name, + qualifiedName: name, + filePath: 'a.ts', + language: 'typescript', + startLine: 1, + endLine: 1, + startColumn: 0, + endColumn: 0, + updatedAt: Date.now(), + }; +} + +describe('getNodesByIds (batch lookup)', () => { + let dir: string; + let db: DatabaseConnection; + let q: QueryBuilder; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'db-perf-batch-')); + db = DatabaseConnection.initialize(path.join(dir, 'test.db')); + q = new QueryBuilder(db.getDb()); + }); + + afterEach(() => { + db.close(); + if (fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('returns a Map keyed by id, with one entry per existing node', () => { + q.insertNodes([makeNode('n1'), makeNode('n2'), makeNode('n3')]); + const out = q.getNodesByIds(['n1', 'n2', 'n3']); + expect(out.size).toBe(3); + expect(out.get('n1')!.name).toBe('n1'); + expect(out.get('n3')!.name).toBe('n3'); + }); + + it('omits missing IDs from the result map (no nulls, no exceptions)', () => { + q.insertNodes([makeNode('n1'), makeNode('n2')]); + const out = q.getNodesByIds(['n1', 'missing', 'n2']); + expect(out.size).toBe(2); + expect(out.has('missing')).toBe(false); + expect(out.has('n1')).toBe(true); + expect(out.has('n2')).toBe(true); + }); + + it('handles an empty input array', () => { + expect(q.getNodesByIds([]).size).toBe(0); + }); + + it('handles batches over the SQLite parameter limit (chunking)', () => { + // Insert 1500 nodes; the helper chunks at 500 internally. + const nodes = Array.from({ length: 1500 }, (_, i) => makeNode(`n${i}`)); + q.insertNodes(nodes); + const ids = nodes.map((n) => n.id); + const out = q.getNodesByIds(ids); + expect(out.size).toBe(1500); + // Spot-check a few from the first / middle / last chunk. + expect(out.has('n0')).toBe(true); + expect(out.has('n750')).toBe(true); + expect(out.has('n1499')).toBe(true); + }); + + it('serves cache hits from memory and queries only the misses', () => { + q.insertNodes([makeNode('n1'), makeNode('n2'), makeNode('n3')]); + // Warm the cache for n1 only. + q.getNodeById('n1'); + // Replace the underlying row to make a miss-vs-cache-hit detectable. + db.getDb().prepare('UPDATE nodes SET name = ? WHERE id = ?').run('changed', 'n1'); + const out = q.getNodesByIds(['n1', 'n2']); + // The cached n1 (still 'n1', not 'changed') must be returned. + expect(out.get('n1')!.name).toBe('n1'); + expect(out.get('n2')!.name).toBe('n2'); + }); +}); + +describe('insertNode cache invalidation', () => { + let dir: string; + let db: DatabaseConnection; + let q: QueryBuilder; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'db-perf-cache-')); + db = DatabaseConnection.initialize(path.join(dir, 'test.db')); + q = new QueryBuilder(db.getDb()); + }); + + afterEach(() => { + db.close(); + if (fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('does not serve a stale cached node after INSERT OR REPLACE', () => { + // Regression: insertNode (which uses INSERT OR REPLACE) used to skip + // cache invalidation, so the next getNodeById returned the pre-replace + // version until LRU eviction. + const original = makeNode('n1', 'oldName'); + q.insertNode(original); + const beforeReplace = q.getNodeById('n1'); + expect(beforeReplace!.name).toBe('oldName'); + + // Replace via insertNode (the bug path). + q.insertNode({ ...original, name: 'newName', updatedAt: Date.now() }); + const afterReplace = q.getNodeById('n1'); + expect(afterReplace!.name).toBe('newName'); + }); +}); + +describe('insertEdges endpoint validation', () => { + let dir: string; + let db: DatabaseConnection; + let q: QueryBuilder; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'db-perf-edges-')); + db = DatabaseConnection.initialize(path.join(dir, 'test.db')); + q = new QueryBuilder(db.getDb()); + }); + + afterEach(() => { + db.close(); + if (fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('skips edges with missing endpoints instead of failing the whole batch', () => { + q.insertNodes([makeNode('source'), makeNode('target'), makeNode('other')]); + + expect(() => + q.insertEdges([ + { source: 'source', target: 'target', kind: 'calls' }, + { source: 'source', target: 'missing-target', kind: 'calls' }, + { source: 'missing-source', target: 'other', kind: 'references' }, + ]) + ).not.toThrow(); + + const edges = q.getOutgoingEdges('source'); + expect(edges).toHaveLength(1); + expect(edges[0]).toMatchObject({ source: 'source', target: 'target', kind: 'calls' }); + }); + + it('does not trust stale cached nodes when validating edge endpoints', () => { + q.insertNodes([makeNode('source'), makeNode('target')]); + expect(q.getNodeById('target')!.id).toBe('target'); + + db.getDb().prepare('DELETE FROM nodes WHERE id = ?').run('target'); + + expect(() => + q.insertEdges([{ source: 'source', target: 'target', kind: 'calls' }]) + ).not.toThrow(); + expect(q.getOutgoingEdges('source')).toEqual([]); + }); +}); + +describe('runMaintenance', () => { + let dir: string; + let db: DatabaseConnection; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'db-perf-maint-')); + db = DatabaseConnection.initialize(path.join(dir, 'test.db')); + }); + + afterEach(() => { + db.close(); + if (fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('runs without throwing on a fresh database', () => { + expect(() => db.runMaintenance()).not.toThrow(); + }); + + it('runs without throwing after writes', () => { + const q = new QueryBuilder(db.getDb()); + q.insertNodes([makeNode('n1'), makeNode('n2')]); + expect(() => db.runMaintenance()).not.toThrow(); + }); + + it('swallows failures rather than propagating (best-effort)', () => { + // Close the DB so the underlying handle would normally throw on any + // exec(). runMaintenance must still not propagate. + db.close(); + expect(() => db.runMaintenance()).not.toThrow(); + }); +}); diff --git a/__tests__/drupal.test.ts b/__tests__/drupal.test.ts new file mode 100644 index 000000000..c4f4421e9 --- /dev/null +++ b/__tests__/drupal.test.ts @@ -0,0 +1,609 @@ +/** + * Tests for Drupal framework resolver. + * + * Unit tests cover drupalResolver.detect(), extract() (routes + hooks), and resolve(). + * Integration tests use a real CodeGraph instance with a temporary Drupal project layout. + */ + +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { afterEach, beforeAll, describe, expect, it } from 'vitest'; +import { CodeGraph } from '../src'; +import { initGrammars, loadAllGrammars } from '../src/extraction/grammars'; +import { drupalResolver } from '../src/resolution/frameworks/drupal'; +import type { ResolutionContext } from '../src/resolution/types'; + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +function makeContext( + overrides: Partial = {}, +): ResolutionContext { + return { + getNodesInFile: () => [], + getNodesByName: () => [], + getNodesByQualifiedName: () => [], + getNodesByKind: () => [], + fileExists: () => false, + readFile: () => null, + getProjectRoot: () => '/project', + getAllFiles: () => [], + getNodesByLowerName: () => [], + getImportMappings: () => [], + ...overrides, + }; +} + +// --------------------------------------------------------------------------- +// detect() +// --------------------------------------------------------------------------- + +describe('drupalResolver.detect', () => { + it('returns true when composer.json has a drupal/ dependency', () => { + const ctx = makeContext({ + readFile: (f) => + f === 'composer.json' + ? JSON.stringify({ + require: { + 'drupal/core-recommended': '~10.5', + 'drush/drush': '^13', + }, + }) + : null, + }); + expect(drupalResolver.detect(ctx)).toBe(true); + }); + + it('returns true when drupal/ dependency is in require-dev', () => { + const ctx = makeContext({ + readFile: (f) => + f === 'composer.json' + ? JSON.stringify({ 'require-dev': { 'drupal/core': '^10' } }) + : null, + }); + expect(drupalResolver.detect(ctx)).toBe(true); + }); + + it('returns false when composer.json has no drupal/ dependencies', () => { + const ctx = makeContext({ + readFile: (f) => + f === 'composer.json' + ? JSON.stringify({ + require: { 'laravel/framework': '^10', php: '>=8.1' }, + }) + : null, + }); + expect(drupalResolver.detect(ctx)).toBe(false); + }); + + it('returns false when composer.json is absent', () => { + const ctx = makeContext({ readFile: () => null }); + expect(drupalResolver.detect(ctx)).toBe(false); + }); + + it('returns false when composer.json is malformed JSON', () => { + const ctx = makeContext({ readFile: () => '{ bad json' }); + expect(drupalResolver.detect(ctx)).toBe(false); + }); + + it('returns true for a contrib module with empty require (composer name/type)', () => { + const ctx = makeContext({ + readFile: (f) => + f === 'composer.json' + ? JSON.stringify({ + name: 'drupal/admin_toolbar', + type: 'drupal-module', + require: {}, + }) + : null, + }); + expect(drupalResolver.detect(ctx)).toBe(true); + }); + + it('returns true via the *.info.yml fallback when composer.json is absent', () => { + const ctx = makeContext({ + readFile: () => null, + getAllFiles: () => [ + 'mymodule/mymodule.info.yml', + 'mymodule/mymodule.routing.yml', + ], + }); + expect(drupalResolver.detect(ctx)).toBe(true); + }); + + it('returns false for a stray *.info.yml with no Drupal PHP/route file', () => { + const ctx = makeContext({ + readFile: () => null, + getAllFiles: () => ['some/unrelated.info.yml'], + }); + expect(drupalResolver.detect(ctx)).toBe(false); + }); +}); + +describe('drupalResolver.claimsReference', () => { + it('claims FQCN handler refs and hook names the pre-filter would drop', () => { + expect(drupalResolver.claimsReference!('\\Drupal\\m\\Form\\SettingsForm')).toBe(true); + expect(drupalResolver.claimsReference!('\\Drupal\\m\\Controller\\C:setNoJsCookie')).toBe(true); + expect(drupalResolver.claimsReference!('hook_form_alter')).toBe(true); + }); + + it('does not claim ordinary identifiers or entity-handler dotted refs', () => { + expect(drupalResolver.claimsReference!('someHelperFunction')).toBe(false); + expect(drupalResolver.claimsReference!('comment.default')).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// extract() — routing.yml +// --------------------------------------------------------------------------- + +describe('drupalResolver.extract — routing.yml', () => { + const routing = ` +mymodule.example: + path: '/mymodule/example' + defaults: + _controller: '\\Drupal\\mymodule\\Controller\\MyController::build' + _title: 'Example page' + requirements: + _permission: 'access content' +`; + + it('emits a route node for each YAML route', () => { + const { nodes } = drupalResolver.extract!( + 'mymodule/mymodule.routing.yml', + routing, + ); + expect(nodes).toHaveLength(1); + expect(nodes[0]!.kind).toBe('route'); + expect(nodes[0]!.name).toBe('/mymodule/example'); + }); + + it('sets qualifiedName to filePath::routeName', () => { + const { nodes } = drupalResolver.extract!( + 'mymodule/mymodule.routing.yml', + routing, + ); + expect(nodes[0]!.qualifiedName).toBe( + 'mymodule/mymodule.routing.yml::mymodule.example', + ); + }); + + it('emits a references edge to the controller FQCN', () => { + const { references } = drupalResolver.extract!( + 'mymodule/mymodule.routing.yml', + routing, + ); + expect(references).toHaveLength(1); + expect(references[0]!.referenceName).toBe( + '\\Drupal\\mymodule\\Controller\\MyController::build', + ); + expect(references[0]!.referenceKind).toBe('references'); + }); + + it('emits a references edge to a _form handler', () => { + const src = ` +mymodule.settings_form: + path: '/admin/config/mymodule' + defaults: + _form: '\\Drupal\\mymodule\\Form\\SettingsForm' + _title: 'MyModule settings' + requirements: + _permission: 'administer site configuration' +`; + const { nodes, references } = drupalResolver.extract!( + 'mymodule/mymodule.routing.yml', + src, + ); + expect(nodes).toHaveLength(1); + expect(references[0]!.referenceName).toBe( + '\\Drupal\\mymodule\\Form\\SettingsForm', + ); + }); + + it('handles multiple routes in one file', () => { + const src = ` +mod.page_one: + path: '/page-one' + defaults: + _controller: '\\Drupal\\mod\\Controller\\PageController::one' + requirements: + _permission: 'access content' + +mod.page_two: + path: '/page-two' + defaults: + _controller: '\\Drupal\\mod\\Controller\\PageController::two' + requirements: + _permission: 'access content' +`; + const { nodes, references } = drupalResolver.extract!( + 'mod/mod.routing.yml', + src, + ); + expect(nodes).toHaveLength(2); + expect(nodes.map((n) => n.name)).toContain('/page-one'); + expect(nodes.map((n) => n.name)).toContain('/page-two'); + expect(references).toHaveLength(2); + }); + + it('skips commented-out lines', () => { + const src = ` +mod.page: + path: '/page' + defaults: + #_controller: '\\Drupal\\mod\\Controller\\Old::build' + _controller: '\\Drupal\\mod\\Controller\\New::build' + requirements: + _permission: 'access content' +`; + const { references } = drupalResolver.extract!('mod/mod.routing.yml', src); + expect(references).toHaveLength(1); + expect(references[0]!.referenceName).toContain('New'); + }); + + it('includes HTTP methods in the route node name when present', () => { + const src = ` +mod.api: + path: '/api/resource' + defaults: + _controller: '\\Drupal\\mod\\Controller\\ApiController::get' + methods: [GET, POST] + requirements: + _permission: 'access content' +`; + const { nodes } = drupalResolver.extract!('mod/mod.routing.yml', src); + expect(nodes[0]!.name).toContain('GET'); + expect(nodes[0]!.name).toContain('POST'); + }); + + it('returns empty result for non-routing-yml files', () => { + const { nodes, references } = drupalResolver.extract!( + 'mymodule.module', + ' { + const { nodes, references } = drupalResolver.extract!( + 'some.routing.yml', + '# empty\n', + ); + expect(nodes).toHaveLength(0); + expect(references).toHaveLength(0); + }); +}); + +// --------------------------------------------------------------------------- +// extract() — hook detection in .module files +// --------------------------------------------------------------------------- + +describe('drupalResolver.extract — hook detection', () => { + it('detects hook implementation via docblock (Strategy A)', () => { + const src = ` r.referenceName === 'hook_form_alter', + ); + expect(hookRef).toBeDefined(); + expect(hookRef!.referenceKind).toBe('references'); + }); + + it('detects hook implementation via name pattern (Strategy B)', () => { + const src = ` r.referenceName === 'hook_views_data', + ); + expect(hookRef).toBeDefined(); + }); + + it('does not emit a hook ref for non-hook helper functions', () => { + // 'other_module_helper' doesn't start with 'mymodule_', so no hook ref + const src = ` { + const src = ` r.referenceName === 'hook_schema'); + expect(hookRef).toBeDefined(); + }); + + it('detects hooks in .theme files', () => { + const src = ` r.referenceName === 'hook_preprocess_node', + ); + expect(hookRef).toBeDefined(); + }); + + it('does not duplicate refs when both docblock and name pattern match', () => { + // Strategy A matches first and adds to docblockMatched set; + // Strategy B skips already-matched functions. + const src = ` r.referenceName === 'hook_form_alter', + ); + expect(hookRefs).toHaveLength(1); + }); +}); + +// --------------------------------------------------------------------------- +// resolve() +// --------------------------------------------------------------------------- + +describe('drupalResolver.resolve', () => { + it('resolves a _controller FQCN with ::method to the method node', () => { + const methodNode = { + id: 'method:abc123', + kind: 'method' as const, + name: 'build', + qualifiedName: 'MyController::build', + filePath: 'web/modules/custom/mymodule/src/Controller/MyController.php', + language: 'php' as const, + startLine: 10, + endLine: 20, + startColumn: 0, + endColumn: 0, + updatedAt: 0, + }; + const classNode = { + id: 'class:def456', + kind: 'class' as const, + name: 'MyController', + qualifiedName: 'MyController', + filePath: 'web/modules/custom/mymodule/src/Controller/MyController.php', + language: 'php' as const, + startLine: 5, + endLine: 30, + startColumn: 0, + endColumn: 0, + updatedAt: 0, + }; + const ctx = makeContext({ + getNodesByName: (name) => (name === 'MyController' ? [classNode] : []), + getNodesInFile: () => [classNode, methodNode], + }); + const ref = { + fromNodeId: 'route:x', + referenceName: '\\Drupal\\mymodule\\Controller\\MyController::build', + referenceKind: 'references' as const, + line: 1, + column: 0, + filePath: 'mymodule.routing.yml', + language: 'yaml' as const, + }; + const resolved = drupalResolver.resolve(ref, ctx); + expect(resolved).not.toBeNull(); + expect(resolved!.targetNodeId).toBe('method:abc123'); + expect(resolved!.confidence).toBeGreaterThanOrEqual(0.85); + }); + + it('resolves a _form FQCN (no ::method) to the class node', () => { + const classNode = { + id: 'class:form123', + kind: 'class' as const, + name: 'SettingsForm', + qualifiedName: 'SettingsForm', + filePath: 'web/modules/custom/mymodule/src/Form/SettingsForm.php', + language: 'php' as const, + startLine: 1, + endLine: 50, + startColumn: 0, + endColumn: 0, + updatedAt: 0, + }; + const ctx = makeContext({ + getNodesByName: (name) => (name === 'SettingsForm' ? [classNode] : []), + }); + const ref = { + fromNodeId: 'route:x', + referenceName: '\\Drupal\\mymodule\\Form\\SettingsForm', + referenceKind: 'references' as const, + line: 1, + column: 0, + filePath: 'mymodule.routing.yml', + language: 'yaml' as const, + }; + const resolved = drupalResolver.resolve(ref, ctx); + expect(resolved).not.toBeNull(); + expect(resolved!.targetNodeId).toBe('class:form123'); + }); + + it('returns null when the target class cannot be found', () => { + const ctx = makeContext({ getNodesByName: () => [] }); + const ref = { + fromNodeId: 'route:x', + referenceName: '\\Drupal\\mymodule\\Controller\\Missing::method', + referenceKind: 'references' as const, + line: 1, + column: 0, + filePath: 'mymodule.routing.yml', + language: 'yaml' as const, + }; + expect(drupalResolver.resolve(ref, ctx)).toBeNull(); + }); + + it('resolves a single-colon controller-service ref (Class:method)', () => { + const methodNode = { + id: 'method:nojs1', + kind: 'method' as const, + name: 'setNoJsCookie', + qualifiedName: 'BigPipeController::setNoJsCookie', + filePath: 'core/modules/big_pipe/src/Controller/BigPipeController.php', + language: 'php' as const, + startLine: 10, + endLine: 20, + startColumn: 0, + endColumn: 0, + updatedAt: 0, + }; + const classNode = { + id: 'class:nojs2', + kind: 'class' as const, + name: 'BigPipeController', + qualifiedName: 'BigPipeController', + filePath: 'core/modules/big_pipe/src/Controller/BigPipeController.php', + language: 'php' as const, + startLine: 5, + endLine: 30, + startColumn: 0, + endColumn: 0, + updatedAt: 0, + }; + const ctx = makeContext({ + getNodesByName: (name) => (name === 'BigPipeController' ? [classNode] : []), + getNodesInFile: () => [classNode, methodNode], + }); + const ref = { + fromNodeId: 'route:x', + referenceName: '\\Drupal\\big_pipe\\Controller\\BigPipeController:setNoJsCookie', + referenceKind: 'references' as const, + line: 1, + column: 0, + filePath: 'big_pipe.routing.yml', + language: 'yaml' as const, + }; + const resolved = drupalResolver.resolve(ref, ctx); + expect(resolved).not.toBeNull(); + expect(resolved!.targetNodeId).toBe('method:nojs1'); + }); +}); + +// --------------------------------------------------------------------------- +// End-to-end integration test +// --------------------------------------------------------------------------- + +beforeAll(async () => { + await initGrammars(); + await loadAllGrammars(); +}); + +describe('Drupal end-to-end — route node linked to controller method', () => { + let tmpDir: string | undefined; + afterEach(() => { + if (tmpDir) fs.rmSync(tmpDir, { recursive: true, force: true }); + tmpDir = undefined; + }); + + it('creates a route→controller edge from routing.yml to PHP class', async () => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-drupal-')); + + // Minimal composer.json to trigger Drupal detection + fs.writeFileSync( + path.join(tmpDir, 'composer.json'), + JSON.stringify({ require: { 'drupal/core-recommended': '~10.5' } }), + ); + + // Module directory structure + const modDir = path.join(tmpDir, 'web', 'modules', 'custom', 'my_module'); + fs.mkdirSync(path.join(modDir, 'src', 'Controller'), { recursive: true }); + + // routing.yml + fs.writeFileSync( + path.join(modDir, 'my_module.routing.yml'), + [ + 'my_module.hello:', + " path: '/hello'", + ' defaults:', + " _controller: '\\Drupal\\my_module\\Controller\\HelloController::build'", + " _title: 'Hello'", + ' requirements:', + " _permission: 'access content'", + ].join('\n') + '\n', + ); + + // PHP controller + fs.writeFileSync( + path.join(modDir, 'src', 'Controller', 'HelloController.php'), + [ + ' 'Hello'];", + ' }', + '}', + ].join('\n') + '\n', + ); + + const cg = CodeGraph.initSync(tmpDir); + await cg.indexAll(); + + // Route node must exist + const routes = cg.getNodesByKind('route'); + expect(routes.length).toBeGreaterThan(0); + const route = routes.find((n) => n.name.includes('/hello')); + expect(route).toBeDefined(); + + // Controller method must be indexed + const methods = cg.getNodesByKind('method'); + const buildMethod = methods.find((n) => n.name === 'build'); + expect(buildMethod).toBeDefined(); + + // Edge: route → build method (or class fallback) + const edges = cg.getOutgoingEdges(route!.id); + expect(edges.length).toBeGreaterThan(0); + + cg.close(); + }); +}); diff --git a/__tests__/explore-output-budget.test.ts b/__tests__/explore-output-budget.test.ts new file mode 100644 index 000000000..cd1a444d5 --- /dev/null +++ b/__tests__/explore-output-budget.test.ts @@ -0,0 +1,248 @@ +/** + * Adaptive output budget for codegraph_explore (#185). + * + * The explore tool used to apply a fixed 35KB output cap regardless of + * project size, which on small codebases was a net loss vs. native + * grep+Read. These tests pin the per-tier budget shape so future tuning + * doesn't silently drift the small-project case back into bloat. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { getExploreOutputBudget, getExploreBudget, ToolHandler } from '../src/mcp/tools'; +import CodeGraph from '../src/index'; + +describe('getExploreOutputBudget', () => { + it('returns a strictly smaller total cap for small projects than for huge ones', () => { + const small = getExploreOutputBudget(100); + const huge = getExploreOutputBudget(30000); + expect(small.maxOutputChars).toBeLessThan(huge.maxOutputChars); + expect(small.defaultMaxFiles).toBeLessThan(huge.defaultMaxFiles); + expect(small.maxCharsPerFile).toBeLessThan(huge.maxCharsPerFile); + }); + + it('caps total output well under 8000 tokens (~32k chars) on small projects', () => { + const small = getExploreOutputBudget(100); + expect(small.maxOutputChars).toBeLessThanOrEqual(20000); + }); + + it('keeps the historical 35k+ ceiling for medium-large projects so existing benchmarks do not regress', () => { + const large = getExploreOutputBudget(10000); + expect(large.maxOutputChars).toBeGreaterThanOrEqual(35000); + }); + + it('uses tier breakpoints matching getExploreBudget so call-count and output-budget agree on a project', () => { + // Very-tiny tier (<150 files) gets a tighter cap than small (150-499) — + // paired with tool gating to handle the MCP-overhead-dominates regime. + const tier0a = getExploreOutputBudget(50); + const tier0b = getExploreOutputBudget(149); + expect(tier0a.maxOutputChars).toBe(tier0b.maxOutputChars); + + const tier1a = getExploreOutputBudget(150); + const tier1b = getExploreOutputBudget(499); + expect(tier1a.maxOutputChars).toBe(tier1b.maxOutputChars); + // The <500 explore-call budget covers both very-tiny and small. + expect(getExploreBudget(50)).toBe(getExploreBudget(499)); + + const tier2a = getExploreOutputBudget(500); + const tier2b = getExploreOutputBudget(4999); + expect(tier2a.maxOutputChars).toBe(tier2b.maxOutputChars); + expect(getExploreBudget(500)).toBe(getExploreBudget(4999)); + + const tier3a = getExploreOutputBudget(5000); + const tier3b = getExploreOutputBudget(14999); + expect(tier3a.maxOutputChars).toBe(tier3b.maxOutputChars); + + // And crossing a breakpoint changes the cap. + expect(tier0a.maxOutputChars).not.toBe(tier1a.maxOutputChars); + expect(tier1a.maxOutputChars).not.toBe(tier2a.maxOutputChars); + expect(tier2a.maxOutputChars).not.toBe(tier3a.maxOutputChars); + }); + + it('gates off "Additional relevant files", completeness signal, and budget note on small projects', () => { + const small = getExploreOutputBudget(100); + expect(small.includeAdditionalFiles).toBe(false); + expect(small.includeCompletenessSignal).toBe(false); + expect(small.includeBudgetNote).toBe(false); + }); + + it('keeps all meta-text on for projects that earn the breadth signal (>=500 files)', () => { + const medium = getExploreOutputBudget(1000); + expect(medium.includeAdditionalFiles).toBe(true); + expect(medium.includeCompletenessSignal).toBe(true); + expect(medium.includeBudgetNote).toBe(true); + }); + + it('keeps the Relationships section on for medium+ tiers — small tiers drop it to maximize body density', () => { + // ITER2: relationships dropped on <500 tiers; on tiny repos the + // per-call payload is the cost driver, so even "cheap" structural + // signal adds up across follow-up turns. Re-enabled at ≥500 where + // body budgets are roomy enough to absorb the 1-2KB overhead. + expect(getExploreOutputBudget(50).includeRelationships).toBe(false); + expect(getExploreOutputBudget(1000).includeRelationships).toBe(true); + expect(getExploreOutputBudget(10000).includeRelationships).toBe(true); + expect(getExploreOutputBudget(30000).includeRelationships).toBe(true); + }); + + it('caps the per-file header symbol list more tightly on small projects', () => { + // Without this cap, a file like Alamofire's Session.swift produced + // a 3.4KB symbol list in the `#### path — sym, sym, ...` header, + // dwarfing the per-file body cap. + const small = getExploreOutputBudget(100); + const huge = getExploreOutputBudget(30000); + expect(small.maxSymbolsInFileHeader).toBeLessThan(huge.maxSymbolsInFileHeader); + expect(small.maxSymbolsInFileHeader).toBeGreaterThan(0); + }); + + it('uses a tighter clustering gap threshold on small projects to break runaway single clusters', () => { + const small = getExploreOutputBudget(100); + const huge = getExploreOutputBudget(30000); + expect(small.gapThreshold).toBeLessThanOrEqual(huge.gapThreshold); + }); + + it('handles the boundary file counts exactly (off-by-one regression guard)', () => { + // 149 -> very-tiny, 150 -> small + expect(getExploreOutputBudget(149).maxOutputChars).toBe(getExploreOutputBudget(50).maxOutputChars); + expect(getExploreOutputBudget(150).maxOutputChars).toBe(getExploreOutputBudget(200).maxOutputChars); + // 499 -> small, 500 -> medium + expect(getExploreOutputBudget(499).maxOutputChars).toBe(getExploreOutputBudget(200).maxOutputChars); + expect(getExploreOutputBudget(500).maxOutputChars).toBe(getExploreOutputBudget(1000).maxOutputChars); + // 4999 -> medium, 5000 -> large + expect(getExploreOutputBudget(4999).maxOutputChars).toBe(getExploreOutputBudget(1000).maxOutputChars); + expect(getExploreOutputBudget(5000).maxOutputChars).toBe(getExploreOutputBudget(10000).maxOutputChars); + // 14999 -> large, 15000 -> xlarge + expect(getExploreOutputBudget(14999).maxOutputChars).toBe(getExploreOutputBudget(10000).maxOutputChars); + expect(getExploreOutputBudget(15000).maxOutputChars).toBe(getExploreOutputBudget(30000).maxOutputChars); + }); +}); + +/** + * End-to-end check that the budget is actually applied by handleExplore. + * + * Builds a tiny synthetic project (<500 files, so the small tier), indexes + * it, and confirms the output: + * - stays under the small-tier maxOutputChars cap + * - omits the meta-text the small tier gates off (completeness signal, + * budget note, "Additional relevant files") + * + * Regression guard for #185 — protects against future edits to handleExplore + * silently re-introducing the fixed 35KB cap on small projects. + */ +describe('codegraph_explore output respects the adaptive budget', () => { + let testDir: string; + let cg: CodeGraph; + let handler: ToolHandler; + + beforeAll(async () => { + testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-explore-budget-')); + const srcDir = path.join(testDir, 'src'); + fs.mkdirSync(srcDir); + + // A handful of files with one fat target file. The fat file mimics the + // Alamofire Session.swift case: many methods stacked on top of each other, + // which collapsed into one giant cluster pre-#185. + const fatLines: string[] = ['export class Session {']; + for (let i = 0; i < 30; i++) { + fatLines.push(` method${i}(arg: string): string {`); + fatLines.push(` return this.helper${i}(arg) + "${i}";`); + fatLines.push(` }`); + fatLines.push(` private helper${i}(arg: string): string {`); + fatLines.push(` return arg.repeat(${i + 1});`); + fatLines.push(` }`); + } + fatLines.push('}'); + fs.writeFileSync(path.join(srcDir, 'session.ts'), fatLines.join('\n')); + + // A few small supporting files so the project has >1 indexed file. + for (let i = 0; i < 5; i++) { + fs.writeFileSync( + path.join(srcDir, `support${i}.ts`), + `import { Session } from './session';\nexport function callSession${i}(s: Session) { return s.method${i}('hi'); }\n` + ); + } + + cg = CodeGraph.initSync(testDir, { + config: { include: ['**/*.ts'], exclude: [] }, + }); + await cg.indexAll(); + handler = new ToolHandler(cg); + }); + + afterAll(() => { + if (cg) cg.destroy(); + if (testDir && fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }); + } + }); + + it('keeps total output under the small-project cap', async () => { + const result = await handler.execute('codegraph_explore', { query: 'Session method helper' }); + const text = result.content?.[0]?.text ?? ''; + const smallBudget = getExploreOutputBudget(100); + // Allow a small overshoot for the trailing markers — the cap is enforced + // per-file rather than as an absolute output ceiling. + expect(text.length).toBeLessThan(smallBudget.maxOutputChars + 500); + }); + + it('omits the meta-text gated off for small projects', async () => { + const result = await handler.execute('codegraph_explore', { query: 'Session method helper' }); + const text = result.content?.[0]?.text ?? ''; + expect(text).not.toContain('### Additional relevant files'); + expect(text).not.toContain('Complete source code is included above'); + expect(text).not.toContain('Explore budget:'); + }); + + it('still includes the Relationships section — it is the cheapest structural signal', async () => { + const result = await handler.execute('codegraph_explore', { query: 'Session method helper' }); + const text = result.content?.[0]?.text ?? ''; + // Either there are relationships, or no edges were significant — both are fine. + // We just want to confirm we did not accidentally gate it off. + const hasRelationships = text.includes('### Relationships'); + const sourceFollowsHeader = text.indexOf('### Source Code') > 0; + expect(hasRelationships || sourceFollowsHeader).toBe(true); + }); + + it('prefixes source lines with line numbers by default (cat -n style)', async () => { + delete process.env.CODEGRAPH_EXPLORE_LINENUMS; + const result = await handler.execute('codegraph_explore', { query: 'Session method helper' }); + const text = result.content?.[0]?.text ?? ''; + // At least one fenced source line should look like `\t`. + expect(/\n\d+\t/.test(text)).toBe(true); + }); + + it('omits line numbers when CODEGRAPH_EXPLORE_LINENUMS=0', async () => { + process.env.CODEGRAPH_EXPLORE_LINENUMS = '0'; + try { + const result = await handler.execute('codegraph_explore', { query: 'Session method helper' }); + const text = result.content?.[0]?.text ?? ''; + // The synthetic source has no tab-prefixed numeric lines of its own, + // so none should appear when the toggle is off. + expect(/\n\d+\t(?:export| )/.test(text)).toBe(false); + } finally { + delete process.env.CODEGRAPH_EXPLORE_LINENUMS; + } + }); + + it('uses language-neutral omission markers (no C-style // in the output)', async () => { + // The gap/trimmed separators must not assume `//` is a comment — that's + // wrong in Python, Ruby, etc. They render inside fenced source blocks. + const result = await handler.execute('codegraph_explore', { query: 'Session method helper' }); + const text = result.content?.[0]?.text ?? ''; + expect(text).not.toContain('// ... (gap)'); + expect(text).not.toContain('// ... trimmed'); + }); + + it('does not collapse a whole-file class into just its header (envelope filter)', async () => { + // The synthetic `Session` class spans the entire file. Without the + // envelope filter it would form one giant cluster that tail-trims to + // the class declaration, hiding the methods. Confirm real method bodies + // make it into the output. Regression guard for the #185 follow-up. + const result = await handler.execute('codegraph_explore', { query: 'Session method helper' }); + const text = result.content?.[0]?.text ?? ''; + // A method body line (`methodN(arg: string)`) should appear, not just + // the `export class Session {` opener. + const hasMethodBody = /method\d+\(arg: string\)/.test(text); + expect(hasMethodBody).toBe(true); + }); +}); diff --git a/__tests__/expo-modules.test.ts b/__tests__/expo-modules.test.ts new file mode 100644 index 000000000..9da68594c --- /dev/null +++ b/__tests__/expo-modules.test.ts @@ -0,0 +1,154 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { expoModulesResolver } from '../src/resolution/frameworks/expo-modules'; + +describe('Expo Modules framework extractor', () => { + it('extracts AsyncFunction / Function / Property literals as method nodes', () => { + const source = ` +import ExpoModulesCore + +public class HapticsModule: Module { + public func definition() -> ModuleDefinition { + Name("ExpoHaptics") + + AsyncFunction("notificationAsync") { (notificationType: NotificationType) in + // body + } + + AsyncFunction("impactAsync") { (style: ImpactStyle) in + // body + } + + Function("synchronousThing") { + return 1 + } + + Property("isAvailable") { + return true + } + } +} +`; + const result = expoModulesResolver.extract?.('ios/HapticsModule.swift', source); + expect(result).toBeDefined(); + const names = result!.nodes.map((n) => n.name); + expect(names).toEqual( + expect.arrayContaining(['notificationAsync', 'impactAsync', 'synchronousThing', 'isAvailable']) + ); + expect(result!.nodes.every((n) => n.kind === 'method')).toBe(true); + expect(result!.nodes.every((n) => n.qualifiedName.includes('ExpoHaptics.'))).toBe(true); + }); + + it('falls back to the class name when the Module has no Name("X") literal', () => { + const source = ` +public class BareModule: Module { + public func definition() -> ModuleDefinition { + Function("doX") { return 1 } + } +} +`; + const result = expoModulesResolver.extract?.('ios/BareModule.swift', source); + // BareModule is used as the qualifier since there's no Name() literal. + expect(result!.nodes[0]?.qualifiedName).toContain('BareModule.doX'); + }); + + it('returns no nodes for a Swift file that is not an Expo Module', () => { + const source = ` +class Helper { + func doX() { } +} +`; + const result = expoModulesResolver.extract?.('Helper.swift', source); + expect(result?.nodes).toHaveLength(0); + }); + + it('also extracts from Kotlin module files', () => { + const source = ` +class FooModule : Module() { + override fun definition() = ModuleDefinition { + Name("ExpoFoo") + AsyncFunction("doAsync") { name: String -> name.uppercase() } + Function("doSync") { 42 } + } +} +`; + const result = expoModulesResolver.extract?.('FooModule.kt', source); + expect(result?.nodes.length).toBe(2); + expect(result?.nodes.map((n) => n.name).sort()).toEqual(['doAsync', 'doSync']); + expect(result?.nodes.every((n) => n.language === 'kotlin')).toBe(true); + }); +}); + +describe('Expo Modules end-to-end — JS caller → native AsyncFunction', () => { + let dir: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'expo-modules-fixture-')); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('JS callsite of a literal AsyncFunction("name") resolves to the native impl node', async () => { + fs.writeFileSync( + path.join(dir, 'package.json'), + '{"dependencies":{"expo-modules-core":"^1.0.0"}}' + ); + fs.mkdirSync(path.join(dir, 'ios')); + fs.writeFileSync( + path.join(dir, 'ios', 'HapticsModule.swift'), + ` +import ExpoModulesCore +public class HapticsModule: Module { + public func definition() -> ModuleDefinition { + Name("ExpoHaptics") + AsyncFunction("uniqueExpoHapticCall") { in /* … */ } + } +} +` + ); + fs.mkdirSync(path.join(dir, 'src')); + fs.writeFileSync( + path.join(dir, 'src', 'index.ts'), + ` +import { requireNativeModule } from 'expo-modules-core'; +const Haptics = requireNativeModule('ExpoHaptics'); +export async function impactAsync() { + return await Haptics.uniqueExpoHapticCall(); +} +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + + // The native method node should exist. + const native = db + .prepare( + "SELECT * FROM nodes WHERE kind='method' AND name='uniqueExpoHapticCall' AND id LIKE 'expo-module:%'" + ) + .all(); + expect(native).toHaveLength(1); + + // And the JS callsite should produce a call edge targeting it. + const callEdge = db + .prepare( + `SELECT t.name target, t.id target_id + FROM edges e + JOIN nodes s ON s.id = e.source + JOIN nodes t ON t.id = e.target + WHERE e.kind = 'calls' + AND s.file_path LIKE '%index.ts' + AND t.name = 'uniqueExpoHapticCall'` + ) + .all(); + cg.close?.(); + expect(callEdge.length).toBeGreaterThanOrEqual(1); + expect(callEdge[0].target_id.startsWith('expo-module:')).toBe(true); + }); +}); diff --git a/__tests__/extraction.test.ts b/__tests__/extraction.test.ts index cb69e2abc..b497af6a9 100644 --- a/__tests__/extraction.test.ts +++ b/__tests__/extraction.test.ts @@ -9,10 +9,9 @@ import * as fs from 'fs'; import * as path from 'path'; import * as os from 'os'; import { CodeGraph } from '../src'; -import { extractFromSource, scanDirectory, shouldIncludeFile } from '../src/extraction'; +import { extractFromSource, scanDirectory } from '../src/extraction'; import { detectLanguage, isLanguageSupported, getSupportedLanguages, initGrammars, loadAllGrammars } from '../src/extraction/grammars'; import { normalizePath } from '../src/utils'; -import { DEFAULT_CONFIG } from '../src/types'; beforeAll(async () => { await initGrammars(); @@ -94,6 +93,14 @@ describe('Language Detection', () => { expect(detectLanguage('main.dart')).toBe('dart'); }); + it('should detect Objective-C files', () => { + expect(detectLanguage('AppDelegate.m')).toBe('objc'); + expect(detectLanguage('ViewController.mm')).toBe('objc'); + const objcHeader = '@interface Foo : NSObject\n@end\n'; + expect(detectLanguage('Foo.h', objcHeader)).toBe('objc'); + expect(detectLanguage('stdio.h', '#ifndef STDIO_H\nvoid printf();\n#endif\n')).toBe('c'); + }); + it('should return unknown for unsupported extensions', () => { expect(detectLanguage('styles.css')).toBe('unknown'); expect(detectLanguage('data.json')).toBe('unknown'); @@ -198,6 +205,38 @@ export interface User { }); }); + it('should extract type references from interface property signatures', () => { + const code = ` +import type { IPage } from '../PromoterList'; +import type { IOrderField } from '../types'; + +interface Hprops { + value?: Partial & Partial; +} +`; + const result = extractFromSource('HeaderFilter.ts', code); + + const refs = result.unresolvedReferences.filter((r) => r.referenceKind === 'references'); + expect(refs.some((r) => r.referenceName === 'IPage')).toBe(true); + expect(refs.some((r) => r.referenceName === 'IOrderField')).toBe(true); + }); + + it('should extract type references from interface method signatures', () => { + const code = ` +import type { IPage } from '../PromoterList'; +import type { IOrderField } from '../types'; + +interface MethodForm { + fetchPage(arg: IPage): IOrderField; +} +`; + const result = extractFromSource('MethodForm.ts', code); + + const refs = result.unresolvedReferences.filter((r) => r.referenceKind === 'references'); + expect(refs.some((r) => r.referenceName === 'IPage')).toBe(true); + expect(refs.some((r) => r.referenceName === 'IOrderField')).toBe(true); + }); + it('should track function calls', () => { const code = ` function main() { @@ -479,6 +518,20 @@ export const authMachine = createMachine({ expect(varNode).toBeDefined(); expect(varNode?.isExported).toBe(true); }); + + it('should extract calls from a top-level variable initializer (issue #425)', () => { + const code = ` +import { getTokenMp } from './api/upload'; + +const token = getTokenMp(); +`; + const result = extractFromSource('app.ts', code); + + const call = result.unresolvedReferences.find( + (ref) => ref.referenceKind === 'calls' && ref.referenceName === 'getTokenMp' + ); + expect(call).toBeDefined(); + }); }); describe('File Node Extraction', () => { @@ -761,6 +814,130 @@ public class Calculator { expect(methodNode).toBeDefined(); expect(methodNode?.isStatic).toBe(true); }); + + it('wraps top-level declarations in a namespace from package_declaration', () => { + const code = ` +package com.example.foo; + +public class Bar { + public String greet() { return "hi"; } +} +`; + const result = extractFromSource('Bar.java', code); + + const ns = result.nodes.find((n) => n.kind === 'namespace'); + expect(ns?.name).toBe('com.example.foo'); + + const cls = result.nodes.find((n) => n.kind === 'class' && n.name === 'Bar'); + expect(cls?.qualifiedName).toBe('com.example.foo::Bar'); + + const greet = result.nodes.find((n) => n.kind === 'method' && n.name === 'greet'); + expect(greet?.qualifiedName).toBe('com.example.foo::Bar::greet'); + }); + + it('does not wrap when no package is declared', () => { + const code = ` +public class Bar { + public String greet() { return "hi"; } +} +`; + const result = extractFromSource('Bar.java', code); + expect(result.nodes.find((n) => n.kind === 'namespace')).toBeUndefined(); + const cls = result.nodes.find((n) => n.kind === 'class' && n.name === 'Bar'); + expect(cls?.qualifiedName).toBe('Bar'); + }); + + it('extracts anonymous-class overrides from `new T() { ... }`', () => { + // The pattern that breaks the trace through `strategy.foo()` in + // libraries like guava's Splitter: the lambda-returned anonymous + // class overrides abstract methods on the base, but without + // extracting those overrides the interface→impl synthesizer has + // nothing to bridge. + const code = ` +package com.example; + +abstract class Base { + abstract int compute(int x); +} + +public class Factory { + public Base make() { + return new Base() { + @Override + int compute(int x) { return x + 1; } + }; + } +} +`; + const result = extractFromSource('Factory.java', code); + + const anon = result.nodes.find((n) => n.kind === 'class' && /Base\$anon@/.test(n.name)); + expect(anon, 'anonymous Base subclass should be extracted as a class').toBeDefined(); + + const compute = result.nodes.find( + (n) => n.kind === 'method' && n.name === 'compute' && n.qualifiedName.includes('$anon@') + ); + expect(compute, 'override method should be a method on the anon class').toBeDefined(); + expect(compute!.qualifiedName).toContain('Factory::make:: r.referenceKind === 'extends' && r.referenceName === 'Base' && r.fromNodeId === anon!.id + ); + expect(extendsRef, 'anon class should carry an `extends Base` reference').toBeDefined(); + + // The enclosing `make` method still emits an instantiates edge to Base — + // anon extraction must not swallow that signal. + const instantiatesRef = result.unresolvedReferences.find( + (r) => r.referenceKind === 'instantiates' && r.referenceName === 'Base' + ); + expect(instantiatesRef, 'enclosing method should still instantiate Base').toBeDefined(); + }); + + it('extracts anonymous-class overrides inside a lambda body', () => { + // The exact guava pattern: a lambda is passed to a constructor, and the + // lambda body returns `new T() { @Override ... }`. The anon class must + // still surface even though it sits inside a lambda_expression node. + const code = ` +package com.example; + +interface Strategy { + java.util.Iterator iterator(String s); +} + +abstract class BaseIter implements java.util.Iterator { + abstract int separatorStart(int start); +} + +public class Splitter { + private final Strategy strategy; + public Splitter(Strategy s) { this.strategy = s; } + + public static Splitter on(char c) { + return new Splitter((seq) -> + new BaseIter() { + @Override + int separatorStart(int start) { return start + 1; } + @Override public boolean hasNext() { return false; } + @Override public String next() { return null; } + }); + } +} +`; + const result = extractFromSource('Splitter.java', code); + + const anon = result.nodes.find((n) => n.kind === 'class' && /BaseIter\$anon@/.test(n.name)); + expect(anon, 'anon BaseIter inside the lambda body should be extracted').toBeDefined(); + + const sepStart = result.nodes.find( + (n) => + n.kind === 'method' && + n.name === 'separatorStart' && + n.qualifiedName.includes('$anon@') + ); + expect(sepStart, 'override inside the lambda-returned anon class should be a method node').toBeDefined(); + }); }); describe('C# Extraction', () => { @@ -1120,6 +1297,54 @@ interface WebSocket { expect(methodNames).toContain('send'); expect(methodNames).toContain('cancel'); }); + + it('wraps top-level declarations in a namespace from package_header', () => { + const code = ` +package com.example.foo + +class Bar { + fun greet(): String = "hi" +} + +fun util(): Int = 42 +`; + const result = extractFromSource('Bar.kt', code); + + const ns = result.nodes.find((n) => n.kind === 'namespace'); + expect(ns?.name).toBe('com.example.foo'); + + const cls = result.nodes.find((n) => n.kind === 'class' && n.name === 'Bar'); + expect(cls?.qualifiedName).toBe('com.example.foo::Bar'); + + const greet = result.nodes.find((n) => n.kind === 'method' && n.name === 'greet'); + expect(greet?.qualifiedName).toBe('com.example.foo::Bar::greet'); + + const util = result.nodes.find((n) => n.kind === 'function' && n.name === 'util'); + expect(util?.qualifiedName).toBe('com.example.foo::util'); + }); + + it('handles a single-segment package', () => { + const code = ` +package foo + +class Bar +`; + const result = extractFromSource('Bar.kt', code); + const cls = result.nodes.find((n) => n.kind === 'class' && n.name === 'Bar'); + expect(cls?.qualifiedName).toBe('foo::Bar'); + }); + + it('does not wrap when no package is declared', () => { + const code = ` +class Bar { + fun greet() = "hi" +} +`; + const result = extractFromSource('Bar.kt', code); + expect(result.nodes.find((n) => n.kind === 'namespace')).toBeUndefined(); + const cls = result.nodes.find((n) => n.kind === 'class' && n.name === 'Bar'); + expect(cls?.qualifiedName).toBe('Bar'); + }); }); describe('Dart Extraction', () => { @@ -1152,6 +1377,11 @@ class UserService { const privateMethod = methodNodes.find((m) => m.name === '_privateMethod'); expect(privateMethod).toBeDefined(); expect(privateMethod?.visibility).toBe('private'); + + // Dart models a method body as a SIBLING of the signature, so the method + // node must be extended to span its body (not just the signature line) — + // required for body-level analysis (callees, the callback synthesizer). + expect(findById!.endLine).toBeGreaterThan(findById!.startLine); }); it('should extract top-level function declarations', () => { @@ -2012,6 +2242,27 @@ end expect(names).toContain('vector'); expect(names).toContain('config.h'); }); + + it('should create unresolved references for local includes', () => { + const code = `#include "myheader.h"`; + const result = extractFromSource('main.cpp', code); + + const importRef = result.unresolvedReferences.find( + (r) => r.referenceKind === 'imports' && r.referenceName === 'myheader.h' + ); + expect(importRef).toBeDefined(); + expect(importRef?.line).toBe(1); + }); + + it('should create unresolved references for system includes', () => { + const code = `#include `; + const result = extractFromSource('main.cpp', code); + + const importRef = result.unresolvedReferences.find( + (r) => r.referenceKind === 'imports' && r.referenceName === 'iostream' + ); + expect(importRef).toBeDefined(); + }); }); describe('Dart imports', () => { @@ -2975,6 +3226,70 @@ export function multiply(a: number, b: number): number { cg.close(); }); + + it('should count file-level tracked YAML files as indexed', async () => { + fs.writeFileSync(path.join(tempDir, 'app.yaml'), 'name: test\n'); + fs.writeFileSync(path.join(tempDir, 'routes.yml'), 'route: value\n'); + + const cg = CodeGraph.initSync(tempDir); + const result = await cg.indexAll(); + + expect(result.success).toBe(true); + expect(result.filesIndexed).toBe(2); + expect(result.filesSkipped).toBe(0); + expect(cg.getFiles().map((f) => f.path).sort()).toEqual(['app.yaml', 'routes.yml']); + + cg.close(); + }); + + it('should count file-level tracked YAML/Twig files as indexed in indexFiles()', async () => { + fs.writeFileSync(path.join(tempDir, 'app.yaml'), 'name: test\n'); + fs.writeFileSync(path.join(tempDir, 'view.twig'), '{{ title }}\n'); + + const cg = CodeGraph.initSync(tempDir); + const result = await cg.indexFiles(['app.yaml', 'view.twig']); + + expect(result.success).toBe(true); + expect(result.filesIndexed).toBe(2); + expect(result.filesSkipped).toBe(0); + + const tracked = cg.getFiles().map((f) => `${f.path}:${f.language}`).sort(); + expect(tracked).toEqual(['app.yaml:yaml', 'view.twig:twig']); + + cg.close(); + }); + + it('should count file-level tracked .properties files as indexed', async () => { + fs.writeFileSync(path.join(tempDir, 'application.properties'), 'server.port=8080\n'); + fs.writeFileSync(path.join(tempDir, 'log.properties'), 'log.level=INFO\n'); + + const cg = CodeGraph.initSync(tempDir); + const result = await cg.indexAll(); + + expect(result.success).toBe(true); + expect(result.filesIndexed).toBe(2); + expect(result.filesSkipped).toBe(0); + + cg.close(); + }); + + it('should count the full file-level tracked class (yaml/twig/properties) in indexFiles()', async () => { + fs.writeFileSync(path.join(tempDir, 'app.yaml'), 'name: test\n'); + fs.writeFileSync(path.join(tempDir, 'view.twig'), '{{ title }}\n'); + fs.writeFileSync(path.join(tempDir, 'application.properties'), 'server.port=8080\n'); + + const cg = CodeGraph.initSync(tempDir); + const result = await cg.indexFiles(['app.yaml', 'view.twig', 'application.properties']); + + expect(result.success).toBe(true); + expect(result.filesIndexed).toBe(3); + expect(result.filesSkipped).toBe(0); + + const tracked = cg.getFiles().map((f) => `${f.path}:${f.language}`).sort(); + expect(tracked).toEqual(['app.yaml:yaml', 'application.properties:properties', 'view.twig:twig']); + + cg.close(); + }); }); describe('Path Normalization', () => { @@ -3003,39 +3318,57 @@ describe('Directory Exclusion', () => { cleanupTempDir(tempDir); }); - it('should exclude node_modules directories', () => { - // Create structure: src/index.ts + node_modules/pkg/index.js + it('should exclude directories listed in .gitignore', () => { + // Create structure: src/index.ts + node_modules/pkg/index.js, gitignore node_modules const srcDir = path.join(tempDir, 'src'); const nmDir = path.join(tempDir, 'node_modules', 'pkg'); fs.mkdirSync(srcDir, { recursive: true }); fs.mkdirSync(nmDir, { recursive: true }); fs.writeFileSync(path.join(srcDir, 'index.ts'), 'export const x = 1;'); fs.writeFileSync(path.join(nmDir, 'index.js'), 'module.exports = {};'); + fs.writeFileSync(path.join(tempDir, '.gitignore'), 'node_modules/\n'); - const config = { ...DEFAULT_CONFIG, rootDir: tempDir }; - const files = scanDirectory(tempDir, config); + const files = scanDirectory(tempDir); expect(files).toContain('src/index.ts'); expect(files.every((f) => !f.includes('node_modules'))).toBe(true); }); - it('should exclude nested node_modules directories', () => { - // Create structure: packages/app/node_modules/pkg/index.js + it('should exclude nested node_modules via a root .gitignore', () => { + // A trailing-slash pattern with no leading slash matches at any depth. const srcDir = path.join(tempDir, 'packages', 'app', 'src'); const nmDir = path.join(tempDir, 'packages', 'app', 'node_modules', 'pkg'); fs.mkdirSync(srcDir, { recursive: true }); fs.mkdirSync(nmDir, { recursive: true }); fs.writeFileSync(path.join(srcDir, 'index.ts'), 'export const x = 1;'); fs.writeFileSync(path.join(nmDir, 'index.js'), 'module.exports = {};'); + fs.writeFileSync(path.join(tempDir, '.gitignore'), 'node_modules/\n'); - const config = { ...DEFAULT_CONFIG, rootDir: tempDir }; - const files = scanDirectory(tempDir, config); + const files = scanDirectory(tempDir); expect(files).toContain('packages/app/src/index.ts'); expect(files.every((f) => !f.includes('node_modules'))).toBe(true); }); - it('should exclude .git directories', () => { + it('should apply a nested .gitignore only to its own subtree', () => { + const appSrc = path.join(tempDir, 'app', 'src'); + fs.mkdirSync(appSrc, { recursive: true }); + fs.writeFileSync(path.join(appSrc, 'keep.ts'), 'export const a = 1;'); + fs.writeFileSync(path.join(appSrc, 'skip.ts'), 'export const b = 2;'); + fs.writeFileSync(path.join(tempDir, 'app', '.gitignore'), 'src/skip.ts\n'); + // A sibling with the same name outside app/ must NOT be ignored. + const otherDir = path.join(tempDir, 'other', 'src'); + fs.mkdirSync(otherDir, { recursive: true }); + fs.writeFileSync(path.join(otherDir, 'skip.ts'), 'export const c = 3;'); + + const files = scanDirectory(tempDir); + + expect(files).toContain('app/src/keep.ts'); + expect(files).not.toContain('app/src/skip.ts'); + expect(files).toContain('other/src/skip.ts'); + }); + + it('should always skip .git directories', () => { const srcDir = path.join(tempDir, 'src'); const gitDir = path.join(tempDir, '.git', 'objects'); fs.mkdirSync(srcDir, { recursive: true }); @@ -3043,8 +3376,7 @@ describe('Directory Exclusion', () => { fs.writeFileSync(path.join(srcDir, 'index.ts'), 'export const x = 1;'); fs.writeFileSync(path.join(gitDir, 'pack.ts'), 'export const y = 2;'); - const config = { ...DEFAULT_CONFIG, rootDir: tempDir }; - const files = scanDirectory(tempDir, config); + const files = scanDirectory(tempDir); expect(files).toContain('src/index.ts'); expect(files.every((f) => !f.includes('.git'))).toBe(true); @@ -3055,29 +3387,12 @@ describe('Directory Exclusion', () => { fs.mkdirSync(srcDir, { recursive: true }); fs.writeFileSync(path.join(srcDir, 'Button.tsx'), 'export function Button() {}'); - const config = { ...DEFAULT_CONFIG, rootDir: tempDir }; - const files = scanDirectory(tempDir, config); + const files = scanDirectory(tempDir); expect(files.length).toBe(1); expect(files[0]).toBe('src/components/Button.tsx'); expect(files[0]).not.toContain('\\'); }); - - it('should respect .codegraphignore marker', () => { - const srcDir = path.join(tempDir, 'src'); - const vendorDir = path.join(tempDir, 'vendor'); - fs.mkdirSync(srcDir, { recursive: true }); - fs.mkdirSync(vendorDir, { recursive: true }); - fs.writeFileSync(path.join(srcDir, 'index.ts'), 'export const x = 1;'); - fs.writeFileSync(path.join(vendorDir, 'lib.ts'), 'export const y = 2;'); - fs.writeFileSync(path.join(vendorDir, '.codegraphignore'), ''); - - const config = { ...DEFAULT_CONFIG, rootDir: tempDir }; - const files = scanDirectory(tempDir, config); - - expect(files).toContain('src/index.ts'); - expect(files.every((f) => !f.includes('vendor'))).toBe(true); - }); }); describe('Git Submodules', () => { @@ -3124,14 +3439,84 @@ describe('Git Submodules', () => { ); git(mainDir, 'commit', '-q', '-m', 'add submodule'); - const config = { ...DEFAULT_CONFIG, rootDir: mainDir }; - const files = scanDirectory(mainDir, config); + const files = scanDirectory(mainDir); expect(files).toContain('app.ts'); expect(files).toContain('libs/lib/lib.ts'); }); }); +describe('Nested non-submodule git repos', () => { + let tempDir: string; + + beforeEach(() => { + tempDir = createTempDir(); + }); + + afterEach(() => { + cleanupTempDir(tempDir); + }); + + it('should index files in embedded git repos run from a git super-repo (issue #193)', async () => { + const { execFileSync } = await import('child_process'); + const git = (cwd: string, ...args: string[]) => + execFileSync('git', args, { cwd, stdio: 'pipe' }); + + // Top-level workspace is itself a git repo, holding no source directly — + // the CMake "super-repo" layout from the issue. + const root = path.join(tempDir, 'root'); + fs.mkdirSync(path.join(root, 'coding'), { recursive: true }); + git(root, 'init', '-q'); + git(root, 'config', 'user.email', 'test@test.com'); + git(root, 'config', 'user.name', 'Test'); + fs.writeFileSync(path.join(root, 'CMakeLists.txt'), 'cmake_minimum_required(VERSION 3.10)\n'); + + // Two independent clones living inside the workspace (NOT submodules): + // one with committed source, one with only untracked source. + const sub1 = path.join(root, 'sub_repo1', 'src'); + fs.mkdirSync(sub1, { recursive: true }); + git(path.join(root, 'sub_repo1'), 'init', '-q'); + git(path.join(root, 'sub_repo1'), 'config', 'user.email', 'test@test.com'); + git(path.join(root, 'sub_repo1'), 'config', 'user.name', 'Test'); + fs.writeFileSync(path.join(sub1, 'one.ts'), 'export const one = 1;'); + git(path.join(root, 'sub_repo1'), 'add', '-A'); + git(path.join(root, 'sub_repo1'), 'commit', '-q', '-m', 'sub1 init'); + + const sub2 = path.join(root, 'sub_repo2', 'src'); + fs.mkdirSync(sub2, { recursive: true }); + git(path.join(root, 'sub_repo2'), 'init', '-q'); + fs.writeFileSync(path.join(sub2, 'two.ts'), 'export const two = 2;'); + + const files = scanDirectory(root); + + // Both committed and untracked source from the nested repos must be found. + expect(files).toContain('sub_repo1/src/one.ts'); + expect(files).toContain('sub_repo2/src/two.ts'); + }); + + it('should respect each embedded repo\'s own .gitignore', async () => { + const { execFileSync } = await import('child_process'); + const git = (cwd: string, ...args: string[]) => + execFileSync('git', args, { cwd, stdio: 'pipe' }); + + const root = path.join(tempDir, 'root'); + fs.mkdirSync(root, { recursive: true }); + git(root, 'init', '-q'); + + const sub = path.join(root, 'sub_repo', 'src'); + fs.mkdirSync(sub, { recursive: true }); + git(path.join(root, 'sub_repo'), 'init', '-q'); + fs.writeFileSync(path.join(root, 'sub_repo', '.gitignore'), 'src/generated.ts\n'); + fs.writeFileSync(path.join(sub, 'real.ts'), 'export const real = 1;'); + fs.writeFileSync(path.join(sub, 'generated.ts'), 'export const generated = 1;'); + + const files = scanDirectory(root); + + expect(files).toContain('sub_repo/src/real.ts'); + expect(files).not.toContain('sub_repo/src/generated.ts'); + }); +}); + // ============================================================================= // Scala // ============================================================================= @@ -3486,6 +3871,53 @@ function increment(): void { } }); + it('should extract calls from top-level +`; + const result = extractFromSource('Issue425Setup.vue', code); + + const call = result.unresolvedReferences.find( + (ref) => ref.referenceKind === 'calls' && ref.referenceName === 'getTokenMp' + ); + expect(call).toBeDefined(); + }); + + it('should extract calls from Vue Options API object methods', () => { + const code = ` + + +`; + const result = extractFromSource('Issue425Options.vue', code); + + const calls = result.unresolvedReferences.filter( + (ref) => ref.referenceKind === 'calls' && ref.referenceName === 'getTokenMp' + ); + expect(calls).toHaveLength(2); + }); + it('should extract from both + + + + diff --git a/site/src/styles/theme.css b/site/src/styles/theme.css new file mode 100644 index 000000000..4a406fac2 --- /dev/null +++ b/site/src/styles/theme.css @@ -0,0 +1,226 @@ +/* ===================================================================== + codegraph — flat / paper editorial theme + Monochrome ink-on-paper, hairline rules, square corners. Shared by the + custom landing page (src/pages/index.astro) and the Starlight docs. + ===================================================================== */ + +/* ---- Fonts ---- */ +:root { + --sl-font: 'Archivo Variable', -apple-system, BlinkMacSystemFont, 'Helvetica Neue', Arial, sans-serif; + --sl-font-mono: 'IBM Plex Mono', ui-monospace, 'SF Mono', Menlo, Consolas, monospace; +} + +/* ---- Starlight colour mapping: light / paper (default) ---- */ +:root, +:root[data-theme='light'] { + --sl-color-accent-low: #e2dfd5; + --sl-color-accent: #16150f; + --sl-color-accent-high: #16150f; + --sl-color-white: #16150f; + --sl-color-gray-1: #2a281f; + --sl-color-gray-2: #56544a; + --sl-color-gray-3: #6f6c61; + --sl-color-gray-4: #87847a; + --sl-color-gray-5: #b4b1a5; + --sl-color-gray-6: #d6d3c8; + --sl-color-gray-7: #e8e6dd; + --sl-color-black: #f7f6f2; + + --sl-color-bg: #f7f6f2; + --sl-color-bg-nav: #f7f6f2; + --sl-color-bg-sidebar: #f7f6f2; + --sl-color-bg-inline-code: #e8e6dd; + --sl-color-bg-accent: #16150f; + + --sl-color-text: #16150f; + --sl-color-text-accent: #16150f; + --sl-color-text-invert: #f7f6f2; + + --sl-color-hairline: #16150f; + --sl-color-hairline-light: #d6d3c8; + --sl-color-hairline-shade: #d6d3c8; + + /* shared tokens */ + --cg-paper: #f7f6f2; + --cg-paper-2: #f1efe8; + --cg-paper-press: #e8e6dd; + --cg-ink: #16150f; + --cg-ink-2: #56544a; + --cg-ink-3: #87847a; + --cg-rule: #16150f; + --cg-rule-soft: #d6d3c8; +} + +/* ---- Starlight colour mapping: dark / ink ---- */ +:root[data-theme='dark'] { + --sl-color-accent-low: #34322a; + --sl-color-accent: #f3f1ea; + --sl-color-accent-high: #f3f1ea; + --sl-color-white: #f3f1ea; + --sl-color-gray-1: #e7e5dc; + --sl-color-gray-2: #c9c6ba; + --sl-color-gray-3: #a7a499; + --sl-color-gray-4: #7c7a70; + --sl-color-gray-5: #57554c; + --sl-color-gray-6: #2c2a23; + --sl-color-gray-7: #1e1c16; + --sl-color-black: #16150f; + + --sl-color-bg: #16150f; + --sl-color-bg-nav: #16150f; + --sl-color-bg-sidebar: #16150f; + --sl-color-bg-inline-code: #23211a; + --sl-color-bg-accent: #f3f1ea; + + --sl-color-text: #f3f1ea; + --sl-color-text-accent: #f3f1ea; + --sl-color-text-invert: #16150f; + + --sl-color-hairline: #f3f1ea; + --sl-color-hairline-light: #34322a; + --sl-color-hairline-shade: #34322a; + + --cg-paper: #16150f; + --cg-paper-2: #1e1c16; + --cg-paper-press: #23211a; + --cg-ink: #f3f1ea; + --cg-ink-2: #b8b5a8; + --cg-ink-3: #87847a; + --cg-rule: #f3f1ea; + --cg-rule-soft: #34322a; +} + +/* ---- Global flat resets ---- */ +*, +*::before, +*::after { + border-radius: 0 !important; /* this design has no rounded corners, anywhere */ +} + +:root { + --sl-shadow-sm: none; + --sl-shadow-md: none; + --sl-shadow-lg: none; +} + +body { + background: var(--cg-paper); + color: var(--cg-ink); + -webkit-font-smoothing: antialiased; + -moz-osx-font-smoothing: grayscale; +} + +:where(h1, h2, h3, h4, h5) { + letter-spacing: -0.02em; +} + +/* ---- Docs chrome ---- */ + +/* Header: one crisp bottom rule. Starlight nests
inside +
, so a bare `.header { border-bottom }` draws two + lines — put the rule on the outer
only and clear the inner div. */ +.header { + background: var(--cg-paper); + -webkit-backdrop-filter: none; + backdrop-filter: none; +} +header.header { + border-bottom: 1px solid var(--cg-rule); +} +.header .header { + border-bottom: 0; +} + +/* Sidebar: crisp right rule */ +#starlight__sidebar, +.sidebar-pane { + border-inline-end: 1px solid var(--cg-rule); + background: var(--cg-paper); +} + +/* Sidebar group labels — small caps, committed editorial direction */ +.sidebar-content details > summary, +.sidebar-content > ul > li > span, +.sidebar-content .large { + font-weight: 700; + letter-spacing: 0.07em; + text-transform: uppercase; + font-size: 0.72rem; + color: var(--cg-ink-2); +} + +/* Sidebar links */ +.sidebar-content a { + color: var(--cg-ink-2); +} +.sidebar-content a:hover { + background: var(--cg-paper-press); + color: var(--cg-ink); +} +.sidebar-content a[aria-current='page'], +.sidebar-content a[aria-current='page']:hover { + background: transparent; + color: var(--cg-ink); + font-weight: 700; + border-inline-start: 2px solid var(--cg-ink); +} + +/* Right "On this page" rail */ +starlight-toc a { + color: var(--cg-ink-3); +} +starlight-toc a[aria-current='true'] { + color: var(--cg-ink); + font-weight: 600; +} + +/* Prev / next pagination: flat bordered boxes */ +.pagination-links a { + border: 1px solid var(--cg-rule); + box-shadow: none; + background: var(--cg-paper); +} +.pagination-links a:hover { + background: var(--cg-paper-press); +} + +/* Inline code */ +.sl-markdown-content :not(pre) > code { + border: 1px solid var(--cg-rule-soft); + background: var(--cg-paper-2); + font-size: 0.875em; +} + +/* Cards / asides: square, hairline */ +.card, +.starlight-aside { + border: 1px solid var(--cg-rule); + box-shadow: none; +} + +/* Search trigger */ +button[data-open-modal] { + border: 1px solid var(--cg-rule); + background: var(--cg-paper); +} + +/* Content horizontal rules */ +.sl-markdown-content hr { + border: 0; + border-top: 1px solid var(--cg-rule); +} + +/* Links in prose */ +.sl-markdown-content a { + color: var(--cg-ink); + text-underline-offset: 3px; +} + +/* On wide screens Starlight right-aligns the content against the TOC + (margin-inline: auto 0), piling all the empty space on the left. Center the + content within its pane so it sits balanced between the sidebar and the TOC. */ +@media (min-width: 72rem) { + .main-pane { + --sl-content-margin-inline: auto; + } +} diff --git a/site/tsconfig.json b/site/tsconfig.json new file mode 100644 index 000000000..8bf91d3bb --- /dev/null +++ b/site/tsconfig.json @@ -0,0 +1,5 @@ +{ + "extends": "astro/tsconfigs/strict", + "include": [".astro/types.d.ts", "**/*"], + "exclude": ["dist"] +} diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 8ee7a6f9e..9e7f98887 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -7,6 +7,7 @@ * Usage: * codegraph Run interactive installer (when no args) * codegraph install Run interactive installer + * codegraph uninstall Remove CodeGraph from your agents * codegraph init [path] Initialize CodeGraph in a project * codegraph uninit [path] Remove CodeGraph from a project * codegraph index [path] Index all files in the project @@ -15,6 +16,9 @@ * codegraph query Search for symbols * codegraph files [options] Show project file structure * codegraph context Build context for a task + * codegraph callers Find what calls a function/method + * codegraph callees Find what a function/method calls + * codegraph impact Analyze what code is affected by changing a symbol * codegraph affected [files] Find test files affected by changes */ @@ -22,9 +26,12 @@ import { Command } from 'commander'; import * as path from 'path'; import * as fs from 'fs'; import { getCodeGraphDir, isInitialized } from '../directory'; +import { detectWorktreeIndexMismatch, worktreeMismatchWarning } from '../sync/worktree'; import { createShimmerProgress } from '../ui/shimmer-progress'; +import { getGlyphs } from '../ui/glyphs'; -import { buildNode25BlockBanner } from './node-version-check'; +import { buildNode25BlockBanner, buildNodeTooOldBanner, MIN_NODE_MAJOR } from './node-version-check'; +import { relaunchWithWasmRuntimeFlagsIfNeeded } from '../extraction/wasm-runtime-flags'; // Lazy-load heavy modules (CodeGraph, runInstaller) to keep CLI startup fast. async function loadCodeGraph(): Promise { @@ -32,7 +39,7 @@ async function loadCodeGraph(): Promise { return await import('../index'); } catch (err) { const msg = err instanceof Error ? err.message : String(err); - console.error('\x1b[31m✗\x1b[0m Failed to load CodeGraph modules.'); + console.error(`\x1b[31m${getGlyphs().err}\x1b[0m Failed to load CodeGraph modules.`); console.error(`\n Node: ${process.version} Platform: ${process.platform} ${process.arch}`); console.error(`\n Error: ${msg}`); console.error('\n Try reinstalling with: npm install -g @colbymchenry/codegraph\n'); @@ -62,6 +69,23 @@ if (nodeMajor >= 25) { } // Override active — banner shown for visibility, continuing. } +// Enforce the supported Node floor. `engines` in package.json only *warns* on +// install (unless engine-strict), so hard-block here to actually keep users off +// unsupported versions. Mirrors the 25+ block above. See package.json `engines`. +if (nodeMajor < MIN_NODE_MAJOR) { + process.stderr.write(buildNodeTooOldBanner(nodeVersion) + '\n'); + if (!process.env.CODEGRAPH_ALLOW_UNSAFE_NODE) { + process.exit(1); + } + // Override active — banner shown for visibility, continuing. +} + +// Re-exec with V8's `--liftoff-only` if it isn't already set, so tree-sitter's +// large WASM grammars never hit the turboshaft Zone OOM (`Fatal process out of +// memory: Zone`) on Node >= 22. No-op under the bundled launcher, which already +// passes the flag. Must run before any grammar (in the parse worker, which +// inherits this process's flags) is compiled. See ../extraction/wasm-runtime-flags. +relaunchWithWasmRuntimeFlagsIfNeeded(__filename); // Check if running with no arguments - run installer if (process.argv.length === 2) { @@ -212,7 +236,7 @@ function createVerboseProgress(): (progress: { phase: string; current: number; t // Log every 5% to keep output manageable if (pct >= lastPct + 5 || progress.current === progress.total) { lastPct = pct; - console.log(`[${elapsed}s] ${progress.current}/${progress.total} (${pct}%)${progress.currentFile ? ` — ${progress.currentFile}` : ''}`); + console.log(`[${elapsed}s] ${progress.current}/${progress.total} (${pct}%)${progress.currentFile ? ` ${getGlyphs().dash} ${progress.currentFile}` : ''}`); } } else if (progress.current > 0) { // Scanning phase (no total yet) — log periodically @@ -227,28 +251,28 @@ function createVerboseProgress(): (progress: { phase: string; current: number; t * Print success message */ function success(message: string): void { - console.log(chalk.green('✓') + ' ' + message); + console.log(chalk.green(getGlyphs().ok) + ' ' + message); } /** * Print error message */ function error(message: string): void { - console.error(chalk.red('✗') + ' ' + message); + console.error(chalk.red(getGlyphs().err) + ' ' + message); } /** * Print info message */ function info(message: string): void { - console.log(chalk.blue('ℹ') + ' ' + message); + console.log(chalk.blue(getGlyphs().info) + ' ' + message); } /** * Print warning message */ function warn(message: string): void { - console.log(chalk.yellow('⚠') + ' ' + message); + console.log(chalk.yellow(getGlyphs().warn) + ' ' + message); } type IndexResult = { @@ -281,7 +305,7 @@ function printIndexResult(clack: typeof import('@clack/prompts'), result: IndexR // continuing to the misleading "No files found" branch or throwing. if (!result.success && !hasErrors && result.filesIndexed === 0) { const generic = result.errors.find((e) => e.severity === 'error'); - clack.log.error(generic?.message ?? 'Indexing failed — no further details available'); + clack.log.error(generic?.message ?? `Indexing failed ${getGlyphs().dash} no further details available`); return; } @@ -293,7 +317,7 @@ function printIndexResult(clack: typeof import('@clack/prompts'), result: IndexR } clack.log.info(`${formatNumber(result.nodesCreated)} nodes, ${formatNumber(result.edgesCreated)} edges in ${formatDuration(result.durationMs)}`); } else if (hasErrors) { - clack.log.error(`Indexing failed — all ${formatNumber(result.filesErrored)} files had errors`); + clack.log.error(`Indexing failed ${getGlyphs().dash} all ${formatNumber(result.filesErrored)} files had errors`); } else { clack.log.warn('No files found to index'); } @@ -327,7 +351,7 @@ function printIndexResult(clack: typeof import('@clack/prompts'), result: IndexR } if (result.filesIndexed > 0) { - clack.log.info('The index is fully usable — only the failed files are missing.'); + clack.log.info(`The index is fully usable ${getGlyphs().dash} only the failed files are missing.`); } } else if (projectPath) { const logPath = path.join(projectPath, '.codegraph', 'errors.log'); @@ -365,7 +389,7 @@ function writeErrorLog(projectPath: string, errors: Array<{ message: string; fil } const lines: string[] = [ - `CodeGraph Error Log — ${new Date().toISOString()}`, + `CodeGraph Error Log - ${new Date().toISOString()}`, `${errorsByFile.size} files with errors`, '', ]; @@ -392,8 +416,8 @@ function writeErrorLog(projectPath: string, errors: Array<{ message: string; fil */ program .command('init [path]') - .description('Initialize CodeGraph in a project directory') - .option('-i, --index', 'Run initial indexing after initialization') + .description('Initialize CodeGraph in a project directory and build the initial index') + .option('-i, --index', 'Deprecated: indexing now runs by default; flag accepted for backward compatibility') .option('-v, --verbose', 'Show detailed worker lifecycle and memory info') .action(async (pathArg: string | undefined, options: { index?: boolean; verbose?: boolean }) => { const projectPath = path.resolve(pathArg || process.cwd()); @@ -405,6 +429,10 @@ program if (isInitialized(projectPath)) { clack.log.warn(`Already initialized in ${projectPath}`); clack.log.info('Use "codegraph index" to re-index or "codegraph sync" to update'); + try { + const { offerWatchFallback } = await import('../installer'); + await offerWatchFallback(clack, projectPath); + } catch { /* non-fatal */ } clack.outro(''); return; } @@ -413,27 +441,29 @@ program const cg = await CodeGraph.init(projectPath, { index: false }); clack.log.success(`Initialized in ${projectPath}`); - if (options.index) { - let result: IndexResult; - - if (options.verbose) { - result = await cg.indexAll({ - onProgress: createVerboseProgress(), - verbose: true, - }); - } else { - process.stdout.write(`${colors.dim}│${colors.reset}\n`); - const progress = createShimmerProgress(); - result = await cg.indexAll({ - onProgress: progress.onProgress, - }); - await progress.stop(); - } - - printIndexResult(clack, result, projectPath); + // Indexing runs by default now. The legacy -i/--index flag is still + // accepted (so existing muscle memory and scripts don't break) but is a + // no-op — initializing always builds the initial index. + let result: IndexResult; + if (options.verbose) { + result = await cg.indexAll({ + onProgress: createVerboseProgress(), + verbose: true, + }); } else { - clack.log.info('Run "codegraph index" to index the project'); + process.stdout.write(`${colors.dim}${getGlyphs().rail}${colors.reset}\n`); + const progress = createShimmerProgress(); + result = await cg.indexAll({ + onProgress: progress.onProgress, + }); + await progress.stop(); } + printIndexResult(clack, result, projectPath); + + try { + const { offerWatchFallback } = await import('../installer'); + await offerWatchFallback(clack, projectPath); + } catch { /* non-fatal */ } clack.outro('Done'); cg.destroy(); @@ -465,7 +495,7 @@ program const rl = readline.createInterface({ input: process.stdin, output: process.stdout }); const answer = await new Promise((resolve) => { rl.question( - chalk.yellow('⚠ This will permanently delete all CodeGraph data. Continue? (y/N) '), + chalk.yellow(`${getGlyphs().warn} This will permanently delete all CodeGraph data. Continue? (y/N) `), resolve ); }); @@ -481,6 +511,15 @@ program const cg = CodeGraph.openSync(projectPath); cg.uninitialize(); + // Clean up any git sync hooks we installed (no-op if none / not a repo). + try { + const { removeGitSyncHook } = await import('../sync/git-hooks'); + const removed = removeGitSyncHook(projectPath); + if (removed.installed.length > 0) { + info(`Removed git ${removed.installed.join(', ')} sync hook${removed.installed.length > 1 ? 's' : ''}`); + } + } catch { /* non-fatal */ } + success(`Removed CodeGraph from ${projectPath}`); } catch (err) { error(`Failed to uninitialize: ${err instanceof Error ? err.message : String(err)}`); @@ -535,7 +574,7 @@ program verbose: true, }); } else { - process.stdout.write(`${colors.dim}│${colors.reset}\n`); + process.stdout.write(`${colors.dim}${getGlyphs().rail}${colors.reset}\n`); const progress = createShimmerProgress(); result = await cg.indexAll({ onProgress: progress.onProgress, @@ -587,7 +626,7 @@ program const clack = await importESM('@clack/prompts'); clack.intro('Syncing CodeGraph'); - process.stdout.write(`${colors.dim}│${colors.reset}\n`); + process.stdout.write(`${colors.dim}${getGlyphs().rail}${colors.reset}\n`); const progress = createShimmerProgress(); const result = await cg.sync({ @@ -606,7 +645,7 @@ program if (result.filesAdded > 0) details.push(`Added: ${result.filesAdded}`); if (result.filesModified > 0) details.push(`Modified: ${result.filesModified}`); if (result.filesRemoved > 0) details.push(`Removed: ${result.filesRemoved}`); - clack.log.info(`${details.join(', ')} — ${formatNumber(result.nodesUpdated)} nodes in ${formatDuration(result.durationMs)}`); + clack.log.info(`${details.join(', ')} ${getGlyphs().dash} ${formatNumber(result.nodesUpdated)} nodes in ${formatDuration(result.durationMs)}`); } clack.outro('Done'); @@ -628,6 +667,11 @@ program .option('-j, --json', 'Output as JSON') .action(async (pathArg: string | undefined, options: { json?: boolean }) => { const projectPath = resolveProjectPath(pathArg); + // The directory the user actually ran from, before walking up to the index + // root. Used to detect when the resolved index lives in a different git + // working tree (e.g. a nested worktree borrowing the main checkout's index). + const startPath = path.resolve(pathArg || process.cwd()); + const worktreeMismatch = detectWorktreeIndexMismatch(startPath, projectPath); try { if (!isInitialized(projectPath)) { @@ -647,6 +691,7 @@ program const stats = cg.getStats(); const changes = cg.getChangedFiles(); const backend = cg.getBackend(); + const journalMode = cg.getJournalMode(); // JSON output mode if (options.json) { @@ -658,6 +703,7 @@ program edgeCount: stats.edgeCount, dbSizeBytes: stats.dbSizeBytes, backend, + journalMode, nodesByKind: stats.nodesByKind, languages: Object.entries(stats.filesByLanguage).filter(([, count]) => count > 0).map(([lang]) => lang), pendingChanges: { @@ -665,6 +711,9 @@ program modified: changes.modified.length, removed: changes.removed.length, }, + worktreeMismatch: worktreeMismatch + ? { worktreeRoot: worktreeMismatch.worktreeRoot, indexRoot: worktreeMismatch.indexRoot } + : null, })); cg.destroy(); return; @@ -674,6 +723,9 @@ program // Project info console.log(chalk.cyan('Project:'), projectPath); + if (worktreeMismatch) { + warn(worktreeMismatchWarning(worktreeMismatch)); + } console.log(); // Index stats @@ -682,14 +734,18 @@ program console.log(` Nodes: ${formatNumber(stats.nodeCount)}`); console.log(` Edges: ${formatNumber(stats.edgeCount)}`); console.log(` DB Size: ${(stats.dbSizeBytes / 1024 / 1024).toFixed(2)} MB`); - // Surface the active SQLite backend so users can spot the silent - // WASM fallback (5-10x slower). better-sqlite3 is in - // `optionalDependencies`, so `npm install` succeeds without it - // when the native build fails. - const backendLabel = backend === 'native' - ? chalk.green('native') - : chalk.yellow('wasm — slower fallback; run `npm rebuild better-sqlite3`'); + // Surface the active SQLite backend (node:sqlite — Node's built-in real + // SQLite, full WAL + FTS5, no native build). + const backendLabel = chalk.green(`node:sqlite ${getGlyphs().dash} built-in (full WAL)`); console.log(` Backend: ${backendLabel}`); + // Effective journal mode: 'wal' means concurrent reads never block on a + // writer; anything else means they can ("database is locked"). node:sqlite + // supports WAL everywhere, so a non-wal mode means the filesystem can't + // (network mounts, WSL2 /mnt). See issue #238. + const journalLabel = journalMode === 'wal' + ? chalk.green('wal') + : chalk.yellow(`${journalMode || 'unknown'} ${getGlyphs().dash} WAL inactive; reads can block on writes`); + console.log(` Journal: ${journalLabel}`); console.log(); // Node breakdown @@ -761,11 +817,21 @@ program const cg = await CodeGraph.open(projectPath); const limit = parseInt(options.limit || '10', 10); - const results = cg.searchNodes(search, { + const rawResults = cg.searchNodes(search, { limit, kinds: options.kind ? [options.kind as any] : undefined, }); + // Mirror the MCP search down-rank so the CLI also surfaces the + // hand-written implementation before protobuf/gRPC scaffolding + // when both share a name. See extraction/generated-detection.ts. + const { isGeneratedFile } = await import('../extraction/generated-detection'); + const results = [...rawResults].sort((a, b) => { + const aGen = isGeneratedFile(a.node.filePath) ? 1 : 0; + const bGen = isGeneratedFile(b.node.filePath) ? 1 : 0; + return aGen - bGen; + }); + if (options.json) { console.log(JSON.stringify(results, null, 2)); } else { @@ -977,8 +1043,9 @@ function printFileTree( const renderNode = (node: TreeNode, prefix: string, isLast: boolean, depth: number): void => { if (maxDepth !== undefined && depth > maxDepth) return; - const connector = isLast ? '└── ' : '├── '; - const childPrefix = isLast ? ' ' : '│ '; + const glyphs = getGlyphs(); + const connector = isLast ? glyphs.treeLast : glyphs.treeBranch; + const childPrefix = isLast ? ' ' : glyphs.treePipe; if (node.name) { let line = prefix + connector + node.name; @@ -1060,9 +1127,16 @@ program .description('Start CodeGraph as an MCP server for AI assistants') .option('-p, --path ', 'Project path (optional for MCP mode, uses rootUri from client)') .option('--mcp', 'Run as MCP server (stdio transport)') - .action(async (options: { path?: string; mcp?: boolean }) => { + .option('--no-watch', 'Disable the file watcher (no auto-sync; useful on slow filesystems like WSL2 /mnt drives)') + .action(async (options: { path?: string; mcp?: boolean; watch?: boolean }) => { const projectPath = options.path ? resolveProjectPath(options.path) : undefined; + // Commander sets watch=false when --no-watch is passed. Route it through + // the same env-var chokepoint the watcher and MCP server already honor. + if (options.watch === false) { + process.env.CODEGRAPH_NO_WATCH = '1'; + } + try { if (options.mcp) { // Start MCP server - it handles initialization lazily based on rootUri from client @@ -1074,7 +1148,7 @@ program // Default: show info about MCP mode. // Use stderr so stdout stays clean for any piped/stdio usage. console.error(chalk.bold('\nCodeGraph MCP Server\n')); - console.error(chalk.blue('ℹ') + ' Use --mcp flag to start the MCP server'); + console.error(chalk.blue(getGlyphs().info) + ' Use --mcp flag to start the MCP server'); console.error('\nTo use with Claude Code, add to your MCP configuration:'); console.error(chalk.dim(` { @@ -1120,7 +1194,7 @@ program const lockPath = path.join(getCodeGraphDir(projectPath), 'codegraph.lock'); if (!fs.existsSync(lockPath)) { - info('No lock file found — nothing to do'); + info(`No lock file found ${getGlyphs().dash} nothing to do`); return; } @@ -1132,6 +1206,264 @@ program } }); +/** + * codegraph callers + * + * CLI parity with the MCP graph tools (codegraph_callers/callees/impact) so the + * traversal queries work in scripts, CI, and git hooks without a running MCP + * server. + */ +program + .command('callers ') + .description('Find all functions/methods that call a specific symbol') + .option('-p, --path ', 'Project path') + .option('-l, --limit ', 'Maximum results', '20') + .option('-j, --json', 'Output as JSON') + .action(async (symbol: string, options: { path?: string; limit?: string; json?: boolean }) => { + const projectPath = resolveProjectPath(options.path); + + try { + if (!isInitialized(projectPath)) { + error(`CodeGraph not initialized in ${projectPath}`); + process.exit(1); + } + + const { default: CodeGraph } = await loadCodeGraph(); + const cg = await CodeGraph.open(projectPath); + const limit = parseInt(options.limit || '20', 10); + + const matches = cg.searchNodes(symbol, { limit: 50 }); + if (matches.length === 0) { + info(`Symbol "${symbol}" not found`); + cg.destroy(); + return; + } + + const seen = new Set(); + const allCallers: Array<{ name: string; kind: string; filePath: string; startLine?: number }> = []; + + for (const match of matches) { + const exactMatch = match.node.name === symbol || match.node.name.endsWith(`.${symbol}`) || match.node.name.endsWith(`::${symbol}`); + if (!exactMatch && matches.length > 1) continue; + for (const c of cg.getCallers(match.node.id)) { + if (!seen.has(c.node.id)) { + seen.add(c.node.id); + allCallers.push({ name: c.node.name, kind: c.node.kind, filePath: c.node.filePath, startLine: c.node.startLine }); + } + } + } + + // Fallback: if exact filter removed everything, use the top match + if (allCallers.length === 0 && matches[0]) { + for (const c of cg.getCallers(matches[0].node.id)) { + if (!seen.has(c.node.id)) { + seen.add(c.node.id); + allCallers.push({ name: c.node.name, kind: c.node.kind, filePath: c.node.filePath, startLine: c.node.startLine }); + } + } + } + + const limited = allCallers.slice(0, limit); + + if (options.json) { + console.log(JSON.stringify({ symbol, callers: limited }, null, 2)); + } else if (limited.length === 0) { + info(`No callers found for "${symbol}"`); + } else { + console.log(chalk.bold(`\nCallers of "${symbol}" (${limited.length}):\n`)); + for (const node of limited) { + const loc = node.startLine ? `:${node.startLine}` : ''; + console.log( + chalk.cyan(node.kind.padEnd(12)) + + chalk.white(node.name) + ); + console.log(chalk.dim(` ${node.filePath}${loc}`)); + console.log(); + } + } + + cg.destroy(); + } catch (err) { + error(`callers failed: ${err instanceof Error ? err.message : String(err)}`); + process.exit(1); + } + }); + +/** + * codegraph callees + */ +program + .command('callees ') + .description('Find all functions/methods that a specific symbol calls') + .option('-p, --path ', 'Project path') + .option('-l, --limit ', 'Maximum results', '20') + .option('-j, --json', 'Output as JSON') + .action(async (symbol: string, options: { path?: string; limit?: string; json?: boolean }) => { + const projectPath = resolveProjectPath(options.path); + + try { + if (!isInitialized(projectPath)) { + error(`CodeGraph not initialized in ${projectPath}`); + process.exit(1); + } + + const { default: CodeGraph } = await loadCodeGraph(); + const cg = await CodeGraph.open(projectPath); + const limit = parseInt(options.limit || '20', 10); + + const matches = cg.searchNodes(symbol, { limit: 50 }); + if (matches.length === 0) { + info(`Symbol "${symbol}" not found`); + cg.destroy(); + return; + } + + const seen = new Set(); + const allCallees: Array<{ name: string; kind: string; filePath: string; startLine?: number }> = []; + + for (const match of matches) { + const exactMatch = match.node.name === symbol || match.node.name.endsWith(`.${symbol}`) || match.node.name.endsWith(`::${symbol}`); + if (!exactMatch && matches.length > 1) continue; + for (const c of cg.getCallees(match.node.id)) { + if (!seen.has(c.node.id)) { + seen.add(c.node.id); + allCallees.push({ name: c.node.name, kind: c.node.kind, filePath: c.node.filePath, startLine: c.node.startLine }); + } + } + } + + if (allCallees.length === 0 && matches[0]) { + for (const c of cg.getCallees(matches[0].node.id)) { + if (!seen.has(c.node.id)) { + seen.add(c.node.id); + allCallees.push({ name: c.node.name, kind: c.node.kind, filePath: c.node.filePath, startLine: c.node.startLine }); + } + } + } + + const limited = allCallees.slice(0, limit); + + if (options.json) { + console.log(JSON.stringify({ symbol, callees: limited }, null, 2)); + } else if (limited.length === 0) { + info(`No callees found for "${symbol}"`); + } else { + console.log(chalk.bold(`\nCallees of "${symbol}" (${limited.length}):\n`)); + for (const node of limited) { + const loc = node.startLine ? `:${node.startLine}` : ''; + console.log( + chalk.cyan(node.kind.padEnd(12)) + + chalk.white(node.name) + ); + console.log(chalk.dim(` ${node.filePath}${loc}`)); + console.log(); + } + } + + cg.destroy(); + } catch (err) { + error(`callees failed: ${err instanceof Error ? err.message : String(err)}`); + process.exit(1); + } + }); + +/** + * codegraph impact + */ +program + .command('impact ') + .description('Analyze what code is affected by changing a symbol') + .option('-p, --path ', 'Project path') + .option('-d, --depth ', 'Traversal depth', '2') + .option('-j, --json', 'Output as JSON') + .action(async (symbol: string, options: { path?: string; depth?: string; json?: boolean }) => { + const projectPath = resolveProjectPath(options.path); + + try { + if (!isInitialized(projectPath)) { + error(`CodeGraph not initialized in ${projectPath}`); + process.exit(1); + } + + const { default: CodeGraph } = await loadCodeGraph(); + const cg = await CodeGraph.open(projectPath); + const depth = Math.min(Math.max(parseInt(options.depth || '2', 10), 1), 10); + + const matches = cg.searchNodes(symbol, { limit: 50 }); + if (matches.length === 0) { + info(`Symbol "${symbol}" not found`); + cg.destroy(); + return; + } + + // Merge impact subgraphs across all exact-matching symbols + const mergedNodes = new Map(); + const seenEdges = new Set(); + let edgeCount = 0; + + for (const match of matches) { + const exactMatch = match.node.name === symbol || match.node.name.endsWith(`.${symbol}`) || match.node.name.endsWith(`::${symbol}`); + if (!exactMatch && matches.length > 1) continue; + const impact = cg.getImpactRadius(match.node.id, depth); + for (const [id, n] of impact.nodes) { + mergedNodes.set(id, { name: n.name, kind: n.kind, filePath: n.filePath, startLine: n.startLine }); + } + for (const e of impact.edges) { + const key = `${e.source}->${e.target}:${e.kind}`; + if (!seenEdges.has(key)) { + seenEdges.add(key); + edgeCount++; + } + } + } + + // Fallback to top match if exact filter removed everything + if (mergedNodes.size === 0 && matches[0]) { + const impact = cg.getImpactRadius(matches[0].node.id, depth); + for (const [id, n] of impact.nodes) { + mergedNodes.set(id, { name: n.name, kind: n.kind, filePath: n.filePath, startLine: n.startLine }); + } + edgeCount = impact.edges.length; + } + + if (options.json) { + console.log(JSON.stringify({ + symbol, + depth, + nodeCount: mergedNodes.size, + edgeCount, + affected: Array.from(mergedNodes.values()), + }, null, 2)); + } else if (mergedNodes.size === 0) { + info(`No affected symbols found for "${symbol}"`); + } else { + console.log(chalk.bold(`\nImpact of changing "${symbol}" — ${mergedNodes.size} affected symbols:\n`)); + + // Group by file + const byFile = new Map>(); + for (const node of mergedNodes.values()) { + const list = byFile.get(node.filePath) || []; + list.push({ name: node.name, kind: node.kind, startLine: node.startLine }); + byFile.set(node.filePath, list); + } + + for (const [file, nodes] of byFile) { + console.log(chalk.cyan(file)); + for (const node of nodes) { + const loc = node.startLine ? `:${node.startLine}` : ''; + console.log(` ${chalk.dim(node.kind.padEnd(12))}${node.name}${chalk.dim(loc)}`); + } + console.log(); + } + } + + cg.destroy(); + } catch (err) { + error(`impact failed: ${err instanceof Error ? err.message : String(err)}`); + process.exit(1); + } + }); + /** * codegraph affected [files...] * @@ -1275,10 +1607,97 @@ program */ program .command('install') - .description('Run interactive installer for Claude Code integration') - .action(async () => { - const { runInstaller } = await import('../installer'); - await runInstaller(); + .description('Install codegraph MCP server into one or more agents (Claude Code, Cursor, Codex CLI, opencode, Hermes Agent)') + .option('-t, --target ', 'Target agent(s): comma-separated ids, or "auto"|"all"|"none". Default: prompt') + .option('-l, --location ', 'Install location: "global" or "local". Default: prompt') + .option('-y, --yes', 'Non-interactive: defaults to --location=global --target=auto, auto-allow on') + .option('--no-permissions', 'Skip writing the auto-allow permissions list (Claude Code only)') + .option('--print-config ', 'Print MCP config snippet for the named agent and exit (no file writes)') + .action(async (opts: { + target?: string; + location?: string; + yes?: boolean; + permissions?: boolean; + printConfig?: string; + }) => { + if (opts.printConfig) { + const { getTarget, listTargetIds } = await import('../installer/targets/registry'); + const target = getTarget(opts.printConfig); + if (!target) { + const known = listTargetIds().join(', '); + error(`Unknown target "${opts.printConfig}". Known: ${known}.`); + process.exit(1); + } + const loc = (opts.location === 'local' ? 'local' : 'global') as 'global' | 'local'; + process.stdout.write(target.printConfig(loc)); + return; + } + + const { runInstallerWithOptions } = await import('../installer'); + if (opts.location && opts.location !== 'global' && opts.location !== 'local') { + error(`--location must be "global" or "local" (got "${opts.location}").`); + process.exit(1); + } + try { + // Commander's `--no-permissions` makes `opts.permissions === false`; + // omitting the flag leaves it `true` (the positive-form default). + // We MUST treat the default-true as "user did not override — let + // the orchestrator prompt" and only forward an explicit `false` + // (or `true` when --yes implies it). Otherwise the auto-allow + // prompt is silently skipped on every interactive run. + const explicitNoPermissions = opts.permissions === false; + const autoAllow: boolean | undefined = explicitNoPermissions + ? false + : opts.yes + ? true + : undefined; + + await runInstallerWithOptions({ + target: opts.target, + location: opts.location as 'global' | 'local' | undefined, + autoAllow, + yes: opts.yes, + }); + } catch (err) { + error(err instanceof Error ? err.message : String(err)); + process.exit(1); + } + }); + +/** + * codegraph uninstall + * + * Inverse of `install`. Removes the codegraph MCP server entry, + * instructions block, and permissions from every agent (or a + * `--target` subset). Prompts global-vs-local when not given. Does NOT + * delete the `.codegraph/` index — that's `codegraph uninit`. + */ +program + .command('uninstall') + .description('Remove codegraph from your agents (Claude Code, Cursor, Codex CLI, opencode, Hermes Agent)') + .option('-t, --target ', 'Target agent(s): comma-separated ids, or "all". Default: all') + .option('-l, --location ', 'Uninstall location: "global" or "local". Default: prompt') + .option('-y, --yes', 'Non-interactive: defaults to --location=global --target=all') + .action(async (opts: { + target?: string; + location?: string; + yes?: boolean; + }) => { + const { runUninstaller } = await import('../installer'); + if (opts.location && opts.location !== 'global' && opts.location !== 'local') { + error(`--location must be "global" or "local" (got "${opts.location}").`); + process.exit(1); + } + try { + await runUninstaller({ + target: opts.target, + location: opts.location as 'global' | 'local' | undefined, + yes: opts.yes, + }); + } catch (err) { + error(err instanceof Error ? err.message : String(err)); + process.exit(1); + } }); // Parse and run diff --git a/src/bin/node-version-check.ts b/src/bin/node-version-check.ts index 6aed1615d..cea0a4351 100644 --- a/src/bin/node-version-check.ts +++ b/src/bin/node-version-check.ts @@ -13,9 +13,12 @@ * unsupported Node.js major version (currently 25+). Pinned via unit * test so the recovery commands and override instructions can't be * silently stripped by future edits. + * + * Uses ASCII glyphs to stay readable on Windows OEM-codepage consoles + * (see ../ui/glyphs.ts for the rationale). */ export function buildNode25BlockBanner(nodeVersion: string): string { - const sep = '─'.repeat(72); + const sep = '-'.repeat(72); return [ sep, `[CodeGraph] Unsupported Node.js version: ${nodeVersion}`, @@ -29,7 +32,44 @@ export function buildNode25BlockBanner(nodeVersion: string): string { ' nvm install 22 && nvm use 22 # nvm', ' brew install node@22 && brew link --overwrite --force node@22 # Homebrew', '', - 'To override (NOT recommended — you will likely OOM):', + 'To override (NOT recommended - you will likely OOM):', + ' CODEGRAPH_ALLOW_UNSAFE_NODE=1 codegraph ...', + sep, + ].join('\n'); +} + +/** + * Lowest supported Node.js major version. Matches the `engines` floor in + * package.json. Below this, CodeGraph relies on language features / native APIs + * that aren't present, and the combination is untested. `engines` alone only + * *warns* on install (unless the user set `engine-strict`), so the CLI bootstrap + * also hard-blocks here to actually enforce the floor. + */ +export const MIN_NODE_MAJOR = 20; + +/** + * Build the bordered banner shown when CodeGraph detects a Node.js major below + * {@link MIN_NODE_MAJOR}. Pinned via unit test so the recovery commands and the + * override env var can't be silently stripped by future edits. + * + * Uses ASCII glyphs to stay readable on Windows OEM-codepage consoles + * (see ../ui/glyphs.ts for the rationale). + */ +export function buildNodeTooOldBanner(nodeVersion: string): string { + const sep = '-'.repeat(72); + return [ + sep, + `[CodeGraph] Unsupported Node.js version: ${nodeVersion}`, + sep, + `CodeGraph requires Node.js ${MIN_NODE_MAJOR} or newer. Older versions lack`, + 'language features and native APIs CodeGraph depends on, and are not', + 'tested or supported.', + '', + 'Fix: install Node.js 22 LTS:', + ' nvm install 22 && nvm use 22 # nvm', + ' brew install node@22 && brew link --overwrite --force node@22 # Homebrew', + '', + 'To override (NOT recommended - unsupported):', ' CODEGRAPH_ALLOW_UNSAFE_NODE=1 codegraph ...', sep, ].join('\n'); diff --git a/src/bin/uninstall.ts b/src/bin/uninstall.ts index 4344a04d6..a168d8075 100644 --- a/src/bin/uninstall.ts +++ b/src/bin/uninstall.ts @@ -2,121 +2,33 @@ /** * CodeGraph preuninstall cleanup script * - * Runs automatically when `npm uninstall -g @colbymchenry/codegraph` is called. - * Removes all CodeGraph configuration from Claude Code: - * - MCP server entry from ~/.claude.json - * - Permissions from ~/.claude/settings.json - * - CodeGraph section from ~/.claude/CLAUDE.md + * Runs automatically when `npm uninstall -g @colbymchenry/codegraph` + * is called. Loops over every known agent target's `uninstall(loc)` + * for the global location only — local-location entries live inside + * project working trees and aren't ours to nuke at npm-uninstall + * time. * - * This script must never throw — a failed cleanup must not block uninstall. + * This script must never throw — a failed cleanup must not block + * uninstall. */ -import * as fs from 'fs'; -import * as path from 'path'; -import * as os from 'os'; - -const CODEGRAPH_SECTION_START = ''; -const CODEGRAPH_SECTION_END = ''; - -function readJson(filePath: string): Record | null { - try { - if (!fs.existsSync(filePath)) return null; - return JSON.parse(fs.readFileSync(filePath, 'utf-8')); - } catch { - return null; - } -} - -function writeJson(filePath: string, data: Record): void { - fs.writeFileSync(filePath, JSON.stringify(data, null, 2) + '\n'); -} - -/** - * Remove CodeGraph MCP server from ~/.claude.json - */ -function removeMcpConfig(): void { - const filePath = path.join(os.homedir(), '.claude.json'); - const config = readJson(filePath); - if (!config?.mcpServers?.codegraph) return; - - delete config.mcpServers.codegraph; - - // Clean up empty mcpServers object - if (Object.keys(config.mcpServers).length === 0) { - delete config.mcpServers; - } - - writeJson(filePath, config); -} - -/** - * Remove CodeGraph permissions from ~/.claude/settings.json - */ -function removeSettings(): void { - const filePath = path.join(os.homedir(), '.claude', 'settings.json'); - const settings = readJson(filePath); - if (!settings) return; - - // Remove codegraph permissions - if (Array.isArray(settings.permissions?.allow)) { - const before = settings.permissions.allow.length; - settings.permissions.allow = settings.permissions.allow.filter( - (p: string) => !p.startsWith('mcp__codegraph__') - ); - if (settings.permissions.allow.length === before) return; - - // Clean up empty allow array - if (settings.permissions.allow.length === 0) { - delete settings.permissions.allow; - } - // Clean up empty permissions object - if (Object.keys(settings.permissions).length === 0) { - delete settings.permissions; - } - - writeJson(filePath, settings); - } -} - -/** - * Remove CodeGraph section from ~/.claude/CLAUDE.md - */ -function removeClaudeMd(): void { - const filePath = path.join(os.homedir(), '.claude', 'CLAUDE.md'); - try { - if (!fs.existsSync(filePath)) return; - let content = fs.readFileSync(filePath, 'utf-8'); - - // Remove marked section - const startIdx = content.indexOf(CODEGRAPH_SECTION_START); - const endIdx = content.indexOf(CODEGRAPH_SECTION_END); - - if (startIdx !== -1 && endIdx > startIdx) { - const before = content.substring(0, startIdx).trimEnd(); - const after = content.substring(endIdx + CODEGRAPH_SECTION_END.length).trimStart(); - content = before + (before && after ? '\n\n' : '') + after; - - if (content.trim() === '') { - // File is empty after removing section — delete it - fs.unlinkSync(filePath); - } else { - fs.writeFileSync(filePath, content.trim() + '\n'); - } +try { + // Lazy require so any module-level error in the registry can't + // bubble out and abort the npm uninstall. + // eslint-disable-next-line @typescript-eslint/no-require-imports + const { ALL_TARGETS } = require('../installer/targets/registry') as + typeof import('../installer/targets/registry'); + + for (const target of ALL_TARGETS) { + if (!target.supportsLocation('global')) continue; + try { + target.uninstall('global'); + } catch { + // Each target is independently safe-to-skip; per-target failure + // must not stop the loop. } - } catch { - // Never fail } +} catch { + // If the registry itself can't be loaded (e.g. partial install), + // we silently skip cleanup. Uninstall still completes. } - -// Run cleanup — never throw -try { - removeMcpConfig(); -} catch { /* ignore */ } - -try { - removeSettings(); -} catch { /* ignore */ } - -try { - removeClaudeMd(); -} catch { /* ignore */ } diff --git a/src/config.ts b/src/config.ts deleted file mode 100644 index 9ab1032a6..000000000 --- a/src/config.ts +++ /dev/null @@ -1,297 +0,0 @@ -/** - * Configuration Management - * - * Load, save, and validate CodeGraph configuration. - */ - -import * as fs from 'fs'; -import * as path from 'path'; -import picomatch from 'picomatch'; -import { CodeGraphConfig, DEFAULT_CONFIG, Language, NodeKind } from './types'; -import { normalizePath } from './utils'; - -/** - * Configuration filename - */ -export const CONFIG_FILENAME = 'config.json'; - -/** - * Get the config file path for a project - */ -export function getConfigPath(projectRoot: string): string { - return path.join(projectRoot, '.codegraph', CONFIG_FILENAME); -} - -/** - * Check if a regex pattern is safe from ReDoS attacks. - * - * Rejects patterns with nested quantifiers (e.g., (a+)+, (a*)*) which - * are the primary source of catastrophic backtracking. Also rejects - * excessively long patterns and validates compilability. - */ -function isSafeRegex(pattern: string): boolean { - // Reject excessively long patterns - if (pattern.length > 500) return false; - - // Reject nested quantifiers: (...)+ followed by +, *, or { - // These are the primary cause of catastrophic backtracking - if (/([+*}])\s*[+*{]/.test(pattern)) return false; - if (/\([^)]*[+*][^)]*\)[+*{]/.test(pattern)) return false; - - // Verify the pattern is a valid regex - try { - new RegExp(pattern); - return true; - } catch { - return false; - } -} - -/** - * Validate a configuration object - */ -export function validateConfig(config: unknown): config is CodeGraphConfig { - if (typeof config !== 'object' || config === null) { - return false; - } - - const c = config as Record; - - // Required fields - if (typeof c.version !== 'number') return false; - if (typeof c.rootDir !== 'string') return false; - if (!Array.isArray(c.include)) return false; - if (!Array.isArray(c.exclude)) return false; - if (!Array.isArray(c.languages)) return false; - if (!Array.isArray(c.frameworks)) return false; - if (typeof c.maxFileSize !== 'number') return false; - if (typeof c.extractDocstrings !== 'boolean') return false; - if (typeof c.trackCallSites !== 'boolean') return false; - - // Validate include/exclude are string arrays - if (!c.include.every((p) => typeof p === 'string')) return false; - if (!c.exclude.every((p) => typeof p === 'string')) return false; - - // Validate languages - const validLanguages: Language[] = [ - 'typescript', - 'javascript', - 'python', - 'go', - 'rust', - 'java', - 'svelte', - 'unknown', - ]; - if (!c.languages.every((l) => validLanguages.includes(l as Language))) return false; - - // Validate frameworks - for (const fw of c.frameworks) { - if (typeof fw !== 'object' || fw === null) return false; - const framework = fw as Record; - if (typeof framework.name !== 'string') return false; - } - - // Validate custom patterns if present - if (c.customPatterns !== undefined) { - if (!Array.isArray(c.customPatterns)) return false; - for (const pattern of c.customPatterns) { - if (typeof pattern !== 'object' || pattern === null) return false; - const p = pattern as Record; - if (typeof p.name !== 'string') return false; - if (typeof p.pattern !== 'string') return false; - if (typeof p.kind !== 'string') return false; - - // Validate regex is compilable and reject patterns with known ReDoS risks - if (!isSafeRegex(p.pattern)) return false; - } - } - - return true; -} - -/** - * Merge configuration with defaults - */ -function mergeConfig( - defaults: CodeGraphConfig, - overrides: Partial -): CodeGraphConfig { - return { - version: overrides.version ?? defaults.version, - rootDir: overrides.rootDir ?? defaults.rootDir, - include: overrides.include ?? defaults.include, - exclude: overrides.exclude ?? defaults.exclude, - languages: overrides.languages ?? defaults.languages, - frameworks: overrides.frameworks ?? defaults.frameworks, - maxFileSize: overrides.maxFileSize ?? defaults.maxFileSize, - extractDocstrings: overrides.extractDocstrings ?? defaults.extractDocstrings, - trackCallSites: overrides.trackCallSites ?? defaults.trackCallSites, - customPatterns: overrides.customPatterns ?? defaults.customPatterns, - }; -} - -/** - * Load configuration from a project - */ -export function loadConfig(projectRoot: string): CodeGraphConfig { - const configPath = getConfigPath(projectRoot); - - if (!fs.existsSync(configPath)) { - // Return default config with adjusted rootDir - return { - ...DEFAULT_CONFIG, - rootDir: projectRoot, - }; - } - - try { - const content = fs.readFileSync(configPath, 'utf-8'); - const parsed = JSON.parse(content) as unknown; - - // Merge with defaults to ensure all fields are present - const merged = mergeConfig(DEFAULT_CONFIG, parsed as Partial); - merged.rootDir = projectRoot; // Always use actual project root - - if (!validateConfig(merged)) { - throw new Error('Invalid configuration format'); - } - - return merged; - } catch (error) { - if (error instanceof SyntaxError) { - throw new Error(`Invalid JSON in config file: ${configPath}`); - } - throw error; - } -} - -/** - * Save configuration to a project - */ -export function saveConfig(projectRoot: string, config: CodeGraphConfig): void { - const configPath = getConfigPath(projectRoot); - const dir = path.dirname(configPath); - - // Ensure directory exists - if (!fs.existsSync(dir)) { - fs.mkdirSync(dir, { recursive: true }); - } - - // Create a copy without rootDir (it's always derived from project path) - const toSave = { ...config }; - delete (toSave as Partial).rootDir; - - const content = JSON.stringify(toSave, null, 2); - - // Atomic write: write to temp file then rename to prevent partial/corrupt configs - const tmpPath = configPath + '.tmp'; - fs.writeFileSync(tmpPath, content, 'utf-8'); - fs.renameSync(tmpPath, configPath); -} - -/** - * Create default configuration for a new project - */ -export function createDefaultConfig(projectRoot: string): CodeGraphConfig { - return { - ...DEFAULT_CONFIG, - rootDir: projectRoot, - }; -} - -/** - * Update specific configuration values - */ -export function updateConfig( - projectRoot: string, - updates: Partial -): CodeGraphConfig { - const current = loadConfig(projectRoot); - const updated = mergeConfig(current, updates); - updated.rootDir = projectRoot; - saveConfig(projectRoot, updated); - return updated; -} - -/** - * Add patterns to include list - */ -export function addIncludePatterns(projectRoot: string, patterns: string[]): CodeGraphConfig { - const config = loadConfig(projectRoot); - const newPatterns = patterns.filter((p) => !config.include.includes(p)); - config.include = [...config.include, ...newPatterns]; - saveConfig(projectRoot, config); - return config; -} - -/** - * Add patterns to exclude list - */ -export function addExcludePatterns(projectRoot: string, patterns: string[]): CodeGraphConfig { - const config = loadConfig(projectRoot); - const newPatterns = patterns.filter((p) => !config.exclude.includes(p)); - config.exclude = [...config.exclude, ...newPatterns]; - saveConfig(projectRoot, config); - return config; -} - -/** - * Add a custom pattern - */ -export function addCustomPattern( - projectRoot: string, - name: string, - pattern: string, - kind: NodeKind -): CodeGraphConfig { - const config = loadConfig(projectRoot); - - if (!config.customPatterns) { - config.customPatterns = []; - } - - // Check for duplicate name - const existing = config.customPatterns.find((p) => p.name === name); - if (existing) { - existing.pattern = pattern; - existing.kind = kind; - } else { - config.customPatterns.push({ name, pattern, kind }); - } - - saveConfig(projectRoot, config); - return config; -} - -/** - * Check if a file path matches the include/exclude patterns - */ -export function shouldIncludeFile(filePath: string, config: CodeGraphConfig): boolean { - // Normalize to forward slashes so Windows backslash paths match glob patterns - filePath = normalizePath(filePath); - - // Simple glob matching (for now, just check if any pattern matches) - // A full implementation would use a proper glob library - - const matchesPattern = (pattern: string, filePath: string): boolean => { - return picomatch.isMatch(filePath, pattern, { dot: true }); - }; - - // Check exclude patterns first - for (const pattern of config.exclude) { - if (matchesPattern(pattern, filePath)) { - return false; - } - } - - // Check include patterns - for (const pattern of config.include) { - if (matchesPattern(pattern, filePath)) { - return true; - } - } - - // Default to not including if no pattern matches - return false; -} diff --git a/src/context/formatter.ts b/src/context/formatter.ts index 37a08ee84..748d17201 100644 --- a/src/context/formatter.ts +++ b/src/context/formatter.ts @@ -5,6 +5,7 @@ */ import { Node, Edge, TaskContext, Subgraph } from '../types'; +import { isGeneratedFile } from '../extraction/generated-detection'; /** * Format context as markdown @@ -21,10 +22,17 @@ export function formatContextAsMarkdown(context: TaskContext): string { lines.push('## Code Context\n'); lines.push(`**Query:** ${context.query}\n`); - // Entry points - compact format - if (context.entryPoints.length > 0) { + // Entry points - compact format. Re-sort so generated files (.pb.go, + // .pulsar.go, mocks, …) rank LAST — a flow query should lead with the + // hand-written implementation, not protobuf scaffolding. + const orderedEntries = [...context.entryPoints].sort((a, b) => { + const aGen = isGeneratedFile(a.filePath) ? 1 : 0; + const bGen = isGeneratedFile(b.filePath) ? 1 : 0; + return aGen - bGen; + }); + if (orderedEntries.length > 0) { lines.push('### Entry Points\n'); - for (const node of context.entryPoints) { + for (const node of orderedEntries) { const location = node.startLine ? `:${node.startLine}` : ''; lines.push(`- **${node.name}** (${node.kind}) - ${node.filePath}${location}`); if (node.signature) { @@ -34,9 +42,14 @@ export function formatContextAsMarkdown(context: TaskContext): string { lines.push(''); } - // Related symbols - compact list (skip verbose structure tree) + // Related symbols - compact list (skip verbose structure tree). Drop nodes + // in generated source files (`.pb.go` / `.pulsar.go` / mocks / …) — agents + // chasing a flow never want to land on protobuf scaffolding (cosmos-Q3 used + // to list `gov.pulsar.go::GetExpeditedThreshold` and `1.pulsar.go::Get` in + // Related Symbols, pure noise that displaced real-flow entries). const otherSymbols = Array.from(context.subgraph.nodes.values()) .filter(n => !context.entryPoints.some(e => e.id === n.id)) + .filter(n => !isGeneratedFile(n.filePath)) .slice(0, 10); // Limit to 10 related symbols if (otherSymbols.length > 0) { @@ -55,10 +68,16 @@ export function formatContextAsMarkdown(context: TaskContext): string { lines.push(''); } - // Code blocks - only for key entry points + // Code blocks - only for key entry points. Re-sort so non-generated blocks + // show first (consistent with Entry Points reordering above). if (context.codeBlocks.length > 0) { + const orderedBlocks = [...context.codeBlocks].sort((a, b) => { + const aGen = isGeneratedFile(a.filePath) ? 1 : 0; + const bGen = isGeneratedFile(b.filePath) ? 1 : 0; + return aGen - bGen; + }); lines.push('### Code\n'); - for (const block of context.codeBlocks) { + for (const block of orderedBlocks) { const nodeName = block.node?.name ?? 'Unknown'; lines.push(`#### ${nodeName} (${block.filePath}:${block.startLine})\n`); lines.push('```' + block.language); diff --git a/src/context/index.ts b/src/context/index.ts index 941923776..7e6619e8b 100644 --- a/src/context/index.ts +++ b/src/context/index.ts @@ -259,7 +259,7 @@ export class ContextBuilder { // Return formatted output or raw context if (opts.format === 'markdown') { - return formatContextAsMarkdown(context); + return formatContextAsMarkdown(context) + this.buildCallPathsSection(subgraph); } else if (opts.format === 'json') { return formatContextAsJson(context); } @@ -267,6 +267,116 @@ export class ContextBuilder { return context; } + /** + * Surface short call-paths among the symbols this context already found, + * derived in-memory from the subgraph's `calls` edges (no extra queries). + * + * This bakes the value of path-finding INTO the always-loaded `context` tool. + * Agents reliably read context's output but do NOT discover/adopt a standalone + * trace tool (in deferred-MCP harnesses they only ToolSearch-select tools they + * already know). Delivering the flow here means "how does X reach Y" is + * answered without the agent needing to find, load, or choose a new tool. + * Chains stop where the static call graph ends (e.g. dynamic dispatch) — that + * truncation is honest, and the agent can codegraph_node the last hop to bridge. + */ + private buildCallPathsSection(subgraph: Subgraph): string { + const adj = new Map(); + for (const e of subgraph.edges) { + if (e.kind !== 'calls') continue; + if (!subgraph.nodes.has(e.source) || !subgraph.nodes.has(e.target)) continue; + const list = adj.get(e.source); + if (list) list.push(e.target); + else adj.set(e.source, [e.target]); + } + if (adj.size === 0) return ''; + + const MAX_HOPS = 6; + const chains: string[][] = []; + let budget = 2000; // bound DFS work on dense subgraphs + const dfs = (id: string, path: string[], seen: Set): void => { + if (budget-- <= 0) return; + const next = (adj.get(id) ?? []).filter((t) => !seen.has(t)); + if (next.length === 0 || path.length >= MAX_HOPS) { + if (path.length >= 3) chains.push([...path]); // >=3 nodes = a real flow, not a single call + return; + } + for (const t of next) { + seen.add(t); + dfs(t, [...path, t], seen); + seen.delete(t); + } + }; + const starts = (subgraph.roots.length > 0 + ? subgraph.roots.filter((id) => adj.has(id)) + : [...adj.keys()] + ).slice(0, 5); + for (const s of starts) dfs(s, [s], new Set([s])); + if (chains.length === 0) return ''; + + // Keep only chains that connect TWO OR MORE query-relevant symbols (roots). + // A chain from a root into an arbitrary callee (render → onMagicFrameGenerate) + // is structurally valid but tangential to the question; requiring ≥2 roots + // keeps the chain anchored to what the user actually asked about. Rank by + // #roots then length, and drop any that are a sub-path of a longer kept chain. + const rootSet = new Set(subgraph.roots); + const rootCount = (c: string[]): number => c.reduce((n, id) => n + (rootSet.has(id) ? 1 : 0), 0); + const relevant = chains.filter((c) => rootCount(c) >= 2); + relevant.sort((a, b) => rootCount(b) - rootCount(a) || b.length - a.length); + const kept: string[][] = []; + for (const c of relevant) { + const key = c.join('>'); + if (kept.some((k) => k.join('>').includes(key))) continue; + kept.push(c); + if (kept.length >= 3) break; + } + if (kept.length === 0) return ''; + const name = (id: string): string => subgraph.nodes.get(id)?.name ?? id; + + // Synthesized (dynamic-dispatch) hops are real `calls` edges but invisible to + // static parsing — mark them inline so the agent sees WHERE the callback was + // wired up (`registered @file:line`) instead of grepping for it. Keyed by + // "source>target". + const synthByPair = new Map(); + for (const e of subgraph.edges) { + if (e.kind !== 'calls' || e.provenance !== 'heuristic') continue; + const m = e.metadata as Record | undefined; + if (!m?.synthesizedBy) continue; + const at = typeof m.registeredAt === 'string' ? ` @${m.registeredAt}` : ''; + const label = m.synthesizedBy === 'callback' + ? `callback via ${m.via ? `\`${String(m.via)}\`` : 'registrar'}${at}` + : m.synthesizedBy === 'react-render' + ? `React re-render via setState${at}` + : m.synthesizedBy === 'jsx-render' + ? `renders <${String(m.via || 'child')}>` + : m.synthesizedBy === 'vue-handler' + ? `Vue @${String(m.event || 'event')} handler` + : `event ${m.event ? `\`${String(m.event)}\`` : ''}${at}`; + synthByPair.set(`${e.source}>${e.target}`, label); + } + const renderChain = (c: string[]): string => { + let s = name(c[0]!); + for (let i = 1; i < c.length; i++) { + const synth = synthByPair.get(`${c[i - 1]}>${c[i]}`); + s += synth ? ` →[${synth}] ${name(c[i]!)}` : ` → ${name(c[i]!)}`; + } + return s; + }; + const hasSynth = kept.some((c) => c.some((_, i) => i > 0 && synthByPair.has(`${c[i - 1]}>${c[i]}`))); + const lines = [ + '', + '## Call paths', + '', + 'Execution flow among the key symbols (traced through the call graph):', + '', + ...kept.map((c) => `- ${renderChain(c)}`), + '', + hasSynth + ? '_Hops marked `[callback/event …]` are dynamic dispatch bridged by codegraph (with the registration site); the rest are direct calls. codegraph_node any symbol for its body._' + : '_codegraph_node any symbol above for its source + its own callers/callees._', + ]; + return '\n' + lines.join('\n') + '\n'; + } + /** * Find relevant subgraph for a query * @@ -477,6 +587,37 @@ export class ContextBuilder { } } + // Iter7 — Core-directory boost. On projects with one file that holds + // the dense majority of internal call edges (e.g. sinatra's + // `lib/sinatra/base.rb` at 85% of all in-file edges), the agent's + // task usually asks about the framework's core. Without this boost, + // ranking favors small focused extension files (e.g. text search + // picks `sinatra-contrib/lib/sinatra/multi_route.rb`'s 10-line + // `route` method over `base.rb`'s `route!` because the extension + // file's `route` matches the query verbatim AND the file is small, + // dwarfing the longer name `route!` in a 1500-line file). Boost + // results that share a directory prefix with the dominant file's + // directory so the core file's siblings outrank sibling-package + // extensions. + try { + const dominant = this.queries.getDominantFile?.(); + if (dominant && dominant.edgeCount >= 3 * dominant.nextEdgeCount) { + // Take the directory of the dominant file (everything up to the + // last slash). For `lib/sinatra/base.rb` → `lib/sinatra/`. + const slash = dominant.filePath.lastIndexOf('/'); + if (slash > 0) { + const coreDir = dominant.filePath.slice(0, slash + 1); + for (const result of searchResults) { + if (result.node.filePath.startsWith(coreDir)) { + result.score += 25; + } + } + } + } + } catch { + // SQL failure — fall through, scoring works without the boost + } + // Step 5a: Multi-term co-occurrence re-ranking (applied BEFORE truncation). // For multi-word queries like "search execution from request to shard", // nodes matching 2+ query terms in their name or path are far more relevant @@ -1006,9 +1147,11 @@ export class ContextBuilder { const code = await this.extractNodeCode(node); if (code) { - // Truncate if too long + // Truncate if too long. Language-neutral marker (no `//` — not a + // comment in Python, Ruby, etc.); this renders inside a fenced + // source block whose language varies. const truncated = code.length > maxBlockSize - ? code.slice(0, maxBlockSize) + '\n// ... truncated ...' + ? code.slice(0, maxBlockSize) + '\n... (truncated) ...' : code; blocks.push({ diff --git a/src/db/index.ts b/src/db/index.ts index 3e490746f..cbc08b8f0 100644 --- a/src/db/index.ts +++ b/src/db/index.ts @@ -10,7 +10,31 @@ import * as path from 'path'; import { SchemaVersion } from '../types'; import { runMigrations, getCurrentVersion, CURRENT_SCHEMA_VERSION } from './migrations'; -export { SqliteDatabase, SqliteBackend, WASM_FALLBACK_FIX_RECIPE } from './sqlite-adapter'; +export { SqliteDatabase, SqliteBackend } from './sqlite-adapter'; + +/** + * Apply connection-level PRAGMAs. Shared by `initialize` and `open` so the two + * paths can't drift. + * + * `busy_timeout` is set FIRST, before any pragma that might touch the database + * file (notably `journal_mode`). If another process holds a write lock at open + * time, the later pragmas — and the connection's first query — then wait out + * the lock instead of throwing "database is locked" immediately. See issue #238. + * + * The 5s window (was 120s) rides out a normal incremental sync; the old + * 2-minute wait presented as a frozen, hung agent. With WAL, reads never block + * on a writer, so this timeout only governs cross-process write contention + * (e.g. the git-hook `codegraph sync` running while the MCP server writes). + */ +function configureConnection(db: SqliteDatabase): void { + db.pragma('busy_timeout = 5000'); // MUST be first — see above + db.pragma('foreign_keys = ON'); + db.pragma('journal_mode = WAL'); // node:sqlite supports WAL on every platform + db.pragma('synchronous = NORMAL'); // safe with WAL mode + db.pragma('cache_size = -64000'); // 64 MB page cache + db.pragma('temp_store = MEMORY'); // temp tables in memory + db.pragma('mmap_size = 268435456'); // 256 MB memory-mapped I/O +} /** * Database connection wrapper with lifecycle management @@ -39,17 +63,7 @@ export class DatabaseConnection { // Create and configure database const { db, backend } = createDatabase(dbPath); - // Enable foreign keys and WAL mode for better performance - db.pragma('foreign_keys = ON'); - db.pragma('journal_mode = WAL'); - // Wait up to 2 minutes if database is locked by another process - // (indexing operations can hold locks for extended periods) - db.pragma('busy_timeout = 120000'); - // Performance tuning - db.pragma('synchronous = NORMAL'); // Safe with WAL mode - db.pragma('cache_size = -64000'); // 64 MB page cache - db.pragma('temp_store = MEMORY'); // Temp tables in memory - db.pragma('mmap_size = 268435456'); // 256 MB memory-mapped I/O + configureConnection(db); // Run schema initialization const schemaPath = path.join(__dirname, 'schema.sql'); @@ -77,17 +91,7 @@ export class DatabaseConnection { const { db, backend } = createDatabase(dbPath); - // Enable foreign keys and WAL mode - db.pragma('foreign_keys = ON'); - db.pragma('journal_mode = WAL'); - // Wait up to 2 minutes if database is locked by another process - // (indexing operations can hold locks for extended periods) - db.pragma('busy_timeout = 120000'); - // Performance tuning - db.pragma('synchronous = NORMAL'); - db.pragma('cache_size = -64000'); - db.pragma('temp_store = MEMORY'); - db.pragma('mmap_size = 268435456'); + configureConnection(db); // Check and run migrations if needed const conn = new DatabaseConnection(db, dbPath, backend); @@ -123,6 +127,25 @@ export class DatabaseConnection { return this.dbPath; } + /** + * The journal mode actually in effect (e.g. 'wal', 'delete'). + * + * SQLite silently keeps the prior mode if WAL can't be enabled — e.g. on + * filesystems without shared-memory support (some network/virtualized mounts, + * WSL2 /mnt), and always on the wasm backend. So the effective mode can differ + * from what `configureConnection` requested. Surfaced in `codegraph status` so + * a "database is locked" report is triageable: 'wal' ⇒ readers never block on a + * writer; anything else ⇒ they can. See issue #238. + */ + getJournalMode(): string { + const raw = this.db.pragma('journal_mode'); + const row = Array.isArray(raw) ? raw[0] : raw; + const mode = row && typeof row === 'object' + ? (row as Record).journal_mode + : row; + return String(mode ?? '').toLowerCase(); + } + /** * Get current schema version */ @@ -163,6 +186,36 @@ export class DatabaseConnection { this.db.exec('ANALYZE'); } + /** + * Lightweight, non-blocking maintenance to run after bulk writes + * (indexAll, sync). Two operations: + * + * - `PRAGMA optimize` — incremental ANALYZE; SQLite only re-analyzes + * tables whose row counts changed materially since the last + * ANALYZE. Without it, the query planner has no statistics on the + * freshly-bulk-loaded tables and can pick suboptimal indexes. + * + * - `PRAGMA wal_checkpoint(PASSIVE)` — fold pending WAL pages back + * into the main database file so the WAL file doesn't grow + * unboundedly between automatic checkpoints (auto-fires at 1000 + * pages by default; large indexAll runs blow past that). + * + * Both operations are silently swallowed on failure — they're a + * best-effort optimization, never load-bearing for correctness. + */ + runMaintenance(): void { + try { + this.db.exec('PRAGMA optimize'); + } catch { + // ignore + } + try { + this.db.exec('PRAGMA wal_checkpoint(PASSIVE)'); + } catch { + // ignore (e.g., not in WAL mode) + } + } + /** * Close the database connection */ diff --git a/src/db/queries.ts b/src/db/queries.ts index db7c6118d..a0ac31eea 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -20,6 +20,34 @@ import { import { safeJsonParse } from '../utils'; import { kindBonus, nameMatchBonus, scorePathRelevance } from '../search/query-utils'; import { parseQuery, boundedEditDistance } from '../search/query-parser'; +import { isGeneratedFile } from '../extraction/generated-detection'; + +/** + * Path-only heuristic for files that should not be candidates for + * "dominant file" detection: test/spec files and tool-generated files. + * Generated files (`*.pb.go`, `*.pulsar.go`, mock outputs, …) often + * have huge in-file edge counts that dwarf the real source — etcd's + * `rpc.pb.go` has 4× the in-file edges of `server.go`. + */ +function isLowValueFile(filePath: string): boolean { + const lp = filePath.toLowerCase(); + return ( + /(?:^|\/)(tests?|__tests?__|spec)\//.test(lp) || + /_test\.go$/.test(lp) || + /(?:^|\/)test_[^/]+\.py$/.test(lp) || + /_test\.py$/.test(lp) || + /_spec\.rb$/.test(lp) || + /_test\.rb$/.test(lp) || + /\.(test|spec)\.[jt]sx?$/.test(lp) || + /(test|spec|tests)\.(java|kt|scala)$/.test(lp) || + /(tests?|spec)\.cs$/.test(lp) || + /tests?\.swift$/.test(lp) || + /_test\.dart$/.test(lp) || + isGeneratedFile(filePath) + ); +} + +const SQLITE_PARAM_CHUNK_SIZE = 500; /** * Database row types (snake_case from SQLite) @@ -180,6 +208,9 @@ export class QueryBuilder { getUnresolvedBatch?: SqliteStatement; getAllFilePaths?: SqliteStatement; getAllNodeNames?: SqliteStatement; + getDominantFile?: SqliteStatement; + getTopRouteFile?: SqliteStatement; + getRoutingManifest?: SqliteStatement; } = {}; constructor(db: SqliteDatabase) { @@ -224,32 +255,34 @@ export class QueryBuilder { return; } - try { - this.stmts.insertNode.run({ - id: node.id, - kind: node.kind, - name: node.name, - qualifiedName: node.qualifiedName ?? node.name, - filePath: node.filePath, - language: node.language, - startLine: node.startLine ?? 0, - endLine: node.endLine ?? 0, - startColumn: node.startColumn ?? 0, - endColumn: node.endColumn ?? 0, - docstring: node.docstring ?? null, - signature: node.signature ?? null, - visibility: node.visibility ?? null, - isExported: node.isExported ? 1 : 0, - isAsync: node.isAsync ? 1 : 0, - isStatic: node.isStatic ? 1 : 0, - isAbstract: node.isAbstract ? 1 : 0, - decorators: node.decorators ? JSON.stringify(node.decorators) : null, - typeParameters: node.typeParameters ? JSON.stringify(node.typeParameters) : null, - updatedAt: node.updatedAt ?? Date.now(), - }); - } catch (error) { - throw error; - } + // INSERT OR REPLACE may overwrite a node we have cached. Drop the + // stale entry so the next getNodeById sees the new row, not the old + // one (matches the cache-invalidation pattern used by updateNode and + // deleteNode below). + this.nodeCache.delete(node.id); + + this.stmts.insertNode.run({ + id: node.id, + kind: node.kind, + name: node.name, + qualifiedName: node.qualifiedName ?? node.name, + filePath: node.filePath, + language: node.language, + startLine: node.startLine ?? 0, + endLine: node.endLine ?? 0, + startColumn: node.startColumn ?? 0, + endColumn: node.endColumn ?? 0, + docstring: node.docstring ?? null, + signature: node.signature ?? null, + visibility: node.visibility ?? null, + isExported: node.isExported ? 1 : 0, + isAsync: node.isAsync ? 1 : 0, + isStatic: node.isStatic ? 1 : 0, + isAbstract: node.isAbstract ? 1 : 0, + decorators: node.decorators ? JSON.stringify(node.decorators) : null, + typeParameters: node.typeParameters ? JSON.stringify(node.typeParameters) : null, + updatedAt: node.updatedAt ?? Date.now(), + }); } /** @@ -380,6 +413,77 @@ export class QueryBuilder { return node; } + /** + * Batch lookup: fetch many nodes by ID in a single SQL round-trip. + * + * Replaces the N+1 pattern in graph traversal where every edge would + * trigger its own `getNodeById` call. For a function with 50 callers + * this collapses 50 point reads into one IN-list query (~10-50x + * faster end-to-end). + * + * Returns a Map keyed by id so callers can preserve their own ordering + * (typically the order edges were returned from the graph). Missing IDs + * are simply absent from the map. + * + * Cache-aware: ids already in the LRU cache are served from memory and + * the SQL query only touches the misses. + */ + getNodesByIds(ids: readonly string[]): Map { + const out = new Map(); + if (ids.length === 0) return out; + + // Serve cache hits first; build the miss list for SQL. + const misses: string[] = []; + for (const id of ids) { + const cached = this.nodeCache.get(id); + if (cached !== undefined) { + // LRU touch + this.nodeCache.delete(id); + this.nodeCache.set(id, cached); + out.set(id, cached); + } else { + misses.push(id); + } + } + if (misses.length === 0) return out; + + // Chunk under SQLite's parameter limit (default 999, raised to 32766 + // in better-sqlite3 builds — chunk at 500 for safety across both + // backends and to keep the query plan simple). + for (let i = 0; i < misses.length; i += SQLITE_PARAM_CHUNK_SIZE) { + const chunk = misses.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); + const placeholders = chunk.map(() => '?').join(','); + const rows = this.db + .prepare(`SELECT * FROM nodes WHERE id IN (${placeholders})`) + .all(...chunk) as NodeRow[]; + for (const row of rows) { + const node = rowToNode(row); + out.set(node.id, node); + this.cacheNode(node); + } + } + return out; + } + + private getExistingNodeIds(ids: readonly string[]): Set { + const out = new Set(); + if (ids.length === 0) return out; + + const uniqueIds = [...new Set(ids)]; + for (let i = 0; i < uniqueIds.length; i += SQLITE_PARAM_CHUNK_SIZE) { + const chunk = uniqueIds.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); + const placeholders = chunk.map(() => '?').join(','); + const rows = this.db + .prepare(`SELECT id FROM nodes WHERE id IN (${placeholders})`) + .all(...chunk) as { id: string }[]; + for (const row of rows) { + out.add(row.id); + } + } + + return out; + } + /** * Add a node to the cache, evicting oldest if needed */ @@ -414,6 +518,158 @@ export class QueryBuilder { return rows.map(rowToNode); } + /** + * Find the file that holds the densest concentration of the project's + * internal call graph — the "core" file. Used by context-builder to + * boost ranking of symbols in that file's directory (so e.g. sinatra + * queries surface `lib/sinatra/base.rb`'s `route!` instead of + * `sinatra-contrib/lib/sinatra/multi_route.rb`'s `route` extension). + * + * Returns null if no file has a meaningful concentration (e.g. spread + * evenly across many files, or empty index). + * + * "Internal" = source and target are in the same file. Cross-file + * edges aren't useful here — they don't tell us which file is the + * functional center. + * + * Excludes test/spec files from candidacy via path-pattern. The agent's + * typical question is "how does X work", not "how is X tested", so + * boosting a test file's directory would be a misfire. + */ + getDominantFile(): { filePath: string; edgeCount: number; nextEdgeCount: number } | null { + if (!this.stmts.getDominantFile) { + // Pull top 20 candidates; we then filter out test/generated files + // in code (regex-grade matching that SQL LIKE can't express). The + // generated-file filter is critical — without it, etcd's + // `api/etcdserverpb/rpc.pb.go` (1916 in-file edges, generated + // protobuf stub) outranks the real `server/etcdserver/server.go` + // (470 edges) by 4×, and the boost would push the agent toward + // generated code. + this.stmts.getDominantFile = this.db.prepare(` + SELECT n.file_path AS file_path, COUNT(*) AS edge_count + FROM edges e + JOIN nodes n ON e.source = n.id + JOIN nodes m ON e.target = m.id + WHERE n.file_path = m.file_path + GROUP BY n.file_path + ORDER BY edge_count DESC + LIMIT 20 + `); + } + const rows = this.stmts.getDominantFile.all() as Array<{ file_path: string; edge_count: number }>; + const filtered = rows.filter(r => !isLowValueFile(r.file_path)); + if (filtered.length === 0 || filtered[0]!.edge_count < 20) return null; + return { + filePath: filtered[0]!.file_path, + edgeCount: filtered[0]!.edge_count, + nextEdgeCount: filtered[1]?.edge_count ?? 0, + }; + } + + /** + * Find the file that holds the densest concentration of the project's + * `route` nodes (framework-emitted: Express/Gin/Flask/Rails/Drupal/etc.). + * Used by handleContext on small repos to inline the project's routing + * config when the agent's query is about request flow — eliminating the + * "Glob + Read routes.rb" pattern that beats codegraph on tiny realworld + * template repos. + * + * Excludes test/generated files from candidacy. Returns null if there + * are fewer than 3 non-test routes total, or if no file holds at least + * 30% of them (diffuse routing → no single answer file). + */ + getTopRouteFile(): { filePath: string; routeCount: number; totalRoutes: number } | null { + if (!this.stmts.getTopRouteFile) { + this.stmts.getTopRouteFile = this.db.prepare(` + SELECT file_path, COUNT(*) AS cnt + FROM nodes + WHERE kind = 'route' + GROUP BY file_path + ORDER BY cnt DESC + LIMIT 20 + `); + } + const rows = this.stmts.getTopRouteFile.all() as Array<{ file_path: string; cnt: number }>; + const filtered = rows.filter(r => !isLowValueFile(r.file_path)); + if (filtered.length === 0) return null; + const totalRoutes = filtered.reduce((sum, r) => sum + r.cnt, 0); + const top = filtered[0]!; + if (totalRoutes < 3 || top.cnt < 3) return null; + if (top.cnt / totalRoutes < 0.30) return null; + return { filePath: top.file_path, routeCount: top.cnt, totalRoutes }; + } + + /** + * Build a URL → handler manifest from the index. Each route node's + * `references` edge points at the function/method that handles the + * request. We join them in one pass; the agent gets the canonical + * routing answer ("POST /users/login → AuthController#login") without + * having to parse the framework's route DSL itself. + * + * Also returns the file with the most handler endpoints — used as the + * "top handler file" to inline source for, so the agent has both the + * mapping AND the handler implementations. + */ + getRoutingManifest(limit: number = 40): { + entries: Array<{ url: string; handler: string; handlerFile: string; handlerLine: number; handlerKind: string }>; + topHandlerFile: string | null; + topHandlerFileCount: number; + totalRoutes: number; + } | null { + if (!this.stmts.getRoutingManifest) { + // Edge kind varies across framework resolvers: Spring/Rails/ + // Laravel/Drupal emit `references`, Express emits `calls`. Accept + // both — the semantic is the same (route → its handler). + this.stmts.getRoutingManifest = this.db.prepare(` + SELECT + r.name AS url, + h.name AS handler, + h.file_path AS handler_file, + h.start_line AS handler_line, + h.kind AS handler_kind + FROM nodes r + JOIN edges e ON e.source = r.id + JOIN nodes h ON e.target = h.id + WHERE r.kind = 'route' + AND e.kind IN ('references', 'calls') + AND h.kind IN ('function', 'method', 'class') + ORDER BY r.file_path, r.start_line + LIMIT ? + `); + } + const rows = this.stmts.getRoutingManifest.all(limit) as Array<{ + url: string; handler: string; handler_file: string; handler_line: number; handler_kind: string; + }>; + // Drop test/generated handlers — same hygiene as elsewhere. + const filtered = rows.filter(r => !isLowValueFile(r.handler_file)); + if (filtered.length < 3) return null; + // Identify the file holding the most handlers (the "primary handler file"). + const fileCounts = new Map(); + for (const r of filtered) { + fileCounts.set(r.handler_file, (fileCounts.get(r.handler_file) ?? 0) + 1); + } + let topHandlerFile: string | null = null; + let topHandlerFileCount = 0; + for (const [file, count] of fileCounts) { + if (count > topHandlerFileCount) { + topHandlerFile = file; + topHandlerFileCount = count; + } + } + return { + entries: filtered.map(r => ({ + url: r.url, + handler: r.handler, + handlerFile: r.handler_file, + handlerLine: r.handler_line, + handlerKind: r.handler_kind, + })), + topHandlerFile, + topHandlerFileCount, + totalRoutes: filtered.length, + }; + } + /** * Get all nodes of a specific kind */ @@ -696,8 +952,14 @@ export class QueryBuilder { const { kinds, languages, limit = 100, offset = 0 } = options; // Add prefix wildcard for better matching (e.g., "auth" matches "AuthService", "authenticate") - // Escape special FTS5 characters and add prefix wildcard + // Escape special FTS5 characters and add prefix wildcard. + // + // `::` is a qualifier separator in Rust/C++/Ruby, not a token char, + // so treat it as whitespace before the strip step. Otherwise queries + // like `stage_apply::run` collapse to `stage_applyrun` (the colons + // are stripped without splitting) and find nothing. See #173. const ftsQuery = query + .replace(/::/g, ' ') // Rust/C++/Ruby qualifier separator .replace(/['"*():^]/g, '') // Remove FTS5 special chars .split(/\s+/) .filter(term => term.length > 0) @@ -978,8 +1240,20 @@ export class QueryBuilder { * Insert multiple edges in a transaction */ insertEdges(edges: Edge[]): void { + if (edges.length === 0) return; + this.db.transaction(() => { + const endpointIds = new Set(); + for (const edge of edges) { + endpointIds.add(edge.source); + endpointIds.add(edge.target); + } + const existingNodeIds = this.getExistingNodeIds([...endpointIds]); + for (const edge of edges) { + if (!existingNodeIds.has(edge.source) || !existingNodeIds.has(edge.target)) { + continue; + } this.insertEdge(edge); } })(); @@ -1352,6 +1626,19 @@ export class QueryBuilder { // Statistics // =========================================================================== + /** + * Lightweight (nodes, edges) count snapshot. Used around an index/sync + * run to compute true additions across extraction + resolution + + * synthesis — the per-phase counter in the orchestrator only sees + * extraction's contribution, which is why the CLI summary under-reported + * the edge count (resolution + synthesizer edges were invisible). + */ + getNodeAndEdgeCount(): { nodes: number; edges: number } { + return this.db + .prepare('SELECT (SELECT COUNT(*) FROM nodes) AS nodes, (SELECT COUNT(*) FROM edges) AS edges') + .get() as { nodes: number; edges: number }; + } + /** * Get graph statistics */ diff --git a/src/db/sqlite-adapter.ts b/src/db/sqlite-adapter.ts index c3d31c8f6..37f0c7902 100644 --- a/src/db/sqlite-adapter.ts +++ b/src/db/sqlite-adapter.ts @@ -1,8 +1,13 @@ /** * SQLite Adapter * - * Provides a unified interface over better-sqlite3 (native) and - * node-sqlite3-wasm (WASM fallback) for universal cross-platform support. + * Thin wrapper over Node's built-in `node:sqlite` (`DatabaseSync`), exposed + * through a small better-sqlite3-shaped interface so the rest of the codebase + * is storage-agnostic. + * + * CodeGraph ships with a bundled Node runtime, so `node:sqlite` (real SQLite, + * with WAL + FTS5) is always available — there is no native build step and no + * wasm fallback. When run from source instead, it requires Node >= 22.5. */ export interface SqliteStatement { @@ -14,123 +19,34 @@ export interface SqliteStatement { export interface SqliteDatabase { prepare(sql: string): SqliteStatement; exec(sql: string): void; - pragma(str: string): any; + pragma(str: string, options?: { simple?: boolean }): any; transaction(fn: (...args: any[]) => T): (...args: any[]) => T; close(): void; readonly open: boolean; } -export type SqliteBackend = 'native' | 'wasm'; - /** - * One-line summary of the recovery steps shown when WASM fallback is - * active. Single source of truth so the recipe can't drift between the - * stderr banner and the MCP status formatter. + * The active SQLite backend. Only one now (`node:sqlite`); kept as a named type + * so `codegraph status` and the per-instance reporting have a stable shape. */ -export const WASM_FALLBACK_FIX_RECIPE = - '`xcode-select --install` (macOS) or `apt install build-essential` (Debian/Ubuntu), ' + - 'then `npm rebuild better-sqlite3`, or `npm install better-sqlite3 --save` to force-include it.'; +export type SqliteBackend = 'node-sqlite'; /** - * Multi-line banner shown to stderr when `createDatabase` falls back to - * WASM. Replaces a one-line `console.warn` that MCP transports (which - * take stdout for the protocol) typically swallow, leaving users on a - * 5-10x slower backend with no signal. + * Wraps Node's built-in `node:sqlite` (`DatabaseSync`) to match the + * better-sqlite3 interface the rest of the code expects. * - * Exported for unit testing — pinning the recipe content prevents - * future edits from silently stripping the recovery commands. + * node:sqlite is real SQLite compiled into Node, so it supports WAL, FTS5, + * mmap, and `@named` params natively — the only shims needed are the + * better-sqlite3 conveniences node:sqlite omits: a `.pragma()` helper, a + * `.transaction()` helper, and `open` (node:sqlite exposes `isOpen`). */ -export function buildWasmFallbackBanner(nativeError?: string): string { - const sep = '─'.repeat(72); - const lines = [ - sep, - '[CodeGraph] WASM SQLite fallback active (better-sqlite3 unavailable)', - sep, - 'Indexing and sync will be 5-10x slower than the native backend.', - '', - 'Fix on macOS:', - ' xcode-select --install # install C build tools', - ' npm rebuild better-sqlite3 # rebuild native binding for current Node', - '', - 'Fix on Linux:', - ' sudo apt install build-essential python3 make # Debian/Ubuntu', - ' # or: sudo yum groupinstall "Development Tools" # RHEL/Fedora', - ' npm rebuild better-sqlite3', - '', - 'Or force-include as a hard dependency on any platform:', - ' npm install better-sqlite3 --save', - '', - 'Verify after fix: `codegraph status` should show `Backend: native`.', - ]; - if (nativeError) { - lines.push('', `Native load error: ${nativeError}`); - } - lines.push(sep); - return lines.join('\n'); -} - -/** - * Translate @named parameters (better-sqlite3 style) to positional ? params - * for node-sqlite3-wasm, which only supports positional binding. - * - * Returns the rewritten SQL and an ordered list of parameter names. - * If no named params are found, returns null for paramOrder (positional mode). - */ -function translateNamedParams(sql: string): { sql: string; paramOrder: string[] | null } { - const paramOrder: string[] = []; - const rewritten = sql.replace(/@(\w+)/g, (_match, name: string) => { - paramOrder.push(name); - return '?'; - }); - if (paramOrder.length === 0) { - return { sql, paramOrder: null }; - } - return { sql: rewritten, paramOrder }; -} - -/** - * Convert better-sqlite3-style params to a positional array for node-sqlite3-wasm. - * - * Handles three calling conventions: - * - Named object: run({ id: '1', name: 'a' }) → positional array via paramOrder - * - Positional args: run('a', 'b') → ['a', 'b'] - * - No args: run() → undefined - */ -function resolveParams(params: any[], paramOrder: string[] | null): any { - if (params.length === 0) return undefined; - - // If paramOrder exists and first arg is a plain object, do named→positional translation - if (paramOrder && params.length === 1 && params[0] !== null && typeof params[0] === 'object' && !Array.isArray(params[0]) && !(params[0] instanceof Buffer) && !(params[0] instanceof Uint8Array)) { - const obj = params[0]; - return paramOrder.map(name => obj[name]); - } - - // Positional: single value or already an array - if (params.length === 1) return params[0]; - return params; -} - -/** - * Wraps node-sqlite3-wasm to match the better-sqlite3 interface. - * - * Key differences handled: - * - better-sqlite3 uses @named params; node-sqlite3-wasm uses positional ? only - * - better-sqlite3 uses variadic args: stmt.run(a, b, c) - * - node-sqlite3-wasm uses a single array/object: stmt.run([a, b, c]) - * - node-sqlite3-wasm has `isOpen` instead of `open` - * - node-sqlite3-wasm doesn't have a `pragma()` method - * - node-sqlite3-wasm doesn't have a `transaction()` method - */ -class WasmDatabaseAdapter implements SqliteDatabase { +class NodeSqliteAdapter implements SqliteDatabase { private _db: any; - // Track raw WASM statements so we can finalize them on close. - // node-sqlite3-wasm won't release its file lock if statements are left open. - private _openStmts = new Set(); constructor(dbPath: string) { // eslint-disable-next-line @typescript-eslint/no-require-imports - const { Database } = require('node-sqlite3-wasm'); - this._db = new Database(dbPath); + const { DatabaseSync } = require('node:sqlite'); + this._db = new DatabaseSync(dbPath); } get open(): boolean { @@ -138,25 +54,23 @@ class WasmDatabaseAdapter implements SqliteDatabase { } prepare(sql: string): SqliteStatement { - const { sql: rewrittenSql, paramOrder } = translateNamedParams(sql); - const stmt = this._db.prepare(rewrittenSql); - this._openStmts.add(stmt); + // node:sqlite matches better-sqlite3's calling convention (variadic + // positional args, or a single object for @named params), so params forward + // through unchanged. + const stmt = this._db.prepare(sql); return { run(...params: any[]) { - const resolved = resolveParams(params, paramOrder); - const result = resolved !== undefined ? stmt.run(resolved) : stmt.run(); + const r = stmt.run(...params); return { - changes: result?.changes ?? 0, - lastInsertRowid: result?.lastInsertRowid ?? 0, + changes: Number(r?.changes ?? 0), + lastInsertRowid: r?.lastInsertRowid ?? 0, }; }, get(...params: any[]) { - const resolved = resolveParams(params, paramOrder); - return resolved !== undefined ? stmt.get(resolved) : stmt.get(); + return stmt.get(...params); }, all(...params: any[]) { - const resolved = resolveParams(params, paramOrder); - return resolved !== undefined ? stmt.all(resolved) : stmt.all(); + return stmt.all(...params); }, }; } @@ -165,41 +79,21 @@ class WasmDatabaseAdapter implements SqliteDatabase { this._db.exec(sql); } - pragma(str: string): any { + pragma(str: string, options?: { simple?: boolean }): any { const trimmed = str.trim(); - - // Write pragma: "key = value" + // Write pragma ("key = value"): node:sqlite is real SQLite, so every pragma + // (WAL, mmap, synchronous, …) applies as-is. if (trimmed.includes('=')) { - const eqIdx = trimmed.indexOf('='); - const key = trimmed.substring(0, eqIdx).trim(); - const value = trimmed.substring(eqIdx + 1).trim(); - - // WAL is not supported in WASM SQLite — use DELETE journal mode - if (key === 'journal_mode' && value.toUpperCase() === 'WAL') { - this._db.exec('PRAGMA journal_mode = DELETE'); - return; - } - - // mmap is not available in WASM — silently skip - if (key === 'mmap_size') { - return; - } - - // synchronous = NORMAL is unsafe without WAL — use FULL - if (key === 'synchronous' && value.toUpperCase() === 'NORMAL') { - this._db.exec('PRAGMA synchronous = FULL'); - return; - } - - this._db.exec(`PRAGMA ${key} = ${value}`); + this._db.exec(`PRAGMA ${trimmed}`); return; } - - // Read pragma: "key" — return the value - const stmt = this._db.prepare(`PRAGMA ${trimmed}`); - const result = stmt.get(); - stmt.finalize(); - return result; + // Read pragma. Default: the row object (e.g. { journal_mode: 'wal' }). + // `{ simple: true }` returns just the single column value, like better-sqlite3. + const row = this._db.prepare(`PRAGMA ${trimmed}`).get(); + if (options?.simple) { + return row && typeof row === 'object' ? Object.values(row)[0] : row; + } + return row; } transaction(fn: (...args: any[]) => T): (...args: any[]) => T { @@ -217,51 +111,29 @@ class WasmDatabaseAdapter implements SqliteDatabase { } close(): void { - // Finalize all tracked statements before closing. - // node-sqlite3-wasm won't release its directory-based file lock - // if any prepared statements remain open. - for (const stmt of this._openStmts) { - try { stmt.finalize(); } catch { /* already finalized */ } - } - this._openStmts.clear(); - this._db.close(); + // node:sqlite's DatabaseSync.close() throws if already closed; make it + // idempotent to match better-sqlite3 (callers may close more than once). + if (this._db.isOpen) this._db.close(); } } /** - * Create a database connection. Tries native better-sqlite3 first, - * falls back to node-sqlite3-wasm. Returns the active backend - * alongside the db so each `DatabaseConnection` can report its own - * backend per-instance — MCP can open multiple project DBs in one - * process (`tools.ts` getCodeGraph cache), so a process-global would - * race / overwrite. + * Create a database connection backed by `node:sqlite`. + * + * Returns the active backend alongside the db so each `DatabaseConnection` can + * report it per-instance — MCP can open multiple project DBs in one process, so + * a process-global would race. */ export function createDatabase(dbPath: string): { db: SqliteDatabase; backend: SqliteBackend } { - let nativeError: string | undefined; - let wasmError: string | undefined; - - // Try native better-sqlite3 first - try { - // eslint-disable-next-line @typescript-eslint/no-require-imports - const Database = require('better-sqlite3'); - const db = new Database(dbPath); - return { db: db as SqliteDatabase, backend: 'native' }; - } catch (error) { - nativeError = error instanceof Error ? error.message : String(error); - } - - // Fall back to WASM try { - const db = new WasmDatabaseAdapter(dbPath); - console.warn(buildWasmFallbackBanner(nativeError)); - return { db, backend: 'wasm' }; + return { db: new NodeSqliteAdapter(dbPath), backend: 'node-sqlite' }; } catch (error) { - wasmError = error instanceof Error ? error.message : String(error); + const msg = error instanceof Error ? error.message : String(error); + throw new Error( + 'Failed to open SQLite via the built-in node:sqlite module.\n' + + 'CodeGraph requires node:sqlite (Node.js 22.5+). Install the self-contained\n' + + 'CodeGraph release (it bundles a compatible Node), or run on Node 22.5+.\n' + + `Underlying error: ${msg}` + ); } - - throw new Error( - `Failed to load any SQLite backend.\n` + - ` Native (better-sqlite3): ${nativeError}\n` + - ` WASM (node-sqlite3-wasm): ${wasmError}` - ); } diff --git a/src/extraction/generated-detection.ts b/src/extraction/generated-detection.ts new file mode 100644 index 000000000..bde190725 --- /dev/null +++ b/src/extraction/generated-detection.ts @@ -0,0 +1,78 @@ +/** + * Generated-file detection for symbol-disambiguation down-ranking. + * + * When a query like "Send" matches 17 symbols across protobuf scaffolding, + * test mocks, and the hand-written implementation, the FTS ranker often + * surfaces the generated stubs first because their names are identical + * to the implementation's name (validated empirically on cosmos-sdk — + * see project_go_multi_module_audit memory). Generated stubs frequently + * have no body to trace from, so the agent ends up reading source anyway. + * + * This helper is a pure path-based classifier consulted at disambiguation + * time (findSymbol / findAllSymbols / codegraph_search formatting), NOT + * a hard filter — generated nodes are still in the graph and remain + * reachable; they just rank LAST when there's a real implementation + * with the same name. + * + * Scope: suffix patterns only. Most generated files follow the + * `..` convention (`.pb.go`, `_grpc.pb.go`, + * `.g.dart`, `_pb2.py`), and that covers ~all of what we saw in the + * Go audit. A future addition would be scanning for the canonical + * `// Code generated by` header during extraction, for the rare files + * that defy the suffix convention. + */ + +const GENERATED_PATTERNS: ReadonlyArray = [ + // Go — protobuf / gRPC / pulsar + /\.pb\.go$/, + /\.pulsar\.go$/, + /_grpc\.pb\.go$/, + // Go — mockgen output. Default emits `mock_.go`; many projects + // (cosmos-sdk uses `expected_*_mocks.go`) rename to `*_mock.go` / + // `*_mocks.go`. Matching either suffix catches both conventions + // without false-positive risk on hand-written sources. + /_mock\.go$/, + /_mocks\.go$/, + /^mock_[^/]+\.go$/, + // TypeScript / JavaScript — common codegen suffixes (Apollo / GraphQL + // codegen, Prisma, Hasura, ts-proto, gRPC-web, swagger-codegen). + /\.generated\.[jt]sx?$/, + /\.gen\.[jt]sx?$/, + /\.pb\.[jt]s$/, + /_pb\.[jt]s$/, + /_grpc_pb\.[jt]s$/, + // Python — protobuf / gRPC / openapi-codegen + /_pb2(_grpc)?\.py$/, + /_pb2\.pyi$/, + // C++ — protobuf + /\.pb\.(cc|h)$/, + // C# — protobuf / gRPC (protoc-gen-csharp puts output under obj/ but + // many projects also commit *.g.cs and *Grpc.cs siblings) + /\.g\.cs$/, + /Grpc\.cs$/, + // Java — protobuf / gRPC: protoc-gen-java emits `*OuterClass.java`, + // protoc-gen-grpc-java emits `*Grpc.java`. The XxxImplBase abstract + // class lives inside Xxx*Grpc.java. + /OuterClass\.java$/, + /Grpc\.java$/, + // Swift — protobuf + /\.pb\.swift$/, + // Dart — build_runner / freezed / json_serializable / chopper + /\.g\.dart$/, + /\.freezed\.dart$/, + /\.pb\.dart$/, + /\.pbgrpc\.dart$/, + /\.chopper\.dart$/, + // Rust — common build.rs OUT_DIR outputs are usually outside the source + // tree, but in-tree generated files often use `*.generated.rs`. + /\.generated\.rs$/, +]; + +/** + * Whether `filePath` looks like a tool-generated source file based on + * its filename. Path-only — does not read content. The result is a + * relevance hint for disambiguation, not a hard claim. + */ +export function isGeneratedFile(filePath: string): boolean { + return GENERATED_PATTERNS.some((p) => p.test(filePath)); +} diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index d15404241..c9a2bcb37 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -10,7 +10,7 @@ import * as path from 'path'; import { Parser, Language as WasmLanguage } from 'web-tree-sitter'; import { Language } from '../types'; -export type GrammarLanguage = Exclude; +export type GrammarLanguage = Exclude; /** * WASM filename map — maps each language to its .wasm grammar file @@ -35,6 +35,9 @@ const WASM_GRAMMAR_FILES: Record = { dart: 'tree-sitter-dart.wasm', pascal: 'tree-sitter-pascal.wasm', scala: 'tree-sitter-scala.wasm', + lua: 'tree-sitter-lua.wasm', + luau: 'tree-sitter-luau.wasm', + objc: 'tree-sitter-objc.wasm', }; /** @@ -61,6 +64,16 @@ export const EXTENSION_MAP: Record = { '.hxx': 'cpp', '.cs': 'csharp', '.php': 'php', + // Drupal-specific PHP file extensions + '.module': 'php', + '.install': 'php', + '.theme': 'php', + '.inc': 'php', + // YAML (used for Drupal routing files; no symbol extraction, file-level tracking only) + '.yml': 'yaml', + '.yaml': 'yaml', + // Twig templates (file-level tracking only, no symbol extraction) + '.twig': 'twig', '.rb': 'ruby', '.rake': 'ruby', '.swift': 'swift', @@ -78,8 +91,44 @@ export const EXTENSION_MAP: Record = { '.fmx': 'pascal', '.scala': 'scala', '.sc': 'scala', + '.lua': 'lua', + '.luau': 'luau', + '.m': 'objc', + '.mm': 'objc', + // XML: file-level tracking; the MyBatis extractor matches `` + // shape and emits SQL-statement nodes (other XML returns empty). + '.xml': 'xml', + // Spring config: `application.properties` / `application-*.properties`. Same + // shape as the `.yml` variants — the YAML/properties extractor emits one node + // per leaf key, and the Spring resolver links `@Value("${k}")` references. + '.properties': 'properties', }; +/** + * Whether a file is one CodeGraph can parse, based purely on its extension. + * This is the single source of truth for "should we index this file" — derived + * from EXTENSION_MAP so parser support and indexing selection never drift. + */ +export function isSourceFile(filePath: string): boolean { + if (isPlayRoutesFile(filePath)) return true; // Play `conf/routes` is extensionless + const dot = filePath.lastIndexOf('.'); + if (dot < 0) return false; + return filePath.slice(dot).toLowerCase() in EXTENSION_MAP; +} + +/** + * Play Framework routes file: the extensionless `conf/routes` (and included + * `conf/*.routes`). No grammar — route extraction is done by the Play framework + * resolver, so it's processed through the no-grammar (`yaml`-style) path. + */ +export function isPlayRoutesFile(filePath: string): boolean { + return ( + filePath === 'conf/routes' || + filePath.endsWith('/conf/routes') || + filePath.endsWith('.routes') + ); +} + /** * Caches for loaded grammars and parsers */ @@ -125,8 +174,12 @@ export async function loadGrammarsForLanguages(languages: Language[]): Promise = new Set([ + // JS / TS — dependency directories + 'node_modules', 'bower_components', 'jspm_packages', 'web_modules', + '.yarn', '.pnpm-store', + // JS / TS — framework & bundler build / cache / deploy output + '.next', '.nuxt', '.svelte-kit', '.turbo', '.vite', '.parcel-cache', '.angular', + '.docusaurus', 'storybook-static', '.vinxi', '.nitro', 'out-tsc', + '.vercel', '.netlify', '.wrangler', + // Build output (common across ecosystems) + 'dist', 'build', 'out', '.output', + // Test / coverage + 'coverage', '.nyc_output', + // Python + '__pycache__', '__pypackages__', '.venv', 'venv', '.pixi', '.pdm-build', + '.mypy_cache', '.pytest_cache', '.ruff_cache', '.tox', '.nox', '.hypothesis', + '.ipynb_checkpoints', '.eggs', + // Rust / JVM (Maven, Gradle, Scala) + 'target', '.gradle', + // .NET + 'obj', + // Vendored deps (Go, PHP/Composer, Ruby/Bundler) + 'vendor', + // Swift / iOS + '.build', 'Pods', 'Carthage', 'DerivedData', '.swiftpm', + // Dart / Flutter + '.dart_tool', '.pub-cache', + // Native (Android NDK, C/C++ deps) + '.cxx', '.externalNativeBuild', 'vcpkg_installed', + // Scala tooling + '.bloop', '.metals', + // Lua / Luau (LuaRocks) + 'lua_modules', '.luarocks', + // Delphi / RAD Studio IDE backups (duplicate .pas source — would double-count) + '__history', '__recovery', + // Generic cache + '.cache', +]); + +/** Gitignore-style patterns for the `ignore` matcher: the dirs above plus a few globs. */ +const DEFAULT_IGNORE_PATTERNS: string[] = [ + ...Array.from(DEFAULT_IGNORE_DIRS, (d) => `${d}/`), + '*.egg-info/', // Python packaging metadata + 'cmake-build-*/', // CLion / CMake build trees + 'bazel-*/', // Bazel output symlink trees +]; + +/** + * An `ignore` matcher seeded with the built-in defaults, merged with the project's + * root .gitignore so a negation there (e.g. `!vendor/`) overrides a default. Shared + * by both enumeration paths so behavior is identical with or without git — and so + * the defaults apply to tracked files too (committing a dependency dir doesn't make + * it project code; the explicit `.gitignore` negation is the only opt-in). + */ +export function buildDefaultIgnore(rootDir: string): Ignore { + const ig = ignore().add(DEFAULT_IGNORE_PATTERNS); + try { + const rootGitignore = path.join(rootDir, '.gitignore'); + if (fs.existsSync(rootGitignore)) ig.add(fs.readFileSync(rootGitignore, 'utf-8')); + } catch { + // Unreadable root .gitignore — the built-in defaults still apply. + } + return ig; } /** - * Check if a file should be included based on config + * Collect git-visible files (tracked + untracked, .gitignore-respected) from the + * git repository rooted at `repoDir`, adding each to `files` with `prefix` + * prepended so paths stay relative to the original scan root. + * + * Recurses into embedded git repositories — nested repos that are NOT submodules + * (independent clones living inside the workspace, common in CMake "super-repo" + * layouts). The parent repo's `git ls-files` cannot see into them: tracked output + * skips them entirely, and untracked output reports them only as an opaque + * "subdir/" entry (trailing slash) rather than expanding their files. Each + * embedded repo is its own git boundary, so we re-run `git ls-files` inside it. + * (See issue #193.) */ -export function shouldIncludeFile( - filePath: string, - config: CodeGraphConfig -): boolean { - // Check exclude patterns first - for (const pattern of config.exclude) { - if (matchesGlob(filePath, pattern)) { - return false; +function collectGitFiles(repoDir: string, prefix: string, files: Set): void { + const gitOpts = { cwd: repoDir, encoding: 'utf-8' as const, timeout: 30000, maxBuffer: 50 * 1024 * 1024, stdio: ['pipe', 'pipe', 'pipe'] as ['pipe', 'pipe', 'pipe'], windowsHide: true }; + + // Tracked files. --recurse-submodules pulls in files from active submodules, + // which the index would otherwise represent only as a commit pointer. + // Without this, monorepos using submodules index 0 files. (See issue #147.) + // Note: --recurse-submodules only supports -c/--cached and --stage modes — it + // can't be combined with -o, so untracked files are gathered separately below. + const tracked = execFileSync('git', ['ls-files', '-c', '--recurse-submodules'], gitOpts); + for (const line of tracked.split('\n')) { + const trimmed = line.trim(); + if (trimmed) { + files.add(normalizePath(prefix + trimmed)); } } - // Check include patterns - for (const pattern of config.include) { - if (matchesGlob(filePath, pattern)) { - return true; + // Untracked files (submodules manage their own untracked state). Embedded git + // repos surface here as a single "subdir/" entry that git refuses to descend + // into — recurse into those as their own repos so their source gets indexed. + const untracked = execFileSync('git', ['ls-files', '-o', '--exclude-standard'], gitOpts); + for (const line of untracked.split('\n')) { + const trimmed = line.trim(); + if (!trimmed) continue; + if (trimmed.endsWith('/')) { + // git only emits a trailing-slash directory entry for an embedded repo. + // Guard with a .git check anyway, and skip anything else exactly as git + // itself skips it (we never descend into a non-repo opaque dir). + const childDir = path.join(repoDir, trimmed); + if (fs.existsSync(path.join(childDir, '.git'))) { + collectGitFiles(childDir, prefix + trimmed, files); + } + continue; } + files.add(normalizePath(prefix + trimmed)); } - - return false; } /** * Get all files visible to git (tracked + untracked but not ignored). - * Respects .gitignore at all levels (root, subdirectories). - * Returns null on failure (non-git project) so callers can fall back. + * Respects .gitignore at all levels (root, subdirectories) and descends into + * embedded (nested, non-submodule) git repos. Returns null on failure + * (non-git project) so callers can fall back to a filesystem walk. */ function getGitVisibleFiles(rootDir: string): Set | null { try { @@ -138,7 +241,7 @@ function getGitVisibleFiles(rootDir: string): Set | null { const gitRoot = execFileSync( 'git', ['rev-parse', '--show-toplevel'], - { cwd: rootDir, encoding: 'utf-8', timeout: 5000, stdio: ['pipe', 'pipe', 'pipe'] } + { cwd: rootDir, encoding: 'utf-8', timeout: 5000, stdio: ['pipe', 'pipe', 'pipe'], windowsHide: true } ).trim(); if (path.resolve(gitRoot) !== path.resolve(rootDir)) { @@ -147,7 +250,7 @@ function getGitVisibleFiles(rootDir: string): Set | null { execFileSync( 'git', ['check-ignore', '-q', path.resolve(rootDir)], - { cwd: rootDir, encoding: 'utf-8', timeout: 5000, stdio: ['pipe', 'pipe', 'pipe'] } + { cwd: rootDir, encoding: 'utf-8', timeout: 5000, stdio: ['pipe', 'pipe', 'pipe'], windowsHide: true } ); // Directory is gitignored by parent repo — fall back to filesystem walk return null; @@ -157,31 +260,12 @@ function getGitVisibleFiles(rootDir: string): Set | null { } const files = new Set(); - const gitOpts = { cwd: rootDir, encoding: 'utf-8' as const, timeout: 30000, maxBuffer: 50 * 1024 * 1024, stdio: ['pipe', 'pipe', 'pipe'] as ['pipe', 'pipe', 'pipe'] }; - - // Tracked files. --recurse-submodules pulls in files from active submodules, - // which the main repo's index would otherwise represent only as a commit pointer. - // Without this, monorepos using submodules index 0 files. (See issue #147.) - // Note: --recurse-submodules only supports -c/--cached and --stage modes — it - // can't be combined with -o, so untracked files are gathered separately below. - const tracked = execFileSync('git', ['ls-files', '-c', '--recurse-submodules'], gitOpts); - for (const line of tracked.split('\n')) { - const trimmed = line.trim(); - if (trimmed) { - files.add(normalizePath(trimmed)); - } - } - - // Untracked files in the main repo (submodules manage their own untracked state). - const untracked = execFileSync('git', ['ls-files', '-o', '--exclude-standard'], gitOpts); - for (const line of untracked.split('\n')) { - const trimmed = line.trim(); - if (trimmed) { - files.add(normalizePath(trimmed)); - } - } - - return files; + collectGitFiles(rootDir, '', files); + // Apply built-in default ignores uniformly — to tracked files too, since + // committing a dependency/build dir doesn't make it project code. A + // `.gitignore` negation (e.g. `!vendor/`) is the explicit opt-in. (issue #407) + const ig = buildDefaultIgnore(rootDir); + return new Set([...files].filter((f) => !ig.ignores(f))); } catch { return null; } @@ -202,12 +286,12 @@ interface GitChanges { * Use `git status` to detect changed files instead of scanning every file. * Returns null on failure so callers fall back to full scan. */ -function getGitChangedFiles(rootDir: string, config: CodeGraphConfig): GitChanges | null { +function getGitChangedFiles(rootDir: string): GitChanges | null { try { const output = execFileSync( 'git', ['status', '--porcelain', '--no-renames'], - { cwd: rootDir, encoding: 'utf-8', timeout: 10000, stdio: ['pipe', 'pipe', 'pipe'] } + { cwd: rootDir, encoding: 'utf-8', timeout: 10000, stdio: ['pipe', 'pipe', 'pipe'], windowsHide: true } ); const modified: string[] = []; @@ -220,8 +304,8 @@ function getGitChangedFiles(rootDir: string, config: CodeGraphConfig): GitChange const statusCode = line.substring(0, 2); const filePath = normalizePath(line.substring(3)); - // Skip files that don't match include/exclude config - if (!shouldIncludeFile(filePath, config)) continue; + // Skip non-source files (git status already omits .gitignored paths). + if (!isSourceFile(filePath)) continue; if (statusCode === '??') { added.push(filePath); @@ -240,20 +324,14 @@ function getGitChangedFiles(rootDir: string, config: CodeGraphConfig): GitChange } /** - * Marker file name that indicates a directory (and all children) should be skipped - */ -const CODEGRAPH_IGNORE_MARKER = '.codegraphignore'; - -/** - * Recursively scan directory for source files. + * Recursively scan a directory for source files. * - * In git repos, uses `git ls-files` to get the file list (inherently - * respects .gitignore at all levels), then filters by config include patterns. - * Falls back to filesystem walk for non-git projects. + * In git repos, uses `git ls-files` (inherently respects .gitignore at all + * levels), then keeps files with a supported source extension. For non-git + * projects, falls back to a filesystem walk that parses .gitignore itself. */ export function scanDirectory( rootDir: string, - config: CodeGraphConfig, onProgress?: (current: number, file: string) => void ): string[] { // Fast path: use git to get all visible files (respects .gitignore everywhere) @@ -262,7 +340,7 @@ export function scanDirectory( const files: string[] = []; let count = 0; for (const filePath of gitFiles) { - if (shouldIncludeFile(filePath, config)) { + if (isSourceFile(filePath)) { files.push(filePath); count++; onProgress?.(count, filePath); @@ -272,7 +350,7 @@ export function scanDirectory( } // Fallback: walk filesystem for non-git projects - return scanDirectoryWalk(rootDir, config, onProgress); + return scanDirectoryWalk(rootDir, onProgress); } /** @@ -281,7 +359,6 @@ export function scanDirectory( */ export async function scanDirectoryAsync( rootDir: string, - config: CodeGraphConfig, onProgress?: (current: number, file: string) => void ): Promise { const gitFiles = getGitVisibleFiles(rootDir); @@ -289,7 +366,7 @@ export async function scanDirectoryAsync( const files: string[] = []; let count = 0; for (const filePath of gitFiles) { - if (shouldIncludeFile(filePath, config)) { + if (isSourceFile(filePath)) { files.push(filePath); count++; onProgress?.(count, filePath); @@ -302,7 +379,7 @@ export async function scanDirectoryAsync( return files; } - return scanDirectoryWalk(rootDir, config, onProgress); + return scanDirectoryWalk(rootDir, onProgress); } /** @@ -310,14 +387,44 @@ export async function scanDirectoryAsync( */ function scanDirectoryWalk( rootDir: string, - config: CodeGraphConfig, onProgress?: (current: number, file: string) => void ): string[] { const files: string[] = []; let count = 0; const visitedDirs = new Set(); - function walk(dir: string): void { + // A .gitignore matcher scoped to the directory that declared it. Patterns in + // a nested .gitignore are relative to that directory, so we keep the dir + // alongside the matcher and test paths relative to it — mirroring how git + // applies .gitignore files at every level. + interface ScopedIgnore { + dir: string; + ig: Ignore; + } + + const loadIgnore = (dir: string): ScopedIgnore | null => { + try { + const giPath = path.join(dir, '.gitignore'); + if (fs.existsSync(giPath)) { + return { dir, ig: ignore().add(fs.readFileSync(giPath, 'utf-8')) }; + } + } catch { + // Unreadable .gitignore — treat as absent. + } + return null; + }; + + const isIgnored = (fullPath: string, isDir: boolean, matchers: ScopedIgnore[]): boolean => { + for (const { dir, ig } of matchers) { + let rel = normalizePath(path.relative(dir, fullPath)); + if (!rel || rel.startsWith('..')) continue; // not under this matcher's dir + if (isDir) rel += '/'; // dir-only rules (e.g. `build/`) only match with the slash + if (ig.ignores(rel)) return true; + } + return false; + }; + + function walk(dir: string, matchers: ScopedIgnore[]): void { let realDir: string; try { realDir = fs.realpathSync(dir); @@ -332,12 +439,11 @@ function scanDirectoryWalk( } visitedDirs.add(realDir); - // Check for .codegraphignore marker file - const ignoreMarker = path.join(dir, CODEGRAPH_IGNORE_MARKER); - if (fs.existsSync(ignoreMarker)) { - logDebug('Skipping directory due to .codegraphignore marker', { dir }); - return; - } + // This directory's own .gitignore (if present) applies to everything below it. + // The root's .gitignore is already merged into the seeded base matcher (so a + // negation there can override a built-in default), so skip it here. + const own = dir === rootDir ? null : loadIgnore(dir); + const active = own ? [...matchers, own] : matchers; let entries: fs.Dirent[]; try { @@ -348,6 +454,9 @@ function scanDirectoryWalk( } for (const entry of entries) { + // Never descend into git internals or our own data directory. + if (entry.name === '.git' || entry.name === '.codegraph') continue; + const fullPath = path.join(dir, entry.name); const relativePath = normalizePath(path.relative(rootDir, fullPath)); @@ -356,19 +465,11 @@ function scanDirectoryWalk( const realTarget = fs.realpathSync(fullPath); const stat = fs.statSync(realTarget); if (stat.isDirectory()) { - const dirPattern = relativePath + '/'; - let excluded = false; - for (const pattern of config.exclude) { - if (matchesGlob(dirPattern, pattern) || matchesGlob(relativePath, pattern)) { - excluded = true; - break; - } - } - if (!excluded) { - walk(fullPath); + if (!isIgnored(fullPath, true, active)) { + walk(fullPath, active); } } else if (stat.isFile()) { - if (shouldIncludeFile(relativePath, config)) { + if (!isIgnored(fullPath, false, active) && isSourceFile(relativePath)) { files.push(relativePath); count++; onProgress?.(count, relativePath); @@ -381,19 +482,11 @@ function scanDirectoryWalk( } if (entry.isDirectory()) { - const dirPattern = relativePath + '/'; - let excluded = false; - for (const pattern of config.exclude) { - if (matchesGlob(dirPattern, pattern) || matchesGlob(relativePath, pattern)) { - excluded = true; - break; - } - } - if (!excluded) { - walk(fullPath); + if (!isIgnored(fullPath, true, active)) { + walk(fullPath, active); } } else if (entry.isFile()) { - if (shouldIncludeFile(relativePath, config)) { + if (!isIgnored(fullPath, false, active) && isSourceFile(relativePath)) { files.push(relativePath); count++; onProgress?.(count, relativePath); @@ -402,7 +495,9 @@ function scanDirectoryWalk( } } - walk(rootDir); + // Seed a base matcher with the built-in default ignores (merged with the root + // .gitignore so a negation can override). Nested .gitignores still layer per-dir. + walk(rootDir, [{ dir: rootDir, ig: buildDefaultIgnore(rootDir) }]); return files; } @@ -411,7 +506,6 @@ function scanDirectoryWalk( */ export class ExtractionOrchestrator { private rootDir: string; - private config: CodeGraphConfig; private queries: QueryBuilder; /** * Names of frameworks detected for this project, populated by indexAll(). @@ -421,9 +515,8 @@ export class ExtractionOrchestrator { */ private detectedFrameworkNames: string[] | null = null; - constructor(rootDir: string, config: CodeGraphConfig, queries: QueryBuilder) { + constructor(rootDir: string, queries: QueryBuilder) { this.rootDir = rootDir; - this.config = config; this.queries = queries; } @@ -462,6 +555,24 @@ export class ExtractionOrchestrator { return null; } }, + // Monorepo support — needed by framework detect()s that probe + // subpackage manifests (e.g. fabric-view looking at + // packages//package.json when the root manifest is just a + // workspace declaration). Matches the resolver-context shape. + listDirectories: (relativePath: string) => { + const target = + relativePath === '.' || relativePath === '' + ? rootDir + : path.join(rootDir, relativePath); + try { + return fs + .readdirSync(target, { withFileTypes: true }) + .filter((entry) => entry.isDirectory()) + .map((entry) => entry.name); + } catch { + return []; + } + }, }; } @@ -472,7 +583,7 @@ export class ExtractionOrchestrator { */ private ensureDetectedFrameworks(files?: string[]): string[] { if (this.detectedFrameworkNames !== null) return this.detectedFrameworkNames; - const fileList = files ?? scanDirectory(this.rootDir, this.config); + const fileList = files ?? scanDirectory(this.rootDir); const context = this.buildDetectionContext(fileList); this.detectedFrameworkNames = detectFrameworks(context).map((r) => r.name); return this.detectedFrameworkNames; @@ -506,7 +617,7 @@ export class ExtractionOrchestrator { total: 0, }); - const files = await scanDirectoryAsync(this.rootDir, this.config, (current, file) => { + const files = await scanDirectoryAsync(this.rootDir, (current, file) => { onProgress?.({ phase: 'scanning', current, @@ -774,18 +885,16 @@ export class ExtractionOrchestrator { continue; } - // Honour config.maxFileSize. Without this check, vendored - // generated headers, minified bundles, and other multi-MB - // files get indexed despite the user setting a size cap — - // wasting WASM heap and the worker recycle budget on inputs - // the user explicitly opted out of. The single-file extractFile - // path already enforces this; the bulk path used to silently - // skip the check. - if (stats.size > this.config.maxFileSize) { + // Honour MAX_FILE_SIZE. Without this check, vendored generated + // headers, minified bundles, and other multi-MB files get indexed, + // wasting WASM heap and the worker recycle budget on inputs with no + // useful symbols. The single-file extractFile path already enforces + // this; the bulk path used to silently skip the check. + if (stats.size > MAX_FILE_SIZE) { processed++; filesSkipped++; errors.push({ - message: `File exceeds max size (${stats.size} > ${this.config.maxFileSize})`, + message: `File exceeds max size (${stats.size} > ${MAX_FILE_SIZE})`, filePath, severity: 'warning', code: 'size_exceeded', @@ -833,7 +942,15 @@ export class ExtractionOrchestrator { } else if (result.errors.some((e) => e.severity === 'error')) { filesErrored++; } else { - filesSkipped++; + // Files with no symbols but no errors (yaml, twig, properties) are + // tracked at the file level — count them as indexed so the CLI + // doesn't misleadingly report "No files found to index". + const lang = detectLanguage(filePath, content); + if (isFileLevelOnlyLanguage(lang)) { + filesIndexed++; + } else { + filesSkipped++; + } } } } @@ -999,7 +1116,12 @@ export class ExtractionOrchestrator { } else if (result.errors.some((e) => e.severity === 'error')) { filesErrored++; } else { - filesSkipped++; + const tracked = this.queries.getFileByPath(filePath); + if (tracked && isFileLevelOnlyLanguage(tracked.language)) { + filesIndexed++; + } else { + filesSkipped++; + } } } @@ -1080,14 +1202,14 @@ export class ExtractionOrchestrator { } // Check file size - if (stats.size > this.config.maxFileSize) { + if (stats.size > MAX_FILE_SIZE) { return { nodes: [], edges: [], unresolvedReferences: [], errors: [ { - message: `File exceeds max size (${stats.size} > ${this.config.maxFileSize})`, + message: `File exceeds max size (${stats.size} > ${MAX_FILE_SIZE})`, filePath: relativePath, severity: 'warning', code: 'size_exceeded', @@ -1197,8 +1319,12 @@ export class ExtractionOrchestrator { } /** - * Sync with current file state. - * Uses git status as a fast path when available, falling back to full scan. + * Sync the index with the current file state. + * + * Change detection is filesystem-based, never git: a (size, mtime) stat + * pre-filter skips unchanged files, then a content-hash compare confirms real + * changes. This works in non-git projects and catches committed changes from + * `git pull`/`checkout`/`merge`/`rebase` that `git status` cannot see. */ async sync(onProgress?: (progress: IndexProgress) => void): Promise { await initGrammars(); // Initialize WASM runtime (grammars loaded lazily below) @@ -1217,96 +1343,75 @@ export class ExtractionOrchestrator { }); const filesToIndex: string[] = []; - const gitChanges = getGitChangedFiles(this.rootDir, this.config); + // === Filesystem reconcile (git-independent) === + // The source of truth for "what changed" is the filesystem vs the indexed + // state — never git. We enumerate the current source files and reconcile + // each against the DB. A cheap (size, mtime) stat pre-filter skips unchanged + // files without reading or hashing them, so the expensive read+hash+parse + // only runs for files that actually changed. This catches edits/adds/deletes + // whether or not the project uses git, and crucially also catches committed + // changes from `git pull`/`checkout`/`merge`/`rebase` — which `git status` + // cannot see, because the working tree is clean afterward. + const currentFiles = scanDirectory(this.rootDir); + filesChecked = currentFiles.length; + const currentSet = new Set(currentFiles); - if (gitChanges) { - // === Git fast path === - // Only inspect the files git reports as changed instead of scanning everything. - filesChecked = gitChanges.modified.length + gitChanges.added.length + gitChanges.deleted.length; + const trackedFiles = this.queries.getAllFiles(); + const trackedMap = new Map(); + for (const f of trackedFiles) { + trackedMap.set(f.path, f); + } - // Handle deleted files - for (const filePath of gitChanges.deleted) { - const tracked = this.queries.getFileByPath(filePath); - if (tracked) { - this.queries.deleteFile(filePath); - filesRemoved++; - } + // Removals: tracked in the DB but no longer a present source file. Check the + // filesystem directly — `scanDirectory` (via `git ls-files`) still lists a + // file deleted from disk but not yet staged, so set membership alone misses it. + for (const tracked of trackedFiles) { + if (!currentSet.has(tracked.path) || !fs.existsSync(path.join(this.rootDir, tracked.path))) { + this.queries.deleteFile(tracked.path); + filesRemoved++; } + } - // Handle modified files — read + hash only these files - for (const filePath of gitChanges.modified) { - const fullPath = path.join(this.rootDir, filePath); - let content: string; + // Adds / modifications. + for (const filePath of currentFiles) { + const fullPath = path.join(this.rootDir, filePath); + const tracked = trackedMap.get(filePath); + + // Cheap pre-filter: an already-indexed file whose size AND mtime both match + // the DB is unchanged — skip it without reading or hashing. (A content + // change that preserves both exactly is the blind spot every mtime-based + // incremental tool accepts; `index --force` is the escape hatch. Git bumps + // mtime on every file it writes during checkout/merge, so pulls are caught.) + if (tracked) { try { - content = fs.readFileSync(fullPath, 'utf-8'); + const stat = fs.statSync(fullPath); + if (stat.size === tracked.size && Math.floor(stat.mtimeMs) === Math.floor(tracked.modifiedAt)) { + continue; + } } catch (error) { - logDebug('Skipping unreadable file during sync', { filePath, error: String(error) }); + logDebug('Skipping unstattable file during sync', { filePath, error: String(error) }); continue; } + } - const contentHash = hashContent(content); - const tracked = this.queries.getFileByPath(filePath); - - if (!tracked) { - filesToIndex.push(filePath); - changedFilePaths.push(filePath); - filesAdded++; - } else if (tracked.contentHash !== contentHash) { - filesToIndex.push(filePath); - changedFilePaths.push(filePath); - filesModified++; - } + // New, or size/mtime changed — read + hash to confirm a real content change. + let content: string; + try { + content = fs.readFileSync(fullPath, 'utf-8'); + } catch (error) { + logDebug('Skipping unreadable file during sync', { filePath, error: String(error) }); + continue; } + const contentHash = hashContent(content); - // Handle added (untracked) files - for (const filePath of gitChanges.added) { + if (!tracked) { filesToIndex.push(filePath); changedFilePaths.push(filePath); filesAdded++; - } - } else { - // === Fallback: full scan (non-git project or git failure) === - const currentFiles = new Set(scanDirectory(this.rootDir, this.config)); - filesChecked = currentFiles.size; - - // Build Map for O(1) lookups instead of .find() per file - const trackedFiles = this.queries.getAllFiles(); - const trackedMap = new Map(); - for (const f of trackedFiles) { - trackedMap.set(f.path, f); - } - - // Find files to remove (in DB but not on disk) - for (const tracked of trackedFiles) { - if (!currentFiles.has(tracked.path)) { - this.queries.deleteFile(tracked.path); - filesRemoved++; - } - } - - // Find files to add or update - for (const filePath of currentFiles) { - const fullPath = path.join(this.rootDir, filePath); - let content: string; - try { - content = fs.readFileSync(fullPath, 'utf-8'); - } catch (error) { - logDebug('Skipping unreadable file during sync', { filePath, error: String(error) }); - continue; - } - - const contentHash = hashContent(content); - const tracked = trackedMap.get(filePath); - - if (!tracked) { - filesToIndex.push(filePath); - changedFilePaths.push(filePath); - filesAdded++; - } else if (tracked.contentHash !== contentHash) { - filesToIndex.push(filePath); - changedFilePaths.push(filePath); - filesModified++; - } + } else if (tracked.contentHash !== contentHash) { + filesToIndex.push(filePath); + changedFilePaths.push(filePath); + filesModified++; } } @@ -1351,7 +1456,7 @@ export class ExtractionOrchestrator { * Uses git status as a fast path when available, falling back to full scan. */ getChangedFiles(): { added: string[]; modified: string[]; removed: string[] } { - const gitChanges = getGitChangedFiles(this.rootDir, this.config); + const gitChanges = getGitChangedFiles(this.rootDir); if (gitChanges) { // === Git fast path === @@ -1367,8 +1472,11 @@ export class ExtractionOrchestrator { } } - // Modified files — read + hash only these, compare with DB - for (const filePath of gitChanges.modified) { + // Modified + added files — read + hash, compare with DB. Untracked (`??`) + // files stay untracked in git even after indexing, so they must be + // hash-compared like modified files instead of always counting as added — + // otherwise status reports them as pending forever. (See issue #206.) + for (const filePath of [...gitChanges.modified, ...gitChanges.added]) { const fullPath = path.join(this.rootDir, filePath); let content: string; try { @@ -1388,16 +1496,11 @@ export class ExtractionOrchestrator { } } - // Added (untracked) files - for (const filePath of gitChanges.added) { - added.push(filePath); - } - return { added, modified, removed }; } // === Fallback: full scan (non-git project or git failure) === - const currentFiles = new Set(scanDirectory(this.rootDir, this.config)); + const currentFiles = new Set(scanDirectory(this.rootDir)); const trackedFiles = this.queries.getAllFiles(); // Build Map for O(1) lookups @@ -1444,4 +1547,4 @@ export class ExtractionOrchestrator { // Re-export useful types and functions export { extractFromSource } from './tree-sitter'; -export { detectLanguage, isLanguageSupported, isGrammarLoaded, getSupportedLanguages, initGrammars, loadGrammarsForLanguages, loadAllGrammars } from './grammars'; +export { detectLanguage, isSourceFile, isLanguageSupported, isGrammarLoaded, getSupportedLanguages, initGrammars, loadGrammarsForLanguages, loadAllGrammars } from './grammars'; diff --git a/src/extraction/languages/c-cpp.ts b/src/extraction/languages/c-cpp.ts index 66219d4fa..fa511150d 100644 --- a/src/extraction/languages/c-cpp.ts +++ b/src/extraction/languages/c-cpp.ts @@ -2,6 +2,51 @@ import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getChildByField, getNodeText } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; +function extractCppQualifiedMethodName(node: SyntaxNode, source: string): string | undefined { + const declarator = getChildByField(node, 'declarator'); + if (!declarator) return undefined; + + const queue: SyntaxNode[] = [declarator]; + while (queue.length > 0) { + const current = queue.shift()!; + if (current.type === 'qualified_identifier') { + const text = getNodeText(current, source).trim(); + const parts = text.split('::').filter(Boolean); + return parts[parts.length - 1]; + } + for (let i = 0; i < current.namedChildCount; i++) { + const child = current.namedChild(i); + if (child) queue.push(child); + } + } + + return undefined; +} + +function extractCppReceiverType(node: SyntaxNode, source: string): string | undefined { + const declarator = getChildByField(node, 'declarator'); + if (!declarator) return undefined; + + const queue: SyntaxNode[] = [declarator]; + while (queue.length > 0) { + const current = queue.shift()!; + if (current.type === 'qualified_identifier') { + const text = getNodeText(current, source).trim(); + const parts = text.split('::').filter(Boolean); + if (parts.length > 1) { + return parts.slice(0, -1).join('::'); + } + return undefined; + } + for (let i = 0; i < current.namedChildCount; i++) { + const child = current.namedChild(i); + if (child) queue.push(child); + } + } + + return undefined; +} + export const cExtractor: LanguageExtractor = { functionTypes: ['function_definition'], classTypes: [], @@ -62,6 +107,8 @@ export const cppExtractor: LanguageExtractor = { nameField: 'declarator', bodyField: 'body', paramsField: 'parameters', + resolveName: extractCppQualifiedMethodName, + getReceiverType: extractCppReceiverType, getVisibility: (node) => { // Check for access specifier in parent const parent = node.parent; diff --git a/src/extraction/languages/csharp.ts b/src/extraction/languages/csharp.ts index 9de537346..100b1b050 100644 --- a/src/extraction/languages/csharp.ts +++ b/src/extraction/languages/csharp.ts @@ -18,7 +18,8 @@ export const csharpExtractor: LanguageExtractor = { propertyTypes: ['property_declaration'], nameField: 'name', bodyField: 'body', - paramsField: 'parameter_list', + paramsField: 'parameters', + returnField: 'type', getVisibility: (node) => { for (let i = 0; i < node.childCount; i++) { const child = node.child(i); diff --git a/src/extraction/languages/go.ts b/src/extraction/languages/go.ts index 898e61651..5b4d7975d 100644 --- a/src/extraction/languages/go.ts +++ b/src/extraction/languages/go.ts @@ -36,6 +36,18 @@ export const goExtractor: LanguageExtractor = { if (typeChild.type === 'interface_type') return 'interface'; return undefined; }, + isExported: (node, source) => { + // Go: a symbol is exported when its identifier starts with an uppercase letter. + // Look at the `name` field directly (works for function_declaration, + // method_declaration, type_spec, and var_spec / const_spec via extractor flow). + const nameNode = getChildByField(node, 'name'); + if (nameNode) { + const text = getNodeText(nameNode, source); + const first = text.charCodeAt(0); + return first >= 65 && first <= 90; // A-Z + } + return false; + }, getReceiverType: (node, source) => { // Go method_declaration has a "receiver" field: func (sl *scrapeLoop) run(...) // The receiver is a parameter_list containing a parameter_declaration diff --git a/src/extraction/languages/index.ts b/src/extraction/languages/index.ts index 1b82262e4..543598b8e 100644 --- a/src/extraction/languages/index.ts +++ b/src/extraction/languages/index.ts @@ -23,6 +23,9 @@ import { kotlinExtractor } from './kotlin'; import { dartExtractor } from './dart'; import { pascalExtractor } from './pascal'; import { scalaExtractor } from './scala'; +import { luaExtractor } from './lua'; +import { luauExtractor } from './luau'; +import { objcExtractor } from './objc'; export const EXTRACTORS: Partial> = { typescript: typescriptExtractor, @@ -43,4 +46,7 @@ export const EXTRACTORS: Partial> = { dart: dartExtractor, pascal: pascalExtractor, scala: scalaExtractor, + lua: luaExtractor, + luau: luauExtractor, + objc: objcExtractor, }; diff --git a/src/extraction/languages/java.ts b/src/extraction/languages/java.ts index 638533f0d..4e8cbc7a1 100644 --- a/src/extraction/languages/java.ts +++ b/src/extraction/languages/java.ts @@ -56,4 +56,12 @@ export const javaExtractor: LanguageExtractor = { } return null; }, + packageTypes: ['package_declaration'], + extractPackage: (node, source) => { + // package_declaration → scoped_identifier or identifier (single-segment) + const id = node.namedChildren.find( + (c: SyntaxNode) => c.type === 'scoped_identifier' || c.type === 'identifier' + ); + return id ? source.substring(id.startIndex, id.endIndex).trim() : null; + }, }; diff --git a/src/extraction/languages/kotlin.ts b/src/extraction/languages/kotlin.ts index 19c386242..e590d4481 100644 --- a/src/extraction/languages/kotlin.ts +++ b/src/extraction/languages/kotlin.ts @@ -235,4 +235,10 @@ export const kotlinExtractor: LanguageExtractor = { } return null; }, + packageTypes: ['package_header'], + extractPackage: (node, source) => { + // package_header → identifier (dotted: `com.example.foo`) + const id = node.namedChildren.find((c: SyntaxNode) => c.type === 'identifier'); + return id ? source.substring(id.startIndex, id.endIndex).trim() : null; + }, }; diff --git a/src/extraction/languages/lua.ts b/src/extraction/languages/lua.ts new file mode 100644 index 000000000..31094dc1a --- /dev/null +++ b/src/extraction/languages/lua.ts @@ -0,0 +1,152 @@ +import type { Node as SyntaxNode } from 'web-tree-sitter'; +import { getNodeText, getChildByField } from '../tree-sitter-helpers'; +import type { LanguageExtractor } from '../tree-sitter-types'; + +// Node names follow the vendored ABI-15 grammar (@tree-sitter-grammars/ +// tree-sitter-lua), NOT the older tree-sitter-wasms build — see grammars.ts. + +/** First descendant of a given type (breadth-first), or null. */ +function findDescendant(node: SyntaxNode, type: string): SyntaxNode | null { + const queue: SyntaxNode[] = [...node.namedChildren]; + while (queue.length) { + const n = queue.shift()!; + if (n.type === type) return n; + queue.push(...n.namedChildren); + } + return null; +} + +/** + * If `callNode` is a `require(...)` call, return the module name; otherwise null. + * Lua/Luau have no import statement — modules are loaded by calling the global + * `require`. Handles both: + * - string requires: `require("net.http")` / `require "net.http"` → "net.http" + * - Roblox/Luau path requires: `require(script.Parent.Signal)` → "Signal" + * (the dominant idiom in Roblox code, where the argument is an instance path + * rather than a string — use the trailing field as the module name). + */ +function requireModule(callNode: SyntaxNode, source: string): string | null { + // function_call > name: , arguments: arguments + const name = getChildByField(callNode, 'name'); + // A dotted/colon callee (e.g. `socket.connect`) is dot/method_index_expression, + // never a bare `require`. + if (!name || name.type !== 'identifier') return null; + if (getNodeText(name, source) !== 'require') return null; + + const args = getChildByField(callNode, 'arguments'); + if (!args) return null; + + // String require — `string > content: string_content` gives the bare name. + const content = findDescendant(args, 'string_content'); + if (content) return getNodeText(content, source).trim() || null; + const str = findDescendant(args, 'string'); + if (str) { + const mod = getNodeText(str, source) + .trim() + .replace(/^\[\[/, '') + .replace(/\]\]$/, '') + .replace(/^["']/, '') + .replace(/["']$/, ''); + if (mod) return mod; + } + + // Roblox/Luau instance-path require: `require(script.Parent.Signal)` → "Signal". + const idx = findDescendant(args, 'dot_index_expression') ?? findDescendant(args, 'method_index_expression'); + if (idx) { + const field = getChildByField(idx, 'field') ?? getChildByField(idx, 'method'); + if (field) return getNodeText(field, source).trim() || null; + } + return null; +} + +export const luaExtractor: LanguageExtractor = { + // function_declaration covers global (`function f`), table (`function t.f`), + // method (`function t:m`), and local (`local function f`) forms — the form is + // distinguished by the `name:` child (identifier / dot_index_expression / + // method_index_expression) and a `local` token, not by separate node types. + // Anonymous `function() ... end` (function_definition) has no name and is + // captured via its enclosing variable instead. + functionTypes: ['function_declaration'], + classTypes: [], // Lua has no classes/structs/interfaces/enums — tables are used for everything + methodTypes: [], + interfaceTypes: [], + structTypes: [], + enumTypes: [], + typeAliasTypes: [], + importTypes: [], // `require` is a function_call — handled in visitNode below + callTypes: ['function_call'], + variableTypes: ['variable_declaration'], // see the `lua` branch in extractVariable + nameField: 'name', + bodyField: 'body', + paramsField: 'parameters', + + getSignature: (node, source) => { + const params = getChildByField(node, 'parameters'); + return params ? getNodeText(params, source) : undefined; + }, + + // `function t.f()` / `function t:m()` are methods on table `t`: return the + // table as the receiver so they extract as methods with a `t::f` qualified + // name. Plain `function f()` / `local function f()` have no receiver and stay + // functions. (For `a.b.c`, the receiver is the nested `a.b`.) + getReceiverType: (node, source) => { + const name = getChildByField(node, 'name'); + if (name && (name.type === 'dot_index_expression' || name.type === 'method_index_expression')) { + const table = getChildByField(name, 'table'); + if (table) return getNodeText(table, source); + } + return undefined; + }, + + // Emit import nodes for `require(...)`. The local-declaration form is handled + // explicitly because the variable branch skips the initializer subtree; bare + // and global `require` calls are caught when the walker reaches the + // function_call node. + visitNode: (node, ctx) => { + const source = ctx.source; + + const emit = (callNode: SyntaxNode): void => { + const mod = requireModule(callNode, source); + if (!mod) return; + const imp = ctx.createNode('import', mod, callNode, { + signature: getNodeText(callNode, source).trim().slice(0, 100), + }); + if (imp && ctx.nodeStack.length > 0) { + const parentId = ctx.nodeStack[ctx.nodeStack.length - 1]; + if (parentId) { + ctx.addUnresolvedReference({ + fromNodeId: parentId, + referenceName: mod, + referenceKind: 'imports', + line: callNode.startPosition.row + 1, + column: callNode.startPosition.column, + }); + } + } + }; + + // Bare / global `require("x")` — claim it so it isn't double-counted as a call. + if (node.type === 'function_call') { + if (requireModule(node, source)) { + emit(node); + return true; + } + return false; + } + + // `local x = require("x")` — variable_declaration wraps an assignment_statement + // whose initializer subtree the variable branch will skip, so dig it out here. + if (node.type === 'variable_declaration') { + const assign = node.namedChildren.find((c) => c.type === 'assignment_statement'); + const exprList = assign?.namedChildren.find((c) => c.type === 'expression_list'); + if (exprList) { + for (const val of exprList.namedChildren) { + if (val.type === 'function_call') emit(val); + } + } + return false; + } + + return false; + }, +}; diff --git a/src/extraction/languages/luau.ts b/src/extraction/languages/luau.ts new file mode 100644 index 000000000..f4f51a1f9 --- /dev/null +++ b/src/extraction/languages/luau.ts @@ -0,0 +1,36 @@ +import { getNodeText, getChildByField } from '../tree-sitter-helpers'; +import type { LanguageExtractor } from '../tree-sitter-types'; +import { luaExtractor } from './lua'; + +// Luau (https://luau.org) is a gradually-typed superset of Lua. The +// tree-sitter-luau grammar reuses the same node names as the vendored Lua +// grammar (function_declaration, variable_declaration, function_call, +// dot/method_index_expression, …), so the Luau extractor extends the Lua one +// and adds the type-system pieces Luau introduces: +// - `type X = ...` / `export type X = ...` → type_definition (type_alias) +// - typed parameters and return types → richer signatures +// +// require detection, receiver-splitting (t.f / t:m → methods), and local +// variable extraction are inherited unchanged from luaExtractor. The shared +// `extractVariable` core branch is gated on `lua` || `luau`. +export const luauExtractor: LanguageExtractor = { + ...luaExtractor, + + // `type X = ...` and `export type X = ...` + typeAliasTypes: ['type_definition'], + + // Only Luau `export type` is exported; the keyword leads the node. + isExported: (node, source) => source.slice(node.startIndex, node.startIndex + 7) === 'export ', + + // Params + Luau return type (the named child after `parameters`, before the body). + getSignature: (node, source) => { + const params = getChildByField(node, 'parameters'); + if (!params) return undefined; + let sig = getNodeText(params, source); + const kids = node.namedChildren; + const idx = kids.findIndex((c) => c.startIndex === params.startIndex); + const ret = idx >= 0 ? kids[idx + 1] : null; + if (ret && ret.type !== 'block') sig += `: ${getNodeText(ret, source)}`; + return sig; + }, +}; diff --git a/src/extraction/languages/objc.ts b/src/extraction/languages/objc.ts new file mode 100644 index 000000000..6671284aa --- /dev/null +++ b/src/extraction/languages/objc.ts @@ -0,0 +1,136 @@ +import type { Node as SyntaxNode } from 'web-tree-sitter'; +import { getChildByField, getNodeText } from '../tree-sitter-helpers'; +import type { ExtractorContext, LanguageExtractor } from '../tree-sitter-types'; + +function findCompoundStatement(node: SyntaxNode): SyntaxNode | null { + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + if (child?.type === 'compound_statement') { + return child; + } + } + return null; +} + +/** Build ObjC selector: `greet`, `doThing:`, or `doThing:with:`. */ +function extractObjcMethodName(node: SyntaxNode, source: string): string | undefined { + if (node.type !== 'method_definition' && node.type !== 'method_declaration') { + return undefined; + } + + const identifiers = node.namedChildren.filter((c) => c.type === 'identifier'); + if (identifiers.length === 0) return undefined; + + const hasParameters = node.namedChildren.some((c) => c.type === 'method_parameter'); + const firstIdentifier = identifiers[0]; + if (!firstIdentifier) return undefined; + if (!hasParameters) { + return getNodeText(firstIdentifier, source); + } + + return identifiers.map((id) => `${getNodeText(id, source)}:`).join(''); +} + +function extractObjcPropertyName(node: SyntaxNode, source: string): string | null { + if (node.type !== 'property_declaration') return null; + + const structDecl = node.namedChildren.find((c) => c.type === 'struct_declaration'); + if (!structDecl) return null; + + const structDeclarator = structDecl.namedChildren.find((c) => c.type === 'struct_declarator'); + if (!structDeclarator) return null; + + let current: SyntaxNode | null = structDeclarator; + while (current) { + const inner: SyntaxNode | undefined = + getChildByField(current, 'declarator') || + current.namedChildren.find((c) => c.type === 'identifier' || c.type === 'pointer_declarator'); + if (!inner) break; + if (inner.type === 'identifier') { + return getNodeText(inner, source); + } + current = inner; + } + + return null; +} + +export const objcExtractor: LanguageExtractor = { + functionTypes: ['function_definition'], + // Only @interface emits a class node; @implementation reuses it via visitNode. + classTypes: ['class_interface'], + methodTypes: ['method_definition'], + interfaceTypes: ['protocol_declaration'], + interfaceKind: 'protocol', + structTypes: ['struct_specifier'], + enumTypes: ['enum_specifier'], + enumMemberTypes: ['enumerator'], + typeAliasTypes: ['type_definition'], + importTypes: ['preproc_include'], + callTypes: ['call_expression', 'message_expression'], + variableTypes: ['declaration'], + propertyTypes: ['property_declaration'], + nameField: 'declarator', + bodyField: 'body', + paramsField: 'parameters', + resolveName: extractObjcMethodName, + extractPropertyName: extractObjcPropertyName, + resolveBody: (node, bodyField) => { + const fromField = getChildByField(node, bodyField); + if (fromField) { + return fromField; + } + return findCompoundStatement(node); + }, + resolveTypeAliasKind: (node, _source) => { + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + if (!child) continue; + if (child.type === 'enum_specifier' && getChildByField(child, 'body')) return 'enum'; + if (child.type === 'struct_specifier' && getChildByField(child, 'body')) return 'struct'; + } + return undefined; + }, + isStatic: (node) => /^\s*\+/.test(node.text), + visitNode: (node, ctx: ExtractorContext) => { + if (node.type !== 'class_implementation') return false; + + const classNameNode = node.namedChildren.find((c) => c.type === 'identifier'); + if (!classNameNode) return true; + + const className = getNodeText(classNameNode, ctx.source); + const classNode = + ctx.nodes.find( + (n) => n.name === className && n.filePath === ctx.filePath && n.kind === 'class' + ) ?? ctx.createNode('class', className, node, {}); + if (!classNode) return true; + + ctx.pushScope(classNode.id); + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + if (child?.type === 'implementation_definition') { + for (let j = 0; j < child.namedChildCount; j++) { + const implChild = child.namedChild(j); + if (implChild) ctx.visitNode(implChild); + } + } + } + ctx.popScope(); + return true; + }, + extractImport: (node, source) => { + const importText = source.substring(node.startIndex, node.endIndex).trim(); + const systemLib = node.namedChildren.find((c: SyntaxNode) => c.type === 'system_lib_string'); + if (systemLib) { + return { moduleName: getNodeText(systemLib, source).replace(/^<|>$/g, ''), signature: importText }; + } + const stringLiteral = node.namedChildren.find((c: SyntaxNode) => c.type === 'string_literal'); + if (stringLiteral) { + const stringContent = stringLiteral.namedChildren.find((c: SyntaxNode) => c.type === 'string_content'); + if (stringContent) { + return { moduleName: getNodeText(stringContent, source), signature: importText }; + } + } + return null; + }, +}; diff --git a/src/extraction/mybatis-extractor.ts b/src/extraction/mybatis-extractor.ts new file mode 100644 index 000000000..fc873e1dc --- /dev/null +++ b/src/extraction/mybatis-extractor.ts @@ -0,0 +1,198 @@ +import { Edge, ExtractionError, ExtractionResult, Node, UnresolvedReference } from '../types'; +import { generateNodeId } from './tree-sitter-helpers'; + +/** + * MyBatisExtractor — parses MyBatis mapper XML files. + * + * MyBatis splits a DAO interface across two files: a Java interface (parsed by + * tree-sitter) declares the method, and an XML mapper file holds the SQL keyed + * by `` (the fully-qualified Java type name) and `id` (the method + * name). Without the XML side in the graph, `trace(Controller, ...DAO.method)` + * dead-ends at the interface method — the SQL it actually runs is invisible, + * and "what does this query touch" / "where is this column written" can't be + * answered. + * + * This extractor emits one method-shaped node per `` and per `` fragment, qualified as `::` so the + * MyBatis framework synthesizer (`src/resolution/frameworks/mybatis.ts`) can + * link the matching Java method → XML statement by suffix-matching qualified + * names. `` inside a statement yields an unresolved + * reference to the SQL fragment, also keyed by `::`. + * + * Non-mapper XML (Maven `pom.xml`, Spring beans XML, `web.xml`, log4j config, + * etc.) is detected by the absence of a `` root and + * returns just a file node — we still need the file row so the watcher can + * track it, but we emit no symbols. + */ +export class MyBatisExtractor { + private filePath: string; + private source: string; + private nodes: Node[] = []; + private edges: Edge[] = []; + private unresolvedReferences: UnresolvedReference[] = []; + private errors: ExtractionError[] = []; + private lineStarts: number[] = []; + + constructor(filePath: string, source: string) { + this.filePath = filePath; + this.source = source; + this.computeLineStarts(); + } + + extract(): ExtractionResult { + const startTime = Date.now(); + + const fileNode = this.createFileNode(); + + try { + const mapperMatch = this.findMapperRoot(); + if (mapperMatch) { + this.extractMapper(fileNode.id, mapperMatch.namespace, mapperMatch.bodyStart, mapperMatch.bodyEnd); + } + } catch (error) { + this.errors.push({ + message: `MyBatis extraction error: ${error instanceof Error ? error.message : String(error)}`, + severity: 'error', + code: 'parse_error', + }); + } + + return { + nodes: this.nodes, + edges: this.edges, + unresolvedReferences: this.unresolvedReferences, + errors: this.errors, + durationMs: Date.now() - startTime, + }; + } + + private createFileNode(): Node { + const lines = this.source.split('\n'); + const id = generateNodeId(this.filePath, 'file', this.filePath, 1); + const node: Node = { + id, + kind: 'file', + name: this.filePath.split('/').pop() || this.filePath, + qualifiedName: this.filePath, + filePath: this.filePath, + language: 'xml', + startLine: 1, + endLine: lines.length || 1, + startColumn: 0, + endColumn: lines[lines.length - 1]?.length ?? 0, + updatedAt: Date.now(), + }; + this.nodes.push(node); + return node; + } + + /** + * Find the `` opening tag. Returns the namespace and + * the byte offsets of the body (between the opening and closing tag) so + * statement extraction can be scoped to mapper contents. + */ + private findMapperRoot(): { namespace: string; bodyStart: number; bodyEnd: number } | null { + const open = /]*)>/.exec(this.source); + if (!open) return null; + const attrs = open[1] ?? ''; + const nsMatch = /\bnamespace\s*=\s*"([^"]+)"/.exec(attrs); + if (!nsMatch) return null; + const bodyStart = open.index + open[0].length; + const closeIdx = this.source.indexOf('', bodyStart); + const bodyEnd = closeIdx >= 0 ? closeIdx : this.source.length; + return { namespace: nsMatch[1]!, bodyStart, bodyEnd }; + } + + private extractMapper(fileNodeId: string, namespace: string, bodyStart: number, bodyEnd: number): void { + const body = this.source.slice(bodyStart, bodyEnd); + // Match each top-level statement-shaped element. The body may have nested + // tags (``, ``, ``), so we scan with a regex that + // pairs an opening tag to its matching close — the simple form below works + // because MyBatis statement elements are not themselves nested. + const stmtRegex = /<(select|insert|update|delete|sql)\b([^>]*)>([\s\S]*?)<\/\1>/g; + let m: RegExpExecArray | null; + while ((m = stmtRegex.exec(body)) !== null) { + const elemType = m[1]!; + const attrs = m[2] ?? ''; + const elemBody = m[3] ?? ''; + const idMatch = /\bid\s*=\s*"([^"]+)"/.exec(attrs); + if (!idMatch) continue; + const id = idMatch[1]!; + const absoluteIndex = bodyStart + m.index; + const startLine = this.getLineNumber(absoluteIndex); + const endLine = this.getLineNumber(absoluteIndex + m[0].length); + const qualified = `${namespace}::${id}`; + const isSqlFragment = elemType === 'sql'; + const nodeId = generateNodeId(this.filePath, 'method', qualified, startLine); + const node: Node = { + id: nodeId, + kind: 'method', + name: id, + qualifiedName: qualified, + filePath: this.filePath, + language: 'xml', + signature: this.buildSignature(elemType, attrs, isSqlFragment), + startLine, + endLine, + startColumn: 0, + endColumn: 0, + docstring: this.previewSql(elemBody), + updatedAt: Date.now(), + }; + this.nodes.push(node); + this.edges.push({ source: fileNodeId, target: nodeId, kind: 'contains' }); + + // → reference to the SQL fragment in this mapper + // (or in another mapper, when the refid is qualified — `ns.X`). + const includeRegex = /]*\brefid\s*=\s*"([^"]+)"/g; + let inc: RegExpExecArray | null; + while ((inc = includeRegex.exec(elemBody)) !== null) { + const refid = inc[1]!; + const refQualified = refid.includes('.') ? refid.replace(/\./g, '::') : `${namespace}::${refid}`; + const includeOffset = absoluteIndex + (m[0].length - m[3]!.length - ``.length) + inc.index; + const line = this.getLineNumber(includeOffset); + this.unresolvedReferences.push({ + fromNodeId: nodeId, + referenceName: refQualified, + referenceKind: 'references', + line, + column: 0, + }); + } + } + } + + private buildSignature(elemType: string, attrs: string, isSqlFragment: boolean): string { + if (isSqlFragment) return ''; + const verb = elemType.toUpperCase(); + const result = /\bresultType\s*=\s*"([^"]+)"/.exec(attrs)?.[1]; + const param = /\bparameterType\s*=\s*"([^"]+)"/.exec(attrs)?.[1]; + const parts = [verb]; + if (param) parts.push(`param=${param}`); + if (result) parts.push(`result=${result}`); + return parts.join(' '); + } + + private previewSql(body: string): string { + return body.replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim().slice(0, 200); + } + + private computeLineStarts(): void { + this.lineStarts = [0]; + for (let i = 0; i < this.source.length; i++) { + if (this.source.charCodeAt(i) === 10) this.lineStarts.push(i + 1); + } + } + + private getLineNumber(offset: number): number { + // Binary search + let lo = 0; + let hi = this.lineStarts.length - 1; + while (lo < hi) { + const mid = (lo + hi + 1) >>> 1; + if (this.lineStarts[mid]! <= offset) lo = mid; + else hi = mid - 1; + } + return lo + 1; + } +} diff --git a/src/extraction/tree-sitter-types.ts b/src/extraction/tree-sitter-types.ts index c3a6b94e8..6c04fbaeb 100644 --- a/src/extraction/tree-sitter-types.ts +++ b/src/extraction/tree-sitter-types.ts @@ -120,6 +120,12 @@ export interface LanguageExtractor { // --- Existing hooks --- + /** Override symbol name extraction (e.g. ObjC multi-part selectors). */ + resolveName?: (node: SyntaxNode, source: string) => string | undefined; + + /** Extract property name when the generic name walk fails (e.g. ObjC @property). */ + extractPropertyName?: (node: SyntaxNode, source: string) => string | null; + /** Extract signature from node */ getSignature?: (node: SyntaxNode, source: string) => string | undefined; /** Extract visibility from node */ @@ -206,4 +212,16 @@ export interface LanguageExtractor { * Returns the callee name if this node is a bare call, or undefined if not. */ extractBareCall?: (node: SyntaxNode, source: string) => string | undefined; + + /** + * Node types representing a file-level package/namespace declaration + * (e.g. Kotlin `package_header`, Java `package_declaration`). When set, + * the core wraps every top-level declaration in an implicit `namespace` + * node carrying the FQN, so cross-file import resolution can match by + * qualifiedName instead of filename (Kotlin filename ≠ class name). + */ + packageTypes?: string[]; + + /** Extract the dotted package name from a package declaration node. */ + extractPackage?: (node: SyntaxNode, source: string) => string | null; } diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 00830ab80..f576839fa 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -15,7 +15,7 @@ import { ExtractionError, UnresolvedReference, } from '../types'; -import { getParser, detectLanguage, isLanguageSupported } from './grammars'; +import { getParser, detectLanguage, isLanguageSupported, isFileLevelOnlyLanguage } from './grammars'; import { generateNodeId, getNodeText, getChildByField, getPrecedingDocstring } from './tree-sitter-helpers'; import type { LanguageExtractor, ExtractorContext } from './tree-sitter-types'; import { EXTRACTORS } from './languages'; @@ -23,6 +23,7 @@ import { LiquidExtractor } from './liquid-extractor'; import { SvelteExtractor } from './svelte-extractor'; import { DfmExtractor } from './dfm-extractor'; import { VueExtractor } from './vue-extractor'; +import { MyBatisExtractor } from './mybatis-extractor'; import { getAllFrameworkResolvers, getApplicableFrameworks, @@ -35,6 +36,9 @@ export { generateNodeId } from './tree-sitter-helpers'; * Extract the name from a node based on language */ function extractName(node: SyntaxNode, source: string, extractor: LanguageExtractor): string { + const hookName = extractor.resolveName?.(node, source); + if (hookName) return hookName; + // Try field name first const nameNode = getChildByField(node, extractor.nameField); if (nameNode) { @@ -50,6 +54,17 @@ function extractName(node: SyntaxNode, source: string, extractor: LanguageExtrac const innerName = getChildByField(resolved, 'declarator') || resolved.namedChild(0); return innerName ? getNodeText(innerName, source) : getNodeText(resolved, source); } + // Lua: `function t.f()` / `function t:m()` — the name node is a dot/method + // index expression; the simple name is the trailing field/method (the table + // receiver is captured separately via getReceiverType). + if (resolved.type === 'dot_index_expression') { + const field = getChildByField(resolved, 'field'); + if (field) return getNodeText(field, source); + } + if (resolved.type === 'method_index_expression') { + const method = getChildByField(resolved, 'method'); + if (method) return getNodeText(method, source); + } return getNodeText(resolved, source); } @@ -200,7 +215,17 @@ export class TreeSitterExtractor { // Push file node onto stack so top-level declarations get contains edges this.nodeStack.push(fileNode.id); + + // File-level package declaration (Kotlin/Java). Creates an implicit + // `namespace` node wrapping every top-level declaration so their + // qualifiedName carries the FQN — required for cross-file import + // resolution on JVM languages where filename ≠ class name. + const packageNodeId = this.extractFilePackage(this.tree.rootNode); + if (packageNodeId) this.nodeStack.push(packageNodeId); + this.visitNode(this.tree.rootNode); + + if (packageNodeId) this.nodeStack.pop(); this.nodeStack.pop(); } catch (error) { const msg = error instanceof Error ? error.message : String(error); @@ -363,6 +388,17 @@ export class TreeSitterExtractor { // their own `calls` refs. else if (INSTANTIATION_KINDS.has(nodeType)) { this.extractInstantiation(node); + // Java/C# `new T(...) { ... }` — anonymous class with body. Without + // extracting it as a class node + its methods, the interface→impl + // synthesizer (Phase 5.5) can't bridge T's abstract methods to the + // anonymous overrides, and an agent investigating a call through T + // (`strategy.iterator(...)` where strategy is a Strategy lambda body) + // has to Read the file to find the actual implementation. + const anonBody = this.findAnonymousClassBody(node); + if (anonBody) { + this.extractAnonymousClass(node, anonBody); + skipChildren = true; + } } // (Decorator handling lives inside the symbol-creating extractors // — extractClass / extractFunction / extractProperty — because the @@ -372,6 +408,22 @@ export class TreeSitterExtractor { else if (nodeType === 'impl_item') { this.extractRustImplItem(node); } + // TypeScript interface members: property_signature (`foo: T`, `foo?: T`) + // and method_signature (`foo(arg: A): R`) both carry type annotations the + // interface walker would otherwise drop. Extract them as `references` + // edges from the interface so resolvers can wire callers/impact for + // types that only appear in interface members. + else if ( + (nodeType === 'property_signature' || nodeType === 'method_signature') && + this.isInsideClassLikeNode() && + this.TYPE_ANNOTATION_LANGUAGES.has(this.language) + ) { + const parentId = this.nodeStack[this.nodeStack.length - 1]; + if (parentId) { + this.extractTypeAnnotations(node, parentId); + } + // don't skipChildren — nested signatures still need traversal + } // Visit children (unless the extract method already visited them) if (!skipChildren) { @@ -401,6 +453,20 @@ export class TreeSitterExtractor { const id = generateNodeId(this.filePath, kind, name, node.startPosition.row + 1); + // Some grammars (e.g. Dart) model a function/method body as a *sibling* of + // the signature node, so the declaration node's own range is just the + // signature line. Extend endLine to the resolved body when it sits beyond + // the node so the node spans its body — required for any body-level analysis + // (callees, the callback synthesizer's body scan, context slices). Guarded to + // only ever extend: for child-body grammars the body is within range (no-op). + let endLine = node.endPosition.row + 1; + if (kind === 'function' || kind === 'method') { + const body = this.extractor?.resolveBody?.(node, this.extractor.bodyField); + if (body && body.endPosition.row + 1 > endLine) { + endLine = body.endPosition.row + 1; + } + } + const newNode: Node = { id, kind, @@ -409,7 +475,7 @@ export class TreeSitterExtractor { filePath: this.filePath, language: this.language, startLine: node.startPosition.row + 1, - endLine: node.endPosition.row + 1, + endLine, startColumn: node.startPosition.column, endColumn: node.endPosition.column, updatedAt: Date.now(), @@ -445,6 +511,33 @@ export class TreeSitterExtractor { return null; } + /** + * Find a `packageTypes` child under the root, create a `namespace` node + * for it, and return its id so the caller can scope top-level + * declarations underneath. Returns null when no package header is + * present (script files, .kts without a package). + */ + private extractFilePackage(rootNode: SyntaxNode): string | null { + const types = this.extractor?.packageTypes; + if (!types || types.length === 0 || !this.extractor?.extractPackage) return null; + + let pkgNode: SyntaxNode | null = null; + for (let i = 0; i < rootNode.namedChildCount; i++) { + const child = rootNode.namedChild(i); + if (child && types.includes(child.type)) { + pkgNode = child; + break; + } + } + if (!pkgNode) return null; + + const pkgName = this.extractor.extractPackage(pkgNode, this.source); + if (!pkgName) return null; + + const ns = this.createNode('namespace', pkgName, pkgNode); + return ns?.id ?? null; + } + /** * Build qualified name from node stack */ @@ -505,7 +598,7 @@ export class TreeSitterExtractor { /** * Extract a function */ - private extractFunction(node: SyntaxNode): void { + private extractFunction(node: SyntaxNode, nameOverride?: string): void { if (!this.extractor) return; // If the language provides getReceiverType and this function has a receiver @@ -515,12 +608,17 @@ export class TreeSitterExtractor { return; } - let name = extractName(node, this.source, this.extractor); + // nameOverride is supplied only for explicitly-named anonymous functions the + // caller resolved itself (e.g. arrow values of exported-const object members + // — SvelteKit actions). Inline-object arrows reached by the general walker + // get no override, so they still fall through to the skip below. + let name = nameOverride ?? extractName(node, this.source, this.extractor); // For arrow functions and function expressions assigned to variables, // resolve the name from the parent variable_declarator. // e.g. `export const useAuth = () => { ... }` — the arrow_function node // has no `name` field; the name lives on the variable_declarator. if ( + !nameOverride && name === '' && (node.type === 'arrow_function' || node.type === 'function_expression') ) { @@ -638,6 +736,11 @@ export class TreeSitterExtractor { // in inline objects). These are ephemeral and create noise (e.g., Svelte context // objects: `ctx.set({ get view() { ... } })`). if (node.parent?.type === 'object' || node.parent?.type === 'object_expression') { + const body = this.extractor.resolveBody?.(node, this.extractor.bodyField) + ?? getChildByField(node, this.extractor.bodyField); + if (body) { + this.visitFunctionBody(body, ''); + } return; } // Not inside a class-like node and no receiver type, treat as function @@ -863,12 +966,12 @@ export class TreeSitterExtractor { const visibility = this.extractor.getVisibility?.(node); const isStatic = this.extractor.isStatic?.(node) ?? false; - // Property name is a direct identifier child - const nameNode = getChildByField(node, 'name') - || node.namedChildren.find(c => c.type === 'identifier'); - if (!nameNode) return; - - const name = getNodeText(nameNode, this.source); + const hookName = this.extractor.extractPropertyName?.(node, this.source); + const nameNode = hookName + ? null + : getChildByField(node, 'name') || node.namedChildren.find(c => c.type === 'identifier'); + const name = hookName ?? (nameNode ? getNodeText(nameNode, this.source) : null); + if (!name) return; // Get property type from the type child (first named child that isn't modifier or identifier) const typeNode = node.namedChildren.find( @@ -890,6 +993,10 @@ export class TreeSitterExtractor { // decorator->target relationship for class properties too. if (propNode) { this.extractDecoratorsFor(node, propNode.id); + // Emit `references` edges from the property to types named in its + // type annotation (#381). The generic walker handles TS-style + // `type_annotation` children; the C# branch walks the `type` field. + this.extractTypeAnnotations(node, propNode.id); } } @@ -972,7 +1079,15 @@ export class TreeSitterExtractor { }); // Java/Kotlin annotations / TS field decorators sit on the // outer field_declaration, not on the individual declarator. - if (fieldNode) this.extractDecoratorsFor(node, fieldNode.id); + if (fieldNode) { + this.extractDecoratorsFor(node, fieldNode.id); + // Same as properties: emit `references` to the field's annotated + // type. The outer `field_declaration` is the right scope to + // search from — C# carries the `type` inside `variable_declaration` + // and the language-aware path in `extractTypeAnnotations` descends + // into that wrapper (#381). + this.extractTypeAnnotations(node, fieldNode.id); + } } } else { // Fallback: try to find an identifier child directly @@ -1046,6 +1161,31 @@ export class TreeSitterExtractor { if (varNode) { this.extractVariableTypeAnnotation(child, varNode.id); } + + if (valueNode && + valueNode.type !== 'object' && + valueNode.type !== 'object_expression') { + this.visitFunctionBody(valueNode, ''); + } + + // Exported const object-of-functions: `export const actions = + // { default: async () => {} }` (SvelteKit form actions / handler maps + // / route tables). Extract each function-valued property as a function + // named by its key + walk its body so its calls (e.g. api.post) are + // captured. Scoped to EXPORTED consts to exclude the inline-object + // noise (`ctx.set({...})`) the object-method skip deliberately avoids. + if (isExported && valueNode && + (valueNode.type === 'object' || valueNode.type === 'object_expression')) { + for (let j = 0; j < valueNode.namedChildCount; j++) { + const pair = valueNode.namedChild(j); + if (pair?.type !== 'pair') continue; + const v = getChildByField(pair, 'value'); + const k = getChildByField(pair, 'key'); + if (k && v && (v.type === 'arrow_function' || v.type === 'function_expression')) { + this.extractFunction(v, getNodeText(k, this.source).replace(/^['"`]|['"`]$/g, '')); + } + } + } } } } @@ -1111,6 +1251,23 @@ export class TreeSitterExtractor { } } } + } else if (this.language === 'lua' || this.language === 'luau') { + // Lua/Luau: variable_declaration → assignment_statement → variable_list + // (name: identifier...) = expression_list. `local x, y = 1, 2` + // declares multiple names; only plain identifiers are locals. + const assign = node.namedChildren.find((c) => c.type === 'assignment_statement') ?? node; + const varList = assign.namedChildren.find((c) => c.type === 'variable_list'); + const exprList = assign.namedChildren.find((c) => c.type === 'expression_list'); + const values = exprList ? exprList.namedChildren : []; + const names = varList ? varList.namedChildren.filter((c) => c.type === 'identifier') : []; + names.forEach((nameNode, i) => { + const name = getNodeText(nameNode, this.source); + if (!name) return; + const valueNode = values[i]; + const initValue = valueNode ? getNodeText(valueNode, this.source).slice(0, 100) : undefined; + const initSignature = initValue ? `= ${initValue}${initValue.length >= 100 ? '...' : ''}` : undefined; + this.createNode(kind, name, nameNode, { docstring, signature: initSignature, isExported }); + }); } else { // Generic fallback for other languages // Try to find identifier children @@ -1220,11 +1377,89 @@ export class TreeSitterExtractor { const value = getChildByField(node, 'value'); if (value) { this.extractTypeRefsFromSubtree(value, typeAliasNode.id); + // `type X = { foo: T; bar(): T }` — make the members first-class + // property/method nodes under the type alias so `recorder.stop()` + // can attach the call edge to `RecorderHandle.stop` instead of + // an unrelated class method picked by path-proximity (#359). + if (this.language === 'typescript' || this.language === 'tsx') { + this.extractTsTypeAliasMembers(value, typeAliasNode); + } } } return false; } + /** + * Surface the members of a TypeScript `type X = { ... }` (or intersection + * thereof) as `property` / `method` nodes under the type-alias node. Only + * walks the immediate object_type / intersection operands so anonymous + * nested object types inside generic arguments (`Promise<{ ok: true }>`) + * don't produce phantom members. + */ + private extractTsTypeAliasMembers(value: SyntaxNode, typeAliasNode: Node): void { + const objectTypes: SyntaxNode[] = []; + if (value.type === 'object_type') { + objectTypes.push(value); + } else if (value.type === 'intersection_type') { + for (let i = 0; i < value.namedChildCount; i++) { + const op = value.namedChild(i); + if (op && op.type === 'object_type') objectTypes.push(op); + } + } else { + return; + } + + this.nodeStack.push(typeAliasNode.id); + for (const objType of objectTypes) { + for (let i = 0; i < objType.namedChildCount; i++) { + const child = objType.namedChild(i); + if (!child) continue; + if (child.type !== 'property_signature' && child.type !== 'method_signature') continue; + + const nameNode = getChildByField(child, 'name'); + const memberName = nameNode ? getNodeText(nameNode, this.source) : ''; + if (!memberName) continue; + + // `foo: () => T` and `foo(): T` are functionally a method on the + // type contract. Treat the property_signature with a function-typed + // annotation as a method too so call sites can resolve to it. + const memberKind: NodeKind = child.type === 'method_signature' + ? 'method' + : this.isTsFunctionTypedProperty(child) ? 'method' : 'property'; + + const docstring = getPrecedingDocstring(child, this.source); + const signature = getNodeText(child, this.source); + this.createNode(memberKind, memberName, child, { + docstring, + signature, + qualifiedName: `${typeAliasNode.name}::${memberName}`, + }); + + // Emit `references` edges from the type alias to types named in the + // member's signature, matching the interface-member behavior added in + // #432. We attach refs to the type-alias parent (consistent with + // interface property_signature treatment). + this.extractTypeAnnotations(child, typeAliasNode.id); + } + } + this.nodeStack.pop(); + } + + /** + * `foo: () => T` → property_signature whose type_annotation contains a + * `function_type`. Treat that as a method-shaped contract member, since + * the call site `obj.foo()` has identical semantics to `bar(): T`. + */ + private isTsFunctionTypedProperty(propertySignature: SyntaxNode): boolean { + const typeAnno = getChildByField(propertySignature, 'type'); + if (!typeAnno) return false; + for (let i = 0; i < typeAnno.namedChildCount; i++) { + const inner = typeAnno.namedChild(i); + if (inner && inner.type === 'function_type') return true; + } + return false; + } + // extractExportedVariables removed — the walker now descends into // export_statement children and the inner declaration's dedicated // extractor (extractVariable, extractFunction, extractClass, etc.) @@ -1384,7 +1619,23 @@ export class TreeSitterExtractor { if (nameField && objectField && (node.type === 'method_invocation' || node.type === 'member_call_expression' || node.type === 'scoped_call_expression')) { // Method call with explicit receiver: receiver.method() / $receiver->method() / ClassName::method() const methodName = getNodeText(nameField, this.source); - let receiverName = getNodeText(objectField, this.source); + // Java `this.userbo.toLogin2()` parses as method_invocation(object=field_access(this, userbo)). + // Without unwrapping, receiverName is `this.userbo` and the name-matcher's + // single-dot receiver regex fails. Pull out the immediate field after `this.` + // so the receiver is the field name (`userbo`), which the resolver can then + // look up in the enclosing class's field declarations. + let receiverName: string; + if (objectField.type === 'field_access') { + const inner = getChildByField(objectField, 'object'); + const fld = getChildByField(objectField, 'field'); + if (inner && fld && (inner.type === 'this' || inner.type === 'this_expression')) { + receiverName = getNodeText(fld, this.source); + } else { + receiverName = getNodeText(objectField, this.source); + } + } else { + receiverName = getNodeText(objectField, this.source); + } // Strip PHP $ prefix from variable names receiverName = receiverName.replace(/^\$/, ''); @@ -1397,14 +1648,49 @@ export class TreeSitterExtractor { calleeName = `${receiverName}.${methodName}`; } } + } else if (node.type === 'message_expression') { + // ObjC message expressions emit one `method` field child per selector + // keyword: `[obj a:1 b:2 c:3]` has three `method=identifier` siblings. + // Joining them with `:` reconstructs the full selector and matches the + // multi-part selector names produced by the ObjC method_definition + // extractor (`extractObjcMethodName` in languages/objc.ts). Without this + // join, multi-keyword call sites only emitted the first keyword and never + // resolved to their target methods (e.g. `GET:parameters:headers:...` had + // zero callers despite obviously being called). + const methodKeywords: string[] = []; + for (let i = 0; i < node.namedChildCount; i++) { + if (node.fieldNameForNamedChild(i) === 'method') { + const kw = node.namedChild(i); + if (kw) methodKeywords.push(getNodeText(kw, this.source)); + } + } + if (methodKeywords.length > 0) { + const methodName: string = + methodKeywords.length === 1 + ? (methodKeywords[0] as string) + : methodKeywords.map((k) => `${k}:`).join(''); + const receiverField = getChildByField(node, 'receiver'); + const SKIP_RECEIVERS = new Set(['self', 'super']); + if (receiverField && receiverField.type !== 'message_expression') { + const receiverName = getNodeText(receiverField, this.source); + if (receiverName && !SKIP_RECEIVERS.has(receiverName)) { + calleeName = `${receiverName}.${methodName}`; + } else { + calleeName = methodName; + } + } else { + calleeName = methodName; + } + } } else { const func = getChildByField(node, 'function') || node.namedChild(0); if (func) { - if (func.type === 'member_expression' || func.type === 'attribute' || func.type === 'selector_expression' || func.type === 'navigation_expression') { + if (func.type === 'member_expression' || func.type === 'attribute' || func.type === 'selector_expression' || func.type === 'navigation_expression' || func.type === 'field_expression') { // Method call: obj.method() or obj.field.method() // Go uses selector_expression with 'field', JS/TS uses member_expression with 'property' // Kotlin uses navigation_expression with navigation_suffix > simple_identifier + // C/C++ use field_expression for both `obj.method()` and `ptr->method()` let property = getChildByField(func, 'property') || getChildByField(func, 'field'); if (!property) { const child1 = func.namedChild(1); @@ -1421,9 +1707,13 @@ export class TreeSitterExtractor { // This helps the resolver distinguish method calls from bare function calls // (e.g., Python's console.print() vs builtin print()) // Skip self/this/cls as they don't aid resolution - const receiver = getChildByField(func, 'object') || getChildByField(func, 'operand') || func.namedChild(0); + const receiver = + getChildByField(func, 'object') || + getChildByField(func, 'operand') || + getChildByField(func, 'argument') || + func.namedChild(0); const SKIP_RECEIVERS = new Set(['self', 'this', 'cls', 'super']); - if (receiver && (receiver.type === 'identifier' || receiver.type === 'simple_identifier')) { + if (receiver && (receiver.type === 'identifier' || receiver.type === 'simple_identifier' || receiver.type === 'field_identifier')) { const receiverName = getNodeText(receiver, this.source); if (!SKIP_RECEIVERS.has(receiverName)) { calleeName = `${receiverName}.${methodName}`; @@ -1505,6 +1795,78 @@ export class TreeSitterExtractor { } } + /** + * Find a `class_body` child of an `object_creation_expression` — the + * marker for an anonymous class (`new T() { ... }`). Returns the body + * node so the caller can walk it as the anon class's members. + */ + private findAnonymousClassBody(node: SyntaxNode): SyntaxNode | null { + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + // Java: `class_body`. C# uses the same node kind. + if (child && (child.type === 'class_body' || child.type === 'declaration_list')) { + return child; + } + } + return null; + } + + /** + * Extract a Java/C# anonymous class — `new T() { ...members }`. Emits a + * `class` node named ``, an `extends` reference to T (so + * Phase 5.5 interface-impl can bridge), and walks the body so its + * `method_declaration` members become method nodes under the anon class. + * + * Why this matters: without anon-class extraction, the overrides inside + * a lambda-returned `new T() { @Override int foo(){...} }` are not nodes, + * so a call through T.foo (the abstract parent method) has no static + * target — the agent has to Read the file to find the implementation. + */ + private extractAnonymousClass(node: SyntaxNode, body: SyntaxNode): void { + if (!this.extractor) return; + + // The instantiated type sits in the same field/position that + // extractInstantiation reads from. Use the same lookup so the anon + // class's `extends` target matches the `instantiates` edge. + const typeNode = + getChildByField(node, 'constructor') || + getChildByField(node, 'type') || + getChildByField(node, 'name') || + node.namedChild(0); + let typeName = typeNode ? getNodeText(typeNode, this.source) : 'Object'; + const ltIdx = typeName.indexOf('<'); + if (ltIdx > 0) typeName = typeName.slice(0, ltIdx); + const lastDot = Math.max(typeName.lastIndexOf('.'), typeName.lastIndexOf('::')); + if (lastDot >= 0) typeName = typeName.slice(lastDot + 1).replace(/^[:.]/, ''); + typeName = typeName.trim() || 'Object'; + + const anonName = `<${typeName}$anon@${node.startPosition.row + 1}>`; + const classNode = this.createNode('class', anonName, node, {}); + if (!classNode) return; + + // The anonymous class implicitly extends/implements the named type. + // We can't tell at extraction time whether T is a class or an interface, + // so emit `extends`. Resolution will still bind T to whatever it is, and + // Phase 5.5 (which already handles both `extends` and `implements`) will + // bridge T's methods to the override names found in the anon body. + this.unresolvedReferences.push({ + fromNodeId: classNode.id, + referenceName: typeName, + referenceKind: 'extends', + line: typeNode?.startPosition.row ?? node.startPosition.row, + column: typeNode?.startPosition.column ?? node.startPosition.column, + }); + + // Walk the body's children so method_declaration nodes inside become + // method nodes scoped to the anon class. + this.nodeStack.push(classNode.id); + for (let i = 0; i < body.namedChildCount; i++) { + const child = body.namedChild(i); + if (child) this.visitNode(child); + } + this.nodeStack.pop(); + } + /** * Scan `declNode` and its preceding siblings (within the parent's * named children) for decorator nodes, emitting a `decorates` @@ -1634,6 +1996,14 @@ export class TreeSitterExtractor { // about `call_expression`, so constructor invocations // produced no graph edges at all. this.extractInstantiation(node); + // Anonymous class with body: `new T() { ... }` (Java/C#). Extract as + // a class so interface-impl synthesis (Phase 5.5) can bridge T's + // methods to the overrides — same rationale as in visitNode. + const anonBody = this.findAnonymousClassBody(node); + if (anonBody) { + this.extractAnonymousClass(node, anonBody); + return; + } } else if (this.extractor!.extractBareCall) { const calleeName = this.extractor!.extractBareCall(node, this.source); if (calleeName && this.nodeStack.length > 0) { @@ -1650,6 +2020,21 @@ export class TreeSitterExtractor { } } + // Nested NAMED functions inside a body — function declarations and named + // function expressions like `.on('mount', function onmount(){})` — become + // their own nodes so the graph can link to them (callback handlers, local + // helpers). Anonymous arrows/expressions fall through to the default + // recursion below, keeping their inner calls attributed to the enclosing + // function: this bounds the new nodes to NAMED functions only (no explosion, + // no lost edges). extractFunction walks the nested body itself, so we return. + if (this.extractor!.functionTypes.includes(nodeType)) { + const nestedName = extractName(node, this.source, this.extractor!); + if (nestedName && nestedName !== '') { + this.extractFunction(node); + return; + } + } + // Extract structural nodes found inside function bodies. // Each extract method visits its own children, so we return after extracting. if (this.extractor!.classTypes.includes(nodeType)) { @@ -1689,6 +2074,42 @@ export class TreeSitterExtractor { * Extract inheritance relationships */ private extractInheritance(node: SyntaxNode, classId: string): void { + // Objective-C @interface MyClass : NSObject + if (node.type === 'class_interface') { + const superclass = getChildByField(node, 'superclass'); + if (superclass) { + const name = getNodeText(superclass, this.source); + this.unresolvedReferences.push({ + fromNodeId: classId, + referenceName: name, + referenceKind: 'extends', + line: superclass.startPosition.row + 1, + column: superclass.startPosition.column, + }); + } + for (let j = 0; j < node.namedChildCount; j++) { + const argList = node.namedChild(j); + if (argList?.type !== 'parameterized_arguments') continue; + for (let k = 0; k < argList.namedChildCount; k++) { + const typeName = argList.namedChild(k); + if (!typeName) continue; + const typeId = typeName.namedChildren.find( + (c: SyntaxNode) => c.type === 'type_identifier' || c.type === 'identifier' + ); + if (!typeId) continue; + const protocolName = getNodeText(typeId, this.source); + this.unresolvedReferences.push({ + fromNodeId: classId, + referenceName: protocolName, + referenceKind: 'implements', + line: typeId.startPosition.row + 1, + column: typeId.startPosition.column, + }); + } + } + return; + } + // Look for extends/implements clauses for (let i = 0; i < node.namedChildCount; i++) { const child = node.namedChild(i); @@ -1718,6 +2139,27 @@ export class TreeSitterExtractor { } } + // C++ base classes: `class Derived : public Base, private Other` → + // base_class_clause holds access specifiers + base type(s). Emit an extends + // ref per base type (skip the public/private/protected keywords). + if (child.type === 'base_class_clause') { + for (const t of child.namedChildren) { + if ( + t.type === 'type_identifier' || + t.type === 'qualified_identifier' || + t.type === 'template_type' + ) { + this.unresolvedReferences.push({ + fromNodeId: classId, + referenceName: getNodeText(t, this.source), + referenceKind: 'extends', + line: t.startPosition.row + 1, + column: t.startPosition.column, + }); + } + } + } + if ( child.type === 'implements_clause' || child.type === 'class_interface_clause' || @@ -2006,6 +2448,17 @@ export class TreeSitterExtractor { if (!this.extractor) return; if (!this.TYPE_ANNOTATION_LANGUAGES.has(this.language)) return; + // C# tree-sitter doesn't produce `type_identifier` leaves — it uses + // `identifier`, `predefined_type`, `qualified_name`, `generic_name`, + // etc. — so the generic walker below emits zero references for it. + // Dispatch to a C#-aware path that only walks type-position subtrees + // (the `type` field of a parameter/method/property/field), so + // parameter NAMES never accidentally surface as type refs (#381). + if (this.language === 'csharp') { + this.extractCsharpTypeRefs(node, nodeId); + return; + } + // Extract parameter type annotations const params = getChildByField(node, this.extractor.paramsField || 'parameters'); if (params) { @@ -2027,6 +2480,113 @@ export class TreeSitterExtractor { } } + /** + * Extract C# type references from a node that owns a type position — + * a method/constructor declaration, a property declaration, or a + * field declaration (which wraps `variable_declaration → type`). + * + * Walks ONLY into known type fields, so parameter names like + * `request` in `Build(UserDto request)` are never mis-emitted as + * type references. Once inside a type subtree, `walkCsharpTypePosition` + * recognizes C#'s actual type-leaf node kinds (`identifier`, + * `qualified_name`, `generic_name`, `array_type`, `nullable_type`, + * `tuple_type`, …) — none of which are `type_identifier`. Closes #381. + */ + private extractCsharpTypeRefs(node: SyntaxNode, nodeId: string): void { + // Return type / property type — the field is named `type`. + const directType = getChildByField(node, 'type'); + if (directType) this.walkCsharpTypePosition(directType, nodeId); + + // Field declarations wrap declarators in a `variable_declaration` + // whose `type` field carries the type. The outer `field_declaration` + // has no `type` field of its own, so the call above is a no-op here + // and we descend one level. + const varDecl = node.namedChildren.find((c: SyntaxNode) => c.type === 'variable_declaration'); + if (varDecl) { + const vdType = getChildByField(varDecl, 'type'); + if (vdType) this.walkCsharpTypePosition(vdType, nodeId); + } + + // Method / constructor parameters. The field name on + // `method_declaration` is `parameters`; it points at a + // `parameter_list` whose `parameter` children each have their own + // `type` field. Walking ONLY the type field skips parameter NAMES, + // which would otherwise mis-emit as type references. + const params = getChildByField(node, 'parameters'); + if (params) { + for (let i = 0; i < params.namedChildCount; i++) { + const child = params.namedChild(i); + if (!child || child.type !== 'parameter') continue; + const paramType = getChildByField(child, 'type'); + if (paramType) this.walkCsharpTypePosition(paramType, nodeId); + } + } + } + + /** + * Walk a C# subtree that is KNOWN to be in a type position + * (return type, parameter type, property type, field type, generic + * argument). Identifiers here are type names, not parameter names. + */ + private walkCsharpTypePosition(node: SyntaxNode, fromNodeId: string): void { + // `predefined_type` is int/string/bool/etc. — never a project ref. + if (node.type === 'predefined_type') return; + + // Bare type name: `Foo` in `Foo bar`, or the `Foo` inside `List`. + if (node.type === 'identifier') { + const name = getNodeText(node, this.source); + if (name && !this.BUILTIN_TYPES.has(name)) { + this.unresolvedReferences.push({ + fromNodeId, + referenceName: name, + referenceKind: 'references', + line: node.startPosition.row + 1, + column: node.startPosition.column, + }); + } + return; + } + + // `Namespace.Foo` → the rightmost identifier is the type. Emit the + // full qualified name as the reference; the resolver can still match + // on the trailing simple name when needed. + if (node.type === 'qualified_name') { + const text = getNodeText(node, this.source); + const last = text.split('.').pop() ?? text; + if (last && !this.BUILTIN_TYPES.has(last)) { + this.unresolvedReferences.push({ + fromNodeId, + referenceName: last, + referenceKind: 'references', + line: node.startPosition.row + 1, + column: node.startPosition.column, + }); + } + return; + } + + // `(int Code, Foo Payload)` — tuple element has BOTH a `type` and a + // `name` field; descending into all named children would mis-emit + // the element name (`Code`, `Payload`) as a type ref. Walk only the + // type field. + if (node.type === 'tuple_element') { + const t = getChildByField(node, 'type'); + if (t) this.walkCsharpTypePosition(t, fromNodeId); + return; + } + + // Composite type nodes — recurse into named children. Covers + // `generic_name` (head identifier + `type_argument_list`), + // `nullable_type`, `array_type`, `pointer_type`, `tuple_type`, + // `ref_type`, and any newer wrapping shapes the grammar adds. + // Identifiers reached here are all type-positional (parameter/field + // names are gated out before we descend). + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + if (child) this.walkCsharpTypePosition(child, fromNodeId); + } + } + /** * Extract type references from a variable's type annotation. */ @@ -2507,6 +3067,17 @@ export function extractFromSource( // Use custom extractor for Liquid const extractor = new LiquidExtractor(filePath, source); result = extractor.extract(); + } else if (detectedLanguage === 'xml') { + // Custom extractor for MyBatis mapper XML. Non-mapper XML returns just a + // file node so the watcher tracks it without emitting symbols. + const extractor = new MyBatisExtractor(filePath, source); + result = extractor.extract(); + } else if (isFileLevelOnlyLanguage(detectedLanguage)) { + // No symbol extraction at this stage — files are tracked at the file-record + // level only. Framework extractors (Drupal routing yml, Spring `@Value` + // resolution against application.yml/application.properties) run later and + // add per-file nodes/references when they apply. + result = { nodes: [], edges: [], unresolvedReferences: [], errors: [], durationMs: 0 }; } else if ( detectedLanguage === 'pascal' && (fileExtension === '.dfm' || fileExtension === '.fmx') diff --git a/src/extraction/wasm-runtime-flags.ts b/src/extraction/wasm-runtime-flags.ts new file mode 100644 index 000000000..c1b30a63e --- /dev/null +++ b/src/extraction/wasm-runtime-flags.ts @@ -0,0 +1,110 @@ +/** + * WASM runtime flags — workaround for the V8 turboshaft WASM Zone OOM. + * + * tree-sitter grammars are large WebAssembly modules. On Node >= 22 the V8 + * "turboshaft" optimizing WASM compiler can exhaust its per-compilation Zone + * arena while compiling these grammars on a background thread, aborting the + * whole process with `Fatal process out of memory: Zone` — even with tens of + * GB of system memory free, because the Zone is a V8-internal arena, not the + * JS heap. Reproduced on Node 22 and 24; Node 25 is already hard-blocked for + * the same crash (see ../bin/node-version-check.ts). See issues #293 and #298. + * + * `--liftoff-only` forces every WASM module to the Liftoff baseline compiler + * and never runs turboshaft, which eliminates the crash. Parsing stays fully + * correct; we only forgo the (marginal, and for grammars rarely reached) + * optimized-tier speedup. + * + * This flag MUST be on node's command line — it is read by V8 at engine init, + * before any of our JS runs. Empirically (Node 24) none of these work: + * - `v8.setFlagsFromString('--liftoff-only')` at runtime — too late. + * - Worker `execArgv: ['--liftoff-only']` — rejected (ERR_WORKER_INVALID_EXEC_ARGV). + * - `NODE_OPTIONS=--liftoff-only` — not on Node's NODE_OPTIONS allowlist. + * Also empirically, `--no-wasm-tier-up` / `--no-wasm-dynamic-tiering` do NOT + * prevent the crash — only disabling the optimizing tier entirely does. + * + * Delivery: the bundled launcher passes the flag directly (see + * scripts/build-bundle.sh and scripts/npm-shim.js); for any other launch path + * (running dist directly, from source, etc.) the CLI re-execs itself once with + * the flag via {@link relaunchWithWasmRuntimeFlagsIfNeeded}. V8 flags are + * PROCESS-global, and the parse worker is created with default (inherited) + * execArgv, so flagging the main process governs the worker's WASM compilation + * too. + */ +import { spawnSync } from 'child_process'; + +/** + * The V8 flag(s) that keep tree-sitter grammar compilation off the turboshaft + * optimizing tier. Single source of truth: the relaunch guard and the test + * suite both read this (a test asserts each is a real flag on the running + * runtime, so a rename can't silently regress the fix). + */ +export const WASM_RUNTIME_FLAGS: readonly string[] = ['--liftoff-only']; + +/** + * Env var set on the relaunched child so a detection slip can never cause an + * infinite re-exec loop. Also lets users force-disable the relaunch. + */ +const RELAUNCH_GUARD_ENV = 'CODEGRAPH_WASM_RELAUNCHED'; + +/** + * Env var carrying the *host* PID (the relauncher's own parent) across the + * re-exec. Without `--liftoff-only` the CLI re-execs itself once, inserting an + * intermediate process between the MCP host and the server. That intermediate + * stays alive (blocked in spawnSync) even after the host is killed, so the + * server's PPID watchdog can't detect the host's death by watching its own + * `process.ppid`. Passing the host PID through lets the watchdog poll it + * directly. Unset on the no-re-exec path (bundled launcher / flag already + * present), where the server is already a direct child of the host. See + * src/mcp/index.ts (#277). + */ +export const HOST_PPID_ENV = 'CODEGRAPH_HOST_PPID'; + +/** True when every required WASM runtime flag is already present in `execArgv`. */ +export function processHasWasmRuntimeFlags( + execArgv: readonly string[] = process.execArgv +): boolean { + return WASM_RUNTIME_FLAGS.every((flag) => execArgv.includes(flag)); +} + +/** + * Build the argv for re-execing node with the WASM runtime flags: our flags + * first, then any node flags already in `execArgv` (deduped), then the script + * and its args. Pure — exported for unit testing. + */ +export function buildRelaunchArgv( + scriptPath: string, + scriptArgs: readonly string[], + execArgv: readonly string[] = process.execArgv +): string[] { + const preserved = execArgv.filter((arg) => !WASM_RUNTIME_FLAGS.includes(arg)); + return [...WASM_RUNTIME_FLAGS, ...preserved, scriptPath, ...scriptArgs]; +} + +/** + * If the current process is missing the WASM runtime flags, re-exec it once + * with them and exit with the child's status. No-op when the flags are already + * present (the normal bundled-launcher path), when already relaunched, or when + * disabled via CODEGRAPH_NO_RELAUNCH. + * + * On spawn failure, returns so the caller runs in-process anyway — risking the + * OOM is still better than refusing to start. + */ +export function relaunchWithWasmRuntimeFlagsIfNeeded(scriptPath: string): void { + if (processHasWasmRuntimeFlags()) return; + if (process.env[RELAUNCH_GUARD_ENV]) return; + if (process.env.CODEGRAPH_NO_RELAUNCH) return; + + const argv = buildRelaunchArgv(scriptPath, process.argv.slice(2)); + const result = spawnSync(process.execPath, argv, { + stdio: 'inherit', + env: { ...process.env, [RELAUNCH_GUARD_ENV]: '1', [HOST_PPID_ENV]: String(process.ppid) }, + windowsHide: true, + }); + + if (result.error) { + // Couldn't relaunch (e.g. execPath unavailable) — fall through and run in + // this process. Degraded (may OOM on huge repos) but not broken. + return; + } + process.exit(result.status ?? (result.signal ? 1 : 0)); +} diff --git a/src/extraction/wasm/tree-sitter-lua.wasm b/src/extraction/wasm/tree-sitter-lua.wasm new file mode 100644 index 000000000..be3231dcd Binary files /dev/null and b/src/extraction/wasm/tree-sitter-lua.wasm differ diff --git a/src/extraction/wasm/tree-sitter-luau.wasm b/src/extraction/wasm/tree-sitter-luau.wasm new file mode 100644 index 000000000..1ed5af18f Binary files /dev/null and b/src/extraction/wasm/tree-sitter-luau.wasm differ diff --git a/src/graph/traversal.ts b/src/graph/traversal.ts index dd5b50296..c366721b8 100644 --- a/src/graph/traversal.ts +++ b/src/graph/traversal.ts @@ -90,29 +90,24 @@ export class GraphTraverser { return priority(a) - priority(b); }); + // Batch-fetch the unvisited neighbors in one query (was N+1 per BFS step). + const wantIds = adjacentEdges + .map((e) => (e.source === node.id ? e.target : e.source)) + .filter((id) => !visited.has(id)); + const neighborNodes = wantIds.length > 0 ? this.queries.getNodesByIds(wantIds) : new Map(); + for (const adjEdge of adjacentEdges) { - // Determine next node: for 'both' direction, edges can be either - // incoming or outgoing, so pick whichever end is not the current node const nextNodeId = adjEdge.source === node.id ? adjEdge.target : adjEdge.source; + if (visited.has(nextNodeId)) continue; - if (visited.has(nextNodeId)) { - continue; - } - - const nextNode = this.queries.getNodeById(nextNodeId); - if (!nextNode) { - continue; - } + const nextNode = neighborNodes.get(nextNodeId); + if (!nextNode) continue; - // Apply node kind filter if (opts.nodeKinds && opts.nodeKinds.length > 0 && !opts.nodeKinds.includes(nextNode.kind)) { continue; } - // Add node to result nodes.set(nextNode.id, nextNode); - - // Queue for further traversal queue.push({ node: nextNode, edge: adjEdge, depth: depth + 1 }); } } @@ -176,19 +171,18 @@ export class GraphTraverser { // Get adjacent edges const adjacentEdges = this.getAdjacentEdges(node.id, opts.direction, opts.edgeKinds); + // Batch-fetch unvisited neighbors (was N+1 per DFS step). + const wantIds = adjacentEdges + .map((e) => (e.source === node.id ? e.target : e.source)) + .filter((id) => !visited.has(id)); + const neighborNodes = wantIds.length > 0 ? this.queries.getNodesByIds(wantIds) : new Map(); + for (const edge of adjacentEdges) { - // Determine next node: for 'both' direction, edges can be either - // incoming or outgoing, so pick whichever end is not the current node const nextNodeId = edge.source === node.id ? edge.target : edge.source; + if (visited.has(nextNodeId)) continue; - if (visited.has(nextNodeId)) { - continue; - } - - const nextNode = this.queries.getNodeById(nextNodeId); - if (!nextNode) { - continue; - } + const nextNode = neighborNodes.get(nextNodeId); + if (!nextNode) continue; // Apply node kind filter if (opts.nodeKinds && opts.nodeKinds.length > 0 && !opts.nodeKinds.includes(nextNode.kind)) { @@ -255,9 +249,15 @@ export class GraphTraverser { visited.add(nodeId); const incomingEdges = this.queries.getIncomingEdges(nodeId, ['calls', 'references', 'imports']); + if (incomingEdges.length === 0) return; + + // Batch-fetch all caller nodes in one round-trip instead of one + // getNodeById per edge (was N+1 — meaningful on functions with many callers). + const sourceIds = incomingEdges.map((e) => e.source); + const callerNodes = this.queries.getNodesByIds(sourceIds); for (const edge of incomingEdges) { - const callerNode = this.queries.getNodeById(edge.source); + const callerNode = callerNodes.get(edge.source); if (callerNode && !visited.has(callerNode.id)) { result.push({ node: callerNode, edge }); this.getCallersRecursive(callerNode.id, maxDepth, currentDepth + 1, result, visited); @@ -294,9 +294,14 @@ export class GraphTraverser { visited.add(nodeId); const outgoingEdges = this.queries.getOutgoingEdges(nodeId, ['calls', 'references', 'imports']); + if (outgoingEdges.length === 0) return; + + // Batch-fetch callee nodes (was N+1 — see getCallersRecursive note). + const targetIds = outgoingEdges.map((e) => e.target); + const calleeNodes = this.queries.getNodesByIds(targetIds); for (const edge of outgoingEdges) { - const calleeNode = this.queries.getNodeById(edge.target); + const calleeNode = calleeNodes.get(edge.target); if (calleeNode && !visited.has(calleeNode.id)) { result.push({ node: calleeNode, edge }); this.getCalleesRecursive(calleeNode.id, maxDepth, currentDepth + 1, result, visited); @@ -388,9 +393,11 @@ export class GraphTraverser { visited.add(nodeId); const outgoingEdges = this.queries.getOutgoingEdges(nodeId, ['extends', 'implements']); + if (outgoingEdges.length === 0) return; + const parents = this.queries.getNodesByIds(outgoingEdges.map((e) => e.target)); for (const edge of outgoingEdges) { - const parentNode = this.queries.getNodeById(edge.target); + const parentNode = parents.get(edge.target); if (parentNode && !nodes.has(parentNode.id)) { nodes.set(parentNode.id, parentNode); edges.push(edge); @@ -411,9 +418,11 @@ export class GraphTraverser { visited.add(nodeId); const incomingEdges = this.queries.getIncomingEdges(nodeId, ['extends', 'implements']); + if (incomingEdges.length === 0) return; + const children = this.queries.getNodesByIds(incomingEdges.map((e) => e.source)); for (const edge of incomingEdges) { - const childNode = this.queries.getNodeById(edge.source); + const childNode = children.get(edge.source); if (childNode && !nodes.has(childNode.id)) { nodes.set(childNode.id, childNode); edges.push(edge); @@ -433,12 +442,13 @@ export class GraphTraverser { // Get all incoming edges (references, calls, type_of, etc.) const incomingEdges = this.queries.getIncomingEdges(nodeId); + if (incomingEdges.length === 0) return result; + // Batch-fetch source nodes (was N+1). + const sources = this.queries.getNodesByIds(incomingEdges.map((e) => e.source)); for (const edge of incomingEdges) { - const sourceNode = this.queries.getNodeById(edge.source); - if (sourceNode) { - result.push({ node: sourceNode, edge }); - } + const sourceNode = sources.get(edge.source); + if (sourceNode) result.push({ node: sourceNode, edge }); } return result; @@ -496,13 +506,16 @@ export class GraphTraverser { const containerKinds = new Set(['class', 'interface', 'struct', 'trait', 'protocol', 'module', 'enum']); if (containerKinds.has(focalNode.kind)) { const containsEdges = this.queries.getOutgoingEdges(nodeId, ['contains']); - for (const edge of containsEdges) { - const childNode = this.queries.getNodeById(edge.target); - if (childNode && !visited.has(childNode.id)) { - nodes.set(childNode.id, childNode); - edges.push(edge); - // Recurse into children at the same depth (they're part of the same symbol) - this.getImpactRecursive(childNode.id, maxDepth, currentDepth, nodes, edges, visited); + if (containsEdges.length > 0) { + const children = this.queries.getNodesByIds(containsEdges.map((e) => e.target)); + for (const edge of containsEdges) { + const childNode = children.get(edge.target); + if (childNode && !visited.has(childNode.id)) { + nodes.set(childNode.id, childNode); + edges.push(edge); + // Recurse into children at the same depth (they're part of the same symbol) + this.getImpactRecursive(childNode.id, maxDepth, currentDepth, nodes, edges, visited); + } } } } @@ -510,9 +523,11 @@ export class GraphTraverser { // Get all incoming edges (things that depend on this node) const incomingEdges = this.queries.getIncomingEdges(nodeId); + if (incomingEdges.length === 0) return; + const sources = this.queries.getNodesByIds(incomingEdges.map((e) => e.source)); for (const edge of incomingEdges) { - const sourceNode = this.queries.getNodeById(edge.source); + const sourceNode = sources.get(edge.source); if (sourceNode && !nodes.has(sourceNode.id)) { nodes.set(sourceNode.id, sourceNode); edges.push(edge); @@ -564,10 +579,17 @@ export class GraphTraverser { nodeId, edgeKinds.length > 0 ? edgeKinds : undefined ); + if (outgoingEdges.length === 0) continue; + + // Batch-fetch only the unvisited targets (was N+1 per BFS frontier). + const wantIds = outgoingEdges + .map((e) => e.target) + .filter((id) => !visited.has(id)); + const nextNodes = wantIds.length > 0 ? this.queries.getNodesByIds(wantIds) : new Map(); for (const edge of outgoingEdges) { if (!visited.has(edge.target)) { - const nextNode = this.queries.getNodeById(edge.target); + const nextNode = nextNodes.get(edge.target); if (nextNode) { queue.push({ nodeId: edge.target, @@ -627,15 +649,15 @@ export class GraphTraverser { */ getChildren(nodeId: string): Node[] { const containsEdges = this.queries.getOutgoingEdges(nodeId, ['contains']); - const children: Node[] = []; + if (containsEdges.length === 0) return []; + // Batch-fetch (was N+1). + const childNodes = this.queries.getNodesByIds(containsEdges.map((e) => e.target)); + const children: Node[] = []; for (const edge of containsEdges) { - const childNode = this.queries.getNodeById(edge.target); - if (childNode) { - children.push(childNode); - } + const childNode = childNodes.get(edge.target); + if (childNode) children.push(childNode); } - return children; } } diff --git a/src/index.ts b/src/index.ts index 7d5867414..ee3bf51fa 100644 --- a/src/index.ts +++ b/src/index.ts @@ -7,7 +7,6 @@ import * as path from 'path'; import { - CodeGraphConfig, Node, Edge, FileRecord, @@ -25,7 +24,6 @@ import { } from './types'; import { DatabaseConnection, getDatabasePath } from './db'; import { QueryBuilder } from './db/queries'; -import { loadConfig, saveConfig, createDefaultConfig } from './config'; import { isInitialized, createDirectory, @@ -48,12 +46,11 @@ import { import { GraphTraverser, GraphQueryManager } from './graph'; import { ContextBuilder, createContextBuilder } from './context'; import { Mutex, FileLock } from './utils'; -import { FileWatcher, WatchOptions } from './sync'; +import { FileWatcher, WatchOptions, PendingFile, LockUnavailableError } from './sync'; // Re-export types for consumers export * from './types'; export { getDatabasePath } from './db'; -export { getConfigPath } from './config'; export { getCodeGraphDir, isInitialized, @@ -78,16 +75,13 @@ export { defaultLogger, } from './errors'; export { Mutex, FileLock, processInBatches, debounce, throttle, MemoryMonitor } from './utils'; -export { FileWatcher, WatchOptions } from './sync'; +export { FileWatcher, WatchOptions, PendingFile, LockUnavailableError } from './sync'; export { MCPServer } from './mcp'; /** * Options for initializing a new CodeGraph project */ export interface InitOptions { - /** Custom configuration overrides */ - config?: Partial; - /** Whether to run initial indexing after init */ index?: boolean; @@ -128,7 +122,6 @@ export interface IndexOptions { export class CodeGraph { private db: DatabaseConnection; private queries: QueryBuilder; - private config: CodeGraphConfig; private projectRoot: string; private orchestrator: ExtractionOrchestrator; private resolver: ReferenceResolver; @@ -148,17 +141,15 @@ export class CodeGraph { private constructor( db: DatabaseConnection, queries: QueryBuilder, - config: CodeGraphConfig, projectRoot: string ) { this.db = db; this.queries = queries; - this.config = config; this.projectRoot = projectRoot; this.fileLock = new FileLock( path.join(projectRoot, '.codegraph', 'codegraph.lock') ); - this.orchestrator = new ExtractionOrchestrator(projectRoot, config, queries); + this.orchestrator = new ExtractionOrchestrator(projectRoot, queries); this.resolver = createResolver(projectRoot, queries); this.graphManager = new GraphQueryManager(queries); this.traverser = new GraphTraverser(queries); @@ -194,19 +185,12 @@ export class CodeGraph { // Create directory structure createDirectory(resolvedRoot); - // Create and save configuration - const config = createDefaultConfig(resolvedRoot); - if (options.config) { - Object.assign(config, options.config); - } - saveConfig(resolvedRoot, config); - // Initialize database const dbPath = getDatabasePath(resolvedRoot); const db = DatabaseConnection.initialize(dbPath); const queries = new QueryBuilder(db.getDb()); - const instance = new CodeGraph(db, queries, config, resolvedRoot); + const instance = new CodeGraph(db, queries, resolvedRoot); // Run initial indexing if requested if (options.index) { @@ -219,7 +203,7 @@ export class CodeGraph { /** * Initialize synchronously (without indexing) */ - static initSync(projectRoot: string, options: Omit = {}): CodeGraph { + static initSync(projectRoot: string): CodeGraph { const resolvedRoot = path.resolve(projectRoot); // Check if already initialized @@ -230,19 +214,12 @@ export class CodeGraph { // Create directory structure createDirectory(resolvedRoot); - // Create and save configuration - const config = createDefaultConfig(resolvedRoot); - if (options.config) { - Object.assign(config, options.config); - } - saveConfig(resolvedRoot, config); - // Initialize database const dbPath = getDatabasePath(resolvedRoot); const db = DatabaseConnection.initialize(dbPath); const queries = new QueryBuilder(db.getDb()); - return new CodeGraph(db, queries, config, resolvedRoot); + return new CodeGraph(db, queries, resolvedRoot); } /** @@ -267,15 +244,12 @@ export class CodeGraph { throw new Error(`Invalid CodeGraph directory: ${validation.errors.join(', ')}`); } - // Load configuration - const config = loadConfig(resolvedRoot); - // Open database const dbPath = getDatabasePath(resolvedRoot); const db = DatabaseConnection.open(dbPath); const queries = new QueryBuilder(db.getDb()); - const instance = new CodeGraph(db, queries, config, resolvedRoot); + const instance = new CodeGraph(db, queries, resolvedRoot); // Sync if requested if (options.sync) { @@ -302,15 +276,12 @@ export class CodeGraph { throw new Error(`Invalid CodeGraph directory: ${validation.errors.join(', ')}`); } - // Load configuration - const config = loadConfig(resolvedRoot); - // Open database const dbPath = getDatabasePath(resolvedRoot); const db = DatabaseConnection.open(dbPath); const queries = new QueryBuilder(db.getDb()); - return new CodeGraph(db, queries, config, resolvedRoot); + return new CodeGraph(db, queries, resolvedRoot); } /** @@ -330,32 +301,6 @@ export class CodeGraph { this.db.close(); } - // =========================================================================== - // Configuration - // =========================================================================== - - /** - * Get the current configuration - */ - getConfig(): CodeGraphConfig { - return { ...this.config }; - } - - /** - * Update configuration - */ - updateConfig(updates: Partial): void { - Object.assign(this.config, updates); - saveConfig(this.projectRoot, this.config); - // Recreate orchestrator and resolver with new config - this.orchestrator = new ExtractionOrchestrator( - this.projectRoot, - this.config, - this.queries - ); - this.resolver = createResolver(this.projectRoot, this.queries); - } - /** * Get the project root directory */ @@ -380,8 +325,23 @@ export class CodeGraph { return { success: false, filesIndexed: 0, filesSkipped: 0, filesErrored: 0, nodesCreated: 0, edgesCreated: 0, errors: [{ message: 'Could not acquire file lock - another process may be indexing', severity: 'error' as const }], durationMs: 0 }; } try { + const before = this.queries.getNodeAndEdgeCount(); const result = await this.orchestrator.indexAll(options.onProgress, options.signal, options.verbose); + // Re-detect frameworks now that the index is populated. The resolver + // is constructed with createResolver() before any files exist, so + // framework resolvers whose detect() consults the indexed file list + // (e.g. UIKit/SwiftUI scanning for imports, swift-objc-bridge looking + // for both Swift and ObjC files) all return false on that initial pass + // and silently drop themselves. Re-initializing here gives them a + // chance to see the actual project before resolution runs. + if (result.success && result.filesIndexed > 0) { + this.resolver.initialize(); + // Cross-file finalization (e.g. NestJS RouterModule prefixes). Runs + // before resolution so updated names show up in subsequent reads. + this.resolver.runPostExtract(); + } + // Resolve references to create call/import/extends edges if (result.success && result.filesIndexed > 0) { // Get count without loading all refs into memory @@ -402,6 +362,21 @@ export class CodeGraph { }); } + // Refresh planner stats + checkpoint the WAL after bulk writes. + // Cheap and non-blocking; never load-bearing for correctness. + if (result.success && result.filesIndexed > 0) { + this.db.runMaintenance(); + } + + // The orchestrator only sees extraction-phase counts; resolution and + // synthesizer edges (often >50% of the graph on JVM repos) come later. + // Recompute against the DB so the CLI summary reports the true totals. + if (result.success && result.filesIndexed > 0) { + const after = this.queries.getNodeAndEdgeCount(); + result.nodesCreated = after.nodes - before.nodes; + result.edgesCreated = after.edges - before.edges; + } + return result; } finally { this.fileLock.release(); @@ -444,6 +419,14 @@ export class CodeGraph { try { const result = await this.orchestrator.sync(options.onProgress); + // Cross-file finalization (e.g. NestJS RouterModule prefixes). Run on + // every sync that touched files so edits to `app.module.ts` propagate + // to controllers in unchanged files. The pass is idempotent and cheap + // (regex over *.module.ts only). + if (result.filesAdded > 0 || result.filesModified > 0) { + this.resolver.runPostExtract(); + } + // Resolve references if files were updated if (result.filesAdded > 0 || result.filesModified > 0) { if (result.changedFilePaths) { @@ -483,6 +466,11 @@ export class CodeGraph { } } + // Refresh planner stats + checkpoint the WAL after bulk writes. + if (result.filesAdded > 0 || result.filesModified > 0 || result.filesRemoved > 0) { + this.db.runMaintenance(); + } + return result; } finally { this.fileLock.release(); @@ -515,9 +503,16 @@ export class CodeGraph { this.watcher = new FileWatcher( this.projectRoot, - this.config, async () => { const result = await this.sync(); + // sync() returns this exact zero-shape iff it failed to acquire the + // file lock (a real empty sync always has filesChecked > 0 because + // scanDirectory ran). Surface that to the watcher as a typed error + // so it keeps pendingFiles + reschedules instead of clearing them + // (#449). + if (result.filesChecked === 0 && result.durationMs === 0) { + throw new LockUnavailableError(); + } const filesChanged = result.filesAdded + result.filesModified + result.filesRemoved; return { filesChanged, durationMs: result.durationMs }; }, @@ -544,6 +539,31 @@ export class CodeGraph { return this.watcher?.isActive() ?? false; } + /** + * Files seen by the file watcher since the last successful sync — + * the per-file "stale" signal MCP tools attach to responses so an agent + * can fall back to {@link Read} for just the affected file without + * waiting for a debounced sync to complete (issue #403). + * + * Returns an empty list when the watcher isn't active, or no events have + * arrived. Each entry includes `firstSeenMs` and `lastSeenMs` (wall-clock + * `Date.now()` values) so callers can render "edited Nms ago", plus an + * `indexing` flag indicating whether the in-flight sync (if any) will + * absorb that file. + */ + getPendingFiles(): PendingFile[] { + return this.watcher?.getPendingFiles() ?? []; + } + + /** + * Resolves once the file watcher has finished its initial chokidar scan. + * Useful for tests that need a deterministic boundary before asserting on + * `getPendingFiles()`. Resolves immediately when no watcher is active. + */ + waitUntilWatcherReady(timeoutMs?: number): Promise { + return this.watcher ? this.watcher.waitUntilReady(timeoutMs) : Promise.resolve(); + } + /** * Get files that have changed since last index */ @@ -613,15 +633,24 @@ export class CodeGraph { } /** - * Active SQLite backend for this project's connection. `wasm` means - * the native better-sqlite3 install failed and the WASM fallback is - * serving requests at 5-10x the latency. Surfaced via `codegraph - * status` and the `codegraph_status` MCP tool. + * Active SQLite backend for this project's connection (`node-sqlite` — Node's + * built-in real-SQLite module). Surfaced via `codegraph status` and the + * `codegraph_status` MCP tool alongside the effective journal mode. */ getBackend(): import('./db').SqliteBackend { return this.db.getBackend(); } + /** + * The journal mode actually in effect ('wal', 'delete', …). 'wal' means + * readers never block on a concurrent writer; anything else means they can, + * which is the precondition for the "database is locked" failures in issue + * #238. Surfaced via `codegraph status` and the `codegraph_status` MCP tool. + */ + getJournalMode(): string { + return this.db.getJournalMode(); + } + // =========================================================================== // Node Operations // =========================================================================== @@ -654,6 +683,33 @@ export class CodeGraph { return this.queries.searchNodes(query, options); } + /** + * Find the project's "primary route file" — the file with the densest + * concentration of framework-emitted `route` nodes (≥3 routes, ≥30% + * of all non-test routes). Used to inline the routing config in + * `codegraph_context` responses on small realworld template repos + * (rails-realworld, laravel-realworld, drupal-admintoolbar, …) where + * Glob+Read of `routes.rb`/`urls.py`/etc. otherwise beats codegraph. + */ + getTopRouteFile(): { filePath: string; routeCount: number; totalRoutes: number } | null { + return this.queries.getTopRouteFile(); + } + + /** + * Build a URL → handler routing manifest from the index. Each entry + * pairs a route node (URL + method) with its handler function/method + * via the `references` edge that framework resolvers emit. Returns + * null when fewer than 3 valid (non-test) routes exist. + */ + getRoutingManifest(limit?: number): { + entries: Array<{ url: string; handler: string; handlerFile: string; handlerLine: number; handlerKind: string }>; + topHandlerFile: string | null; + topHandlerFileCount: number; + totalRoutes: number; + } | null { + return this.queries.getRoutingManifest(limit); + } + // =========================================================================== // Edge Operations // =========================================================================== diff --git a/src/installer/clack.d.ts b/src/installer/clack.d.ts index 08e96874b..29d08ad4d 100644 --- a/src/installer/clack.d.ts +++ b/src/installer/clack.d.ts @@ -24,6 +24,13 @@ declare module '@clack/prompts' { initialValue?: Value; }): Promise; + export function multiselect(opts: { + message: string; + options: { value: Value; label: string; hint?: string }[]; + initialValues?: Value[]; + required?: boolean; + }): Promise; + export function spinner(): { start(message?: string): void; stop(message?: string): void; diff --git a/src/installer/claude-md-template.ts b/src/installer/claude-md-template.ts deleted file mode 100644 index 130d4472b..000000000 --- a/src/installer/claude-md-template.ts +++ /dev/null @@ -1,44 +0,0 @@ -/** - * CLAUDE.md template for CodeGraph instructions - * - * This template is injected into ~/.claude/CLAUDE.md (global) or ./.claude/CLAUDE.md (local) - * Keep this in sync with the README.md "Recommended: Add Global Instructions" section - */ - -// Markers to identify CodeGraph section for updates -export const CODEGRAPH_SECTION_START = ''; -export const CODEGRAPH_SECTION_END = ''; - -export const CLAUDE_MD_TEMPLATE = `${CODEGRAPH_SECTION_START} -## CodeGraph - -CodeGraph builds a semantic knowledge graph of codebases for faster, smarter code exploration. - -### If \`.codegraph/\` exists in the project - -**NEVER call \`codegraph_explore\` or \`codegraph_context\` directly in the main session.** These tools return large amounts of source code that fills up main session context. Instead, ALWAYS spawn an Explore agent for any exploration question (e.g., "how does X work?", "explain the Y system", "where is Z implemented?"). - -**When spawning Explore agents**, include this instruction in the prompt: - -> This project has CodeGraph initialized (.codegraph/ exists). Use \`codegraph_explore\` as your PRIMARY tool — it returns full source code sections from all relevant files in one call. -> -> **Rules:** -> 1. Follow the explore call budget in the \`codegraph_explore\` tool description — it scales automatically based on project size. -> 2. Do NOT re-read files that codegraph_explore already returned source code for. The source sections are complete and authoritative. -> 3. Only fall back to grep/glob/read for files listed under "Additional relevant files" if you need more detail, or if codegraph returned no results. - -**The main session may only use these lightweight tools directly** (for targeted lookups before making edits, not for exploration): - -| Tool | Use For | -|------|---------| -| \`codegraph_search\` | Find symbols by name | -| \`codegraph_callers\` / \`codegraph_callees\` | Trace call flow | -| \`codegraph_impact\` | Check what's affected before editing | -| \`codegraph_node\` | Get a single symbol's details | - -### If \`.codegraph/\` does NOT exist - -At the start of a session, ask the user if they'd like to initialize CodeGraph: - -"I notice this project doesn't have CodeGraph initialized. Would you like me to run \`codegraph init -i\` to build a code knowledge graph?" -${CODEGRAPH_SECTION_END}`; diff --git a/src/installer/config-writer.ts b/src/installer/config-writer.ts index 5e019909d..acf5f4cfa 100644 --- a/src/installer/config-writer.ts +++ b/src/installer/config-writer.ts @@ -1,292 +1,60 @@ /** - * Config file writing for the CodeGraph installer - * Writes to claude.json, settings.json, and CLAUDE.md + * Backwards-compat shim — original Claude-only writer functions. + * + * The installer now uses the multi-target architecture in + * `./targets/`. This file is preserved so existing imports (the test + * suite, downstream tooling) keep working unchanged. Each function + * delegates to the Claude target. New code should import the target + * registry from `./targets/registry` directly. + * + * @deprecated Use `targets/registry.ts` and the `AgentTarget` + * abstraction instead. */ -import * as fs from 'fs'; import * as path from 'path'; import * as os from 'os'; -export type InstallLocation = 'global' | 'local'; import { - CLAUDE_MD_TEMPLATE, - CODEGRAPH_SECTION_START, - CODEGRAPH_SECTION_END, -} from './claude-md-template'; - -/** - * Get the path to the Claude config directory - */ -function getClaudeConfigDir(location: InstallLocation): string { - if (location === 'global') { - return path.join(os.homedir(), '.claude'); - } - return path.join(process.cwd(), '.claude'); -} - -/** - * Get the path to the claude.json file - * - Global: ~/.claude.json (root level) - * - Local: ./.claude.json (project root) - */ -function getClaudeJsonPath(location: InstallLocation): string { - if (location === 'global') { - return path.join(os.homedir(), '.claude.json'); - } - return path.join(process.cwd(), '.claude.json'); -} - -/** - * Get the path to the settings.json file - * - Global: ~/.claude/settings.json - * - Local: ./.claude/settings.json - */ -function getSettingsJsonPath(location: InstallLocation): string { - const configDir = getClaudeConfigDir(location); - return path.join(configDir, 'settings.json'); -} - -/** - * Read a JSON file, returning an empty object if it doesn't exist. - * Distinguishes between missing files (returns {}) and corrupted - * files (logs warning, returns {}). - */ -function readJsonFile(filePath: string): Record { - if (!fs.existsSync(filePath)) { - return {}; - } - try { - const content = fs.readFileSync(filePath, 'utf-8'); - return JSON.parse(content); - } catch (err) { - const msg = err instanceof Error ? err.message : String(err); - console.warn(` Warning: Could not parse ${path.basename(filePath)}: ${msg}`); - console.warn(` A backup will be created before overwriting.`); - // Create a backup of the corrupted file - try { - const backupPath = filePath + '.backup'; - fs.copyFileSync(filePath, backupPath); - } catch { /* ignore backup failure */ } - return {}; - } -} - -/** - * Write a file atomically by writing to a temp file then renaming. - * Prevents corruption if the process crashes mid-write. - */ -function atomicWriteFileSync(filePath: string, content: string): void { - const dir = path.dirname(filePath); - if (!fs.existsSync(dir)) { - fs.mkdirSync(dir, { recursive: true }); - } - const tmpPath = filePath + '.tmp.' + process.pid; - try { - fs.writeFileSync(tmpPath, content); - fs.renameSync(tmpPath, filePath); - } catch (err) { - // Clean up temp file on failure - try { fs.unlinkSync(tmpPath); } catch { /* ignore */ } - throw err; - } -} - -/** - * Write a JSON file, creating parent directories if needed - */ -function writeJsonFile(filePath: string, data: Record): void { - atomicWriteFileSync(filePath, JSON.stringify(data, null, 2) + '\n'); -} + writeMcpEntry, + writePermissionsEntry, +} from './targets/claude'; +import { readJsonFile } from './targets/shared'; -/** - * Get the MCP server configuration - */ -function getMcpServerConfig(): Record { - return { - type: 'stdio', - command: 'codegraph', - args: ['serve', '--mcp'], - }; -} +export type InstallLocation = 'global' | 'local'; /** - * Write the MCP server configuration to claude.json + * Each shim calls ONLY the named per-file helper — writeMcpConfig + * writes only the MCP JSON, writePermissions only settings.json. The + * full multi-file install lives in `claudeTarget.install()` which the + * new orchestrator uses. + * + * There is no `writeClaudeMd` shim anymore: codegraph stopped writing a + * CLAUDE.md instructions block (issue #529) now that the MCP server's + * `initialize` instructions are the single source of truth. */ export function writeMcpConfig(location: InstallLocation): void { - const claudeJsonPath = getClaudeJsonPath(location); - const config = readJsonFile(claudeJsonPath); - - // Ensure mcpServers object exists - if (!config.mcpServers) { - config.mcpServers = {}; - } - - // Add or update codegraph server - config.mcpServers.codegraph = getMcpServerConfig(); - - writeJsonFile(claudeJsonPath, config); -} - -/** - * Get the list of permissions for CodeGraph tools - */ -function getCodeGraphPermissions(): string[] { - return [ - 'mcp__codegraph__codegraph_search', - 'mcp__codegraph__codegraph_context', - 'mcp__codegraph__codegraph_callers', - 'mcp__codegraph__codegraph_callees', - 'mcp__codegraph__codegraph_impact', - 'mcp__codegraph__codegraph_node', - 'mcp__codegraph__codegraph_status', - ]; + writeMcpEntry(location); } -/** - * Write permissions to settings.json - */ export function writePermissions(location: InstallLocation): void { - const settingsPath = getSettingsJsonPath(location); - const settings = readJsonFile(settingsPath); - - // Ensure permissions object exists - if (!settings.permissions) { - settings.permissions = {}; - } - - // Ensure allow array exists - if (!Array.isArray(settings.permissions.allow)) { - settings.permissions.allow = []; - } - - // Add CodeGraph permissions (avoiding duplicates) - const codegraphPermissions = getCodeGraphPermissions(); - for (const permission of codegraphPermissions) { - if (!settings.permissions.allow.includes(permission)) { - settings.permissions.allow.push(permission); - } - } - - writeJsonFile(settingsPath, settings); + writePermissionsEntry(location); } -/** - * Check if MCP config already exists for CodeGraph - */ export function hasMcpConfig(location: InstallLocation): boolean { - const claudeJsonPath = getClaudeJsonPath(location); - const config = readJsonFile(claudeJsonPath); + // local scope lives in ./.mcp.json (project scope); global is the + // user-scope ~/.claude.json. Mirrors the Claude target's paths. + const file = location === 'global' + ? path.join(os.homedir(), '.claude.json') + : path.join(process.cwd(), '.mcp.json'); + const config = readJsonFile(file); return !!config.mcpServers?.codegraph; } -/** - * Check if permissions already exist for CodeGraph - */ export function hasPermissions(location: InstallLocation): boolean { - const settingsPath = getSettingsJsonPath(location); - const settings = readJsonFile(settingsPath); - const permissions = settings.permissions?.allow; - if (!Array.isArray(permissions)) { - return false; - } - // Check if at least one CodeGraph permission exists - return permissions.some((p: string) => p.startsWith('mcp__codegraph__')); -} - -/** - * Get the path to CLAUDE.md - * - Global: ~/.claude/CLAUDE.md - * - Local: ./.claude/CLAUDE.md - */ -function getClaudeMdPath(location: InstallLocation): string { - const configDir = getClaudeConfigDir(location); - return path.join(configDir, 'CLAUDE.md'); -} - -/** - * Check if CLAUDE.md has CodeGraph section - */ -export function hasClaudeMdSection(location: InstallLocation): boolean { - const claudeMdPath = getClaudeMdPath(location); - try { - if (fs.existsSync(claudeMdPath)) { - const content = fs.readFileSync(claudeMdPath, 'utf-8'); - return content.includes(CODEGRAPH_SECTION_START) || content.includes('## CodeGraph'); - } - } catch { - // Ignore errors - } - return false; -} - -/** - * Write or update CLAUDE.md with CodeGraph instructions - * - * If the file exists and has a CodeGraph section (marked or unmarked), - * it will be replaced. Otherwise, the template is appended. - */ -export function writeClaudeMd(location: InstallLocation): { created: boolean; updated: boolean } { - const claudeMdPath = getClaudeMdPath(location); - const configDir = getClaudeConfigDir(location); - - // Ensure directory exists - if (!fs.existsSync(configDir)) { - fs.mkdirSync(configDir, { recursive: true }); - } - - // Check if file exists - if (!fs.existsSync(claudeMdPath)) { - // Create new file with just the CodeGraph section - atomicWriteFileSync(claudeMdPath, CLAUDE_MD_TEMPLATE + '\n'); - return { created: true, updated: false }; - } - - // Read existing content - let content = fs.readFileSync(claudeMdPath, 'utf-8'); - - // Check for marked section (from previous installer) - if (content.includes(CODEGRAPH_SECTION_START)) { - // Replace the marked section - const startIdx = content.indexOf(CODEGRAPH_SECTION_START); - const endIdx = content.indexOf(CODEGRAPH_SECTION_END); - - if (endIdx > startIdx) { - // Replace existing marked section - const before = content.substring(0, startIdx); - const after = content.substring(endIdx + CODEGRAPH_SECTION_END.length); - content = before + CLAUDE_MD_TEMPLATE + after; - atomicWriteFileSync(claudeMdPath, content); - return { created: false, updated: true }; - } - } - - // Check for unmarked "## CodeGraph" section (from manual setup) - const codegraphHeaderRegex = /\n## CodeGraph\n/; - const match = content.match(codegraphHeaderRegex); - - if (match && match.index !== undefined) { - // Find the end of the CodeGraph section (next h2 header or end of file) - // Use negative lookahead (?!#) to match "## X" but not "### X" - const sectionStart = match.index; - const afterSection = content.substring(sectionStart + 1); - const nextHeaderMatch = afterSection.match(/\n## (?!#)/); - - let sectionEnd: number; - if (nextHeaderMatch && nextHeaderMatch.index !== undefined) { - sectionEnd = sectionStart + 1 + nextHeaderMatch.index; - } else { - sectionEnd = content.length; - } - - // Replace the section - const before = content.substring(0, sectionStart); - const after = content.substring(sectionEnd); - content = before + '\n' + CLAUDE_MD_TEMPLATE + after; - atomicWriteFileSync(claudeMdPath, content); - return { created: false, updated: true }; - } - - // No existing section, append to end - content = content.trimEnd() + '\n\n' + CLAUDE_MD_TEMPLATE + '\n'; - atomicWriteFileSync(claudeMdPath, content); - return { created: false, updated: false }; + const file = location === 'global' + ? path.join(os.homedir(), '.claude', 'settings.json') + : path.join(process.cwd(), '.claude', 'settings.json'); + const settings = readJsonFile(file); + const allow = settings.permissions?.allow; + if (!Array.isArray(allow)) return false; + return allow.some((p: string) => p.startsWith('mcp__codegraph__')); } diff --git a/src/installer/index.ts b/src/installer/index.ts index 7d01af9b8..edd48ecaf 100644 --- a/src/installer/index.ts +++ b/src/installer/index.ts @@ -1,18 +1,44 @@ /** * CodeGraph Interactive Installer * - * Uses @clack/prompts for a polished interactive CLI experience. + * Multi-target: writes MCP server config + instructions for the + * agents the user picks (Claude Code, Cursor, Codex CLI, opencode, + * Hermes Agent, Gemini CLI, Antigravity IDE). + * Defaults to the Claude-only behavior for backwards compatibility + * when no targets are explicitly chosen and nothing else is detected. + * + * Uses @clack/prompts for the interactive UI; `runInstallerWithOptions` + * is the non-interactive entry point used by the `--target` / + * `--print-config` CLI flags. */ import { execSync } from 'child_process'; import * as path from 'path'; import * as fs from 'fs'; import { - writeMcpConfig, writePermissions, writeClaudeMd, - hasMcpConfig, hasPermissions, + ALL_TARGETS, + detectAll, + getTarget, + resolveTargetFlag, +} from './targets/registry'; +import type { AgentTarget, Location, TargetId } from './targets/types'; +import { getGlyphs } from '../ui/glyphs'; +// Import the lightweight submodules directly (not the ../sync barrel, which +// re-exports FileWatcher and would transitively pull in ../extraction — the +// installer must stay importable even when native modules can't load). +import { watchDisabledReason } from '../sync/watch-policy'; +import { isGitRepo, isSyncHookInstalled, installGitSyncHook } from '../sync/git-hooks'; + +// Backwards-compat: keep these named exports — downstream code may +// import them. The shim in `config-writer.ts` continues to re-export +// them too. +export { + writeMcpConfig, + writePermissions, + hasMcpConfig, + hasPermissions, } from './config-writer'; - -import type { InstallLocation } from './config-writer'; +export type { InstallLocation } from './config-writer'; // Dynamic import helper — tsc compiles import() to require() in CJS mode, // which fails for ESM-only packages. This bypasses the transformation. @@ -20,16 +46,10 @@ import type { InstallLocation } from './config-writer'; const importESM = new Function('specifier', 'return import(specifier)') as (specifier: string) => Promise; -/** - * Format a number with commas - */ function formatNumber(n: number): string { return n.toLocaleString(); } -/** - * Get the package version - */ function getVersion(): string { try { const packageJsonPath = path.join(__dirname, '..', '..', 'package.json'); @@ -40,125 +60,395 @@ function getVersion(): string { } } +export interface RunInstallerOptions { + /** Comma-separated target list, or `auto` / `all` / `none`. */ + target?: string; + /** Skip the location prompt; use this value directly. */ + location?: Location; + /** Skip the auto-allow prompt; use this value directly. */ + autoAllow?: boolean; + /** + * Skip every confirm and use defaults: location=global, + * autoAllow=true, target=auto. For scripting / CI. + */ + yes?: boolean; +} + /** - * Run the interactive installer + * Interactive entry point — preserves the historical UX (`codegraph + * install` with no args goes through the prompts), but now starts + * the targets multi-select pre-populated with detected agents. */ export async function runInstaller(): Promise { + return runInstallerWithOptions({}); +} + +export async function runInstallerWithOptions(opts: RunInstallerOptions): Promise { const clack = await importESM('@clack/prompts'); clack.intro(`CodeGraph v${getVersion()}`); - // Step 1: Install globally - const shouldInstallGlobally = await clack.confirm({ - message: 'Install codegraph globally? (Required for MCP server)', - initialValue: true, - }); + // --yes implies all defaults; explicit flags still win. + const useDefaults = opts.yes === true; + + // Step 1: which agent targets? Asked FIRST so the user knows what + // they're committing to before we touch npm or disk. Detection + // probes the user-provided location if known, else 'global' as the + // most common default — labels are a hint, not load-bearing. + const detectionLocation: Location = opts.location ?? 'global'; + const targets = await resolveTargets(clack, opts, detectionLocation, useDefaults); + if (targets.length === 0) { + clack.outro('No agent targets selected — nothing to do.'); + return; + } - if (clack.isCancel(shouldInstallGlobally)) { - clack.cancel('Installation cancelled.'); - process.exit(0); + // Step 2: install the codegraph npm package on PATH (always offered; + // matches existing behavior). Skipped when --yes (assume present). + if (!useDefaults) { + const shouldInstallGlobally = await clack.confirm({ + message: 'Install the codegraph CLI on your PATH? (Required so agents can launch the MCP server)', + initialValue: true, + }); + if (clack.isCancel(shouldInstallGlobally)) { + clack.cancel('Installation cancelled.'); + process.exit(0); + } + if (shouldInstallGlobally) { + const s = clack.spinner(); + s.start('Installing codegraph CLI...'); + try { + execSync('npm install -g @colbymchenry/codegraph', { stdio: 'pipe', windowsHide: true }); + s.stop('Installed codegraph CLI on PATH'); + } catch { + s.stop('Could not install (permission denied)'); + clack.log.warn('Try: sudo npm install -g @colbymchenry/codegraph'); + } + } else { + clack.log.info('Skipped CLI install — agents will not be able to launch the MCP server without it'); + } } - if (shouldInstallGlobally) { - const s = clack.spinner(); - s.start('Installing codegraph globally...'); - try { - execSync('npm install -g @colbymchenry/codegraph', { stdio: 'pipe' }); - s.stop('Installed codegraph globally'); - } catch { - s.stop('Could not install globally (permission denied)'); - clack.log.warn('Try: sudo npm install -g @colbymchenry/codegraph'); + // Step 3: where the per-agent config files should land. + let location: Location; + if (opts.location) { + location = opts.location; + } else if (useDefaults) { + location = 'global'; + } else { + // If every selected target is global-only (e.g. Codex), skip the + // prompt and force user-wide — project-local would just produce + // skip warnings. + const allGlobalOnly = targets.every((t) => !t.supportsLocation('local')); + if (allGlobalOnly) { + location = 'global'; + clack.log.info('Writing user-wide configs (selected agents have no project-local config).'); + } else { + const sel = await clack.select({ + message: 'Apply agent configs to all your projects, or just this one?', + options: [ + { value: 'global' as const, label: 'All projects', hint: '~/.claude, ~/.cursor, etc.' }, + { value: 'local' as const, label: 'Just this project', hint: './.claude, ./.cursor, etc.' }, + ], + initialValue: 'global' as const, + }); + if (clack.isCancel(sel)) { + clack.cancel('Installation cancelled.'); + process.exit(0); + } + location = sel; } + } + + // Step 4: auto-allow permissions (only meaningful for Claude; + // skipped silently by other targets). + let autoAllow: boolean; + if (opts.autoAllow !== undefined) { + autoAllow = opts.autoAllow; + } else if (useDefaults) { + autoAllow = true; + } else if (targets.some((t) => t.id === 'claude')) { + const ans = await clack.confirm({ + message: 'Auto-allow CodeGraph commands? (Skips permission prompts in Claude Code)', + initialValue: true, + }); + if (clack.isCancel(ans)) { + clack.cancel('Installation cancelled.'); + process.exit(0); + } + autoAllow = ans; } else { - clack.log.info('Skipped global install — MCP server may not work without it'); + autoAllow = false; } - // Step 2: Installation location - const location = await clack.select({ - message: 'Where would you like to install?', - options: [ - { value: 'global' as const, label: 'Global', hint: '~/.claude — available in all projects' }, - { value: 'local' as const, label: 'Local', hint: './.claude — this project only' }, - ], - initialValue: 'global' as const, - }); + // Step 5: per-target install loop. + for (const target of targets) { + if (!target.supportsLocation(location)) { + clack.log.warn( + `${target.displayName}: skipped — does not support --location=${location}.`, + ); + continue; + } + const result = target.install(location, { autoAllow }); + for (const file of result.files) { + const verb = file.action === 'unchanged' + ? 'Unchanged' + : file.action === 'created' ? 'Created' + : file.action === 'removed' ? 'Removed' + : 'Updated'; + clack.log.success(`${target.displayName}: ${verb} ${tildify(file.path)}`); + } + for (const note of result.notes ?? []) { + clack.log.info(`${target.displayName}: ${note}`); + } + } - if (clack.isCancel(location)) { - clack.cancel('Installation cancelled.'); - process.exit(0); + // Step 6: for local install, initialize the project. + if (location === 'local') { + await initializeLocalProject(clack, useDefaults); } - // Step 3: Auto-allow permissions - const autoAllow = await clack.confirm({ - message: 'Auto-allow CodeGraph commands? (Skips permission prompts)', - initialValue: true, + if (location === 'global') { + clack.note('cd your-project\ncodegraph init -i', 'Quick start'); + } + + const finalNote = targets.length > 0 + ? `Done! Restart your agent${targets.length > 1 ? 's' : ''} to use CodeGraph.` + : 'Done!'; + clack.outro(finalNote); +} + +export interface RunUninstallerOptions { + /** + * Comma-separated target list, or `auto` / `all` / `none`. Defaults + * to `all` — uninstall sweeps every known agent and reports which + * ones it actually touched, so the user doesn't have to know where + * they configured it. + */ + target?: string; + /** Skip the location prompt; use this value directly. */ + location?: Location; + /** Non-interactive: location=global, target=all, no prompts. */ + yes?: boolean; +} + +export type UninstallStatus = 'removed' | 'not-configured' | 'unsupported'; + +/** + * Per-target outcome of an uninstall sweep. `removed` means we deleted + * at least one thing; `not-configured` means the agent had no codegraph + * config at this location (nothing to do); `unsupported` means the + * agent has no config concept for this location (e.g. Codex is + * global-only, so a `local` uninstall skips it). + */ +export interface UninstallReport { + id: TargetId; + displayName: string; + status: UninstallStatus; + /** Absolute paths we actually edited/removed (action === 'removed'). */ + removedPaths: string[]; + /** Verbatim notes from the target (rare for uninstall). */ + notes: string[]; +} + +/** + * Pure uninstall sweep — no prompts, no I/O beyond the targets' own + * file edits. Exposed (and unit-tested) separately from the clack UI in + * `runUninstaller` so the aggregation logic can be asserted directly. + * + * Each target's `uninstall()` is already safe to call when nothing was + * installed (it returns `not-found` actions), so this is safe to run + * across every target unconditionally. + */ +export function uninstallTargets( + targets: readonly AgentTarget[], + location: Location, +): UninstallReport[] { + return targets.map((target) => { + if (!target.supportsLocation(location)) { + const only: Location = location === 'local' ? 'global' : 'local'; + return { + id: target.id, + displayName: target.displayName, + status: 'unsupported' as const, + removedPaths: [], + notes: [`no ${location} config — this agent is ${only}-only`], + }; + } + const result = target.uninstall(location); + const removedPaths = result.files + .filter((f) => f.action === 'removed') + .map((f) => f.path); + return { + id: target.id, + displayName: target.displayName, + status: removedPaths.length > 0 ? ('removed' as const) : ('not-configured' as const), + removedPaths, + notes: result.notes ?? [], + }; }); +} - if (clack.isCancel(autoAllow)) { - clack.cancel('Installation cancelled.'); - process.exit(0); +/** + * Interactive uninstaller — the inverse of `runInstallerWithOptions`. + * Asks global-vs-local first (unless `--location`/`--yes` is given), + * then sweeps every agent target (or the `--target` subset) and prints + * one block per agent so the user sees exactly which providers it hit. + * + * Removes only what install wrote (MCP server entry, instructions + * block, permissions) — never the `.codegraph/` index, which `codegraph + * uninit` owns. + */ +export async function runUninstaller(opts: RunUninstallerOptions): Promise { + const clack = await importESM('@clack/prompts'); + + clack.intro(`CodeGraph v${getVersion()} — uninstall`); + + const useDefaults = opts.yes === true; + + // Step 1: which location — asked FIRST, the one decision the user + // must make. Global sweeps ~/.claude, ~/.codex, etc.; local sweeps + // the configs in this project directory. + let location: Location; + if (opts.location) { + location = opts.location; + } else if (useDefaults) { + location = 'global'; + } else { + const sel = await clack.select({ + message: 'Remove CodeGraph from all your projects, or just this one?', + options: [ + { value: 'global' as const, label: 'All projects (global)', hint: '~/.claude, ~/.cursor, ~/.codex, ~/.config/opencode, ~/.hermes, ~/.gemini, ~/.kiro' }, + { value: 'local' as const, label: 'Just this project (local)', hint: './.claude, ./.cursor, ./opencode.jsonc, ./.gemini, ./.kiro' }, + ], + initialValue: 'global' as const, + }); + if (clack.isCancel(sel)) { + clack.cancel('Uninstall cancelled.'); + process.exit(0); + } + location = sel; } - // Step 4: Write configuration files - writeConfigs(clack, location, autoAllow); + // Step 2: which agents. Default is every agent, so the user doesn't + // have to remember where they installed it — unconfigured agents are + // reported as "nothing to remove" and left untouched. An explicit + // --target subsets this. + let targets: AgentTarget[]; + if (opts.target !== undefined) { + targets = resolveTargetFlag(opts.target, location); + } else { + targets = [...ALL_TARGETS]; + } + if (targets.length === 0) { + clack.outro('No agent targets selected — nothing to do.'); + return; + } - // Step 5: For local install, initialize the project - if (location === 'local') { - await initializeLocalProject(clack); + // Step 3: sweep + per-agent feedback. + const reports = uninstallTargets(targets, location); + const removed = reports.filter((r) => r.status === 'removed'); + + for (const r of reports) { + if (r.status === 'removed') { + for (const p of r.removedPaths) { + clack.log.success(`${r.displayName}: removed ${tildify(p)}`); + } + } else if (r.status === 'not-configured') { + clack.log.info(`${r.displayName}: not configured — nothing to remove`); + } else { + clack.log.info(`${r.displayName}: skipped — ${r.notes[0] ?? 'unsupported location'}`); + } } - // Done - if (location === 'global') { - clack.note( - 'cd your-project\ncodegraph init -i', - 'Quick start', - ); + // Step 4: for local uninstall, the index dir is separate — point at + // `uninit` so the user knows it's still there (and how to remove it). + if (location === 'local' && fs.existsSync(path.join(process.cwd(), '.codegraph'))) { + clack.log.info('The .codegraph/ index for this project is still here. Run `codegraph uninit` to delete it.'); } - clack.outro('Done! Restart Claude Code to use CodeGraph.'); + // Step 5: summary. + if (removed.length > 0) { + const names = removed.map((r) => r.displayName).join(', '); + clack.outro( + `Removed CodeGraph from ${removed.length} agent${removed.length > 1 ? 's' : ''}: ${names}. ` + + `Restart ${removed.length > 1 ? 'them' : 'it'} to apply.`, + ); + } else { + clack.outro(`CodeGraph was not configured in any ${location} agent — nothing to remove.`); + } } /** - * Write all configuration files and log results + * Replace home-directory prefix in a path with `~/` for cleaner log + * lines. Pure cosmetic. */ -function writeConfigs( +function tildify(p: string): string { + const home = require('os').homedir(); + if (p.startsWith(home + path.sep)) return '~' + p.substring(home.length); + return p; +} + +async function resolveTargets( clack: typeof import('@clack/prompts'), - location: InstallLocation, - autoAllow: boolean, -): void { - const locationLabel = location === 'global' ? '~/.claude' : './.claude'; - - // MCP config - const mcpAction = hasMcpConfig(location) ? 'Updated' : 'Added'; - writeMcpConfig(location); - clack.log.success(`${mcpAction} MCP server in ${locationLabel}.json`); - - // Permissions - if (autoAllow) { - const permAction = hasPermissions(location) ? 'Updated' : 'Added'; - writePermissions(location); - clack.log.success(`${permAction} permissions in ${locationLabel}/settings.json`); - } - - // CLAUDE.md - const claudeMdResult = writeClaudeMd(location); - const claudeMdPath = `${locationLabel}/CLAUDE.md`; - if (claudeMdResult.created) { - clack.log.success(`Created ${claudeMdPath}`); - } else if (claudeMdResult.updated) { - clack.log.success(`Updated ${claudeMdPath}`); - } else { - clack.log.success(`Added CodeGraph instructions to ${claudeMdPath}`); + opts: RunInstallerOptions, + location: Location, + useDefaults: boolean, +): Promise { + // Explicit --target flag wins. + if (opts.target !== undefined) { + return resolveTargetFlag(opts.target, location); + } + + // --yes implies auto-detect. + if (useDefaults) { + return resolveTargetFlag('auto', location); + } + + // Interactive multi-select. + const detected = detectAll(location); + const initialValues = detected + .filter(({ detection }) => detection.installed) + .map(({ target }) => target.id); + // If nothing detected, default to Claude alone (matches the + // historical default and the smallest-surprise outcome). + const initial = initialValues.length > 0 ? initialValues : ['claude']; + + const choice = await clack.multiselect({ + message: 'Which agents should CodeGraph configure?', + options: ALL_TARGETS.map((t) => { + const det = detected.find(({ target }) => target.id === t.id)!.detection; + const flag = det.installed ? '(detected)' : '(not found)'; + const globalOnly = !t.supportsLocation('local') ? ' — global only' : ''; + return { + value: t.id, + label: `${t.displayName} ${flag}${globalOnly}`, + }; + }), + initialValues: initial, + required: false, + }); + + if (clack.isCancel(choice)) { + clack.cancel('Installation cancelled.'); + process.exit(0); } + + return choice + .map((id) => getTarget(id)) + .filter((t): t is AgentTarget => t !== undefined); } /** - * Initialize CodeGraph in the current project (for local installs) + * Initialize CodeGraph in the current project (for local installs), then + * offer the watch fallback when the live watcher won't run here (see + * offerWatchFallback). Agent-agnostic by nature. */ -async function initializeLocalProject(clack: typeof import('@clack/prompts')): Promise { +async function initializeLocalProject( + clack: typeof import('@clack/prompts'), + useDefaults = false, +): Promise { const projectPath = process.cwd(); - // Lazy-load CodeGraph (requires native modules) let CodeGraph: typeof import('../index').default; try { CodeGraph = (await import('../index')).default; @@ -172,6 +462,7 @@ async function initializeLocalProject(clack: typeof import('@clack/prompts')): P // Check if already initialized if (CodeGraph.isInitialized(projectPath)) { clack.log.info('CodeGraph already initialized in this project'); + await offerWatchFallback(clack, projectPath, { yes: useDefaults }); return; } @@ -181,7 +472,7 @@ async function initializeLocalProject(clack: typeof import('@clack/prompts')): P // Index the project with shimmer progress (worker thread for smooth animation) const { createShimmerProgress } = await import('../ui/shimmer-progress'); - process.stdout.write(`\x1b[2m│\x1b[0m\n`); + process.stdout.write(`\x1b[2m${getGlyphs().rail}\x1b[0m\n`); const progress = createShimmerProgress(); const result = await cg.indexAll({ @@ -197,7 +488,77 @@ async function initializeLocalProject(clack: typeof import('@clack/prompts')): P } cg.close(); + + await offerWatchFallback(clack, projectPath, { yes: useDefaults }); } -// Re-export for CLI -export type { InstallLocation }; +/** + * When the live file watcher will be disabled for this project (e.g. WSL2 + * /mnt drives, or CODEGRAPH_NO_WATCH), the index would silently go stale. + * Explain that, and offer to keep it fresh automatically via git hooks + * (commit / pull / checkout) instead of manual `codegraph sync`. + * + * No-op on environments where the watcher runs normally, so it's safe to + * call unconditionally after init. + */ +export async function offerWatchFallback( + clack: typeof import('@clack/prompts'), + projectPath: string, + opts: { yes?: boolean } = {}, +): Promise { + const reason = watchDisabledReason(projectPath); + if (!reason) return; // Watcher runs normally — nothing to set up. + + clack.log.warn(`Live file watching is disabled here — ${reason}.`); + clack.log.info('Until you re-sync, the CodeGraph index stays frozen — it will not pick up edits on its own.'); + + // No git repo → the commit-hook path doesn't apply; point at manual sync. + if (!isGitRepo(projectPath)) { + clack.log.info('Run `codegraph sync` after changing files to refresh the index.'); + return; + } + + // Already wired up on a previous run — confirm and move on without nagging. + if (isSyncHookInstalled(projectPath)) { + clack.log.info('Git sync hooks are already installed — the index refreshes after commit / pull / checkout.'); + return; + } + + let choice: 'hook' | 'manual'; + if (opts.yes) { + choice = 'hook'; + } else { + const sel = await clack.select({ + message: 'How should CodeGraph keep its index fresh?', + options: [ + { value: 'hook' as const, label: 'Sync on git commit / pull / checkout', hint: 'installs git hooks (recommended)' }, + { value: 'manual' as const, label: 'I\'ll run `codegraph sync` myself', hint: 'fully manual' }, + ], + initialValue: 'hook' as const, + }); + if (clack.isCancel(sel)) { + clack.log.info('Skipped — run `codegraph sync` after changes to refresh the index.'); + return; + } + choice = sel; + } + + if (choice === 'manual') { + clack.log.info('Run `codegraph sync` after changing files to refresh the index.'); + return; + } + + const result = installGitSyncHook(projectPath); + if (result.installed.length > 0) { + clack.log.success( + `Installed git ${result.installed.join(', ')} hook${result.installed.length > 1 ? 's' : ''} — ` + + 'the index refreshes in the background after each.', + ); + clack.log.info('Run `codegraph sync` anytime to refresh immediately.'); + } else { + clack.log.warn( + `Could not install git hooks${result.skipped ? ` (${result.skipped})` : ''}. ` + + 'Run `codegraph sync` after changes instead.', + ); + } +} diff --git a/src/installer/instructions-template.ts b/src/installer/instructions-template.ts new file mode 100644 index 000000000..e4040927c --- /dev/null +++ b/src/installer/instructions-template.ts @@ -0,0 +1,18 @@ +/** + * Marker constants for the legacy agent-instructions block. + * + * Codegraph used to write a `## CodeGraph` usage guide into each + * agent's instructions file (CLAUDE.md / AGENTS.md / GEMINI.md / + * codegraph.mdc / Kiro steering doc). That duplicated the guidance the + * MCP server already emits in its `initialize` response — every agent + * read the same playbook twice each turn (issue #529). The installer no + * longer writes an instructions file; the MCP server instructions in + * `mcp/server-instructions.ts` are the single source of truth. + * + * These markers are retained so install (self-heal on upgrade) and + * uninstall can find and strip the block a previous install wrote. + */ + +/** Markers used by the marker-based section removal. */ +export const CODEGRAPH_SECTION_START = ''; +export const CODEGRAPH_SECTION_END = ''; diff --git a/src/installer/targets/antigravity.ts b/src/installer/targets/antigravity.ts new file mode 100644 index 000000000..1c128491a --- /dev/null +++ b/src/installer/targets/antigravity.ts @@ -0,0 +1,289 @@ +/** + * Google Antigravity IDE target. Antigravity is Google's VS Code-derived + * multi-agent IDE; the Gemini CLI is in the process of consolidating with + * it under a single agent platform. Antigravity reads MCP server + * definitions from a separate config file from the CLI. + * + * ## Config path: unified vs legacy + * + * Antigravity recently migrated to a **unified** MCP config path shared + * across all Antigravity tools: + * + * - **Unified** (post-migration, current): `~/.gemini/config/mcp_config.json` + * — signalled by the `~/.gemini/config/.migrated` marker file. + * - **Legacy** (pre-migration): `~/.gemini/antigravity/mcp_config.json` + * — what the github-mcp-server install guide still documents. + * + * We detect the marker at install time and write to the right path. On + * uninstall we sweep BOTH — so a user who installed on the legacy path, + * was then auto-migrated by Antigravity, and re-ran `codegraph install` + * doesn't end up with stale codegraph entries in two files. + * + * ## Entry shape: no `type: stdio` field + * + * Antigravity rejects MCP entries that carry the `type: "stdio"` field + * the rest of our targets use — the working entries it manages itself + * (e.g. `code-review-graph`) omit it, and dropping it was load-bearing + * to get codegraph to appear in the Customizations UI. We build the + * entry locally instead of routing through `getMcpServerConfig()`. + * + * ## macOS GUI app PATH resolution + * + * Antigravity is a GUI Electron app. macOS gives Dock/Finder-launched + * apps a stripped PATH (`/usr/bin:/bin:/usr/sbin:/sbin`) — nvm-managed + * tools live outside that, so a bare `codegraph` command fails to spawn + * even when `which codegraph` resolves in the user's shell. We resolve + * `codegraph` to its absolute path on macOS at install time. (Linux GUI + * apps inherit user PATH; Windows uses `PATH` env directly — both are + * fine with the bare command.) + * + * ## Shared instructions (no GEMINI.md from here) + * + * The IDE shares `~/.gemini/GEMINI.md` with Gemini CLI for instructions + * — written by the `./gemini.ts` target. We deliberately don't touch it + * here so uninstalling Antigravity without uninstalling Gemini CLI + * leaves CLI instructions intact. Users who install only Antigravity + * still get a working MCP integration; the prefer-codegraph-over-grep + * guidance just won't be present unless they also install the gemini + * target. + * + * ## Location + * + * `supportsLocation('local')` returns false — Antigravity has no + * project-scoped config concept as of 2026-05. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { execSync } from 'child_process'; +import { + AgentTarget, + DetectionResult, + InstallOptions, + Location, + WriteResult, +} from './types'; +import { + jsonDeepEqual, + readJsonFile, + writeJsonFile, +} from './shared'; + +function unifiedConfigDir(): string { + return path.join(os.homedir(), '.gemini', 'config'); +} +function unifiedMcpConfigPath(): string { + return path.join(unifiedConfigDir(), 'mcp_config.json'); +} +function legacyConfigDir(): string { + return path.join(os.homedir(), '.gemini', 'antigravity'); +} +function legacyMcpConfigPath(): string { + return path.join(legacyConfigDir(), 'mcp_config.json'); +} +function migratedMarkerPath(): string { + return path.join(unifiedConfigDir(), '.migrated'); +} + +/** + * Pick the right MCP config path to write to. + * + * Prefers the unified `~/.gemini/config/mcp_config.json` when Antigravity + * has signalled it's migrated (`.migrated` marker present, OR the + * unified file already exists — Antigravity creates it on first + * launch post-migration). Falls back to the legacy + * `~/.gemini/antigravity/mcp_config.json` for users on a pre-migration + * Antigravity build. + */ +function preferredMcpConfigPath(): string { + if (fs.existsSync(migratedMarkerPath())) return unifiedMcpConfigPath(); + if (fs.existsSync(unifiedMcpConfigPath())) return unifiedMcpConfigPath(); + return legacyMcpConfigPath(); +} + +/** + * Resolve the on-disk path of the `codegraph` binary so a Mac GUI app + * launched from Dock/Finder (with a stripped PATH) can find it. Falls + * back to the bare `codegraph` name when: + * + * - we're not on macOS (Linux GUI apps inherit user PATH; Windows + * uses env PATH directly), OR + * - the lookup fails for any reason (preserving install in restricted + * environments where `which`/`command -v` aren't available). + * + * Resolution prefers `command -v` (built-in, no PATH manipulation), + * with `which` as a fallback. Both are read via the user's interactive + * shell PATH at install time — that's the right PATH for finding + * nvm-managed tools like ours. + */ +function resolveCodegraphCommand(): string { + if (process.platform !== 'darwin') return 'codegraph'; + try { + const resolved = execSync('command -v codegraph || which codegraph', { + encoding: 'utf-8', + stdio: ['ignore', 'pipe', 'ignore'], + shell: '/bin/bash', + windowsHide: true, + }).trim(); + if (resolved && fs.existsSync(resolved)) return resolved; + } catch { + /* fall through to bare name */ + } + return 'codegraph'; +} + +/** + * Build the codegraph MCP-server entry for Antigravity. Distinct from + * `getMcpServerConfig()` because Antigravity (a) rejects the `type` + * field and (b) needs an absolute command path on macOS — see file + * header. + */ +function buildAntigravityEntry(): { command: string; args: string[] } { + return { + command: resolveCodegraphCommand(), + args: ['serve', '--mcp'], + }; +} + +class AntigravityTarget implements AgentTarget { + readonly id = 'antigravity' as const; + readonly displayName = 'Antigravity IDE'; + readonly docsUrl = 'https://antigravity.google'; + + supportsLocation(loc: Location): boolean { + return loc === 'global'; + } + + detect(loc: Location): DetectionResult { + if (loc !== 'global') { + return { installed: false, alreadyConfigured: false }; + } + const file = preferredMcpConfigPath(); + const config = readJsonFile(file); + const alreadyConfigured = !!config.mcpServers?.codegraph; + // "Installed" heuristic: either the unified config dir, the legacy + // config dir, or one of the config files exists. Antigravity creates + // ~/.gemini/ on first launch even before MCP configs. + const installed = + fs.existsSync(unifiedConfigDir()) || + fs.existsSync(legacyConfigDir()) || + fs.existsSync(file); + return { installed, alreadyConfigured, configPath: file }; + } + + install(loc: Location, _opts: InstallOptions): WriteResult { + if (loc !== 'global') { + return { + files: [], + notes: ['Antigravity IDE has no project-local config — re-run with --location=global.'], + }; + } + const files: WriteResult['files'] = []; + files.push(writeMcpEntry()); + // If the user originally installed on the legacy path and Antigravity + // has since migrated, strip the stale legacy entry so they don't + // wind up with two competing codegraph configs. + const legacyCleanup = cleanupLegacyEntry(); + if (legacyCleanup) files.push(legacyCleanup); + return { + files, + notes: ['Restart Antigravity for MCP changes to take effect.'], + }; + } + + uninstall(loc: Location): WriteResult { + if (loc !== 'global') return { files: [] }; + const files: WriteResult['files'] = []; + + // Remove from the preferred path. + const preferred = preferredMcpConfigPath(); + files.push(removeCodegraphFromFile(preferred)); + + // Also sweep the OTHER path (legacy when preferred is unified, and + // vice versa) — handles the migration-half-state case where codegraph + // got written to one file but Antigravity now reads from the other. + const other = preferred === unifiedMcpConfigPath() + ? legacyMcpConfigPath() + : unifiedMcpConfigPath(); + if (preferred !== other) { + const otherResult = removeCodegraphFromFile(other); + // Only surface the secondary file if we actually touched it — + // a `not-found` on a file the user never had is noise. + if (otherResult.action === 'removed') files.push(otherResult); + } + + return { files }; + } + + printConfig(loc: Location): string { + if (loc !== 'global') { + return '# Antigravity IDE has no project-local config — use --location=global.\n'; + } + const file = preferredMcpConfigPath(); + const snippet = JSON.stringify({ mcpServers: { codegraph: buildAntigravityEntry() } }, null, 2); + return `# Add to ${file}\n\n${snippet}\n`; + } + + describePaths(loc: Location): string[] { + if (loc !== 'global') return []; + return [preferredMcpConfigPath()]; + } +} + +function writeMcpEntry(): WriteResult['files'][number] { + const file = preferredMcpConfigPath(); + const dir = path.dirname(file); + if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true }); + + const existing = readJsonFile(file); + const before = existing.mcpServers?.codegraph; + const after = buildAntigravityEntry(); + + if (jsonDeepEqual(before, after)) { + return { path: file, action: 'unchanged' }; + } + const action: 'created' | 'updated' = + before ? 'updated' : (fs.existsSync(file) ? 'updated' : 'created'); + if (!existing.mcpServers) existing.mcpServers = {}; + existing.mcpServers.codegraph = after; + writeJsonFile(file, existing); + return { path: file, action }; +} + +/** + * Strip the codegraph entry from the legacy `~/.gemini/antigravity/mcp_config.json` + * if it's present AND we're writing to the unified path. Used by install + * to migrate users who had codegraph configured on the legacy path + * before Antigravity migrated their config. Returns the file action for + * reporting, or `null` when there's nothing to clean up. + */ +function cleanupLegacyEntry(): WriteResult['files'][number] | null { + if (preferredMcpConfigPath() !== unifiedMcpConfigPath()) return null; + const legacy = legacyMcpConfigPath(); + if (!fs.existsSync(legacy)) return null; + const config = readJsonFile(legacy); + if (!config.mcpServers?.codegraph) return null; + delete config.mcpServers.codegraph; + if (Object.keys(config.mcpServers).length === 0) { + delete config.mcpServers; + } + writeJsonFile(legacy, config); + return { path: legacy, action: 'removed' }; +} + +function removeCodegraphFromFile(file: string): WriteResult['files'][number] { + if (!fs.existsSync(file)) return { path: file, action: 'not-found' }; + const config = readJsonFile(file); + if (!config.mcpServers?.codegraph) return { path: file, action: 'not-found' }; + delete config.mcpServers.codegraph; + if (Object.keys(config.mcpServers).length === 0) { + delete config.mcpServers; + } + // Leave a now-empty `{}` in place — Antigravity manages this file and + // a stray empty file is less surprising than a deletion. + writeJsonFile(file, config); + return { path: file, action: 'removed' }; +} + +export const antigravityTarget: AgentTarget = new AntigravityTarget(); diff --git a/src/installer/targets/claude.ts b/src/installer/targets/claude.ts new file mode 100644 index 000000000..3259dea1b --- /dev/null +++ b/src/installer/targets/claude.ts @@ -0,0 +1,379 @@ +/** + * Claude Code target. Writes: + * + * - MCP server entry to `~/.claude.json` (global = user scope, loads + * in every project) or `./.mcp.json` (local = project scope, the + * file Claude Code actually reads for a single project). See the + * scope table at https://code.claude.com/docs/en/mcp. + * - Permissions to `~/.claude/settings.json` (global) or + * `./.claude/settings.json` (local), gated on `autoAllow`. + * - Instructions to `~/.claude/CLAUDE.md` (global) or + * `./.claude/CLAUDE.md` (local). + * + * Earlier versions wrote the local MCP entry to `./.claude.json` — a + * file Claude Code never reads — so the server silently never loaded + * until the user manually renamed it to `.mcp.json` (issue #207). We + * now write `./.mcp.json` and migrate any stale `./.claude.json` entry + * out of the way on install and uninstall. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { + AgentTarget, + DetectionResult, + InstallOptions, + Location, + WriteResult, +} from './types'; +import { + getCodeGraphPermissions, + getMcpServerConfig, + jsonDeepEqual, + readJsonFile, + removeMarkedSection, + writeJsonFile, +} from './shared'; +import { + CODEGRAPH_SECTION_END, + CODEGRAPH_SECTION_START, +} from '../instructions-template'; + +function configDir(loc: Location): string { + return loc === 'global' + ? path.join(os.homedir(), '.claude') + : path.join(process.cwd(), '.claude'); +} +function mcpJsonPath(loc: Location): string { + // global → ~/.claude.json (user scope: visible in every project). + // local → ./.mcp.json (project scope: the ONLY project-level MCP + // file Claude Code reads — NOT ./.claude.json, which it ignores). + return loc === 'global' + ? path.join(os.homedir(), '.claude.json') + : path.join(process.cwd(), '.mcp.json'); +} +/** + * Where pre-#207 installers wrote the local MCP entry. Claude Code + * never reads a project-level `./.claude.json`, so we migrate the + * codegraph entry out of it on install and strip it on uninstall. + * Only the project-local path is legacy — global `~/.claude.json` is + * the correct user-scope location and is left untouched. + */ +function legacyLocalMcpPath(): string { + return path.join(process.cwd(), '.claude.json'); +} +function settingsJsonPath(loc: Location): string { + return path.join(configDir(loc), 'settings.json'); +} +function instructionsPath(loc: Location): string { + return path.join(configDir(loc), 'CLAUDE.md'); +} + +class ClaudeCodeTarget implements AgentTarget { + readonly id = 'claude' as const; + readonly displayName = 'Claude Code'; + readonly docsUrl = 'https://docs.claude.com/en/docs/claude-code'; + + supportsLocation(_loc: Location): boolean { + return true; + } + + detect(loc: Location): DetectionResult { + const mcpPath = mcpJsonPath(loc); + const config = readJsonFile(mcpPath); + const alreadyConfigured = !!config.mcpServers?.codegraph; + // For "installed" we infer from the existence of either the dir + // (global) or the project marker file (local). Cheap and avoids + // shelling out to `claude --version`. + const installed = loc === 'global' + ? fs.existsSync(configDir(loc)) || fs.existsSync(mcpPath) + : fs.existsSync(mcpPath) || fs.existsSync(configDir(loc)); + return { installed, alreadyConfigured, configPath: mcpPath }; + } + + install(loc: Location, opts: InstallOptions): WriteResult { + const files: WriteResult['files'] = []; + + // 1. MCP server entry + files.push(writeMcpEntry(loc)); + + // 1b. Migrate away any stale ./.claude.json left by a pre-#207 + // local install, so the project isn't left with two competing + // (one dead) MCP configs. + if (loc === 'local') { + const migrated = cleanupLegacyLocalMcp(); + if (migrated) files.push(migrated); + } + + // 2. Permissions (only when autoAllow) + if (opts.autoAllow) { + files.push(writePermissionsEntry(loc)); + } + + // 2b. Strip stale auto-sync hooks left by a pre-0.8 install. Those + // versions wrote `codegraph mark-dirty` / `sync-if-dirty` hooks to + // settings.json; both subcommands are gone from the CLI, so the + // Stop hook now fails every turn with "unknown command + // 'sync-if-dirty'". Cleaning up on install makes an upgrade + // self-healing. Only surfaced when something was actually removed. + const hookCleanup = cleanupLegacyHooks(loc); + if (hookCleanup.action === 'removed') files.push(hookCleanup); + + // 3. CLAUDE.md instructions — no longer written. The codegraph + // usage guidance now ships solely in the MCP server's `initialize` + // response (see `mcp/server-instructions.ts`), which Claude Code + // surfaces in the system prompt automatically. Writing it into + // CLAUDE.md as well meant the agent read the same playbook twice + // every turn (issue #529). Strip any block a previous install left + // behind so an upgrade self-heals — same idiom as the hook cleanup. + const instrCleanup = removeInstructionsEntry(loc); + if (instrCleanup.action === 'removed') files.push(instrCleanup); + + return { files }; + } + + uninstall(loc: Location): WriteResult { + const files: WriteResult['files'] = []; + + // 1. MCP server entry + const mcpPath = mcpJsonPath(loc); + const config = readJsonFile(mcpPath); + if (config.mcpServers?.codegraph) { + delete config.mcpServers.codegraph; + if (Object.keys(config.mcpServers).length === 0) { + delete config.mcpServers; + } + writeJsonFile(mcpPath, config); + files.push({ path: mcpPath, action: 'removed' }); + } else { + files.push({ path: mcpPath, action: 'not-found' }); + } + + // 1b. Also strip the codegraph entry from a legacy ./.claude.json + // so uninstall fully reverses a pre-#207 local install. + if (loc === 'local') { + const migrated = cleanupLegacyLocalMcp(); + if (migrated) files.push(migrated); + } + + // 2. Permissions + const settingsPath = settingsJsonPath(loc); + const settings = readJsonFile(settingsPath); + if (Array.isArray(settings.permissions?.allow)) { + const before = settings.permissions.allow.length; + settings.permissions.allow = settings.permissions.allow.filter( + (p: string) => !p.startsWith('mcp__codegraph__'), + ); + if (settings.permissions.allow.length !== before) { + if (settings.permissions.allow.length === 0) { + delete settings.permissions.allow; + } + if (Object.keys(settings.permissions).length === 0) { + delete settings.permissions; + } + writeJsonFile(settingsPath, settings); + files.push({ path: settingsPath, action: 'removed' }); + } else { + files.push({ path: settingsPath, action: 'not-found' }); + } + } else { + files.push({ path: settingsPath, action: 'not-found' }); + } + + // 2b. Strip any stale auto-sync hooks a pre-0.8 install left in + // settings.json. The hook-cleanup step was lost when the installer + // moved to the per-target architecture; restoring it here means + // uninstall — and the npm `preuninstall` hook that drives it — fully + // reverses a legacy install. + const hookCleanup = cleanupLegacyHooks(loc); + if (hookCleanup.action === 'removed') files.push(hookCleanup); + + // 3. Instructions — strip the legacy CodeGraph block if present. + files.push(removeInstructionsEntry(loc)); + + return { files }; + } + + printConfig(loc: Location): string { + const target = mcpJsonPath(loc); + const snippet = JSON.stringify({ mcpServers: { codegraph: getMcpServerConfig() } }, null, 2); + return `# Add to ${target}\n\n${snippet}\n`; + } + + describePaths(loc: Location): string[] { + return [mcpJsonPath(loc), settingsJsonPath(loc), instructionsPath(loc)]; + } +} + +/** + * Per-file write helpers, exported so the legacy `config-writer.ts` + * shim can call only the named operation (writeMcpConfig writes ONLY + * the MCP entry, etc.) instead of `claudeTarget.install()` which + * writes all three files. Without this split the shims silently + * cause side effects callers don't expect. + */ +export function writeMcpEntry(loc: Location): WriteResult['files'][number] { + const file = mcpJsonPath(loc); + const existing = readJsonFile(file); + const before = existing.mcpServers?.codegraph; + const after = getMcpServerConfig(); + + if (jsonDeepEqual(before, after)) { + // Already exactly what we'd write — preserve byte-identical file. + return { path: file, action: 'unchanged' }; + } + // 'created' here means: the file itself did not exist before this + // write. A pre-existing MCP JSON file (`~/.claude.json` globally, + // `./.mcp.json` locally) containing other MCP servers (no + // `codegraph` key) is 'updated', not 'created' — we're adding an + // entry to a file that was already there. Codex uses a different + // idiom (empty-content => 'created') because its config.toml is + // ours alone to manage. + const action: 'created' | 'updated' = before ? 'updated' : (fs.existsSync(file) ? 'updated' : 'created'); + if (!existing.mcpServers) existing.mcpServers = {}; + existing.mcpServers.codegraph = after; + writeJsonFile(file, existing); + return { path: file, action }; +} + +/** + * Strip the codegraph entry from a legacy project-local + * `./.claude.json` (written by pre-#207 installers, which Claude Code + * never read). Surgical: only our `codegraph` key is removed; sibling + * MCP servers and any unrelated keys are preserved, and the file is + * deleted only when removal leaves it completely empty. Returns the + * file action for reporting, or `null` when there's nothing to migrate. + */ +function cleanupLegacyLocalMcp(): WriteResult['files'][number] | null { + const file = legacyLocalMcpPath(); + if (!fs.existsSync(file)) return null; + const config = readJsonFile(file); + if (!config.mcpServers?.codegraph) return null; + delete config.mcpServers.codegraph; + if (Object.keys(config.mcpServers).length === 0) delete config.mcpServers; + if (Object.keys(config).length === 0) { + try { fs.unlinkSync(file); } catch { /* ignore */ } + } else { + writeJsonFile(file, config); + } + return { path: file, action: 'removed' }; +} + +/** + * True when a Claude Code hook `command` is one of the auto-sync hooks + * a pre-0.8 install wrote. Those installers added + * `PostToolUse(Edit|Write) → codegraph mark-dirty` and + * `Stop → codegraph sync-if-dirty` (local builds used the + * `npx @colbymchenry/codegraph …` form, which still contains the + * `codegraph ` substring). Both subcommands were later + * removed from the CLI, so the Stop hook fails every turn with + * "unknown command 'sync-if-dirty'". Matching on the codegraph-scoped + * subcommand keeps unrelated user hooks (e.g. GitKraken's + * `gk ai hook run`) untouched. + */ +function isLegacyCodegraphHookCommand(command: unknown): boolean { + if (typeof command !== 'string') return false; + return ( + command.includes('codegraph mark-dirty') || + command.includes('codegraph sync-if-dirty') + ); +} + +/** + * Remove stale codegraph auto-sync hooks from Claude `settings.json`. + * + * Surgical at the individual-command level: only entries matching + * `isLegacyCodegraphHookCommand` are dropped, so a sibling hook sharing + * a matcher group (or the Stop event) with ours survives. We prune a + * matcher group only once its `hooks` array is empty, an event only + * once it has no groups left, and `hooks` itself only once every event + * is gone — and none of that runs unless we actually removed a + * codegraph command, so a settings.json with no legacy hooks is left + * byte-for-byte untouched and reported `unchanged`. + * + * Exported so it can be unit-tested directly and reused by both + * `install` (an upgrade self-heals) and `uninstall`. + */ +export function cleanupLegacyHooks(loc: Location): WriteResult['files'][number] { + const file = settingsJsonPath(loc); + if (!fs.existsSync(file)) return { path: file, action: 'not-found' }; + + const settings = readJsonFile(file); + const hooks = settings.hooks; + if (!hooks || typeof hooks !== 'object' || Array.isArray(hooks)) { + return { path: file, action: 'unchanged' }; + } + + // Pass 1: drop the legacy command(s) from inside every matcher group. + let removedAny = false; + for (const event of Object.keys(hooks)) { + const groups = hooks[event]; + if (!Array.isArray(groups)) continue; + for (const group of groups) { + if (!group || !Array.isArray(group.hooks)) continue; + const before = group.hooks.length; + group.hooks = group.hooks.filter( + (h: any) => !isLegacyCodegraphHookCommand(h?.command), + ); + if (group.hooks.length !== before) removedAny = true; + } + } + + if (!removedAny) return { path: file, action: 'unchanged' }; + + // Pass 2: prune empty matcher groups, then events with no groups + // left, then an empty top-level `hooks`. Guarded by `removedAny` so + // we never restructure a settings.json that had no codegraph hooks. + for (const event of Object.keys(hooks)) { + const groups = hooks[event]; + if (!Array.isArray(groups)) continue; + hooks[event] = groups.filter( + (g: any) => !(g && Array.isArray(g.hooks) && g.hooks.length === 0), + ); + if (hooks[event].length === 0) delete hooks[event]; + } + if (Object.keys(hooks).length === 0) delete settings.hooks; + + writeJsonFile(file, settings); + return { path: file, action: 'removed' }; +} + +export function writePermissionsEntry(loc: Location): WriteResult['files'][number] { + const file = settingsJsonPath(loc); + const settings = readJsonFile(file); + const created = !fs.existsSync(file); + + if (!settings.permissions) settings.permissions = {}; + if (!Array.isArray(settings.permissions.allow)) settings.permissions.allow = []; + + const want = getCodeGraphPermissions(); + const before = [...settings.permissions.allow]; + for (const perm of want) { + if (!settings.permissions.allow.includes(perm)) { + settings.permissions.allow.push(perm); + } + } + if (jsonDeepEqual(before, settings.permissions.allow) && !created) { + return { path: file, action: 'unchanged' }; + } + writeJsonFile(file, settings); + return { path: file, action: created ? 'created' : 'updated' }; +} + +/** + * Strip the marker-delimited CodeGraph block from CLAUDE.md if a prior + * install wrote one. Codegraph no longer maintains an instructions file + * (issue #529) — the MCP server's `initialize` instructions are the + * single source of truth — so both install (self-heal on upgrade) and + * uninstall call this. `removeMarkedSection` returns `not-found`/`kept` + * when there's nothing to strip; the install caller drops those from + * the report so a fresh install stays quiet. + */ +export function removeInstructionsEntry(loc: Location): WriteResult['files'][number] { + const file = instructionsPath(loc); + const action = removeMarkedSection(file, CODEGRAPH_SECTION_START, CODEGRAPH_SECTION_END); + return { path: file, action }; +} + +export const claudeTarget: AgentTarget = new ClaudeCodeTarget(); diff --git a/src/installer/targets/codex.ts b/src/installer/targets/codex.ts new file mode 100644 index 000000000..ccd9bf64e --- /dev/null +++ b/src/installer/targets/codex.ts @@ -0,0 +1,175 @@ +/** + * OpenAI Codex CLI target. + * + * - MCP server entry to `~/.codex/config.toml` as the dotted-key + * table `[mcp_servers.codegraph]`. TOML — not JSON — handled by + * the narrow serializer in `./toml.ts`. + * - Instructions to `~/.codex/AGENTS.md`. + * + * Codex CLI as of 2026-05 has no project-local config concept — + * everything lives under `~/.codex/`. `supportsLocation('local')` + * returns false; the orchestrator skips Codex when the user picks + * the local install location. + * + * No permissions concept. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { + AgentTarget, + DetectionResult, + InstallOptions, + Location, + WriteResult, +} from './types'; +import { + atomicWriteFileSync, + getMcpServerConfig, + removeMarkedSection, +} from './shared'; +import { + CODEGRAPH_SECTION_END, + CODEGRAPH_SECTION_START, +} from '../instructions-template'; +import { buildTomlTable, removeTomlTable, upsertTomlTable } from './toml'; + +const TOML_HEADER = 'mcp_servers.codegraph'; + +function configDir(): string { + return path.join(os.homedir(), '.codex'); +} +function tomlConfigPath(): string { + return path.join(configDir(), 'config.toml'); +} +function instructionsPath(): string { + return path.join(configDir(), 'AGENTS.md'); +} + +class CodexTarget implements AgentTarget { + readonly id = 'codex' as const; + readonly displayName = 'Codex CLI'; + readonly docsUrl = 'https://github.com/openai/codex'; + + supportsLocation(loc: Location): boolean { + return loc === 'global'; + } + + detect(loc: Location): DetectionResult { + if (loc !== 'global') { + return { installed: false, alreadyConfigured: false }; + } + const tomlPath = tomlConfigPath(); + let alreadyConfigured = false; + if (fs.existsSync(tomlPath)) { + try { + const content = fs.readFileSync(tomlPath, 'utf-8'); + alreadyConfigured = content.includes(`[${TOML_HEADER}]`); + } catch { /* ignore */ } + } + const installed = fs.existsSync(configDir()); + return { installed, alreadyConfigured, configPath: tomlPath }; + } + + install(loc: Location, _opts: InstallOptions): WriteResult { + if (loc !== 'global') { + return { + files: [], + notes: ['Codex CLI has no project-local config — re-run with --location=global to install.'], + }; + } + const files: WriteResult['files'] = []; + + files.push(writeMcpEntry()); + + // AGENTS.md is no longer written — the codegraph usage guidance + // ships in the MCP server's `initialize` response (issue #529). + // Strip a block a previous install left so an upgrade self-heals. + const instrCleanup = removeInstructionsEntry(); + if (instrCleanup.action === 'removed') files.push(instrCleanup); + + return { files }; + } + + uninstall(loc: Location): WriteResult { + if (loc !== 'global') return { files: [] }; + const files: WriteResult['files'] = []; + + const tomlPath = tomlConfigPath(); + if (fs.existsSync(tomlPath)) { + const content = fs.readFileSync(tomlPath, 'utf-8'); + const { content: nextContent, action } = removeTomlTable(content, TOML_HEADER); + if (action === 'removed') { + if (nextContent.trim() === '') { + try { fs.unlinkSync(tomlPath); } catch { /* ignore */ } + } else { + atomicWriteFileSync(tomlPath, nextContent.trimEnd() + '\n'); + } + files.push({ path: tomlPath, action: 'removed' }); + } else { + files.push({ path: tomlPath, action: 'not-found' }); + } + } else { + files.push({ path: tomlPath, action: 'not-found' }); + } + + files.push(removeInstructionsEntry()); + + return { files }; + } + + printConfig(loc: Location): string { + if (loc !== 'global') { + return '# Codex CLI has no project-local config — use --location=global.\n'; + } + const block = buildCodegraphBlock(); + return `# Add to ${tomlConfigPath()}\n\n${block}\n`; + } + + describePaths(loc: Location): string[] { + if (loc !== 'global') return []; + return [tomlConfigPath(), instructionsPath()]; + } +} + +function buildCodegraphBlock(): string { + const mcp = getMcpServerConfig(); + return buildTomlTable(TOML_HEADER, { + command: mcp.command, + args: mcp.args, + }); +} + +function writeMcpEntry(): WriteResult['files'][number] { + const file = tomlConfigPath(); + const dir = path.dirname(file); + if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true }); + + const block = buildCodegraphBlock(); + // Single read — `existing === ''` derives both "is the file empty + // or absent" and "what was its content," avoiding a TOCTOU window + // between two `fs.existsSync` calls. + const existing = fs.existsSync(file) ? fs.readFileSync(file, 'utf-8') : ''; + const created = existing.length === 0; + const { content: nextContent, action } = upsertTomlTable(existing, TOML_HEADER, block); + + if (action === 'unchanged') { + return { path: file, action: 'unchanged' }; + } + atomicWriteFileSync(file, nextContent); + return { path: file, action: created ? 'created' : 'updated' }; +} + +/** + * Strip the marker-delimited CodeGraph block from `~/.codex/AGENTS.md` + * if a prior install wrote one. Used by both install (self-heal on + * upgrade) and uninstall — see issue #529. + */ +function removeInstructionsEntry(): WriteResult['files'][number] { + const file = instructionsPath(); + const action = removeMarkedSection(file, CODEGRAPH_SECTION_START, CODEGRAPH_SECTION_END); + return { path: file, action }; +} + +export const codexTarget: AgentTarget = new CodexTarget(); diff --git a/src/installer/targets/cursor.ts b/src/installer/targets/cursor.ts new file mode 100644 index 000000000..c2d3595ca --- /dev/null +++ b/src/installer/targets/cursor.ts @@ -0,0 +1,247 @@ +/** + * Cursor target. + * + * - MCP server entry to `~/.cursor/mcp.json` (global) or + * `./.cursor/mcp.json` (local). Same `{mcpServers: {...}}` shape + * as Claude. + * - Instructions to `./.cursor/rules/codegraph.mdc` (project-local + * ONLY). Cursor's rules system is a project-scoped surface; + * global cursor rules aren't a stable convention as of 2026-05. + * For `--location=global`, only mcp.json is written. + * + * ## Why we hardcode `--path` for Cursor + * + * Cursor launches MCP-server subprocesses with a working directory + * that ISN'T the workspace root AND doesn't pass `rootUri` / + * `workspaceFolders` in the MCP initialize call. The codegraph MCP + * server's `process.cwd()` fallback therefore misses the workspace's + * `.codegraph/` and reports "not initialized" on every tool call. + * + * So we inject `--path` into the args ourselves: + * + * - `local` install: absolute path (we know it at install time). + * - `global` install: `${workspaceFolder}` — Cursor expands this to + * the open workspace's root, giving us per-workspace behavior + * from a single global config. + * + * Codex and Claude do not need this — they launch MCP servers with + * `cwd = workspace` and pass `rootUri`, respectively. + * + * No permissions concept — Cursor doesn't have an auto-allow list + * the installer can populate. `autoAllow` is silently ignored. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { + AgentTarget, + DetectionResult, + InstallOptions, + Location, + WriteResult, +} from './types'; +import { + atomicWriteFileSync, + getMcpServerConfig, + jsonDeepEqual, + readJsonFile, + writeJsonFile, +} from './shared'; +import { + CODEGRAPH_SECTION_END, + CODEGRAPH_SECTION_START, +} from '../instructions-template'; + +function mcpJsonPath(loc: Location): string { + return loc === 'global' + ? path.join(os.homedir(), '.cursor', 'mcp.json') + : path.join(process.cwd(), '.cursor', 'mcp.json'); +} +/** + * Cursor "rules" file. Only meaningful for the project-local + * location — Cursor reads `.cursor/rules/*.mdc` from the workspace + * root. There is no global equivalent. + */ +function rulesPath(): string { + return path.join(process.cwd(), '.cursor', 'rules', 'codegraph.mdc'); +} + +/** + * Cursor `.mdc` rules use YAML-ish frontmatter. `alwaysApply: true` + * makes the rule load on every conversation regardless of file + * patterns — appropriate for a tool-usage guide that's relevant + * whenever the user is asking the agent to navigate code. + */ +const MDC_FRONTMATTER = [ + '---', + 'description: CodeGraph MCP usage guide — when to use which tool', + 'alwaysApply: true', + '---', + '', +].join('\n'); + +class CursorTarget implements AgentTarget { + readonly id = 'cursor' as const; + readonly displayName = 'Cursor'; + readonly docsUrl = 'https://docs.cursor.com/context/model-context-protocol'; + + supportsLocation(_loc: Location): boolean { + // Both supported, but `local` writes more files (mcp.json + rules); + // `global` writes only mcp.json. The orchestrator surfaces the + // difference via describePaths. + return true; + } + + detect(loc: Location): DetectionResult { + const mcpPath = mcpJsonPath(loc); + const config = readJsonFile(mcpPath); + const alreadyConfigured = !!config.mcpServers?.codegraph; + // "Installed" heuristic: does ~/.cursor exist (global) or has the + // user opted into a project-local cursor config dir? + const installed = loc === 'global' + ? fs.existsSync(path.join(os.homedir(), '.cursor')) + : fs.existsSync(path.join(process.cwd(), '.cursor')); + return { installed, alreadyConfigured, configPath: mcpPath }; + } + + install(loc: Location, _opts: InstallOptions): WriteResult { + const files: WriteResult['files'] = []; + + files.push(writeMcpEntry(loc)); + + // We no longer write `.cursor/rules/codegraph.mdc` — the codegraph + // usage guidance ships in the MCP server's `initialize` response, + // the single source of truth (issue #529). Strip a rules file a + // previous install created so an upgrade self-heals. + if (loc === 'local') { + const rulesCleanup = removeRulesEntry(); + if (rulesCleanup.action === 'removed') files.push(rulesCleanup); + } + + return { + files, + notes: ['Restart Cursor for MCP changes to take effect.'], + }; + } + + uninstall(loc: Location): WriteResult { + const files: WriteResult['files'] = []; + + const mcpPath = mcpJsonPath(loc); + const config = readJsonFile(mcpPath); + if (config.mcpServers?.codegraph) { + delete config.mcpServers.codegraph; + if (Object.keys(config.mcpServers).length === 0) { + delete config.mcpServers; + } + writeJsonFile(mcpPath, config); + files.push({ path: mcpPath, action: 'removed' }); + } else { + files.push({ path: mcpPath, action: 'not-found' }); + } + + if (loc === 'local') { + files.push(removeRulesEntry()); + } + + return { files }; + } + + printConfig(loc: Location): string { + const target = mcpJsonPath(loc); + const snippet = JSON.stringify({ mcpServers: { codegraph: buildCursorMcpConfig(loc) } }, null, 2); + return `# Add to ${target}\n\n${snippet}\n`; + } + + describePaths(loc: Location): string[] { + return loc === 'local' + ? [mcpJsonPath(loc), rulesPath()] + : [mcpJsonPath(loc)]; + } +} + +/** + * Build the codegraph MCP-server config for Cursor at the given + * location. Inherits the shared shape ({type, command, args}) and + * appends `--path` so the spawned MCP server resolves the workspace + * correctly regardless of Cursor's launch cwd. See file header for + * the full rationale. + */ +function buildCursorMcpConfig(loc: Location): { type: string; command: string; args: string[] } { + const base = getMcpServerConfig(); + const pathArg = loc === 'local' ? process.cwd() : '${workspaceFolder}'; + return { ...base, args: [...base.args, '--path', pathArg] }; +} + +function writeMcpEntry(loc: Location): WriteResult['files'][number] { + const file = mcpJsonPath(loc); + const existing = readJsonFile(file); + const before = existing.mcpServers?.codegraph; + const after = buildCursorMcpConfig(loc); + + if (jsonDeepEqual(before, after)) { + return { path: file, action: 'unchanged' }; + } + const action: 'created' | 'updated' = before ? 'updated' : (fs.existsSync(file) ? 'updated' : 'created'); + if (!existing.mcpServers) existing.mcpServers = {}; + existing.mcpServers.codegraph = after; + writeJsonFile(file, existing); + return { path: file, action }; +} + +/** + * Remove the Cursor rules file on uninstall (and as a self-heal on + * install — see issue #529). + * + * Unlike the shared CLAUDE.md / AGENTS.md files (where codegraph owns + * only a marker-delimited section), `.cursor/rules/codegraph.mdc` is a + * file we create OUTRIGHT — the frontmatter is ours too. So a plain + * `removeMarkedSection` is wrong here: it would strip our instruction + * block but leave the orphaned `description: CodeGraph ...` frontmatter + * behind, so the file lingers and still "mentions" codegraph. + * + * Instead: strip our block, and if nothing but our own frontmatter + * remains, delete the whole file. Only when the user has added their + * own content outside our markers do we keep the file (minus our block). + */ +function removeRulesEntry(): WriteResult['files'][number] { + const file = rulesPath(); + if (!fs.existsSync(file)) return { path: file, action: 'not-found' }; + + let content: string; + try { + content = fs.readFileSync(file, 'utf-8'); + } catch { + return { path: file, action: 'not-found' }; + } + + const ourFrontmatter = MDC_FRONTMATTER.trim(); + const startIdx = content.indexOf(CODEGRAPH_SECTION_START); + const endIdx = content.indexOf(CODEGRAPH_SECTION_END); + + // Our marked block is present — strip it, then decide what's left. + if (startIdx !== -1 && endIdx > startIdx) { + const before = content.substring(0, startIdx).trimEnd(); + const after = content.substring(endIdx + CODEGRAPH_SECTION_END.length).trimStart(); + const remainder = (before + (before && after ? '\n\n' : '') + after).trim(); + if (remainder === '' || remainder === ourFrontmatter) { + try { fs.unlinkSync(file); } catch { /* ignore */ } + } else { + atomicWriteFileSync(file, remainder + '\n'); + } + return { path: file, action: 'removed' }; + } + + // No block, but the file is still our pristine frontmatter-only file + // — it's ours, so remove it. + if (content.trim() === ourFrontmatter) { + try { fs.unlinkSync(file); } catch { /* ignore */ } + return { path: file, action: 'removed' }; + } + + // Foreign content we don't recognize — leave it alone. + return { path: file, action: 'not-found' }; +} + +export const cursorTarget: AgentTarget = new CursorTarget(); diff --git a/src/installer/targets/gemini.ts b/src/installer/targets/gemini.ts new file mode 100644 index 000000000..b6cc3bdd5 --- /dev/null +++ b/src/installer/targets/gemini.ts @@ -0,0 +1,162 @@ +/** + * Gemini CLI target (also covers the rebranded "Antigravity CLI" — + * Google is in the middle of unifying its CLI tools under + * Antigravity, and the new CLI continues to read `~/.gemini/settings.json` + * + project-local `.gemini/settings.json`). Writes: + * + * - MCP server entry to `~/.gemini/settings.json` (global) or + * `./.gemini/settings.json` (local) under the standard + * `mcpServers.codegraph` key. Same shape as Claude / Cursor. + * - Instructions to `~/.gemini/GEMINI.md` (global) or `./GEMINI.md` + * (local — Gemini reads the project root file directly, not + * under `.gemini/`). + * + * No permissions concept — Gemini CLI gates tool invocations through + * the `trust` field per server, not an external allowlist. We leave + * `trust` unset so the user controls confirmation prompts. + * + * The Antigravity IDE shares `~/.gemini/GEMINI.md` for instructions + * but uses a separate MCP config file (`~/.gemini/antigravity/mcp_config.json`) + * — see `./antigravity.ts`. Both targets writing to GEMINI.md is + * safe: the marker-based section replacement makes the second write + * a byte-identical no-op. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { + AgentTarget, + DetectionResult, + InstallOptions, + Location, + WriteResult, +} from './types'; +import { + getMcpServerConfig, + jsonDeepEqual, + readJsonFile, + removeMarkedSection, + writeJsonFile, +} from './shared'; +import { + CODEGRAPH_SECTION_END, + CODEGRAPH_SECTION_START, +} from '../instructions-template'; + +function configDir(loc: Location): string { + return loc === 'global' + ? path.join(os.homedir(), '.gemini') + : path.join(process.cwd(), '.gemini'); +} +function settingsJsonPath(loc: Location): string { + return path.join(configDir(loc), 'settings.json'); +} +function instructionsPath(loc: Location): string { + // Global GEMINI.md lives under ~/.gemini/; project-local GEMINI.md + // lives at the project root (NOT under .gemini/), matching how + // Gemini CLI's hierarchical context loader searches. + return loc === 'global' + ? path.join(configDir('global'), 'GEMINI.md') + : path.join(process.cwd(), 'GEMINI.md'); +} + +class GeminiTarget implements AgentTarget { + readonly id = 'gemini' as const; + readonly displayName = 'Gemini CLI'; + readonly docsUrl = 'https://geminicli.com/docs/tools/mcp-server/'; + + supportsLocation(_loc: Location): boolean { + return true; + } + + detect(loc: Location): DetectionResult { + const file = settingsJsonPath(loc); + const config = readJsonFile(file); + const alreadyConfigured = !!config.mcpServers?.codegraph; + const installed = loc === 'global' + ? fs.existsSync(configDir('global')) || fs.existsSync(file) + : fs.existsSync(file) || fs.existsSync(configDir('local')); + return { installed, alreadyConfigured, configPath: file }; + } + + install(loc: Location, _opts: InstallOptions): WriteResult { + const files: WriteResult['files'] = []; + files.push(writeMcpEntry(loc)); + + // GEMINI.md is no longer written — the codegraph usage guidance + // ships in the MCP server's `initialize` response (issue #529). + // Strip a block a previous install left so an upgrade self-heals. + const instrCleanup = removeInstructionsEntry(loc); + if (instrCleanup.action === 'removed') files.push(instrCleanup); + + return { files }; + } + + uninstall(loc: Location): WriteResult { + const files: WriteResult['files'] = []; + + const file = settingsJsonPath(loc); + const config = readJsonFile(file); + if (config.mcpServers?.codegraph) { + delete config.mcpServers.codegraph; + if (Object.keys(config.mcpServers).length === 0) { + delete config.mcpServers; + } + // If the file is now an empty `{}` we still leave it — other + // (top-level) Gemini settings the user might add later can + // share the file; deleting it would be surprising. + writeJsonFile(file, config); + files.push({ path: file, action: 'removed' }); + } else { + files.push({ path: file, action: 'not-found' }); + } + + files.push(removeInstructionsEntry(loc)); + + return { files }; + } + + printConfig(loc: Location): string { + const target = settingsJsonPath(loc); + const snippet = JSON.stringify({ mcpServers: { codegraph: getMcpServerConfig() } }, null, 2); + return `# Add to ${target}\n\n${snippet}\n`; + } + + describePaths(loc: Location): string[] { + return [settingsJsonPath(loc), instructionsPath(loc)]; + } +} + +function writeMcpEntry(loc: Location): WriteResult['files'][number] { + const file = settingsJsonPath(loc); + const dir = path.dirname(file); + if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true }); + + const existing = readJsonFile(file); + const before = existing.mcpServers?.codegraph; + const after = getMcpServerConfig(); + + if (jsonDeepEqual(before, after)) { + return { path: file, action: 'unchanged' }; + } + const action: 'created' | 'updated' = + before ? 'updated' : (fs.existsSync(file) ? 'updated' : 'created'); + if (!existing.mcpServers) existing.mcpServers = {}; + existing.mcpServers.codegraph = after; + writeJsonFile(file, existing); + return { path: file, action }; +} + +/** + * Strip the marker-delimited CodeGraph block from GEMINI.md if a prior + * install wrote one. Used by both install (self-heal on upgrade) and + * uninstall — see issue #529. + */ +function removeInstructionsEntry(loc: Location): WriteResult['files'][number] { + const file = instructionsPath(loc); + const action = removeMarkedSection(file, CODEGRAPH_SECTION_START, CODEGRAPH_SECTION_END); + return { path: file, action }; +} + +export const geminiTarget: AgentTarget = new GeminiTarget(); diff --git a/src/installer/targets/hermes.ts b/src/installer/targets/hermes.ts new file mode 100644 index 000000000..eacab6d3d --- /dev/null +++ b/src/installer/targets/hermes.ts @@ -0,0 +1,356 @@ +/** + * Hermes Agent target. + * + * Hermes reads MCP servers from `$HERMES_HOME/config.yaml` under the + * top-level `mcp_servers` key, and exposes discovered MCP tools through + * dynamic toolsets named `mcp-`. We add: + * + * mcp_servers.codegraph -> `codegraph serve --mcp` + * platform_toolsets.cli -> `mcp-codegraph` + * + * The second entry matters because Hermes CLI profiles often enable an + * explicit `platform_toolsets.cli` list. Without `mcp-codegraph` in that + * list, the MCP server can be configured and connected but its tools may + * still be filtered out of normal CLI sessions. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { + AgentTarget, + DetectionResult, + InstallOptions, + Location, + WriteResult, +} from './types'; +import { atomicWriteFileSync } from './shared'; + +type LineRange = { start: number; end: number }; + +class HermesTarget implements AgentTarget { + readonly id = 'hermes' as const; + readonly displayName = 'Hermes Agent'; + readonly docsUrl = 'https://hermes-agent.nousresearch.com'; + + supportsLocation(loc: Location): boolean { + return loc === 'global'; + } + + detect(loc: Location): DetectionResult { + if (loc !== 'global') { + return { installed: false, alreadyConfigured: false }; + } + const file = configPath(); + const content = readText(file); + const installed = fs.existsSync(hermesHome()) || fs.existsSync(file); + return { + installed, + alreadyConfigured: hasCodeGraphMcpServer(content), + configPath: file, + }; + } + + install(loc: Location, _opts: InstallOptions): WriteResult { + if (loc !== 'global') { + return { + files: [], + notes: ['Hermes Agent uses $HERMES_HOME/config.yaml; re-run with --location=global.'], + }; + } + return { + files: [writeHermesConfig()], + notes: ['Start a new Hermes session for MCP changes to take effect.'], + }; + } + + uninstall(loc: Location): WriteResult { + if (loc !== 'global') return { files: [] }; + const file = configPath(); + if (!fs.existsSync(file)) { + return { files: [{ path: file, action: 'not-found' }] }; + } + + const before = readText(file); + const after = removeCodeGraphToolset(removeCodeGraphMcpServer(before)); + if (after === before) { + return { files: [{ path: file, action: 'not-found' }] }; + } + atomicWriteFileSync(file, ensureTrailingNewline(after)); + return { files: [{ path: file, action: 'removed' }] }; + } + + printConfig(loc: Location): string { + if (loc !== 'global') { + return '# Hermes Agent uses $HERMES_HOME/config.yaml; use --location=global.\n'; + } + return [ + `# Add to ${configPath()}`, + '', + renderCodeGraphMcpBlock().join('\n'), + '', + 'platform_toolsets:', + ' cli:', + ' - hermes-cli', + ' - mcp-codegraph', + '', + ].join('\n'); + } + + describePaths(loc: Location): string[] { + return loc === 'global' ? [configPath()] : []; + } +} + +function hermesHome(): string { + return process.env.HERMES_HOME + ? path.resolve(process.env.HERMES_HOME) + : path.join(os.homedir(), '.hermes'); +} + +function configPath(): string { + return path.join(hermesHome(), 'config.yaml'); +} + +function readText(file: string): string { + try { + return fs.readFileSync(file, 'utf-8'); + } catch { + return ''; + } +} + +function writeHermesConfig(): WriteResult['files'][number] { + const file = configPath(); + const existed = fs.existsSync(file); + const before = readText(file); + const afterMcp = upsertCodeGraphMcpServer(before); + const after = upsertCodeGraphToolset(afterMcp); + + if (after === before) { + return { path: file, action: 'unchanged' }; + } + atomicWriteFileSync(file, ensureTrailingNewline(after)); + return { path: file, action: existed ? 'updated' : 'created' }; +} + +function ensureTrailingNewline(text: string): string { + return text.endsWith('\n') ? text : text + '\n'; +} + +function splitLines(content: string): string[] { + return content.replace(/\r\n/g, '\n').replace(/\r/g, '\n').split('\n'); +} + +function joinLines(lines: string[]): string { + while (lines.length > 0 && lines[lines.length - 1] === '') lines.pop(); + return lines.join('\n') + '\n'; +} + +function topLevelRange(lines: string[], key: string): LineRange | null { + const start = lines.findIndex((line) => line.trim() === `${key}:`); + if (start === -1) return null; + let end = lines.length; + for (let i = start + 1; i < lines.length; i++) { + const line = lines[i] ?? ''; + if (line.trim() === '') continue; + if (/^[A-Za-z_][A-Za-z0-9_-]*:\s*(?:#.*)?$/.test(line)) { + end = i; + break; + } + } + return { start, end }; +} + +function childRange(lines: string[], parent: LineRange, child: string): LineRange | null { + const startPattern = new RegExp(`^ ${escapeRegExp(child)}:\\s*(?:#.*)?$`); + let start = -1; + for (let i = parent.start + 1; i < parent.end; i++) { + if (startPattern.test(lines[i] ?? '')) { + start = i; + break; + } + } + if (start === -1) return null; + + let end = parent.end; + for (let i = start + 1; i < parent.end; i++) { + const line = lines[i] ?? ''; + if (line.trim() === '') continue; + if (/^ \S/.test(line)) { + end = i; + break; + } + } + while (end > start + 1 && (lines[end - 1] ?? '').trim() === '') { + end--; + } + return { start, end }; +} + +/** + * Block-range for a 2-space-indented child whose value is a YAML block list. + * + * Unlike `childRange`, this handles PyYAML's default `default_flow_style=False` + * serialization, where list items sit at the SAME indent as the parent key: + * + * cli: + * - hermes-cli # indent 2 — belongs to cli, not a sibling + * - browser + * + * `childRange`'s `^ \S` heuristic mistakes that first ` - hermes-cli` line + * for the next sibling key and truncates the block, causing inserts to land + * before the existing items at a different indent (issue #456). This helper + * recognizes a ` - ` line as part of the block instead, and reports back + * the actual indent used by existing items so the inserter matches it. + */ +function listChildBlock( + lines: string[], + parent: LineRange, + child: string, +): (LineRange & { itemIndent: string }) | null { + const startPattern = new RegExp(`^ ${escapeRegExp(child)}:\\s*(?:#.*)?$`); + let start = -1; + for (let i = parent.start + 1; i < parent.end; i++) { + if (startPattern.test(lines[i] ?? '')) { + start = i; + break; + } + } + if (start === -1) return null; + + let end = parent.end; + for (let i = start + 1; i < parent.end; i++) { + const line = lines[i] ?? ''; + if (line.trim() === '') continue; + const indentMatch = line.match(/^( *)/); + const indent = indentMatch?.[1]?.length ?? 0; + if (indent >= 4) continue; + if (indent === 2 && /^ - /.test(line)) continue; + end = i; + break; + } + while (end > start + 1 && (lines[end - 1] ?? '').trim() === '') { + end--; + } + + let itemIndent = ' '; + for (let i = start + 1; i < end; i++) { + const m = (lines[i] ?? '').match(/^( +)- /); + if (m && m[1]) { + itemIndent = m[1]; + break; + } + } + return { start, end, itemIndent }; +} + +function escapeRegExp(value: string): string { + return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +function renderCodeGraphMcpChild(): string[] { + return [ + ' codegraph:', + ' command: codegraph', + ' args:', + ' - serve', + ' - --mcp', + ' timeout: 120', + ' connect_timeout: 60', + ' enabled: true', + ]; +} + +function renderCodeGraphMcpBlock(): string[] { + return ['mcp_servers:', ...renderCodeGraphMcpChild()]; +} + +function hasCodeGraphMcpServer(content: string): boolean { + const lines = splitLines(content); + const parent = topLevelRange(lines, 'mcp_servers'); + return !!parent && !!childRange(lines, parent, 'codegraph'); +} + +function upsertCodeGraphMcpServer(content: string): string { + const lines = splitLines(content); + const parent = topLevelRange(lines, 'mcp_servers'); + const child = parent ? childRange(lines, parent, 'codegraph') : null; + const replacement = renderCodeGraphMcpChild(); + + if (!parent) { + if (lines.length > 0 && lines[lines.length - 1] === '') lines.pop(); + if (lines.length > 0) lines.push(''); + lines.push(...renderCodeGraphMcpBlock()); + return joinLines(lines); + } + + if (child) { + const existing = lines.slice(child.start, child.end); + if (arrayEqual(existing, replacement)) return joinLines(lines); + lines.splice(child.start, child.end - child.start, ...replacement); + return joinLines(lines); + } + + lines.splice(parent.end, 0, ...replacement); + return joinLines(lines); +} + +function removeCodeGraphMcpServer(content: string): string { + const lines = splitLines(content); + const parent = topLevelRange(lines, 'mcp_servers'); + const child = parent ? childRange(lines, parent, 'codegraph') : null; + if (!child) return content; + lines.splice(child.start, child.end - child.start); + return joinLines(lines); +} + +function upsertCodeGraphToolset(content: string): string { + const lines = splitLines(content); + const parent = topLevelRange(lines, 'platform_toolsets'); + const cli = parent ? listChildBlock(lines, parent, 'cli') : null; + + if (!parent) { + if (lines.length > 0 && lines[lines.length - 1] === '') lines.pop(); + if (lines.length > 0) lines.push(''); + lines.push('platform_toolsets:', ' cli:', ' - hermes-cli', ' - mcp-codegraph'); + return joinLines(lines); + } + + if (!cli) { + lines.splice(parent.end, 0, ' cli:', ' - hermes-cli', ' - mcp-codegraph'); + return joinLines(lines); + } + + const hasEntry = lines + .slice(cli.start + 1, cli.end) + .some((line) => line.trim() === '- mcp-codegraph'); + if (hasEntry) return joinLines(lines); + + lines.splice(cli.end, 0, `${cli.itemIndent}- mcp-codegraph`); + return joinLines(lines); +} + +function removeCodeGraphToolset(content: string): string { + const lines = splitLines(content); + const parent = topLevelRange(lines, 'platform_toolsets'); + const cli = parent ? listChildBlock(lines, parent, 'cli') : null; + if (!cli) return content; + + const hasEntry = lines + .slice(cli.start + 1, cli.end) + .some((line) => line.trim() === '- mcp-codegraph'); + if (!hasEntry) return content; + + const next = lines.filter((line, idx) => { + if (idx <= cli.start || idx >= cli.end) return true; + return line.trim() !== '- mcp-codegraph'; + }); + return joinLines(next); +} + +function arrayEqual(a: string[], b: string[]): boolean { + return a.length === b.length && a.every((value, idx) => value === b[idx]); +} + +export const hermesTarget: AgentTarget = new HermesTarget(); diff --git a/src/installer/targets/kiro.ts b/src/installer/targets/kiro.ts new file mode 100644 index 000000000..5658dd0ee --- /dev/null +++ b/src/installer/targets/kiro.ts @@ -0,0 +1,165 @@ +/** + * Kiro CLI / IDE target. Writes: + * + * - MCP server entry to `~/.kiro/settings/mcp.json` (global) or + * `./.kiro/settings/mcp.json` (local). Standard `mcpServers.codegraph` + * shape, same as Claude / Cursor / Gemini. + * - Instructions to `~/.kiro/steering/codegraph.md` (global) or + * `./.kiro/steering/codegraph.md` (local). Kiro's "steering" system + * loads every `*.md` file in the steering dir as agent context, so + * a dedicated `codegraph.md` is the natural surface — we own the + * whole file outright (no marker-based merging needed) and delete + * it on uninstall. + * + * No permissions concept — Kiro gates tool invocations through its own + * UI prompts rather than an external allowlist. `autoAllow` is silently + * ignored. + * + * Paths are identical on macOS / Linux / Windows because Kiro resolves + * its config root from `os.homedir()` on all three (Windows `~` → + * `%USERPROFILE%\.kiro`). + * + * Docs: https://kiro.dev/docs/cli/mcp/ + * https://kiro.dev/docs/cli/steering/ + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { + AgentTarget, + DetectionResult, + InstallOptions, + Location, + WriteResult, +} from './types'; +import { + getMcpServerConfig, + jsonDeepEqual, + readJsonFile, + writeJsonFile, +} from './shared'; + +function configDir(loc: Location): string { + return loc === 'global' + ? path.join(os.homedir(), '.kiro') + : path.join(process.cwd(), '.kiro'); +} +function mcpJsonPath(loc: Location): string { + return path.join(configDir(loc), 'settings', 'mcp.json'); +} +function steeringPath(loc: Location): string { + return path.join(configDir(loc), 'steering', 'codegraph.md'); +} + +class KiroTarget implements AgentTarget { + readonly id = 'kiro' as const; + readonly displayName = 'Kiro'; + readonly docsUrl = 'https://kiro.dev/docs/cli/mcp/'; + + supportsLocation(_loc: Location): boolean { + return true; + } + + detect(loc: Location): DetectionResult { + const file = mcpJsonPath(loc); + const config = readJsonFile(file); + const alreadyConfigured = !!config.mcpServers?.codegraph; + const installed = loc === 'global' + ? fs.existsSync(configDir('global')) || fs.existsSync(file) + : fs.existsSync(file) || fs.existsSync(configDir('local')); + return { installed, alreadyConfigured, configPath: file }; + } + + install(loc: Location, _opts: InstallOptions): WriteResult { + const files: WriteResult['files'] = []; + files.push(writeMcpEntry(loc)); + + // The steering doc is no longer written — the codegraph usage + // guidance ships in the MCP server's `initialize` response (issue + // #529). Delete a `codegraph.md` a previous install created so an + // upgrade self-heals. + const steeringCleanup = removeSteeringEntry(loc); + if (steeringCleanup.action === 'removed') files.push(steeringCleanup); + + return { + files, + // The IDE-only enable-MCP step is load-bearing: Kiro IDE ships + // with MCP support disabled by default, so even a valid + // `~/.kiro/settings/mcp.json` at the documented path is ignored + // until the user flips the toggle. Kiro CLI reads the same file + // without a gate, so we call out which audience this applies to. + notes: [ + 'Restart Kiro for MCP changes to take effect.', + 'Kiro IDE: also enable MCP in Settings (search "MCP" → "Enabled"). Kiro CLI users can skip this step.', + ], + }; + } + + uninstall(loc: Location): WriteResult { + const files: WriteResult['files'] = []; + + const file = mcpJsonPath(loc); + const config = readJsonFile(file); + if (config.mcpServers?.codegraph) { + delete config.mcpServers.codegraph; + if (Object.keys(config.mcpServers).length === 0) { + delete config.mcpServers; + } + writeJsonFile(file, config); + files.push({ path: file, action: 'removed' }); + } else { + files.push({ path: file, action: 'not-found' }); + } + + files.push(removeSteeringEntry(loc)); + + return { files }; + } + + printConfig(loc: Location): string { + const target = mcpJsonPath(loc); + const snippet = JSON.stringify({ mcpServers: { codegraph: getMcpServerConfig() } }, null, 2); + return `# Add to ${target}\n\n${snippet}\n`; + } + + describePaths(loc: Location): string[] { + return [mcpJsonPath(loc), steeringPath(loc)]; + } +} + +function writeMcpEntry(loc: Location): WriteResult['files'][number] { + const file = mcpJsonPath(loc); + const dir = path.dirname(file); + if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true }); + + const existing = readJsonFile(file); + const before = existing.mcpServers?.codegraph; + const after = getMcpServerConfig(); + + if (jsonDeepEqual(before, after)) { + return { path: file, action: 'unchanged' }; + } + const action: 'created' | 'updated' = + before ? 'updated' : (fs.existsSync(file) ? 'updated' : 'created'); + if (!existing.mcpServers) existing.mcpServers = {}; + existing.mcpServers.codegraph = after; + writeJsonFile(file, existing); + return { path: file, action }; +} + +/** + * Delete the steering file we own. If a user has hand-edited the file + * out of recognition we still remove it — codegraph.md is a name we + * claim, and a partial install leaving the file behind is worse than + * a clean delete. Used by both install (self-heal on upgrade — see + * issue #529) and uninstall. + */ +function removeSteeringEntry(loc: Location): WriteResult['files'][number] { + const file = steeringPath(loc); + if (!fs.existsSync(file)) return { path: file, action: 'not-found' }; + try { fs.unlinkSync(file); } catch { /* ignore */ } + return { path: file, action: 'removed' }; +} + +export const kiroTarget: AgentTarget = new KiroTarget(); diff --git a/src/installer/targets/opencode.ts b/src/installer/targets/opencode.ts new file mode 100644 index 000000000..5ec97436d --- /dev/null +++ b/src/installer/targets/opencode.ts @@ -0,0 +1,239 @@ +/** + * opencode target. + * + * - MCP server entry to `~/.config/opencode/opencode.jsonc` (global, + * XDG-style; `%APPDATA%/opencode/opencode.jsonc` on Windows) or + * `./opencode.jsonc` (local). Falls back to `opencode.json` when a + * `.json` file already exists; defaults new installs to `.jsonc` + * because that's what opencode itself creates on first run. + * - Instructions to `~/.config/opencode/AGENTS.md` (global) or + * `./AGENTS.md` (local). opencode reads AGENTS.md for agent + * instructions — same convention Codex CLI uses. + * - No permissions concept. + * + * Config shape uses opencode's wrapper: + * { + * "$schema": "https://opencode.ai/config.json", + * "mcp": { "codegraph": { "type": "local", "command": [...], "enabled": true } } + * } + * + * The shape differs from Claude/Cursor — opencode uses `mcp.` + * (not `mcpServers`), takes `command` as a string array combining + * binary + args, and includes an explicit `enabled` flag. + * + * Reads + writes go through `jsonc-parser` so any `//` and `/* *\/` + * comments the user has added to their `.jsonc` survive idempotent + * re-runs. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { parse as parseJsonc, modify, applyEdits } from 'jsonc-parser'; +import { + AgentTarget, + DetectionResult, + InstallOptions, + Location, + WriteResult, +} from './types'; +import { + atomicWriteFileSync, + jsonDeepEqual, + removeMarkedSection, +} from './shared'; +import { + CODEGRAPH_SECTION_END, + CODEGRAPH_SECTION_START, +} from '../instructions-template'; + +function globalConfigDir(): string { + if (process.platform === 'win32') { + const appData = process.env.APPDATA ?? path.join(os.homedir(), 'AppData', 'Roaming'); + return path.join(appData, 'opencode'); + } + // XDG_CONFIG_HOME if set, else ~/.config — matches opencode's docs. + const xdg = process.env.XDG_CONFIG_HOME && process.env.XDG_CONFIG_HOME.trim().length > 0 + ? process.env.XDG_CONFIG_HOME + : path.join(os.homedir(), '.config'); + return path.join(xdg, 'opencode'); +} + +function configBaseDir(loc: Location): string { + return loc === 'global' ? globalConfigDir() : process.cwd(); +} + +// Pick existing .jsonc, then .json, default to .jsonc for new files. +// opencode auto-creates .jsonc on first run, so that's the dominant +// real-world case and the sensible default for greenfield installs. +function configPath(loc: Location): string { + const dir = configBaseDir(loc); + const jsonc = path.join(dir, 'opencode.jsonc'); + const json = path.join(dir, 'opencode.json'); + if (fs.existsSync(jsonc)) return jsonc; + if (fs.existsSync(json)) return json; + return jsonc; +} + +function instructionsPath(loc: Location): string { + return path.join(configBaseDir(loc), 'AGENTS.md'); +} + +function readConfigText(file: string): string { + if (!fs.existsSync(file)) return ''; + return fs.readFileSync(file, 'utf-8'); +} + +function parseConfig(text: string): Record { + if (!text.trim()) return {}; + const errors: any[] = []; + const result = parseJsonc(text, errors, { allowTrailingComma: true }); + if (result == null || typeof result !== 'object' || Array.isArray(result)) { + return {}; + } + return result as Record; +} + +function getOpencodeServerEntry(): { type: string; command: string[]; enabled: boolean } { + return { + type: 'local', + command: ['codegraph', 'serve', '--mcp'], + enabled: true, + }; +} + +const FORMATTING = { tabSize: 2, insertSpaces: true, eol: '\n' }; + +class OpencodeTarget implements AgentTarget { + readonly id = 'opencode' as const; + readonly displayName = 'opencode'; + readonly docsUrl = 'https://opencode.ai/docs/config'; + + supportsLocation(_loc: Location): boolean { + return true; + } + + detect(loc: Location): DetectionResult { + const file = configPath(loc); + const config = parseConfig(readConfigText(file)); + const alreadyConfigured = !!config.mcp?.codegraph; + const installed = loc === 'global' + ? fs.existsSync(globalConfigDir()) + : fs.existsSync(file); + return { installed, alreadyConfigured, configPath: file }; + } + + install(loc: Location, _opts: InstallOptions): WriteResult { + const files: WriteResult['files'] = []; + files.push(writeMcpEntry(loc)); + + // AGENTS.md is no longer written — the codegraph usage guidance + // ships in the MCP server's `initialize` response (issue #529). + // Strip a block a previous install left so an upgrade self-heals. + const instrCleanup = removeInstructionsEntry(loc); + if (instrCleanup.action === 'removed') files.push(instrCleanup); + + return { files }; + } + + uninstall(loc: Location): WriteResult { + const files: WriteResult['files'] = []; + const file = configPath(loc); + + if (!fs.existsSync(file)) { + files.push({ path: file, action: 'not-found' }); + } else { + const text = readConfigText(file); + const config = parseConfig(text); + if (!config.mcp?.codegraph) { + files.push({ path: file, action: 'not-found' }); + } else { + // Drop our key surgically. Leaves siblings + comments untouched. + let edits = modify(text, ['mcp', 'codegraph'], undefined, { + formattingOptions: FORMATTING, + }); + let updated = applyEdits(text, edits); + + // If `mcp` is now an empty object, drop the wrapper too. + const afterParsed = parseConfig(updated); + if (afterParsed.mcp && typeof afterParsed.mcp === 'object' && + Object.keys(afterParsed.mcp).length === 0) { + edits = modify(updated, ['mcp'], undefined, { formattingOptions: FORMATTING }); + updated = applyEdits(updated, edits); + } + + atomicWriteFileSync(file, updated); + files.push({ path: file, action: 'removed' }); + } + } + + files.push(removeInstructionsEntry(loc)); + + return { files }; + } + + printConfig(loc: Location): string { + const target = configPath(loc); + const snippet = JSON.stringify({ + $schema: 'https://opencode.ai/config.json', + mcp: { codegraph: getOpencodeServerEntry() }, + }, null, 2); + return `# Add to ${target}\n\n${snippet}\n`; + } + + describePaths(loc: Location): string[] { + return [configPath(loc), instructionsPath(loc)]; + } +} + +function writeMcpEntry(loc: Location): WriteResult['files'][number] { + const file = configPath(loc); + const existed = fs.existsSync(file); + let text = readConfigText(file); + + // Seed a minimal opencode config when the file is brand-new so + // the result is a complete, schema-tagged file (not just a bare + // `{ "mcp": {...} }`). + if (!text.trim()) { + text = '{\n "$schema": "https://opencode.ai/config.json"\n}\n'; + } + + const config = parseConfig(text); + const before = config.mcp?.codegraph; + const after = getOpencodeServerEntry(); + + if (jsonDeepEqual(before, after)) { + return { path: file, action: 'unchanged' }; + } + + // Add $schema if the user's existing file is missing it. + if (!config.$schema) { + const schemaEdits = modify(text, ['$schema'], 'https://opencode.ai/config.json', { + formattingOptions: FORMATTING, + }); + text = applyEdits(text, schemaEdits); + } + + // Surgical edit — preserves comments, formatting, and order of + // every key we don't touch. + const edits = modify(text, ['mcp', 'codegraph'], after, { + formattingOptions: FORMATTING, + }); + const updated = applyEdits(text, edits); + atomicWriteFileSync(file, updated); + + return { path: file, action: existed ? 'updated' : 'created' }; +} + +/** + * Strip the marker-delimited CodeGraph block from AGENTS.md if a prior + * install wrote one. Used by both install (self-heal on upgrade) and + * uninstall — see issue #529. + */ +function removeInstructionsEntry(loc: Location): WriteResult['files'][number] { + const file = instructionsPath(loc); + const action = removeMarkedSection(file, CODEGRAPH_SECTION_START, CODEGRAPH_SECTION_END); + return { path: file, action }; +} + +export const opencodeTarget: AgentTarget = new OpencodeTarget(); diff --git a/src/installer/targets/registry.ts b/src/installer/targets/registry.ts new file mode 100644 index 000000000..5e929d468 --- /dev/null +++ b/src/installer/targets/registry.ts @@ -0,0 +1,91 @@ +/** + * Registry of all known agent targets. + * + * Adding a new target = create `targets/.ts` exporting an + * `AgentTarget`, then add it to the array below. Order here is the + * order they appear in the multiselect prompt, in `--target=all`, + * and in `--print-config`'s help listing — keep it stable. + */ + +import { AgentTarget, Location, TargetId } from './types'; +import { claudeTarget } from './claude'; +import { cursorTarget } from './cursor'; +import { codexTarget } from './codex'; +import { opencodeTarget } from './opencode'; +import { hermesTarget } from './hermes'; +import { geminiTarget } from './gemini'; +import { antigravityTarget } from './antigravity'; +import { kiroTarget } from './kiro'; + +export const ALL_TARGETS: readonly AgentTarget[] = Object.freeze([ + claudeTarget, + cursorTarget, + codexTarget, + opencodeTarget, + hermesTarget, + geminiTarget, + antigravityTarget, + kiroTarget, +]); + +export function getTarget(id: string): AgentTarget | undefined { + return ALL_TARGETS.find((t) => t.id === id); +} + +export function listTargetIds(): TargetId[] { + return ALL_TARGETS.map((t) => t.id); +} + +/** + * Run `detect()` for every target at the given location. Returns the + * full registry zipped with detection results — orchestrator uses + * this to seed the multiselect prompt with installed agents + * pre-checked. + */ +export function detectAll(loc: Location): Array<{ + target: AgentTarget; + detection: ReturnType; +}> { + return ALL_TARGETS.map((target) => ({ + target, + detection: target.detect(loc), + })); +} + +/** + * Resolve a `--target=` flag value to a list of `AgentTarget` + * instances. Accepts: + * + * - `auto` — return all targets whose `detect().installed` is true, + * or `['claude']` as a fallback if none detected (least-surprise + * for existing users). + * - `all` — every target in the registry. + * - `none` — empty list (caller skips agent writes entirely). + * - csv list — `'claude,cursor'` etc. Unknown ids throw. + */ +export function resolveTargetFlag(value: string, loc: Location): AgentTarget[] { + if (value === 'none') return []; + if (value === 'all') return [...ALL_TARGETS]; + if (value === 'auto') { + const detected = detectAll(loc).filter(({ detection }) => detection.installed); + if (detected.length > 0) return detected.map(({ target }) => target); + const fallback = getTarget('claude'); + return fallback ? [fallback] : []; + } + + const ids = value.split(',').map((s) => s.trim()).filter(Boolean); + const resolved: AgentTarget[] = []; + const unknown: string[] = []; + for (const id of ids) { + const t = getTarget(id); + if (t) resolved.push(t); + else unknown.push(id); + } + if (unknown.length > 0) { + const known = listTargetIds().join(', '); + throw new Error( + `Unknown --target id(s): ${unknown.join(', ')}. Known: ${known}, plus 'auto' / 'all' / 'none'.`, + ); + } + return resolved; +} diff --git a/src/installer/targets/shared.ts b/src/installer/targets/shared.ts new file mode 100644 index 000000000..6d54ab570 --- /dev/null +++ b/src/installer/targets/shared.ts @@ -0,0 +1,206 @@ +/** + * Helpers shared across `AgentTarget` implementations. + * + * Lifted from the original `config-writer.ts` so each target can + * compose them without inheritance. Kept deliberately small — the + * targets are different enough (JSON vs TOML vs Markdown, varying + * idempotency markers) that a base class would force the awkward + * shape onto everyone. + */ + +import * as fs from 'fs'; +import * as path from 'path'; + +/** + * The MCP-server config block codegraph injects. Same shape across + * all JSON-shaped agent configs (Claude, Cursor, opencode), only the + * surrounding wrapper differs. Codex (TOML) builds its own block. + */ +export function getMcpServerConfig(): { type: string; command: string; args: string[] } { + return { + type: 'stdio', + command: 'codegraph', + args: ['serve', '--mcp'], + }; +} + +/** + * Permissions list for Claude `settings.json`. Other targets that + * have a permissions concept can compose this list directly. The + * permission strings follow Claude's `mcp____` format. + */ +export function getCodeGraphPermissions(): string[] { + return [ + 'mcp__codegraph__codegraph_search', + 'mcp__codegraph__codegraph_context', + 'mcp__codegraph__codegraph_callers', + 'mcp__codegraph__codegraph_callees', + 'mcp__codegraph__codegraph_impact', + 'mcp__codegraph__codegraph_node', + 'mcp__codegraph__codegraph_status', + ]; +} + +/** + * Read a JSON file, returning `{}` when missing or unparseable. + * + * Unparseable files are backed up to `.backup` BEFORE we return + * `{}` — so an idempotent re-run never silently deletes a user's + * existing config that happened to break JSON parse temporarily. + */ +export function readJsonFile(filePath: string): Record { + if (!fs.existsSync(filePath)) { + return {}; + } + try { + return JSON.parse(fs.readFileSync(filePath, 'utf-8')); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + console.warn(` Warning: Could not parse ${path.basename(filePath)}: ${msg}`); + console.warn(` A backup will be created before overwriting.`); + try { + fs.copyFileSync(filePath, filePath + '.backup'); + } catch { /* ignore backup failure */ } + return {}; + } +} + +/** + * Write a file atomically: write to `.tmp.`, then rename. + * + * Prevents corruption if the process crashes mid-write. The temp + * file is cleaned up on rename failure. + */ +export function atomicWriteFileSync(filePath: string, content: string): void { + const dir = path.dirname(filePath); + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }); + } + const tmpPath = filePath + '.tmp.' + process.pid; + try { + fs.writeFileSync(tmpPath, content); + fs.renameSync(tmpPath, filePath); + } catch (err) { + try { fs.unlinkSync(tmpPath); } catch { /* ignore */ } + throw err; + } +} + +/** + * Atomic JSON write. Trailing newline matches the convention every + * existing target had — preserves diff-friendly file shape. + */ +export function writeJsonFile(filePath: string, data: Record): void { + atomicWriteFileSync(filePath, JSON.stringify(data, null, 2) + '\n'); +} + +/** + * Compare two JSON values for deep equality, ignoring key order. + * + * Used for idempotency: when the on-disk config already exactly + * matches what we'd write, return action=`unchanged` instead of + * re-writing (and emitting a confusing "Updated" log line). + */ +export function jsonDeepEqual(a: unknown, b: unknown): boolean { + if (a === b) return true; + if (typeof a !== typeof b) return false; + if (a === null || b === null) return a === b; + if (typeof a !== 'object') return false; + if (Array.isArray(a) !== Array.isArray(b)) return false; + if (Array.isArray(a) && Array.isArray(b)) { + if (a.length !== b.length) return false; + return a.every((v, i) => jsonDeepEqual(v, b[i])); + } + const ao = a as Record; + const bo = b as Record; + const ak = Object.keys(ao).sort(); + const bk = Object.keys(bo).sort(); + if (ak.length !== bk.length) return false; + if (!ak.every((k, i) => k === bk[i])) return false; + return ak.every((k) => jsonDeepEqual(ao[k], bo[k])); +} + +/** + * Replace or append a marker-delimited section in a markdown-ish file. + * + * Used by Claude / Codex for the ` ... ` block. Preserves all content outside the + * markers verbatim. + * + * Returns `created` when the file didn't exist; `updated` when + * markers were found and content swapped; `appended` when markers + * weren't found and section was added at end. `unchanged` when the + * existing block already matches `body`. + */ +export function replaceOrAppendMarkedSection( + filePath: string, + body: string, + startMarker: string, + endMarker: string, +): 'created' | 'updated' | 'appended' | 'unchanged' { + if (!fs.existsSync(filePath)) { + atomicWriteFileSync(filePath, body + '\n'); + return 'created'; + } + + const content = fs.readFileSync(filePath, 'utf-8'); + const startIdx = content.indexOf(startMarker); + const endIdx = content.indexOf(endMarker); + + if (startIdx !== -1 && endIdx > startIdx) { + const existingBlock = content.substring(startIdx, endIdx + endMarker.length); + if (existingBlock === body) { + return 'unchanged'; + } + const before = content.substring(0, startIdx); + const after = content.substring(endIdx + endMarker.length); + atomicWriteFileSync(filePath, before + body + after); + return 'updated'; + } + + // No markers — append. Preserve existing content with a separating + // blank line. + const trimmed = content.trimEnd(); + const sep = trimmed.length > 0 ? '\n\n' : ''; + atomicWriteFileSync(filePath, trimmed + sep + body + '\n'); + return 'appended'; +} + +/** + * Inverse of `replaceOrAppendMarkedSection`. Strips the marker + * block from `filePath` if present. If the file becomes empty after + * removal, deletes the file entirely (matches the existing Claude + * uninstall behavior). + * + * Returns `removed` when content was stripped, `not-found` when + * the markers weren't present, `kept` when the file didn't exist. + */ +export function removeMarkedSection( + filePath: string, + startMarker: string, + endMarker: string, +): 'removed' | 'not-found' | 'kept' { + if (!fs.existsSync(filePath)) return 'kept'; + + let content: string; + try { + content = fs.readFileSync(filePath, 'utf-8'); + } catch { + return 'kept'; + } + + const startIdx = content.indexOf(startMarker); + const endIdx = content.indexOf(endMarker); + if (startIdx === -1 || endIdx <= startIdx) return 'not-found'; + + const before = content.substring(0, startIdx).trimEnd(); + const after = content.substring(endIdx + endMarker.length).trimStart(); + const joined = before + (before && after ? '\n\n' : '') + after; + + if (joined.trim() === '') { + try { fs.unlinkSync(filePath); } catch { /* ignore */ } + } else { + atomicWriteFileSync(filePath, joined.trim() + '\n'); + } + return 'removed'; +} diff --git a/src/installer/targets/toml.ts b/src/installer/targets/toml.ts new file mode 100644 index 000000000..29348a7c9 --- /dev/null +++ b/src/installer/targets/toml.ts @@ -0,0 +1,154 @@ +/** + * Tiny TOML helpers — just enough to inject / replace / remove a + * single dotted-key table block (`[mcp_servers.codegraph]`) inside an + * existing `~/.codex/config.toml`. We deliberately do NOT try to be a + * general TOML parser/serializer; that would mean pulling in a + * dependency (~50KB) for ~6 lines of output. + * + * Strategy: treat the file as text. Find the `[mcp_servers.codegraph]` + * header line, splice it (and the lines that follow it until the next + * `[...]` header or EOF) in or out. Everything outside that block is + * preserved verbatim, byte-for-byte. + * + * Limitations (acceptable for our narrow use): + * - Only handles top-level table headers; not array-of-tables or + * subtables nested inside `[mcp_servers]` itself (we always write + * the full dotted key `[mcp_servers.codegraph]`). + * - Doesn't validate sibling TOML — if the file is malformed + * elsewhere, our injection won't fix it but won't make it worse. + * - Quotes string values with double quotes; escapes `\` and `"`. + */ + +/** + * Serialize a record into the body lines of a TOML table. Values + * supported: string, string[]. Other types throw — the codex MCP + * config only needs these two. + */ +export function serializeTomlTableBody(values: Record): string { + const lines: string[] = []; + for (const [key, value] of Object.entries(values)) { + if (typeof value === 'string') { + lines.push(`${key} = ${quoteString(value)}`); + } else if (Array.isArray(value) && value.every((v) => typeof v === 'string')) { + const parts = value.map(quoteString).join(', '); + lines.push(`${key} = [${parts}]`); + } else { + throw new Error(`Unsupported TOML value type for key "${key}"`); + } + } + return lines.join('\n'); +} + +function quoteString(s: string): string { + // TOML basic strings: backslash and double-quote escapes; control + // chars not expected in our payload (paths/args). + return '"' + s.replace(/\\/g, '\\\\').replace(/"/g, '\\"') + '"'; +} + +/** + * Build a full table block: header line + body. Suitable for direct + * insertion into a TOML file. + */ +export function buildTomlTable(header: string, values: Record): string { + return `[${header}]\n${serializeTomlTableBody(values)}`; +} + +/** + * Insert or replace a top-level dotted-key TOML table block in the + * given file content. Preserves all other content verbatim. + * + * Returns `'inserted'` when the table was newly added, `'replaced'` + * when an existing one was rewritten, `'unchanged'` when the + * existing block already matches `block` byte-for-byte. + */ +export function upsertTomlTable( + fileContent: string, + header: string, + block: string, +): { content: string; action: 'inserted' | 'replaced' | 'unchanged' } { + const headerLine = `[${header}]`; + const headerIdx = findHeaderIndex(fileContent, headerLine); + + if (headerIdx === -1) { + // Insert at end with separating blank line if there's existing content. + const trimmed = fileContent.trimEnd(); + const sep = trimmed.length > 0 ? '\n\n' : ''; + return { + content: trimmed + sep + block + '\n', + action: 'inserted', + }; + } + + // Find the end of this block: next `[...]` header (at line start) or EOF. + const blockEnd = findNextTableHeader(fileContent, headerIdx + headerLine.length); + const existingBlock = fileContent.substring(headerIdx, blockEnd).replace(/\n+$/, ''); + + if (existingBlock === block) { + return { content: fileContent, action: 'unchanged' }; + } + + const before = fileContent.substring(0, headerIdx); + const after = fileContent.substring(blockEnd); + // Trim trailing blank lines from `before` (we'll re-add one) and + // leading blank lines from `after` so the file shape stays clean. + const beforeClean = before.replace(/\n+$/, ''); + const afterClean = after.replace(/^\n+/, ''); + const sepBefore = beforeClean.length > 0 ? '\n\n' : ''; + const sepAfter = afterClean.length > 0 ? '\n\n' : '\n'; + return { + content: beforeClean + sepBefore + block + sepAfter + afterClean, + action: 'replaced', + }; +} + +/** + * Remove a top-level dotted-key TOML table block. Returns the + * possibly-empty new content + an action flag. + */ +export function removeTomlTable( + fileContent: string, + header: string, +): { content: string; action: 'removed' | 'not-found' } { + const headerLine = `[${header}]`; + const headerIdx = findHeaderIndex(fileContent, headerLine); + if (headerIdx === -1) return { content: fileContent, action: 'not-found' }; + + const blockEnd = findNextTableHeader(fileContent, headerIdx + headerLine.length); + const before = fileContent.substring(0, headerIdx).replace(/\n+$/, ''); + const after = fileContent.substring(blockEnd).replace(/^\n+/, ''); + const joined = before + (before && after ? '\n\n' : '') + after; + return { content: joined, action: 'removed' }; +} + +/** + * Locate the byte index of a header line (`[foo.bar]`) when it + * appears at the start of a line. Returns -1 if not found. + */ +function findHeaderIndex(content: string, headerLine: string): number { + // Search BOL or right after a newline. + if (content.startsWith(headerLine)) return 0; + const needle = '\n' + headerLine; + const idx = content.indexOf(needle); + return idx === -1 ? -1 : idx + 1; +} + +/** + * Find the byte index of the next top-level `[...]` table header + * (excluding array-of-tables `[[...]]`) starting from `from`, or + * return content length when none. + */ +function findNextTableHeader(content: string, from: number): number { + // Look for "\n[" but skip "\n[[" (array of tables). + let i = from; + while (i < content.length) { + const nlIdx = content.indexOf('\n[', i); + if (nlIdx === -1) return content.length; + if (content[nlIdx + 2] === '[') { + // [[...]] — keep searching past it. + i = nlIdx + 2; + continue; + } + return nlIdx + 1; + } + return content.length; +} diff --git a/src/installer/targets/types.ts b/src/installer/targets/types.ts new file mode 100644 index 000000000..4b3267e97 --- /dev/null +++ b/src/installer/targets/types.ts @@ -0,0 +1,106 @@ +/** + * Agent target abstraction for the installer. + * + * Each MCP-capable agent (Claude Code, Cursor, Codex CLI, opencode, ...) + * implements this interface so the installer orchestrator can write the + * right MCP-server config + instructions file + permissions for that + * agent without baking client-specific paths into core code. Adding a + * new agent = one new file in `targets/` + one entry in `registry.ts`. + * + * Closes the Claude-locked installer issue (upstream #137). The + * runtime MCP server is already agent-agnostic; this brings the + * installer to the same surface. + */ + +export type Location = 'global' | 'local'; + +/** + * Stable string id used in the `--target` CLI flag and the registry + * lookup. New targets add a value here when they're added to the + * registry. Keep these short and lowercase. + */ +export type TargetId = 'claude' | 'cursor' | 'codex' | 'opencode' | 'hermes' | 'gemini' | 'antigravity' | 'kiro'; + +/** + * Result of `target.detect(location)`. + * + * `installed` is a best-effort heuristic that the agent's CLI / app / + * config dir is present on this system — used to default the + * multiselect prompt to "what's actually here." False positives are + * acceptable (we still write); false negatives just mean the user + * has to opt in manually. + * + * `alreadyConfigured` reports whether codegraph has already been + * wired into this target at this location — drives the + * "Updated"-vs-"Added" log line and lets `--check` exit 0/1. + */ +export interface DetectionResult { + installed: boolean; + alreadyConfigured: boolean; + /** Path inspected; surfaced in diagnostic / dry-run output. */ + configPath?: string; +} + +/** + * What `target.install(location)` actually changed on disk. The + * orchestrator renders one log line per file using `action`. + * + * `unchanged` means we touched the file but its contents were already + * what we'd write — used for byte-identical idempotent re-runs. + */ +export interface WriteResult { + files: Array<{ + path: string; + action: 'created' | 'updated' | 'unchanged' | 'removed' | 'not-found' | 'kept'; + }>; + /** + * Optional one-line notes the orchestrator surfaces verbatim — e.g. + * "Restart Cursor to apply." Keep these short; multi-line goes in + * the README. + */ + notes?: string[]; +} + +export interface InstallOptions { + /** + * Whether to write the agent's permissions / auto-allow surface + * (Claude `settings.json`, others where applicable). When the + * target has no permissions concept this option is a no-op. + */ + autoAllow: boolean; +} + +export interface AgentTarget { + /** Stable id; matches the `TargetId` union. */ + readonly id: TargetId; + /** Human-readable name shown in clack prompts and log lines. */ + readonly displayName: string; + /** Optional URL for "where do I learn more about this agent." */ + readonly docsUrl?: string; + /** + * Whether this target supports the given install location. + * + * Some agents (Codex CLI as of 2026-05) have no project-local + * config concept — only a single `~/.codex/` dir. Returning false + * for an unsupported (target, location) pair lets the orchestrator + * skip cleanly with a clear message. + */ + supportsLocation(loc: Location): boolean; + detect(loc: Location): DetectionResult; + install(loc: Location, opts: InstallOptions): WriteResult; + /** + * Inverse of install. Removes only what install would have written; + * preserves sibling MCP servers, sibling permissions, and unrelated + * markdown sections. Must be safe to call when nothing was ever + * installed (returns `not-found` actions). + */ + uninstall(loc: Location): WriteResult; + /** + * Print the MCP-server snippet a user would paste manually for this + * target. Used by `codegraph install --print-config ` and by + * the README. Must NOT touch the filesystem. + */ + printConfig(loc: Location): string; + /** Filesystem paths this target would write to at this location. */ + describePaths(loc: Location): string[]; +} diff --git a/src/mcp/daemon-paths.ts b/src/mcp/daemon-paths.ts new file mode 100644 index 000000000..b1afc40df --- /dev/null +++ b/src/mcp/daemon-paths.ts @@ -0,0 +1,99 @@ +/** + * Daemon socket + lockfile path helpers — issue #411. + * + * One shared `codegraph serve --mcp` daemon per project root means we need a + * stable, project-keyed rendezvous between cooperating processes. The IPC + * surface area is just two file paths: + * + * - `daemon.sock` — Unix domain socket / named pipe the daemon listens on. + * - `daemon.pid` — atomic-create lockfile holding the daemon's pid + version. + * + * Both live under `.codegraph/` so the project-scoped uninstall (`codegraph + * uninit`) sweeps them up for free. + * + * Special-case: Unix domain socket paths have a hard length limit (~104 on + * macOS, ~108 on Linux); when the in-project path exceeds it we fall back to + * an absolute-path hash under `os.tmpdir()`. The pidfile always stays in the + * project (it doesn't have a length limit) — and acts as the authoritative + * pointer to the socket path the daemon chose. + */ + +import * as crypto from 'crypto'; +import * as os from 'os'; +import * as path from 'path'; +import { getCodeGraphDir } from '../directory'; + +/** Soft upper bound for in-project socket paths. */ +const POSIX_SOCKET_PATH_LIMIT = 100; + +/** Short stable identifier for a project root — used in tmpdir/pipe names. */ +function projectHash(projectRoot: string): string { + return crypto.createHash('sha256').update(path.resolve(projectRoot)).digest('hex').slice(0, 16); +} + +/** + * Compute the socket / named-pipe path the daemon should listen on (and the + * proxy should connect to) for `projectRoot`. Deterministic given a project + * root, so independent processes converge without coordination. + */ +export function getDaemonSocketPath(projectRoot: string): string { + if (process.platform === 'win32') { + return `\\\\.\\pipe\\codegraph-${projectHash(projectRoot)}`; + } + const inProject = path.join(getCodeGraphDir(projectRoot), 'daemon.sock'); + if (inProject.length <= POSIX_SOCKET_PATH_LIMIT) return inProject; + // Long project paths (deep monorepos, Bazel out dirs) need tmpdir fallback + // or `bind` returns EADDRINUSE / ENAMETOOLONG. Hash keeps it project-scoped. + return path.join(os.tmpdir(), `codegraph-${projectHash(projectRoot)}.sock`); +} + +/** Absolute path to the daemon pid lockfile for `projectRoot`. */ +export function getDaemonPidPath(projectRoot: string): string { + return path.join(getCodeGraphDir(projectRoot), 'daemon.pid'); +} + +/** Structured contents of the pid lockfile. */ +export interface DaemonLockInfo { + pid: number; + version: string; + socketPath: string; + startedAt: number; +} + +/** + * Serialize a {@link DaemonLockInfo} for writing to the pidfile. JSON for + * human readability — operators occasionally `cat` this when debugging. + */ +export function encodeLockInfo(info: DaemonLockInfo): string { + return JSON.stringify(info, null, 2) + '\n'; +} + +/** + * Parse a pidfile body. Tolerant of old-format pidfiles (plain decimal pid) so + * a 0.10.x daemon doesn't trip over a 0.9.x lockfile if that ever happens — + * we treat such a lockfile as "process is unknown version, refuse to share." + */ +export function decodeLockInfo(raw: string): DaemonLockInfo | null { + const trimmed = raw.trim(); + if (!trimmed) return null; + try { + const parsed = JSON.parse(trimmed); + if ( + parsed && + typeof parsed.pid === 'number' && + typeof parsed.version === 'string' && + typeof parsed.socketPath === 'string' && + typeof parsed.startedAt === 'number' + ) { + return parsed as DaemonLockInfo; + } + return null; + } catch { + // Fall through to legacy plain-pid handling. + } + const pid = Number(trimmed); + if (Number.isFinite(pid) && pid > 0) { + return { pid, version: 'unknown', socketPath: '', startedAt: 0 }; + } + return null; +} diff --git a/src/mcp/daemon.ts b/src/mcp/daemon.ts new file mode 100644 index 000000000..ffd262476 --- /dev/null +++ b/src/mcp/daemon.ts @@ -0,0 +1,397 @@ +/** + * Shared MCP daemon — issue #411. + * + * One detached `codegraph serve --mcp` daemon process per project root, + * accepting N concurrent MCP clients over a Unix-domain socket (or named pipe + * on Windows). Each incoming connection gets its own {@link MCPSession}; all + * sessions share a single {@link MCPEngine}, which means a single file watcher + * (one inotify set), a single SQLite connection (one WAL writer), and a single + * tree-sitter warm-up — paid once, amortized across every agent talking to the + * project. + * + * Lifecycle (see also `./index.ts` and `./proxy.ts`): + * - The daemon is spawned **detached** (its own session/process group, stdio + * decoupled) by the first launcher that finds no daemon running. It is NOT + * a child of any MCP host, so closing one terminal / Ctrl-C'ing one session + * can't take it down and sever the others. That's why this process has no + * PPID watchdog: it deliberately outlives every individual client. + * - Every MCP host talks to the daemon through a thin `proxy` process (the + * thing the host actually spawned). The proxy keeps the #277 PPID watchdog, + * so a SIGKILL'd host still reaps its proxy promptly; the proxy's socket + * close then decrements the daemon's refcount. + * - When the last client disconnects the daemon lingers for + * `CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS` (default 300s) so back-to-back agent + * runs in the same project don't repay startup, then exits cleanly. This is + * what keeps a single-agent session from leaking a daemon forever (#277). + * + * What this file owns: + * - Listening on the daemon socket and spawning per-connection sessions. + * - The handshake "hello" line that lets a proxy verify it found a + * same-version daemon before piping any JSON-RPC through it. + * - The lockfile (`.codegraph/daemon.pid`) competing daemons arbitrate + * against — atomic `O_EXCL` create with the full record written in the same + * breath (no empty-file window) + cleanup on exit. + * - Reference counting + idle timeout. + * - Graceful shutdown on SIGTERM/SIGINT and idle exit. + * + * What this file does NOT own: + * - The proxy side (`./proxy.ts`). + * - The decision of *whether* to run as daemon at all — that's `MCPServer`. + * - The MCP protocol state machine — that's `./session.ts`. + */ + +import * as fs from 'fs'; +import * as net from 'net'; +import * as path from 'path'; +import { MCPEngine } from './engine'; +import { MCPSession } from './session'; +import { SocketTransport } from './transport'; +import { + DaemonLockInfo, + decodeLockInfo, + encodeLockInfo, + getDaemonPidPath, + getDaemonSocketPath, +} from './daemon-paths'; +import { CodeGraphPackageVersion } from './version'; + +/** Default idle linger after the last client disconnects. */ +const DEFAULT_IDLE_TIMEOUT_MS = 300_000; + +/** Bytes/parse-window for an oversized hello line — bounded against a malicious peer. */ +const MAX_HELLO_LINE_BYTES = 4096; + +/** + * Wire format for the one-shot hello line the daemon emits on every new + * connection. Versioned with the package's own semver so a 0.9.x proxy never + * pipes through a 0.10.x daemon (or vice-versa) — the proxy falls back to + * direct mode on mismatch rather than risk subtle wire incompatibilities. + */ +export interface DaemonHello { + codegraph: string; // package version (must match the proxy's own version) + pid: number; // daemon pid (informational; for `ps` debugging) + socketPath: string; // echoed back so the proxy can log it + protocol: 1; // bump if the hello shape changes +} + +export interface DaemonStartResult { + /** Always-non-null for a successfully-started daemon. */ + socketPath: string; + /** Lockfile contents as written. */ + lock: DaemonLockInfo; +} + +/** + * Run as the shared daemon for `projectRoot`. Resolves once the socket is + * listening. The Daemon owns the socket, the engine, and the lockfile until + * `stop()` is called or it exits on idle/signal. + * + * Race-safe: callers must first call `tryAcquireDaemonLock(projectRoot)` and + * only construct a Daemon if they got the lock (`kind: 'acquired'`). The atomic + * `O_EXCL` create inside the acquire helper — which now also writes the full + * record before returning — is the only synchronization between competing + * daemons. + */ +export class Daemon { + private server: net.Server | null = null; + private clients = new Set(); + private idleTimer: NodeJS.Timeout | null = null; + private idleTimeoutMs: number; + private engine: MCPEngine; + private stopping = false; + private socketPath: string; + private pidPath: string; + + constructor( + private projectRoot: string, + opts: { idleTimeoutMs?: number } = {}, + ) { + this.socketPath = getDaemonSocketPath(projectRoot); + this.pidPath = getDaemonPidPath(projectRoot); + this.idleTimeoutMs = opts.idleTimeoutMs ?? resolveIdleTimeoutMs(); + this.engine = new MCPEngine(); + this.engine.setProjectPathHint(projectRoot); + } + + /** + * Bind the socket, kick off engine init, and register signal handlers. The + * lockfile body was already written atomically by `tryAcquireDaemonLock`, so + * there is nothing to write here. The promise resolves once the server is + * listening — the daemon then sticks around until idle/shutdown. + */ + async start(): Promise { + // Engine init is deliberately backgrounded — see #172. The first session + // to land waits on `ensureInitialized` either way, and unloaded sessions + // (cross-project tool calls only) shouldn't pay any open cost. + void this.engine.ensureInitialized(this.projectRoot); + + // Stale socket file (left over from a SIGKILL'd previous daemon) will + // wedge `listen` with EADDRINUSE. We arrived here holding the lockfile, + // which means there's no live daemon, so it's safe to clear. + if (process.platform !== 'win32') { + try { fs.unlinkSync(this.socketPath); } catch { /* not-exists is fine */ } + } + + await new Promise((resolve, reject) => { + const server = net.createServer((socket) => this.handleConnection(socket)); + server.once('error', (err) => reject(err)); + server.listen(this.socketPath, () => { + // POSIX: tighten permissions to user-only — the socket lives under + // `.codegraph/`, which is git-ignored but may be on a shared FS. + if (process.platform !== 'win32') { + try { fs.chmodSync(this.socketPath, 0o600); } catch { /* best-effort */ } + } + this.server = server; + resolve(); + }); + }); + + const lock: DaemonLockInfo = { + pid: process.pid, + version: CodeGraphPackageVersion, + socketPath: this.socketPath, + startedAt: Date.now(), + }; + + process.stderr.write( + `[CodeGraph daemon] Listening on ${this.socketPath} (pid ${process.pid}, v${CodeGraphPackageVersion}). Idle timeout ${this.idleTimeoutMs}ms.\n` + ); + + // No clients yet: arm the idle timer immediately so a daemon that nobody + // ever connects to (e.g. spawned then abandoned because the launcher died) + // doesn't pin resources forever. + this.armIdleTimer(); + + process.on('SIGINT', () => this.stop('SIGINT')); + process.on('SIGTERM', () => this.stop('SIGTERM')); + + return { socketPath: this.socketPath, lock }; + } + + /** Currently-connected client count. Exposed for tests / status output. */ + getClientCount(): number { + return this.clients.size; + } + + /** The socket path the daemon is (or will be) listening on. */ + getSocketPath(): string { + return this.socketPath; + } + + /** Graceful shutdown: close all sessions, the engine, and clean up the lock. */ + async stop(reason: string = 'stop'): Promise { + if (this.stopping) return; + this.stopping = true; + if (this.idleTimer) { + clearTimeout(this.idleTimer); + this.idleTimer = null; + } + process.stderr.write(`[CodeGraph daemon] Shutting down (${reason}; clients=${this.clients.size}).\n`); + for (const session of [...this.clients]) { + try { session.stop(); } catch { /* best-effort */ } + } + this.clients.clear(); + if (this.server) { + await new Promise((resolve) => this.server!.close(() => resolve())); + this.server = null; + } + this.engine.stop(); + this.cleanupLockfile(); + if (process.platform !== 'win32') { + try { fs.unlinkSync(this.socketPath); } catch { /* may already be gone */ } + } + process.exit(0); + } + + private handleConnection(socket: net.Socket): void { + // Hello first so the proxy can verify versions before piping any + // application bytes. The proxy reads exactly one line, then forwards. + const hello: DaemonHello = { + codegraph: CodeGraphPackageVersion, + pid: process.pid, + socketPath: this.socketPath, + protocol: 1, + }; + socket.write(JSON.stringify(hello) + '\n'); + + const transport = new SocketTransport(socket); + const session = new MCPSession(transport, this.engine, { + explicitProjectPath: this.projectRoot, + }); + transport.onClose(() => this.dropClient(session)); + this.clients.add(session); + this.disarmIdleTimer(); + session.start(); + } + + private dropClient(session: MCPSession): void { + if (!this.clients.delete(session)) return; + if (this.clients.size === 0) this.armIdleTimer(); + } + + private armIdleTimer(): void { + if (this.idleTimer || this.stopping) return; + if (this.idleTimeoutMs <= 0) return; // 0 = never idle-exit + this.idleTimer = setTimeout(() => { + this.idleTimer = null; + // Last-second sanity check: if a connection landed between the timer + // firing and now, don't exit. (setImmediate-ordering is the only way + // this races; cheap to defend against.) + if (this.clients.size > 0) { + this.armIdleTimer(); + return; + } + void this.stop('idle timeout'); + }, this.idleTimeoutMs); + // Don't keep the event loop alive just for this — the net.Server keeps the + // loop alive while listening, so the timer still fires; once we stop() the + // loop should drain naturally. + this.idleTimer.unref?.(); + } + + private disarmIdleTimer(): void { + if (!this.idleTimer) return; + clearTimeout(this.idleTimer); + this.idleTimer = null; + } + + private cleanupLockfile(): void { + try { + if (fs.existsSync(this.pidPath)) { + // Only remove if it still belongs to us — another daemon may have + // already taken over while we were shutting down (extremely rare). + const raw = fs.readFileSync(this.pidPath, 'utf8'); + const info = decodeLockInfo(raw); + if (info && info.pid === process.pid) { + fs.unlinkSync(this.pidPath); + } + } + } catch { /* best-effort; we're exiting anyway */ } + } +} + +/** + * Result of `tryAcquireDaemonLock`. Either we got the lockfile (caller becomes + * the daemon), or it already existed (caller should connect to the existing + * daemon as a proxy, or — if the holder is dead — clear it and retry). + */ +export type AcquireResult = + | { kind: 'acquired'; pidPath: string; info: DaemonLockInfo } + | { kind: 'taken'; existing: DaemonLockInfo | null; pidPath: string }; + +/** + * Atomically create the daemon pidfile with its full record already in place. + * Returns either an `acquired` result (the caller is the daemon-elect and may + * construct a {@link Daemon}) or a `taken` result. + * + * must-fix 1 (issue #411 review): the lockfile must appear in ONE atomic step, + * already complete — never empty, even momentarily. The first attempt at this + * (`O_EXCL` create then a separate `writeSync`) left a microsecond window where + * the file existed but was empty; under concurrent daemon startup a third + * candidate could read that empty file, decode it as `null`, and `unlink` the + * winner's lock → two daemons (two watchers, two writers). The window was + * normally too small to hit, but the chokidar watcher's extra startup time made + * concurrent daemons overlap enough to reproduce it reliably. + * + * The fix writes the complete record to a private temp file, then hard-links it + * into place: `link()` is atomic AND exclusive (EEXIST if the target exists), so + * the pidfile becomes visible in one step already containing a full record. + * Whoever links first wins; everyone else gets EEXIST and reads a complete file. + * There is no empty-file window at all. + */ +export function tryAcquireDaemonLock(projectRoot: string): AcquireResult { + const pidPath = getDaemonPidPath(projectRoot); + // Make sure the .codegraph/ directory exists — the daemon may be the first + // thing to touch it on a fresh-clone-but-already-initialized checkout. + fs.mkdirSync(path.dirname(pidPath), { recursive: true }); + + const info: DaemonLockInfo = { + pid: process.pid, + version: CodeGraphPackageVersion, + socketPath: getDaemonSocketPath(projectRoot), + startedAt: Date.now(), + }; + + // Temp name is pid-scoped so racing candidates never collide on it. + const tmp = `${pidPath}.${process.pid}.tmp`; + let acquired = false; + try { + fs.writeFileSync(tmp, encodeLockInfo(info), { mode: 0o600 }); + try { + fs.linkSync(tmp, pidPath); // atomic + exclusive + acquired = true; + } catch (err: unknown) { + if ((err as NodeJS.ErrnoException).code !== 'EEXIST') throw err; + } + } finally { + try { fs.unlinkSync(tmp); } catch { /* temp already gone */ } + } + + if (acquired) return { kind: 'acquired', pidPath, info }; + + // Taken. Because the pidfile was link'd atomically it always holds a complete + // record — `existing` is null only for a genuinely corrupt leftover, never a + // mid-write race. + let existing: DaemonLockInfo | null = null; + try { + existing = decodeLockInfo(fs.readFileSync(pidPath, 'utf8')); + } catch { /* unreadable lockfile — treat as malformed */ } + return { kind: 'taken', existing, pidPath }; +} + +/** + * Remove a stale pidfile, but only if it still names a dead process. Re-reads + * the file immediately before unlinking so we never delete a lock that a live + * daemon (re)acquired in the meantime. + * + * must-fix 1 (issue #411 review): the original unconditionally `unlink`'d, + * which let a racing candidate delete a healthy daemon's lock. Passing + * `expectedDeadPid` (the pid the caller believed was dead) makes the clear a + * compare-and-delete: bail if the file now holds a different pid, or any live + * pid. Returns true when the stale lock is gone (or was already gone). + */ +export function clearStaleDaemonLock(pidPath: string, expectedDeadPid?: number): boolean { + try { + const raw = fs.readFileSync(pidPath, 'utf8'); + const info = decodeLockInfo(raw); + if (info) { + // A different pid took over since we read it — not ours to clear. + if (expectedDeadPid !== undefined && info.pid !== expectedDeadPid) return false; + // Holder is actually alive — never clear a live daemon's lock. + if (info.pid > 0 && isProcessAlive(info.pid)) return false; + } + fs.unlinkSync(pidPath); + return true; + } catch (err: unknown) { + const e = err as NodeJS.ErrnoException; + if (e.code === 'ENOENT') return true; // already gone + return false; + } +} + +/** + * Probe whether `pid` is currently alive (signal-0). Treats EPERM as alive on + * every platform (the process exists, it's just not ours to signal) so we never + * mistake a live daemon for a dead one and clear its lock. + */ +export function isProcessAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (err: unknown) { + const e = err as NodeJS.ErrnoException; + if (e.code === 'EPERM') return true; // exists, just not ours to signal + return false; + } +} + +function resolveIdleTimeoutMs(): number { + const raw = process.env.CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS; + if (raw === undefined || raw === '') return DEFAULT_IDLE_TIMEOUT_MS; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed < 0) return DEFAULT_IDLE_TIMEOUT_MS; + return Math.floor(parsed); +} + +/** Exported for test stubs that need to bound the hello-line read. */ +export { MAX_HELLO_LINE_BYTES }; diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts new file mode 100644 index 000000000..193f2bbd0 --- /dev/null +++ b/src/mcp/engine.ts @@ -0,0 +1,269 @@ +/** + * MCP shared engine — the heavyweight, *shared* state for an MCP server: + * the project's {@link CodeGraph} instance, file watcher, and the + * {@link ToolHandler} cache for cross-project queries. + * + * One engine, many sessions: + * - direct mode (single stdio session) instantiates one engine + one session; + * - daemon mode instantiates one engine and a new session per socket + * connection. Every session reads from the same SQLite WAL and the same + * inotify watch set — that's the entire point of issue #411. + */ + +import CodeGraph, { findNearestCodeGraphRoot } from '../index'; +import { watchDisabledReason } from '../sync'; +import { ToolHandler } from './tools'; + +export interface MCPEngineOptions { + /** + * Whether to start the file watcher when initializing. Daemon and direct + * modes both want this true; tests may set it false to keep the engine + * cheap. Honors {@link watchDisabledReason} regardless. + */ + watch?: boolean; +} + +/** + * Shared MCP engine. Thread-safe in the sense that multiple sessions can + * call its methods concurrently — internally it serializes initialization + * through a single promise so multiple sessions racing each other on first + * connect never double-open the SQLite file. + */ +export class MCPEngine { + private cg: CodeGraph | null = null; + private toolHandler: ToolHandler; + // Project root we resolved to. Null until `ensureInitialized` succeeds + // (or null forever if no .codegraph/ ever turned up — that's a valid + // state for the engine, since cross-project queries still work). + private projectPath: string | null = null; + // Set on first `ensureInitialized` so subsequent sessions don't redo work. + private initPromise: Promise | null = null; + private watcherStarted = false; + private opts: Required; + private closed = false; + + constructor(opts: MCPEngineOptions = {}) { + this.opts = { watch: opts.watch ?? true }; + this.toolHandler = new ToolHandler(null); + } + + /** + * Convenience for {@link MCPServer} compatibility: pre-seed an explicit + * project path (from the `--path` CLI flag) without yet opening it. This + * keeps the synchronous constructor cheap; the actual open happens on the + * first `ensureInitialized` call. + */ + setProjectPathHint(projectPath: string): void { + this.projectPath = projectPath; + this.toolHandler.setDefaultProjectHint(projectPath); + } + + /** Project root that the engine resolved on first init (null if none). */ + getProjectPath(): string | null { + return this.projectPath; + } + + /** Shared ToolHandler — sessions delegate tool dispatch through this. */ + getToolHandler(): ToolHandler { + return this.toolHandler; + } + + /** Whether the default project's CodeGraph is open. */ + hasDefaultCodeGraph(): boolean { + return this.toolHandler.hasDefaultCodeGraph(); + } + + /** + * Walk up from `searchFrom` to find the nearest `.codegraph/` and open it. + * Idempotent: concurrent callers share one in-flight init; subsequent + * callers after success are no-ops. + * + * The original `MCPServer.tryInitializeDefault` carried the same retry-on- + * subsequent-tool-call semantics; we preserve them by NOT throwing when the + * search misses (just leaves `cg` null so the next call can retry). + */ + async ensureInitialized(searchFrom: string): Promise { + if (this.closed) return; + if (this.toolHandler.hasDefaultCodeGraph()) return; + if (this.initPromise) { + try { await this.initPromise; } catch { /* let caller retry */ } + return; + } + + this.initPromise = this.doInitialize(searchFrom).finally(() => { + this.initPromise = null; + }); + try { + await this.initPromise; + } catch { + // Init errors are logged inside `doInitialize`; falling through here + // matches MCPServer's previous "retry on next tool call" behavior. + } + } + + /** + * Synchronous last-resort init used by the per-session retry loop when the + * background `ensureInitialized` already finished (or failed) and we need + * to pick up a project that appeared *after* the engine started. + */ + retryInitializeSync(searchFrom: string): void { + if (this.closed) return; + if (this.toolHandler.hasDefaultCodeGraph()) return; + this.toolHandler.setDefaultProjectHint(searchFrom); + const resolvedRoot = findNearestCodeGraphRoot(searchFrom); + if (!resolvedRoot) return; + try { + // Close any previously failed instance to avoid leaking resources. + if (this.cg) { + try { this.cg.close(); } catch { /* ignore */ } + this.cg = null; + } + this.cg = CodeGraph.openSync(resolvedRoot); + this.projectPath = resolvedRoot; + this.toolHandler.setDefaultCodeGraph(this.cg); + this.startWatching(); + this.catchUpSync(); + } catch { + // Still failing — caller will try again on the next tool call. + } + } + + /** + * Close everything. Used on graceful daemon shutdown (SIGTERM/idle timeout) + * and on direct-mode stop. Idempotent. + */ + stop(): void { + if (this.closed) return; + this.closed = true; + this.toolHandler.closeAll(); + if (this.cg) { + try { this.cg.close(); } catch { /* ignore */ } + this.cg = null; + } + } + + private async doInitialize(searchFrom: string): Promise { + this.toolHandler.setDefaultProjectHint(searchFrom); + + const resolvedRoot = findNearestCodeGraphRoot(searchFrom); + if (!resolvedRoot) { + // No .codegraph/ above searchFrom. Sessions may still discover one later via roots/list + this.projectPath = searchFrom; + return; + } + + this.projectPath = resolvedRoot; + try { + this.cg = await CodeGraph.open(resolvedRoot); + this.toolHandler.setDefaultCodeGraph(this.cg); + this.startWatching(); + this.catchUpSync(); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + process.stderr.write(`[CodeGraph MCP] Failed to open project at ${resolvedRoot}: ${msg}\n`); + } + } + + /** + * Start file watching on the active CodeGraph instance. Idempotent — the + * watcher is per-engine, not per-session, which is why the daemon path + * collapses N inotify sets to one. The wording of the disabled-reason log + * exactly matches the prior in-tree implementation so log-driven dashboards + * keep working. + */ + private startWatching(): void { + if (!this.cg || this.watcherStarted || !this.opts.watch) return; + + const disabledReason = watchDisabledReason(this.projectPath ?? process.cwd()); + if (disabledReason) { + process.stderr.write( + `[CodeGraph MCP] File watcher disabled — ${disabledReason}. ` + + `The graph will not auto-update; run \`codegraph sync\` (or install the git sync hooks via \`codegraph init\`) to refresh.\n` + ); + this.watcherStarted = true; + return; + } + + // Optional override for the debounce window via env var (issue #403). + // Useful for workspaces with bursty writes (formatter-on-save chains, + // large generated outputs) where the 2s default fires too often. Clamped + // to [100ms, 60s]; out-of-range / non-numeric values fall back to the + // FileWatcher default. We log the active value so it's discoverable. + const debounceMs = parseDebounceEnv(process.env.CODEGRAPH_WATCH_DEBOUNCE_MS); + if (debounceMs !== undefined) { + process.stderr.write(`[CodeGraph MCP] File watcher debounce: ${debounceMs}ms (CODEGRAPH_WATCH_DEBOUNCE_MS)\n`); + } + + const started = this.cg.watch({ + debounceMs, + onSyncComplete: (result) => { + if (result.filesChanged > 0) { + process.stderr.write( + `[CodeGraph MCP] Auto-synced ${result.filesChanged} file(s) in ${result.durationMs}ms\n` + ); + } + }, + onSyncError: (err) => { + process.stderr.write(`[CodeGraph MCP] Auto-sync error: ${err.message}\n`); + }, + }); + + this.watcherStarted = true; + if (started) { + process.stderr.write('[CodeGraph MCP] File watcher active — graph will auto-sync on changes\n'); + } else { + process.stderr.write( + '[CodeGraph MCP] File watcher unavailable on this platform — run `codegraph sync` to refresh the graph after changes.\n' + ); + } + } + + /** + * Reconcile the index with the current filesystem once, right after open — + * catches edits, adds, deletes, and `git pull`/`checkout` changes made while + * no watcher was running. Runs in the background, but the returned promise + * is pushed into the ToolHandler as a one-shot gate so the *first* tool + * call awaits completion before serving (without this, a tool call that + * races past sync returns rows for files that no longer exist on disk — + * and the per-file staleness banner can't help because `getPendingFiles()` + * is populated by the watcher, not by catch-up). + */ + private catchUpSync(): void { + const cg = this.cg; + if (!cg) return; + const p = cg + .sync() + .then((result) => { + const changed = result.filesAdded + result.filesModified + result.filesRemoved; + if (changed > 0) { + process.stderr.write(`[CodeGraph MCP] Caught up ${changed} file(s) changed since last run\n`); + } + }) + .catch((err) => { + const msg = err instanceof Error ? err.message : String(err); + process.stderr.write(`[CodeGraph MCP] Catch-up sync failed: ${msg}\n`); + }); + this.toolHandler.setCatchUpGate(p); + } +} + +/** + * Parse and clamp the CODEGRAPH_WATCH_DEBOUNCE_MS env override. + * + * Issue #403: workspaces with bursty writes (formatter-on-save, multi-file + * refactors) sometimes want a longer quiet window before sync. Returns + * `undefined` for unset / empty / non-numeric / out-of-range values so the + * FileWatcher default (2000ms) takes over — never throws. + * + * Clamp range: 100ms (faster would mean a sync per keystroke) to 60s (longer + * and the watcher feels broken). Out-of-range values are treated as "ignore + * this misconfiguration" rather than capped, since silently capping a 0 or + * a typoed value would mask a real config bug. + */ +export function parseDebounceEnv(raw: string | undefined): number | undefined { + if (!raw || !raw.trim()) return undefined; + const n = Number(raw); + if (!Number.isFinite(n) || !Number.isInteger(n)) return undefined; + if (n < 100 || n > 60000) return undefined; + return n; +} diff --git a/src/mcp/index.ts b/src/mcp/index.ts index e516631a0..85c3949e2 100644 --- a/src/mcp/index.ts +++ b/src/mcp/index.ts @@ -13,328 +13,441 @@ * const server = new MCPServer('/path/to/project'); * await server.start(); * ``` + * + * Runtime modes (decided in {@link MCPServer.start}): + * + * - **Direct** — one process serves one MCP client over stdio. The pre-#411 + * behavior; used when the user opts out (`CODEGRAPH_NO_DAEMON=1`), no + * `.codegraph/` is reachable, or the daemon machinery fails for any reason. + * - **Proxy** — what an MCP host actually talks to when sharing is on: a thin + * stdio↔socket pipe to the shared daemon. The proxy carries the #277 PPID + * watchdog, so a SIGKILL'd host reaps its proxy promptly. See {@link ./proxy.ts}. + * - **Daemon** — a *detached* background process (its own session/process + * group) that serves N proxies over a Unix-domain socket / named pipe, + * sharing one CodeGraph + watcher + SQLite handle. Spawned on demand; never a + * child of any host, so it survives individual sessions and is reaped by + * client-refcount + idle timeout. See {@link ./daemon.ts} and issue #411. + * + * The detached-daemon + always-proxy split is the fix for the review finding + * that the original in-process daemon (a) was the first host's child, so closing + * that terminal severed every other client, and (b) disabled the PPID watchdog, + * regressing #277 (orphaned daemons on host SIGKILL). */ +import * as fs from 'fs'; import * as path from 'path'; -import CodeGraph, { findNearestCodeGraphRoot } from '../index'; -import { StdioTransport, JsonRpcRequest, JsonRpcNotification, ErrorCodes } from './transport'; -import { tools, ToolHandler } from './tools'; -import { SERVER_INSTRUCTIONS } from './server-instructions'; +import { spawn, StdioOptions } from 'child_process'; +import { findNearestCodeGraphRoot } from '../index'; +import { getCodeGraphDir } from '../directory'; +import { StdioTransport } from './transport'; +import { MCPEngine } from './engine'; +import { MCPSession } from './session'; +import { + Daemon, + clearStaleDaemonLock, + isProcessAlive, + tryAcquireDaemonLock, +} from './daemon'; +import { runProxy } from './proxy'; +import { getDaemonSocketPath } from './daemon-paths'; +import { HOST_PPID_ENV } from '../extraction/wasm-runtime-flags'; /** - * Convert a file:// URI to a filesystem path. - * Handles URL encoding and Windows drive letter paths. + * How often to poll `process.ppid` to detect parent process death (see #277). + * 5s is a deliberate trade-off: the failure mode being guarded against is rare + * (parent SIGKILL'd), and longer poll = less wakeup overhead while idle. */ -function fileUriToPath(uri: string): string { - try { - const url = new URL(uri); - let filePath = decodeURIComponent(url.pathname); - // On Windows, file:///C:/path produces pathname /C:/path — strip leading / - if (process.platform === 'win32' && /^\/[a-zA-Z]:/.test(filePath)) { - filePath = filePath.slice(1); - } - return path.resolve(filePath); - } catch { - // Fallback for non-standard URIs - return uri.replace(/^file:\/\/\/?/, ''); - } +const DEFAULT_PPID_POLL_MS = 5000; + +/** + * Env var that marks a process as the *detached daemon* itself (set by + * {@link spawnDetachedDaemon} when it re-invokes the CLI). Without it a + * `serve --mcp` invocation is a launcher that connects-or-spawns; with it, the + * process IS the daemon and must never try to spawn another (infinite spawn). + */ +const DAEMON_INTERNAL_ENV = 'CODEGRAPH_DAEMON_INTERNAL'; + +/** + * Retries for the detached daemon arbitrating the O_EXCL lock against a racing + * sibling. Tiny — the lock resolves on the first round in practice; the retries + * only cover clearing a genuinely stale (dead-pid) lockfile. + */ +const TAKEOVER_MAX_RETRIES = 5; +const TAKEOVER_RETRY_DELAY_MS = 100; + +/** + * How long a launcher waits for a freshly-spawned daemon to bind its socket + * before giving up and running in-process. The daemon binds the socket *before* + * the (backgrounded) engine/grammar warm-up, so this only needs to cover node + * process startup. 60 × 100ms = 6s of headroom for a cold/slow box; on the + * common path the socket appears within a few rounds. + */ +const DAEMON_CONNECT_MAX_RETRIES = 60; +const DAEMON_CONNECT_RETRY_DELAY_MS = 100; + +/** + * Resolve the PPID watchdog poll interval from an env override. A value of + * `0` disables the watchdog entirely (escape hatch for embedded scenarios + * where the parent legitimately re-parents the server on purpose). Anything + * non-numeric or negative falls back to the default. + */ +function parsePpidPollMs(raw: string | undefined): number { + if (raw === undefined || raw === '') return DEFAULT_PPID_POLL_MS; + const parsed = Number(raw); + if (!Number.isFinite(parsed)) return DEFAULT_PPID_POLL_MS; + if (parsed < 0) return DEFAULT_PPID_POLL_MS; + return Math.floor(parsed); } /** - * MCP Server Info + * Parse the host PID propagated across the `--liftoff-only` re-exec + * ({@link HOST_PPID_ENV}). Returns a positive integer PID, or null when + * unset/invalid — the direct-launch path, where the watchdog falls back to + * `process.ppid` divergence. PIDs of 0/1 are rejected (0 = unknown, 1 = init, + * i.e. already orphaned), so the watchdog doesn't latch onto init. */ -const SERVER_INFO = { - name: 'codegraph', - version: '0.1.0', -}; +function parseHostPpid(raw: string | undefined): number | null { + if (raw === undefined || raw === '') return null; + const parsed = Number(raw); + if (!Number.isInteger(parsed) || parsed <= 1) return null; + return parsed; +} + +/** Whether `CODEGRAPH_NO_DAEMON` was set to a truthy value. */ +function daemonOptOutSet(): boolean { + const raw = process.env.CODEGRAPH_NO_DAEMON; + if (!raw) return false; + return raw !== '0' && raw.toLowerCase() !== 'false'; +} + +/** Whether this process was spawned to BE the detached daemon. */ +function daemonInternalSet(): boolean { + const raw = process.env[DAEMON_INTERNAL_ENV]; + return !!raw && raw !== '0' && raw.toLowerCase() !== 'false'; +} /** - * MCP Protocol Version + * Resolve the project root the daemon machinery should key on. Returns + * `null` when no `.codegraph/` is reachable from the candidate path — in + * that case the caller must run in direct mode, since the daemon lockfile + * and socket both live under `.codegraph/`. + * + * The result is canonicalized with `realpathSync` so every client converges on + * the same socket/lock path regardless of how it expressed the path: a client + * launched with cwd under a symlink (e.g. macOS `/var` → `/private/var`, where + * spawned `process.cwd()` is already realpath'd) and one that passed a + * symlinked `rootUri` would otherwise hash to different sockets and silently + * fail to share the daemon. */ -const PROTOCOL_VERSION = '2024-11-05'; +function resolveDaemonRoot(explicitPath: string | null): string | null { + const candidate = explicitPath ?? process.cwd(); + const root = findNearestCodeGraphRoot(candidate); + if (!root) return null; + try { return fs.realpathSync(root); } catch { return root; } +} + +/** + * Spawn the shared daemon as a fully detached background process: its own + * session/process group (so a SIGHUP/SIGINT to the launcher's terminal can't + * reach it) with stdio decoupled from the launcher (logs to + * `.codegraph/daemon.log`). Re-invokes the *same* CLI faithfully across dev and + * bundled launches by reusing `process.argv[0]` (the right node), the current + * `process.execArgv` (carries `--liftoff-only`, so the daemon never re-execs) + * and `process.argv[1]` (this script). The spawned process self-arbitrates the + * O_EXCL lock, so racing launchers may each spawn one — losers exit and every + * launcher proxies through the single winner. + */ +function spawnDetachedDaemon(root: string): void { + const scriptPath = process.argv[1]; + if (!scriptPath) { + // No resolvable CLI entry point to re-invoke — let the caller fall back to + // direct mode rather than spawn something broken. + throw new Error('cannot resolve CLI script path to spawn the daemon'); + } + + let logFd: number | null = null; + let stdio: StdioOptions = 'ignore'; + try { + logFd = fs.openSync(path.join(getCodeGraphDir(root), 'daemon.log'), 'a'); + stdio = ['ignore', logFd, logFd]; + } catch { + stdio = 'ignore'; // no log file — discard daemon output rather than fail + } + try { + const child = spawn( + process.execPath, + [...process.execArgv, scriptPath, 'serve', '--mcp', '--path', root], + { + detached: true, + stdio, + windowsHide: true, + env: { ...process.env, [DAEMON_INTERNAL_ENV]: '1' }, + }, + ); + child.unref(); + } finally { + // The child holds its own dup of the log fd now; the launcher doesn't need it. + if (logFd !== null) { + try { fs.closeSync(logFd); } catch { /* ignore */ } + } + } +} /** * MCP Server for CodeGraph * * Implements the Model Context Protocol to expose CodeGraph * functionality as tools that can be called by AI assistants. + * + * Backwards-compatible constructor and `start()` signature with the + * pre-issue-#411 implementation: callers continue to do + * `new MCPServer(path).start()`. Internally we now pick from direct / proxy / + * daemon at start time. */ export class MCPServer { - private transport: StdioTransport; - private cg: CodeGraph | null = null; - private toolHandler: ToolHandler; private projectPath: string | null; + // Direct-mode-only state. In daemon mode the per-connection sessions live + // inside the Daemon class; in proxy mode there is no session at all. + private session: MCPSession | null = null; + private engine: MCPEngine | null = null; + private daemon: Daemon | null = null; + private ppidWatchdog: ReturnType | null = null; + // PPID watchdog baseline — captured at construction so we always have a + // baseline, even if start() runs after a fork-style reparent. + private originalPpid: number = process.ppid; + private hostPpid: number | null = parseHostPpid(process.env[HOST_PPID_ENV]); + // Idempotency guard for stop(). + private stopped = false; + private mode: 'unstarted' | 'direct' | 'proxy' | 'daemon' = 'unstarted'; constructor(projectPath?: string) { this.projectPath = projectPath || null; - this.transport = new StdioTransport(); - // Create ToolHandler eagerly — cross-project queries work even without a default project - this.toolHandler = new ToolHandler(null); } /** - * Start the MCP server + * Start the MCP server. * - * Note: CodeGraph initialization is deferred until the initialize request - * is received, which includes the rootUri from the client. - */ - async start(): Promise { - // Start listening for messages immediately - don't check initialization yet - // We'll get the project path from the initialize request's rootUri - this.transport.start(this.handleMessage.bind(this)); - - // Keep the process running - process.on('SIGINT', () => this.stop()); - process.on('SIGTERM', () => this.stop()); - - // When the parent process (Claude Code) exits, stdin closes. - // Detect this and shut down gracefully to prevent orphaned processes. - process.stdin.on('end', () => this.stop()); - process.stdin.on('close', () => this.stop()); - } - - /** - * Try to initialize CodeGraph for the default project. - * - * Walks up parent directories to find the nearest .codegraph/ folder, - * similar to how git finds .git/ directories. + * Decision order: + * 1. `CODEGRAPH_NO_DAEMON=1` → direct mode (unchanged pre-#411 behavior). + * 2. `CODEGRAPH_DAEMON_INTERNAL=1` → we ARE the detached daemon; listen. + * 3. No `.codegraph/` reachable → direct mode (the daemon's lockfile and + * socket both live under `.codegraph/`). + * 4. Otherwise connect to (or spawn) the shared daemon and proxy to it. * - * If initialization fails, the error is recorded but the server continues - * to work — cross-project queries and retries on subsequent tool calls - * are still possible. + * On any unexpected failure in step 4 we transparently fall back to direct + * mode — a misbehaving daemon must never block a session from starting. */ - private async tryInitializeDefault(projectPath: string): Promise { - // Walk up parent directories to find nearest .codegraph/ - const resolvedRoot = findNearestCodeGraphRoot(projectPath); + async start(): Promise { + // The detached daemon process itself. Checked before the opt-out so the + // daemon honors the same env it was spawned with (it never sets NO_DAEMON). + if (daemonInternalSet()) { + return this.startDaemonProcess(); + } - if (!resolvedRoot) { - this.projectPath = projectPath; - return; + // Direct mode if the user opted out. Setting the env var is sufficient to + // get the pre-#411 single-process behavior. + if (daemonOptOutSet()) { + return this.startDirect('CODEGRAPH_NO_DAEMON set'); } - this.projectPath = resolvedRoot; + const root = resolveDaemonRoot(this.projectPath); + if (!root) { + // No initialized project found — daemon mode has nowhere to put its + // socket. The fresh-checkout / outside-project case; behave as before. + return this.startDirect('no .codegraph/ root found'); + } try { - this.cg = await CodeGraph.open(resolvedRoot); - this.toolHandler.setDefaultCodeGraph(this.cg); - this.startWatching(); + const mode = await this.connectOrSpawnDaemon(root); + if (mode === 'fallback') { + return this.startDirect('daemon unavailable; fallback to direct'); + } + // 'proxy': connectOrSpawnDaemon ran the stdio↔socket pipe to completion + // (it only returns once the host disconnected). The process is now + // expected to terminate naturally — the proxy installed its own watchdog. + this.mode = 'proxy'; + return; } catch (err) { - // Log the error so transient failures are diagnosable (see issue #47) + // Belt-and-braces: if anything throws inside the daemon machinery, + // never wedge the user — fall back to a working direct-mode session. const msg = err instanceof Error ? err.message : String(err); - process.stderr.write(`[CodeGraph MCP] Failed to open project at ${resolvedRoot}: ${msg}\n`); + process.stderr.write(`[CodeGraph MCP] Daemon path failed (${msg}); falling back to direct mode.\n`); + return this.startDirect('daemon path threw'); } } /** - * Retry initialization of the default project if it previously failed. - * Called lazily on tool calls that need the default project. - * Re-walks parent directories each time so it picks up projects - * initialized after the MCP server started. + * Stop the server. In daemon mode this triggers graceful shutdown of every + * connected session; in direct mode it mirrors the pre-#411 behavior (close + * cg, exit). Proxy mode never routes through here — the proxy exits itself. */ - private retryInitIfNeeded(): void { - // Already initialized successfully - if (this.toolHandler.hasDefaultCodeGraph()) return; - // No project path to retry with - if (!this.projectPath) return; - - const resolvedRoot = findNearestCodeGraphRoot(this.projectPath); - if (!resolvedRoot) return; - - try { - // Close any previously failed instance to avoid leaking resources - if (this.cg) { - try { this.cg.close(); } catch { /* ignore */ } - this.cg = null; - } - this.cg = CodeGraph.openSync(resolvedRoot); - this.projectPath = resolvedRoot; - this.toolHandler.setDefaultCodeGraph(this.cg); - this.startWatching(); - } catch { - // Still failing — will retry on next tool call + stop(): void { + if (this.stopped) return; + this.stopped = true; + if (this.ppidWatchdog) { + clearInterval(this.ppidWatchdog); + this.ppidWatchdog = null; + } + if (this.daemon) { + void this.daemon.stop('stop()'); + // Daemon.stop calls process.exit; nothing else to do. + return; + } + if (this.session) { + this.session.stop(); + this.session = null; } + if (this.engine) { + this.engine.stop(); + this.engine = null; + } + process.exit(0); } - /** - * Start file watching on the active CodeGraph instance. - * Logs sync activity to stderr for diagnostics. - */ - private startWatching(): void { - if (!this.cg) return; - - const started = this.cg.watch({ - onSyncComplete: (result) => { - if (result.filesChanged > 0) { - process.stderr.write( - `[CodeGraph MCP] Auto-synced ${result.filesChanged} file(s) in ${result.durationMs}ms\n` - ); - } - }, - onSyncError: (err) => { - process.stderr.write(`[CodeGraph MCP] Auto-sync error: ${err.message}\n`); - }, + /** Single-process stdio MCP session — the pre-issue-#411 code path. */ + private async startDirect(reason: string): Promise { + if (reason && process.env.CODEGRAPH_MCP_DEBUG) { + process.stderr.write(`[CodeGraph MCP] Direct mode: ${reason}.\n`); + } + this.engine = new MCPEngine(); + const transport = new StdioTransport(); + this.session = new MCPSession(transport, this.engine, { + explicitProjectPath: this.projectPath, }); - if (started) { - process.stderr.write('[CodeGraph MCP] File watcher active — graph will auto-sync on changes\n'); + if (this.projectPath) { + // Background init so the initialize response stays fast (#172). + void this.engine.ensureInitialized(this.projectPath); } - } - /** - * Stop the server - */ - stop(): void { - // Close all cached cross-project connections first - this.toolHandler.closeAll(); - // Close the main CodeGraph instance - if (this.cg) { - this.cg.close(); - this.cg = null; - } - this.transport.stop(); - process.exit(0); + this.session.start(); + + // Detect parent-process death — same logic as pre-refactor. When stdin + // closes we go through StdioTransport's `process.exit(0)` already, but + // SIGKILL of the parent doesn't reliably close stdin on Linux (#277). + process.stdin.on('end', () => this.stop()); + process.stdin.on('close', () => this.stop()); + + this.mode = 'direct'; + this.installSignalHandlers(); + this.installPpidWatchdog(); } /** - * Handle incoming JSON-RPC messages + * Run as the detached shared daemon (process spawned with + * `CODEGRAPH_DAEMON_INTERNAL=1`). Arbitrate the O_EXCL lock, then either + * become the daemon (bind the socket, serve forever) or — if a live daemon + * already holds the lock — exit so we don't leak a redundant process. + * + * No PPID watchdog and no stdin handlers: the daemon is detached on purpose + * and reaps itself via client-refcount + idle timeout (see {@link Daemon}). */ - private async handleMessage(message: JsonRpcRequest | JsonRpcNotification): Promise { - // Check if it's a request (has id) or notification (no id) - const isRequest = 'id' in message; - - switch (message.method) { - case 'initialize': - if (isRequest) { - await this.handleInitialize(message as JsonRpcRequest); - } - break; - - case 'initialized': - // Notification that client has finished initialization - // No action needed - the client is ready - break; - - case 'tools/list': - if (isRequest) { - await this.handleToolsList(message as JsonRpcRequest); - } - break; - - case 'tools/call': - if (isRequest) { - await this.handleToolsCall(message as JsonRpcRequest); - } - break; - - case 'ping': - if (isRequest) { - this.transport.sendResult((message as JsonRpcRequest).id, {}); - } - break; - - default: - if (isRequest) { - this.transport.sendError( - (message as JsonRpcRequest).id, - ErrorCodes.MethodNotFound, - `Method not found: ${message.method}` - ); - } + private async startDaemonProcess(): Promise { + const root = resolveDaemonRoot(this.projectPath) ?? this.projectPath ?? process.cwd(); + for (let attempt = 0; attempt < TAKEOVER_MAX_RETRIES; attempt++) { + const lock = tryAcquireDaemonLock(root); + + if (lock.kind === 'acquired') { + const daemon = new Daemon(root); + await daemon.start(); + this.daemon = daemon; + this.mode = 'daemon'; + return; // the net.Server keeps the process alive + } + + // Taken. If the holder is alive, another daemon already serves (or is + // binding) — we're redundant; exit cleanly so the launcher proxies to it. + const existing = lock.existing; + if (existing && existing.pid > 0 && isProcessAlive(existing.pid)) { + process.stderr.write( + `[CodeGraph daemon] Another daemon (pid ${existing.pid}) already holds the lock; exiting.\n` + ); + process.exit(0); + } + + // Holder is dead (or the record is unreadable) — clear it (pid-verified, + // so we never delete a live daemon's lock) and retry the acquire. + clearStaleDaemonLock(lock.pidPath, existing?.pid); + await sleep(TAKEOVER_RETRY_DELAY_MS); } + + process.stderr.write('[CodeGraph daemon] Could not acquire the daemon lock; exiting.\n'); + process.exit(0); } /** - * Handle initialize request + * Become a proxy to the shared daemon, spawning the daemon first if none is + * reachable. Returns 'proxy' once the proxied session has run to completion + * (the host disconnected), or 'fallback' if the caller should run in-process. */ - private async handleInitialize(request: JsonRpcRequest): Promise { - const params = request.params as { - rootUri?: string; - workspaceFolders?: Array<{ uri: string; name: string }>; - } | undefined; - - // Extract project path from rootUri or workspaceFolders - let projectPath = this.projectPath; - - if (params?.rootUri) { - projectPath = fileUriToPath(params.rootUri); - } else if (params?.workspaceFolders?.[0]?.uri) { - projectPath = fileUriToPath(params.workspaceFolders[0].uri); + private async connectOrSpawnDaemon(root: string): Promise<'proxy' | 'fallback'> { + const socketPath = getDaemonSocketPath(root); + + // Fast path: a daemon may already be listening. On success runProxy pipes + // stdio until the host disconnects, so a 'proxied' outcome means this + // process has finished its entire job. + let probe = await runProxy(socketPath); + if (probe.outcome === 'proxied') return 'proxy'; + if (probe.reason === 'version mismatch') return 'fallback'; + + // No reachable daemon — spawn one (detached) and wait for it to bind. + spawnDetachedDaemon(root); + + for (let attempt = 0; attempt < DAEMON_CONNECT_MAX_RETRIES; attempt++) { + await sleep(DAEMON_CONNECT_RETRY_DELAY_MS); + probe = await runProxy(socketPath); + if (probe.outcome === 'proxied') return 'proxy'; + if (probe.reason === 'version mismatch') return 'fallback'; } - // Fall back to current working directory if no path provided - if (!projectPath) { - projectPath = process.cwd(); - } - - // Try to initialize the default project (non-fatal if it fails) - await this.tryInitializeDefault(projectPath); - - // We accept the client's protocol version but respond with our supported version. - // The `instructions` field is surfaced by MCP clients in the agent's system - // prompt automatically — it's the right place for the universal tool-selection - // playbook, ahead of individual tool descriptions. - this.transport.sendResult(request.id, { - protocolVersion: PROTOCOL_VERSION, - capabilities: { - tools: {}, - }, - serverInfo: SERVER_INFO, - instructions: SERVER_INSTRUCTIONS, - }); + // Daemon never came up in time — run in-process so the user is never blocked. + return 'fallback'; } - /** - * Handle tools/list request - */ - private async handleToolsList(request: JsonRpcRequest): Promise { - this.retryInitIfNeeded(); - this.transport.sendResult(request.id, { - tools: this.toolHandler.getTools(), - }); + /** Standard SIGINT/SIGTERM handlers that route to our `stop()` (direct mode). */ + private installSignalHandlers(): void { + process.on('SIGINT', () => this.stop()); + process.on('SIGTERM', () => this.stop()); } /** - * Handle tools/call request + * PPID watchdog (#277) — direct mode only. Daemon mode is detached on purpose + * and reaps via idle timeout; proxy mode installs its own watchdog inside + * {@link runProxy}. So this only ever runs for an in-process direct session. */ - private async handleToolsCall(request: JsonRpcRequest): Promise { - const params = request.params as { - name: string; - arguments?: Record; - }; - - if (!params || !params.name) { - this.transport.sendError( - request.id, - ErrorCodes.InvalidParams, - 'Missing tool name' - ); - return; - } - - const toolName = params.name; - const toolArgs = params.arguments || {}; - - // Validate tool exists - const tool = tools.find(t => t.name === toolName); - if (!tool) { - this.transport.sendError( - request.id, - ErrorCodes.InvalidParams, - `Unknown tool: ${toolName}` - ); - return; - } - - // If the default project isn't initialized yet, retry in case it was - // initialized after the MCP server started (e.g. user ran codegraph init) - this.retryInitIfNeeded(); - - const result = await this.toolHandler.execute(toolName, toolArgs); - - this.transport.sendResult(request.id, result); + private installPpidWatchdog(): void { + if (this.mode !== 'direct') return; + const pollMs = parsePpidPollMs(process.env.CODEGRAPH_PPID_POLL_MS); + if (pollMs <= 0) return; + this.ppidWatchdog = setInterval(() => { + const current = process.ppid; + const ppidChanged = current !== this.originalPpid; + const hostGone = this.hostPpid !== null && !isProcessAlive(this.hostPpid); + if (ppidChanged || hostGone) { + const reason = ppidChanged + ? `ppid ${this.originalPpid} -> ${current}` + : `host pid ${this.hostPpid} exited`; + process.stderr.write( + `[CodeGraph MCP] Parent process exited (${reason}); shutting down.\n` + ); + this.stop(); + } + }, pollMs); + this.ppidWatchdog.unref(); } } +function sleep(ms: number): Promise { + // Deliberately NOT unref'd. During the daemon connect/takeover retry loop we + // may be between processes — no socket bound yet, no transport, no listener + // pinning the event loop. An unref'd timer would let Node drain the loop and + // exit silently before we get a chance to try again. + return new Promise((resolve) => { setTimeout(resolve, ms); }); +} + // Export for use in CLI export { StdioTransport } from './transport'; export { tools, ToolHandler } from './tools'; +// Surface a few daemon-mode bits for tests + diagnostics. +export { Daemon } from './daemon'; +export { CodeGraphPackageVersion } from './version'; diff --git a/src/mcp/proxy.ts b/src/mcp/proxy.ts new file mode 100644 index 000000000..938b135ef --- /dev/null +++ b/src/mcp/proxy.ts @@ -0,0 +1,246 @@ +/** + * MCP proxy mode — issue #411. + * + * The proxy is a near-transparent stdio↔socket pipe. Once it has verified + * the daemon's hello line (same major.minor.patch as ours), it does no + * protocol parsing of its own: every byte the MCP host writes to the proxy's + * stdin goes straight to the daemon socket, and every byte the daemon emits + * goes straight to the host's stdout. Server-initiated JSON-RPC requests + * (e.g. `roots/list`) flow through the same pipe transparently. + * + * Lifecycle expectations: + * - The proxy exits when *either* stream closes (host stdin closed → + * daemon socket end, or daemon-side socket close → host stdout end). + * - Closing the socket on the proxy side is what tells the daemon to + * decrement its connected-clients refcount. + * - On a parent-process death we can't detect via stdin close (e.g. SIGKILL + * of the MCP host), the proxy's PPID watchdog catches it — same logic + * the direct-mode server uses; see issue #277. + */ + +import * as fs from 'fs'; +import * as net from 'net'; +import { HOST_PPID_ENV } from '../extraction/wasm-runtime-flags'; +import { DaemonHello, MAX_HELLO_LINE_BYTES } from './daemon'; +import { CodeGraphPackageVersion } from './version'; + +/** Default poll cadence for the PPID watchdog (same as the direct server). */ +const DEFAULT_PPID_POLL_MS = 5000; + +export interface ProxyResult { + /** + * `proxied` — successfully attached to a same-version daemon and piped + * stdio. The proxy stays alive until either end closes. + * `fallback-needed` — the daemon rejected us (version mismatch / unreachable + * socket) and the caller should run the server in direct mode. + */ + outcome: 'proxied' | 'fallback-needed'; + reason?: string; +} + +/** + * Attempt to connect to the daemon at `socketPath` and pipe stdio through it. + * + * Returns a promise that resolves when either: + * - the connection succeeded and one of stdin/socket has now closed + * (after which the process should exit), or + * - the connection failed early enough that the caller can still fall + * back to direct mode. + * + * The `expectedVersion` param defaults to the package's own version — daemon + * and proxy MUST match exactly. Mismatch resolves with + * `outcome: 'fallback-needed'` so the caller can transparently start its own + * server. (We accept the cost of two concurrent servers in this case as the + * price of never silently running a stale daemon against newer client code.) + */ +export async function runProxy( + socketPath: string, + expectedVersion: string = CodeGraphPackageVersion, +): Promise { + // POSIX: refuse to connect to a stale socket file that points at no + // listening process. `fs.existsSync` is a cheap pre-check; a real + // ECONNREFUSED below catches the rare "exists but unbound" race. + if (process.platform !== 'win32' && !fs.existsSync(socketPath)) { + return { outcome: 'fallback-needed', reason: 'socket file missing' }; + } + + const socket = net.createConnection(socketPath); + socket.setEncoding('utf8'); + + const hello = await readHelloLine(socket).catch((err) => { + socket.destroy(); + return new Error(String(err)); + }); + if (hello instanceof Error) { + return { outcome: 'fallback-needed', reason: hello.message }; + } + + if (hello.codegraph !== expectedVersion) { + process.stderr.write( + `[CodeGraph MCP] Found a daemon on ${socketPath} but version (${hello.codegraph}) ` + + `differs from ours (${expectedVersion}); falling back to direct mode.\n` + ); + socket.destroy(); + return { outcome: 'fallback-needed', reason: 'version mismatch' }; + } + + process.stderr.write( + `[CodeGraph MCP] Attached to shared daemon on ${socketPath} (pid ${hello.pid}, v${hello.codegraph}).\n` + ); + + startPpidWatchdog(socket); + await pipeUntilClose(socket); + // Host disconnected (or the daemon went away). The proxy's only job is the + // pipe; exit now so we don't linger — process.stdin's 'data' listener would + // otherwise keep the event loop alive and leave a zombie launcher behind. + process.exit(0); +} + +/** + * Read one CRLF/LF-terminated JSON line from the socket, parse it as the + * daemon hello, and return it. Bounded to {@link MAX_HELLO_LINE_BYTES} so a + * malicious or broken peer can't OOM us. Times out at 3s — a healthy daemon + * sends hello immediately on accept. + */ +function readHelloLine(socket: net.Socket): Promise { + return new Promise((resolve, reject) => { + let buffer = ''; + const cleanup = () => { + socket.removeListener('data', onData); + socket.removeListener('error', onError); + socket.removeListener('close', onClose); + clearTimeout(timer); + }; + const onData = (chunk: string | Buffer) => { + buffer += typeof chunk === 'string' ? chunk : chunk.toString('utf8'); + const idx = buffer.indexOf('\n'); + if (idx === -1) { + if (buffer.length > MAX_HELLO_LINE_BYTES) { + cleanup(); + reject(new Error('daemon hello line exceeded size limit')); + } + return; + } + const line = buffer.slice(0, idx); + // Re-emit anything past the newline so the pipe-stage sees it. + const tail = buffer.slice(idx + 1); + cleanup(); + if (tail.length > 0) { + // Push back via unshift — Node's net.Socket supports it on readable streams. + socket.unshift(tail); + } + try { + const parsed = JSON.parse(line) as DaemonHello; + if (typeof parsed.codegraph !== 'string' || typeof parsed.pid !== 'number') { + reject(new Error('daemon hello missing required fields')); + return; + } + resolve(parsed); + } catch (err) { + reject(new Error(`daemon hello not JSON: ${err instanceof Error ? err.message : String(err)}`)); + } + }; + const onError = (err: Error) => { cleanup(); reject(err); }; + const onClose = () => { cleanup(); reject(new Error('daemon closed connection before hello')); }; + const timer = setTimeout(() => { + cleanup(); + reject(new Error('timed out waiting for daemon hello')); + }, 3000); + timer.unref?.(); + socket.on('data', onData); + socket.on('error', onError); + socket.on('close', onClose); + }); +} + +/** + * Pipe stdin → socket and socket → stdout. Resolves once either end closes + * so the process can exit. Note: we deliberately do NOT use + * `process.stdin.pipe(socket)` because pipe propagates 'end' onto the + * downstream, which would close the socket prematurely if stdin happens to + * end early — the MCP spec allows it to stay open across reconnects. + */ +function pipeUntilClose(socket: net.Socket): Promise { + return new Promise((resolve) => { + let resolved = false; + const done = () => { if (!resolved) { resolved = true; resolve(); } }; + + process.stdin.on('data', (chunk) => { + try { socket.write(chunk); } catch { /* socket may have errored — close path catches it */ } + }); + process.stdin.on('end', () => { + try { socket.end(); } catch { /* ignore */ } + done(); + }); + process.stdin.on('close', () => { + try { socket.destroy(); } catch { /* ignore */ } + done(); + }); + + socket.on('data', (chunk) => { + try { process.stdout.write(chunk); } catch { /* ignore */ } + }); + socket.on('end', () => done()); + socket.on('close', () => done()); + socket.on('error', (err) => { + process.stderr.write(`[CodeGraph MCP] daemon socket error: ${err.message}\n`); + done(); + }); + }); +} + +/** + * PPID watchdog mirroring the one in `MCPServer.start` — kills the proxy if + * the MCP host (or its proxy of a host, see HOST_PPID_ENV) goes away without + * closing stdin. Issue #277 documents why we can't rely on stdin EOF on + * Linux: the parent may be SIGKILL'd and reparenting doesn't close pipes. + * + * The proxy's "kill" is just a socket close + process.exit — no SQLite or + * watchers to clean up, so this is cheap. + */ +function startPpidWatchdog(socket: net.Socket): void { + const pollMs = parsePollMs(process.env.CODEGRAPH_PPID_POLL_MS); + if (pollMs <= 0) return; + const originalPpid = process.ppid; + const hostPpid = parseHostPpid(process.env[HOST_PPID_ENV]); + const timer = setInterval(() => { + const current = process.ppid; + const ppidChanged = current !== originalPpid; + const hostGone = hostPpid !== null && !isProcessAliveLocal(hostPpid); + if (ppidChanged || hostGone) { + const reason = ppidChanged + ? `ppid ${originalPpid} -> ${current}` + : `host pid ${hostPpid} exited`; + process.stderr.write(`[CodeGraph MCP] Parent process exited (${reason}); shutting down.\n`); + try { socket.destroy(); } catch { /* ignore */ } + process.exit(0); + } + }, pollMs); + timer.unref?.(); +} + +function parsePollMs(raw: string | undefined): number { + if (raw === undefined || raw === '') return DEFAULT_PPID_POLL_MS; + const parsed = Number(raw); + if (!Number.isFinite(parsed)) return DEFAULT_PPID_POLL_MS; + if (parsed < 0) return DEFAULT_PPID_POLL_MS; + return Math.floor(parsed); +} + +function parseHostPpid(raw: string | undefined): number | null { + if (raw === undefined || raw === '') return null; + const parsed = Number(raw); + if (!Number.isInteger(parsed) || parsed <= 1) return null; + return parsed; +} + +function isProcessAliveLocal(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (err: unknown) { + const e = err as NodeJS.ErrnoException; + if (e.code === 'EPERM') return true; + return false; + } +} diff --git a/src/mcp/server-instructions.ts b/src/mcp/server-instructions.ts index 0c715ea81..db9949a74 100644 --- a/src/mcp/server-instructions.ts +++ b/src/mcp/server-instructions.ts @@ -22,33 +22,49 @@ in the workspace. Reads are sub-millisecond; the index lags writes by about a second through the file watcher. Consult it BEFORE writing or editing code, not during. +## Answer directly — don't delegate exploration + +For "how does X work", architecture, trace, or where-is-X questions, +answer DIRECTLY using 2-3 codegraph calls: \`codegraph_context\` first, +then ONE \`codegraph_explore\` for the source of the symbols it surfaces. +Codegraph IS the pre-built search index — so delegating the lookup to a +separate file-reading sub-task/agent, or running your own grep + read +loop, repeats work codegraph already did and costs more for the same +answer. Reach for raw Read/Grep only to confirm a specific detail +codegraph didn't cover. A direct codegraph answer is typically a handful +of calls; a grep/read exploration is dozens. + ## Tool selection by intent - **"What is the symbol named X?"** → \`codegraph_search\` - **"What's the deal with this task / feature / area?"** → \`codegraph_context\` (PRIMARY — composes search + node + callers + callees in one call) +- **"How does X reach/become Y? / trace the flow / the path from X to Y"** → \`codegraph_trace\` (ONE call returns the whole call path, including dynamic-dispatch hops — callbacks, React re-render, JSX children — that grep can't follow) - **"What calls this?"** → \`codegraph_callers\` - **"What does this call?"** → \`codegraph_callees\` - **"What would changing this break?"** → \`codegraph_impact\` - **"Show me this symbol's source / signature / docstring."** → \`codegraph_node\` -- **"Survey an unfamiliar topic / pattern / module."** → \`codegraph_explore\` (heavier; deep dive) +- **"Show me several related symbols' source / survey an area."** → \`codegraph_explore\` (ONE capped call; prefer over many codegraph_node/Read) - **"What's in directory X?"** → \`codegraph_files\` - **"Is the index ready / what's its size?"** → \`codegraph_status\` ## Common chains +- **Flow / "how does X reach Y"**: \`codegraph_trace\` from→to FIRST — one call returns the entire path with dynamic-dispatch hops bridged. Then ONE \`codegraph_explore\` for the hop bodies if you need them. Do NOT reconstruct the path with \`codegraph_search\` + \`codegraph_callers\` — that's exactly what trace does in a single call. - **Onboarding**: \`codegraph_context\` first. If still unclear, \`codegraph_explore\` for breadth, then \`codegraph_node\` on specific symbols. - **Refactor planning**: \`codegraph_search\` → \`codegraph_callers\` → \`codegraph_impact\`. The blast-radius answer comes from impact, not from walking callers manually. - **Debugging a regression**: \`codegraph_callers\` of the suspected symbol; widen with \`codegraph_impact\` if an unexpected call appears. ## Anti-patterns +- **Trust codegraph's results — don't re-verify them with grep.** They come from a full AST parse; re-checking with grep is slower, less accurate, and wastes context. - **Don't grep first** when looking up a symbol by name — \`codegraph_search\` is faster and returns kind + location + signature. - **Don't chain \`codegraph_search\` + \`codegraph_node\`** when you just want context — \`codegraph_context\` is one round-trip. -- **Don't use \`codegraph_explore\` for narrow questions** — it's a multi-call deep dive, expensive in tokens. Save it for genuine "I'm new here" surveys. -- **Don't query the index immediately after editing a file** — the watcher needs ~500ms to debounce + sync. Wait for the next turn. +- **Don't loop \`codegraph_node\` over many symbols** — one \`codegraph_explore\` call returns them all grouped by file, while each separate call re-reads the whole context and costs far more. Use \`codegraph_node\` for a single symbol. +- **After editing, check the staleness banner.** When a tool response starts with "⚠️ Some files referenced below were edited since the last index sync…", the listed files are pending re-index — Read those specific files for accurate content. Every file NOT in that banner is fresh, so still trust codegraph. \`codegraph_status\` also lists pending files under "Pending sync". ## Limitations +- If a tool reports the project isn't initialized, \`.codegraph/\` doesn't exist yet — offer to run \`codegraph init -i\` to build the index. - Index lags file writes by ~1 second. - Cross-file resolution is best-effort name matching; ambiguous calls may return multiple candidates. - No live correctness validation — that's still the TypeScript compiler / test suite / linter's job. Codegraph supplements those with structural context they don't have. diff --git a/src/mcp/session.ts b/src/mcp/session.ts new file mode 100644 index 000000000..157dc17db --- /dev/null +++ b/src/mcp/session.ts @@ -0,0 +1,272 @@ +/** + * MCP per-connection session — speaks the JSON-RPC protocol (initialize, + * tools/list, tools/call) over a single {@link JsonRpcTransport}. It owns + * per-client state only (which protocol version the client asked for, whether + * it advertised `roots`, the one-shot roots/list latch); the heavyweight + * resources (CodeGraph, watcher, ToolHandler) live in the shared + * {@link MCPEngine} so daemon mode can collapse N inotify sets / DB handles + * to one. + * + * The state-machine itself mirrors what `MCPServer` used to do inline before + * issue #411 split it out — the same regression tests in + * `__tests__/mcp-initialize.test.ts` still drive this code path. + */ + +import * as path from 'path'; +import { JsonRpcRequest, JsonRpcNotification, JsonRpcTransport, ErrorCodes } from './transport'; +import { MCPEngine } from './engine'; +import { tools } from './tools'; +import { SERVER_INSTRUCTIONS } from './server-instructions'; +import { CodeGraphPackageVersion } from './version'; + +/** + * MCP Server Info — kept on the session because some clients log it. The + * version tracks the real package version (was a hard-coded '0.1.0'). + */ +const SERVER_INFO = { + name: 'codegraph', + version: CodeGraphPackageVersion, +}; + +/** MCP Protocol Version (latest the server claims). */ +const PROTOCOL_VERSION = '2024-11-05'; + +/** + * How long to wait for the client's `roots/list` response before giving up + * and falling back to the process cwd. + */ +const ROOTS_LIST_TIMEOUT_MS = 5000; + +/** + * Convert a file:// URI to a filesystem path. Handles URL encoding and + * Windows drive letter paths. + */ +function fileUriToPath(uri: string): string { + try { + const url = new URL(uri); + let filePath = decodeURIComponent(url.pathname); + if (process.platform === 'win32' && /^\/[a-zA-Z]:/.test(filePath)) { + filePath = filePath.slice(1); + } + return path.resolve(filePath); + } catch { + return uri.replace(/^file:\/\/\/?/, ''); + } +} + +/** First usable filesystem path from a `roots/list` result, or null. */ +function firstRootPath(result: unknown): string | null { + if (!result || typeof result !== 'object') return null; + const roots = (result as { roots?: unknown }).roots; + if (!Array.isArray(roots) || roots.length === 0) return null; + const first = roots[0] as { uri?: unknown }; + if (typeof first?.uri !== 'string') return null; + return fileUriToPath(first.uri); +} + +export interface MCPSessionOptions { + /** + * Explicit project path from the `--path` CLI flag. When set, the session + * will not bother asking the client for `roots/list` — we already know + * where the project lives. + */ + explicitProjectPath?: string | null; +} + +/** + * One MCP client's view of the server. Created fresh per stdio launch + * (direct mode) or per socket connection (daemon mode). + */ +export class MCPSession { + private clientSupportsRoots = false; + private rootsAttempted = false; + private resolvePromise: Promise | null = null; + private explicitProjectPath: string | null; + + constructor( + private transport: JsonRpcTransport, + private engine: MCPEngine, + opts: MCPSessionOptions = {}, + ) { + this.explicitProjectPath = opts.explicitProjectPath ?? null; + } + + /** + * Start handling messages from the transport. Returns immediately — the + * session lives for as long as the transport is open. + */ + start(): void { + this.transport.start(this.handleMessage.bind(this)); + } + + /** + * Tear down the session. Does NOT touch the engine (the engine may serve + * other sessions) or call `process.exit` (the daemon decides when to exit). + */ + stop(): void { + this.transport.stop(); + } + + /** Underlying transport — exposed for daemon-side close hooks. */ + getTransport(): JsonRpcTransport { + return this.transport; + } + + private async handleMessage(message: JsonRpcRequest | JsonRpcNotification): Promise { + const isRequest = 'id' in message; + switch (message.method) { + case 'initialize': + if (isRequest) await this.handleInitialize(message as JsonRpcRequest); + break; + case 'initialized': + // Notification that client has finished initialization — no action needed. + break; + case 'tools/list': + if (isRequest) await this.handleToolsList(message as JsonRpcRequest); + break; + case 'tools/call': + if (isRequest) await this.handleToolsCall(message as JsonRpcRequest); + break; + case 'ping': + if (isRequest) this.transport.sendResult((message as JsonRpcRequest).id, {}); + break; + default: + if (isRequest) { + this.transport.sendError( + (message as JsonRpcRequest).id, + ErrorCodes.MethodNotFound, + `Method not found: ${message.method}`, + ); + } + } + } + + private async handleInitialize(request: JsonRpcRequest): Promise { + const params = request.params as { + rootUri?: string; + workspaceFolders?: Array<{ uri: string; name: string }>; + capabilities?: { roots?: unknown }; + } | undefined; + + this.clientSupportsRoots = !!params?.capabilities?.roots; + + // Explicit project signal, strongest first: client-provided rootUri / + // workspaceFolders (LSP-style), else the --path the server was launched + // with. cwd is NOT used here — we defer it so a roots/list answer can + // win over it. See issue #196. + let explicitPath: string | null = null; + if (params?.rootUri) { + explicitPath = fileUriToPath(params.rootUri); + } else if (params?.workspaceFolders?.[0]?.uri) { + explicitPath = fileUriToPath(params.workspaceFolders[0].uri); + } else if (this.explicitProjectPath) { + explicitPath = this.explicitProjectPath; + } + + // Respond to the handshake BEFORE doing any heavy init — see issue #172. + this.transport.sendResult(request.id, { + protocolVersion: PROTOCOL_VERSION, + capabilities: { tools: {} }, + serverInfo: SERVER_INFO, + instructions: SERVER_INSTRUCTIONS, + }); + + if (explicitPath) { + // Kick off engine init in the background. If another session in the + // same daemon already opened the project, `ensureInitialized` is a + // ~free no-op — N concurrent clients pay exactly one open. + this.resolvePromise = this.engine.ensureInitialized(explicitPath); + } + } + + private async handleToolsList(request: JsonRpcRequest): Promise { + await this.retryInitIfNeeded(); + this.transport.sendResult(request.id, { + tools: this.engine.getToolHandler().getTools(), + }); + } + + private async handleToolsCall(request: JsonRpcRequest): Promise { + const params = request.params as { + name: string; + arguments?: Record; + }; + + if (!params || !params.name) { + this.transport.sendError(request.id, ErrorCodes.InvalidParams, 'Missing tool name'); + return; + } + + const toolName = params.name; + const toolArgs = params.arguments || {}; + + const tool = tools.find((t) => t.name === toolName); + if (!tool) { + this.transport.sendError( + request.id, + ErrorCodes.InvalidParams, + `Unknown tool: ${toolName}`, + ); + return; + } + + await this.retryInitIfNeeded(); + + const result = await this.engine.getToolHandler().execute(toolName, toolArgs); + this.transport.sendResult(request.id, result); + } + + /** + * Lazy default-project resolution. Three layers: + * 1. await the in-flight init kicked off from `handleInitialize` (if any); + * 2. if still uninitialized and we never asked the client for its roots, + * do so now (one-shot); fall back to cwd if the client lacks roots; + * 3. last-resort: re-walk from the best candidate — picks up projects + * that were `codegraph init`'d *after* the server started. + */ + private async retryInitIfNeeded(): Promise { + if (this.resolvePromise) { + try { await this.resolvePromise; } catch { /* fall through to retry */ } + this.resolvePromise = null; + } + + if (this.engine.hasDefaultCodeGraph()) return; + + const hint = this.explicitProjectPath ?? this.engine.getProjectPath(); + if (!hint && !this.rootsAttempted) { + this.rootsAttempted = true; + this.resolvePromise = this.clientSupportsRoots + ? this.initFromRoots() + : this.engine.ensureInitialized(process.cwd()); + try { await this.resolvePromise; } catch { /* fall through */ } + this.resolvePromise = null; + if (this.engine.hasDefaultCodeGraph()) return; + } + + // Last resort: walk from the best candidate (sync open). Picks up + // projects that appeared after the server started. + const candidate = hint ?? process.cwd(); + this.engine.retryInitializeSync(candidate); + } + + /** + * Ask the client for its workspace root via `roots/list` and open the + * first one. Falls back to `process.cwd()` on timeout or empty answer. + */ + private async initFromRoots(): Promise { + let target = process.cwd(); + try { + const result = await this.transport.request('roots/list', undefined, ROOTS_LIST_TIMEOUT_MS); + const rootPath = firstRootPath(result); + if (rootPath) { + target = rootPath; + } else { + process.stderr.write('[CodeGraph MCP] Client returned no workspace roots; falling back to process cwd.\n'); + } + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + process.stderr.write(`[CodeGraph MCP] roots/list request failed (${msg}); falling back to process cwd.\n`); + } + await this.engine.ensureInitialized(target); + } +} diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index e796cfc74..2e9c6f816 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -5,17 +5,75 @@ */ import CodeGraph, { findNearestCodeGraphRoot } from '../index'; +import { + detectWorktreeIndexMismatch, + worktreeMismatchWarning, + worktreeMismatchNotice, + type WorktreeIndexMismatch, +} from '../sync/worktree'; +import type { PendingFile } from '../sync'; import type { Node, Edge, SearchResult, Subgraph, TaskContext, NodeKind } from '../types'; import { createHash } from 'crypto'; -import { writeFileSync, readFileSync, existsSync } from 'fs'; -import { clamp, validatePathWithinRoot } from '../utils'; +import { + constants as fsConstants, + closeSync, + existsSync, + lstatSync, + openSync, + readFileSync, + statSync, + writeSync, +} from 'fs'; +import { clamp, validatePathWithinRoot, validateProjectPath } from '../utils'; +import { isGeneratedFile } from '../extraction/generated-detection'; import { tmpdir } from 'os'; -import { join } from 'path'; -import { WASM_FALLBACK_FIX_RECIPE } from '../db'; +import * as pathModule from 'path'; +import { join, resolve as resolvePath } from 'path'; /** Maximum output length to prevent context bloat (characters) */ const MAX_OUTPUT_LENGTH = 15000; +/** + * Maximum length for free-form string inputs (query, task, symbol). + * Bounds memory and CPU when a buggy or hostile MCP client sends a + * huge payload — without this an attacker could ship a 100MB string + * and force a full FTS5 scan / OOM the server. 10 000 characters is + * far beyond any realistic legitimate query. + */ +const MAX_INPUT_LENGTH = 10_000; + +/** + * Maximum length for path-like string inputs (projectPath, path + * filter, glob pattern). Paths beyond a few thousand chars are + * never legitimate and signal abuse or a bug upstream. + */ +const MAX_PATH_LENGTH = 4_096; + +/** + * Rust path roots that have no file-system equivalent — `crate` is the + * current crate, `super` is the parent module, `self` is the current + * module. Used by `matchesSymbol` to strip these before file-path + * matching so `crate::configurator::stage_apply::run` resolves the + * same as `configurator::stage_apply::run`. + */ +const RUST_PATH_PREFIXES = new Set(['crate', 'super', 'self']); + +/** + * Node kinds that contain other symbols. For these, `codegraph_node` with + * `includeCode=true` returns a structural outline (member names + signatures + * + line numbers) instead of the full body, which for a large class is a + * multi-thousand-character wall of source that bloats the agent's context. + */ +const CONTAINER_NODE_KINDS = new Set([ + 'class', 'struct', 'interface', 'trait', 'protocol', 'enum', 'namespace', 'module', +]); + +/** Last `::` / `.` / `/`-separated segment of a qualified symbol. */ +function lastQualifierPart(symbol: string): string { + const parts = symbol.split(/::|[./]/).filter((p) => p.length > 0); + return parts[parts.length - 1] ?? symbol; +} + /** * Calculate the recommended number of codegraph_explore calls based on project size. * Larger codebases need more exploration calls to cover their surface area, @@ -29,20 +87,279 @@ export function getExploreBudget(fileCount: number): number { return 5; } +/** + * Adaptive output budget for `codegraph_explore`, scaled to project size. + * + * Smaller codebases get a tighter total cap, fewer default files, smaller + * per-file cap, and tighter clustering — so a focused query on a 100-file + * project doesn't dump a whole file's worth of source into the agent's + * context. Larger codebases keep the generous defaults because the + * agent's native discovery cost (grep + find + many Reads) genuinely + * dwarfs a fat explore call at that scale. + * + * Meta-text (relationships map, "additional relevant files" list, + * completeness signal, budget note) is gated off for tiny projects + * where one rich call is the whole story and the extra prose is just + * overhead. + * + * Tier breakpoints mirror `getExploreBudget` so a project sits in the + * same tier across both knobs. + */ +export interface ExploreOutputBudget { + /** Hard cap on total output characters. */ + maxOutputChars: number; + /** Default `maxFiles` when the caller didn't specify one. */ + defaultMaxFiles: number; + /** Cap on contiguous source returned per file (across all its clusters). */ + maxCharsPerFile: number; + /** Cluster gap threshold in lines — tighter clustering on small projects. */ + gapThreshold: number; + /** Max symbols listed in the per-file header (`#### path — sym(kind), ...`). */ + maxSymbolsInFileHeader: number; + /** Max edges shown per relationship kind in the Relationships section. */ + maxEdgesPerRelationshipKind: number; + /** Include the "Relationships" section. */ + includeRelationships: boolean; + /** Include the "Additional relevant files (not shown)" trailing list. */ + includeAdditionalFiles: boolean; + /** Include the "Complete source code is included above…" reminder. */ + includeCompletenessSignal: boolean; + /** Include the explore-budget reminder at the end. */ + includeBudgetNote: boolean; + /** + * Hard-drop test/spec/icon/i18n files from the relevant-file set unless + * the query itself mentions tests. Today they're only deprioritized in + * the sort, which on tiny repos still lets one slip into the top N (e.g. + * cobra's `command_test.go` displaced `args.go` and contributed ~10KB of + * pure noise to "How does cobra parse commands?"). Off by default; on + * for the very-tiny tier where one slip dominates the budget. + */ + excludeLowValueFiles: boolean; +} + +export function getExploreOutputBudget(fileCount: number): ExploreOutputBudget { + if (fileCount < 150) { + return { + // ITER3: revert iter2's aggressive body shrink (forced Read fallback — + // the per-file 2.5K cap pushed the agent to Read instead of node). + // Back to the iter1 shape (13K/4/3.8K) but keep the test-file + // hard-exclude. The cost lever for this tier lives in handleContext + // (steering the agent to stop after 1-2 calls), not in this budget. + maxOutputChars: 13000, + defaultMaxFiles: 4, + maxCharsPerFile: 3800, + gapThreshold: 7, + maxSymbolsInFileHeader: 5, + maxEdgesPerRelationshipKind: 4, + includeRelationships: false, + includeAdditionalFiles: false, + includeCompletenessSignal: false, + includeBudgetNote: false, + excludeLowValueFiles: true, + }; + } + if (fileCount < 500) { + return { + // ITER3: same revert/keep-filter pattern as <150. + maxOutputChars: 18000, + defaultMaxFiles: 5, + maxCharsPerFile: 3800, + gapThreshold: 8, + maxSymbolsInFileHeader: 6, + maxEdgesPerRelationshipKind: 6, + includeRelationships: false, + includeAdditionalFiles: false, + includeCompletenessSignal: false, + includeBudgetNote: false, + excludeLowValueFiles: true, + }; + } + if (fileCount < 5000) { + return { + // Sized so ONE explore can cover a flow that centers on a god-file (e.g. + // excalidraw's 415 KB App.tsx): the previous 2500/file returned <1% of such + // a file, forcing the agent to Read it anyway. Per-file must also stay ≥ the + // smaller <500 tier (3800) — the old 2500 was non-monotonic. Tokens are + // cheap relative to a 5–10 Read round-trip spiral; favor sufficiency. + maxOutputChars: 28000, + defaultMaxFiles: 10, + maxCharsPerFile: 6500, + gapThreshold: 12, + maxSymbolsInFileHeader: 10, + maxEdgesPerRelationshipKind: 10, + includeRelationships: true, + includeAdditionalFiles: true, + includeCompletenessSignal: true, + includeBudgetNote: true, + excludeLowValueFiles: false, + }; + } + if (fileCount < 15000) { + return { + maxOutputChars: 35000, + defaultMaxFiles: 12, + maxCharsPerFile: 7000, + gapThreshold: 15, + maxSymbolsInFileHeader: 15, + maxEdgesPerRelationshipKind: 15, + includeRelationships: true, + includeAdditionalFiles: true, + includeCompletenessSignal: true, + includeBudgetNote: true, + excludeLowValueFiles: false, + }; + } + return { + maxOutputChars: 38000, + defaultMaxFiles: 14, + maxCharsPerFile: 7000, + gapThreshold: 15, + maxSymbolsInFileHeader: 15, + maxEdgesPerRelationshipKind: 15, + includeRelationships: true, + includeAdditionalFiles: true, + includeCompletenessSignal: true, + includeBudgetNote: true, + excludeLowValueFiles: false, + }; +} + +/** + * Whether `codegraph_explore` should prefix source lines with their line + * numbers (cat -n style: `\t`). + * + * Line numbers let the agent cite `file:line` straight from the explore + * payload instead of re-Reading the file just to find a line number — the + * dominant residual cost on precise-tracing questions (#185 follow-up). + * + * Defaults ON. Set `CODEGRAPH_EXPLORE_LINENUMS=0` to disable (used by the + * A/B harness to measure the payload-cost vs. read-savings tradeoff). + */ +function exploreLineNumbersEnabled(): boolean { + return process.env.CODEGRAPH_EXPLORE_LINENUMS !== '0'; +} + +/** + * Adaptive explore sizing (default ON). `codegraph_explore` skeletonizes OFF-SPINE + * polymorphic-sibling files — a file whose class is one of ≥3 interchangeable + * implementations of a shared interface (e.g. OkHttp's `: Interceptor` classes) — + * to class + member signatures (bodies elided), keeping the on-spine exemplar full. + * This sizes the response to the answer instead of the budget cap on sibling-heavy + * flows (OkHttp interceptor-chain explore 28.5k→16.6k, ~28% cheaper than native + * search, reads flat). It is PROVABLY INERT elsewhere: distinct pipeline steps (no + * ≥3-implementer supertype, e.g. Excalidraw's `renderStaticScene`) and on-spine + * files keep full source — output is byte-identical to shipped on excalidraw / + * tokio / django / vscode / gin. Set `CODEGRAPH_ADAPTIVE_EXPLORE=0` to disable. + */ +function adaptiveExploreEnabled(): boolean { + return process.env.CODEGRAPH_ADAPTIVE_EXPLORE !== '0' && process.env.CODEGRAPH_ADAPTIVE_EXPLORE !== 'false'; +} + +/** + * Prefix each line of a source slice with its 1-based line number, matching + * the Read tool's `cat -n` convention (number + tab) so the agent treats it + * the same way it treats Read output. + * + * @param slice contiguous source text (already extracted from the file) + * @param firstLineNumber the 1-based line number of the slice's first line + */ +function numberSourceLines(slice: string, firstLineNumber: number): string { + const out: string[] = []; + const split = slice.split('\n'); + for (let i = 0; i < split.length; i++) { + out.push(`${firstLineNumber + i}\t${split[i]}`); + } + return out.join('\n'); +} + /** * Mark a Claude session as having consulted MCP tools. * This enables Grep/Glob/Bash commands that would otherwise be blocked. + * + * Why the explicit openSync + O_NOFOLLOW dance instead of plain writeFileSync: + * tmpdir() is world-writable on Linux (mode 1777), so on a shared multi-user + * machine any other local user can pre-create `codegraph-consulted-` as + * a symlink pointing at a file the victim owns. The old `writeFileSync` would + * happily follow that link and overwrite the target's contents with the ISO + * timestamp string (CWE-59). The session-id hash provides the predictability + * gate, but it's defense-in-depth: if a session id ever surfaces in logs, + * argv, or telemetry the attack becomes trivial, and the right fix is to not + * follow links from /tmp paths in the first place. */ function markSessionConsulted(sessionId: string): void { try { const hash = createHash('md5').update(sessionId).digest('hex').slice(0, 16); const markerPath = join(tmpdir(), `codegraph-consulted-${hash}`); - writeFileSync(markerPath, new Date().toISOString(), 'utf8'); + // Refuse to follow a pre-planted symlink at the marker path (CWE-59). + // O_NOFOLLOW (below) is the atomic, TOCTOU-free guard on POSIX, but it is + // `undefined` on Windows (libuv ignores it), so the bitwise-OR silently + // drops it and openSync would follow the link. This lstat check closes that + // gap cross-platform; ENOENT (path is free) falls through to create it. + try { + if (lstatSync(markerPath).isSymbolicLink()) return; + } catch { + // No existing entry (or stat failed) — nothing to refuse; proceed. + } + // O_NOFOLLOW makes openSync throw ELOOP if markerPath is already a symlink. + // O_CREAT + O_TRUNC keep the original "create-or-overwrite" semantics, and + // mode 0o600 prevents readback by other local users (the marker payload is + // benign, but narrowing the exposure costs nothing). + const flags = fsConstants.O_WRONLY | fsConstants.O_CREAT | fsConstants.O_TRUNC | fsConstants.O_NOFOLLOW; + const fd = openSync(markerPath, flags, 0o600); + try { + writeSync(fd, new Date().toISOString()); + } finally { + closeSync(fd); + } } catch { - // Silently fail - don't break MCP on marker write failure + // Silently fail - don't break MCP on marker write failure. ELOOP from a + // planted symlink lands here too, which is the intended behavior: refuse + // to write rather than overwrite an attacker-chosen target. } } +/** + * Per-file staleness banner emitted at the top of a tool response when the + * file watcher has pending events for files referenced by the response. + * The agent uses this to fall back to Read for those specific files + * without waiting for the debounced sync (issue #403). + */ +export function formatStaleBanner(stale: PendingFile[]): string { + const now = Date.now(); + const lines = stale.map((p) => { + const ageMs = Math.max(0, now - p.lastSeenMs); + const label = p.indexing ? 'indexing in progress' : 'pending sync'; + return ` - ${p.path} (edited ${ageMs}ms ago, ${label})`; + }); + return ( + '⚠️ Some files referenced below were edited since the last index sync — ' + + 'their codegraph entries may be stale:\n' + + lines.join('\n') + + '\nFor accurate content of those specific files, Read them directly. ' + + 'The rest of this response is fresh.' + ); +} + +/** + * Compact footer listing pending files that are NOT referenced in this + * response. Gives the agent a complete project-wide freshness picture + * without bloating the main banner. + */ +export function formatStaleFooter(stale: PendingFile[]): string { + const MAX = 5; + const now = Date.now(); + const shown = stale.slice(0, MAX); + const lines = shown.map((p) => { + const ageMs = Math.max(0, now - p.lastSeenMs); + return ` - ${p.path} (edited ${ageMs}ms ago)`; + }); + const more = stale.length > MAX ? `\n - …and ${stale.length - MAX} more` : ''; + return ( + `(Note: ${stale.length} file(s) elsewhere in this project are pending index ` + + `sync but were not referenced above:\n${lines.join('\n')}${more})` + ); +} + /** * MCP Tool definition */ @@ -118,7 +435,7 @@ export const tools: ToolDefinition[] = [ }, { name: 'codegraph_context', - description: 'PRIMARY TOOL: Build comprehensive context for a task. Returns entry points, related symbols, and key code - often enough to understand the codebase without additional tool calls. NOTE: This provides CODE context, not product requirements. For new features, still clarify UX/behavior questions with the user before implementing.', + description: 'PRIMARY TOOL — call FIRST for any "how does X work"/architecture/bug question. Returns entry points + related symbols + key code in one call; usually answers without further search/Read/Grep. Provides CODE context, not product requirements.', inputSchema: { type: 'object', properties: { @@ -143,7 +460,7 @@ export const tools: ToolDefinition[] = [ }, { name: 'codegraph_callers', - description: 'Find all functions/methods that call a specific symbol. Useful for understanding usage patterns and impact of changes.', + description: 'List functions that call . For deep flow use codegraph_trace.', inputSchema: { type: 'object', properties: { @@ -163,7 +480,7 @@ export const tools: ToolDefinition[] = [ }, { name: 'codegraph_callees', - description: 'Find all functions/methods that a specific symbol calls. Useful for understanding dependencies and code flow.', + description: 'List functions that calls. For deep flow use codegraph_trace.', inputSchema: { type: 'object', properties: { @@ -183,7 +500,7 @@ export const tools: ToolDefinition[] = [ }, { name: 'codegraph_impact', - description: 'Analyze the impact radius of changing a symbol. Shows what code could be affected by modifications.', + description: 'List symbols affected by changing . Use before a refactor.', inputSchema: { type: 'object', properties: { @@ -203,7 +520,7 @@ export const tools: ToolDefinition[] = [ }, { name: 'codegraph_node', - description: 'Get detailed information about a specific code symbol. Use includeCode=true only when you need the full source code - otherwise just get location and signature to minimize context usage.', + description: 'One symbol\'s location, signature, callers/callees trail. includeCode=true returns the verbatim body. Use codegraph_trace for full paths instead of chaining nodes.', inputSchema: { type: 'object', properties: { @@ -223,7 +540,7 @@ export const tools: ToolDefinition[] = [ }, { name: 'codegraph_explore', - description: 'Deep exploration tool — returns comprehensive context for a topic in a SINGLE call. Groups all relevant source code by file (contiguous sections, not snippets), includes a relationship map, and uses deeper graph traversal. Designed to replace multiple codegraph_node + file Read calls. Use this instead of codegraph_context when you need thorough understanding. IMPORTANT: Use specific symbol names, file names, or short code terms in your query — NOT natural language sentences. Before calling this, use codegraph_search to discover relevant symbol names, then include those names in your query. Bad: "how are agent prompts loaded and passed to the CLI". Good: "readAgentsFromDirectory createClaudeSession chat-manager agents.ts".', + description: 'Source of SEVERAL related symbols grouped by file, in one capped call. Query is a bag of symbol/file names (not a question). Returned source is verbatim Read-equivalent — do not re-open shown files. Prefer over chained codegraph_node.', inputSchema: { type: 'object', properties: { @@ -243,7 +560,7 @@ export const tools: ToolDefinition[] = [ }, { name: 'codegraph_status', - description: 'Get the status of the CodeGraph index, including statistics about indexed files, nodes, and edges.', + description: 'Index health check (files / nodes / edges). Skip unless debugging.', inputSchema: { type: 'object', properties: { @@ -253,7 +570,7 @@ export const tools: ToolDefinition[] = [ }, { name: 'codegraph_files', - description: 'REQUIRED for file/folder exploration. Get the project file structure from the CodeGraph index. Returns a tree view of all indexed files with metadata (language, symbol count). Much faster than Glob/filesystem scanning. Use this FIRST when exploring project structure, finding files, or understanding codebase organization.', + description: 'Indexed file tree with language + symbol counts. Faster than Glob for project layout.', inputSchema: { type: 'object', properties: { @@ -284,6 +601,25 @@ export const tools: ToolDefinition[] = [ }, }, }, + { + name: 'codegraph_trace', + description: 'Call path between two symbols — "how does reach ?" Returns the chain with each hop\'s body inlined plus the destination\'s callees, in ONE call. Ideal for flow questions (update→render, request→handler, QuerySet→SQL). If no static path exists the chain broke at dynamic dispatch — the failure response inlines both endpoints + their TO-file siblings.', + inputSchema: { + type: 'object', + properties: { + from: { + type: 'string', + description: 'Symbol the flow starts at (e.g., "QuerySet", "handleRequest", "mutateElement")', + }, + to: { + type: 'string', + description: 'Symbol the flow should reach (e.g., "execute_sql", "render", "setState")', + }, + projectPath: projectPathProperty, + }, + required: ['from', 'to'], + }, + }, ]; /** @@ -295,6 +631,23 @@ export const tools: ToolDefinition[] = [ export class ToolHandler { // Cache of opened CodeGraph instances for cross-project queries private projectCache: Map = new Map(); + // The directory the server last searched for a default project. Surfaced in + // the "not initialized" error so users can see why detection missed. + private defaultProjectHint: string | null = null; + // Per-start-path cache of the git worktree/index mismatch (issue #155). The + // mismatch is a fixed property of (where the request came from → which + // .codegraph/ it resolves to), so the up-to-two `git rev-parse` spawns run + // once and every later tool call reuses the result — never shelling out to + // git on the hot path. `undefined` = not computed yet; `null` = no mismatch. + private worktreeMismatchCache: Map = new Map(); + // Gate that the MCP engine pokes after `cg.open()` so the first tool call + // blocks on the post-open filesystem reconcile (catch-up sync). Without + // this, a tool call that races past `catchUpSync()` serves rows for files + // that were deleted (or edited) while no MCP server was running — and the + // per-file staleness banner can't help, because `getPendingFiles()` is + // populated by the watcher, not by catch-up. Cleared on first await so + // subsequent calls don't pay any cost. + private catchUpGate: Promise | null = null; constructor(private cg: CodeGraph | null) {} @@ -305,6 +658,25 @@ export class ToolHandler { this.cg = cg; } + /** + * Engine-only: register the catch-up sync promise so the next `execute()` + * call awaits it before serving. The handler swallows rejections (the + * engine logs them) so a sync failure never propagates as a tool error; + * we still want to serve a best-effort result over the same potentially- + * stale data, which is what would have happened without the gate. + */ + setCatchUpGate(p: Promise | null): void { + this.catchUpGate = p; + } + + /** + * Record the directory the server tried to resolve the default project from. + * Used only to make the "no default project" error actionable. + */ + setDefaultProjectHint(searchedPath: string): void { + this.defaultProjectHint = searchedPath; + } + /** * Whether a default CodeGraph instance is available */ @@ -312,19 +684,80 @@ export class ToolHandler { return this.cg !== null; } + /** + * Optional allowlist of exposed tools, parsed from the CODEGRAPH_MCP_TOOLS + * env var (comma-separated short names, e.g. "trace,search,node,context"). + * Unset/empty → every tool is exposed. Lets an operator (or an A/B harness) + * trim the tool surface without rebuilding the client config; the ablated + * tool is then truly absent from ListTools rather than merely denied on call. + * Matching is on the short form, so "trace" and "codegraph_trace" both work. + */ + private toolAllowlist(): Set | null { + const raw = process.env.CODEGRAPH_MCP_TOOLS; + if (!raw || !raw.trim()) return null; + const short = (s: string) => s.trim().replace(/^codegraph_/, ''); + const set = new Set(raw.split(',').map(short).filter(Boolean)); + return set.size ? set : null; + } + + /** Whether a tool name passes the CODEGRAPH_MCP_TOOLS allowlist (if any). */ + private isToolAllowed(name: string): boolean { + const allow = this.toolAllowlist(); + return !allow || allow.has(name.replace(/^codegraph_/, '')); + } + /** * Get tool definitions with dynamic descriptions based on project size. * The codegraph_explore tool description includes a budget recommendation - * scaled to the number of indexed files. + * scaled to the number of indexed files. Honors the CODEGRAPH_MCP_TOOLS + * allowlist so a trimmed surface is reflected in ListTools. */ getTools(): ToolDefinition[] { - if (!this.cg) return tools; + const allow = this.toolAllowlist(); + let visible = allow + ? tools.filter(t => allow.has(t.name.replace(/^codegraph_/, ''))) + : tools; + if (!this.cg) return visible; try { const stats = this.cg.getStats(); const budget = getExploreBudget(stats.fileCount); - return tools.map(tool => { + // Tiny-repo tool gating: on projects under TINY_REPO_FILE_THRESHOLD + // files, only expose the 5 core tools (search, context, node, + // explore, trace). The 5 omitted tools (callers, callees, impact, + // status, files) reduce to one grep at this scale. + // + // n=2 audits ruled out cutting below 5 tools: + // - 3-tool gate (search + context + trace): cost regressed on + // cobra/ky/sinatra. The agent fell back to raw Reads to cover + // what codegraph_node + codegraph_explore would have answered. + // - 1-tool gate (search only): catastrophic regression — express + // went from -43% WIN to +107% LOSS. With only search, the agent + // can't navigate the call graph structurally and reads everything. + // + // 5 is the empirical lower bound. Tools beyond search/context/ + // node/explore/trace pay overhead that the agent doesn't recoup + // on tiny-repo flow questions. + // ITER4: raise threshold 150 → 500 so single-file frameworks + // (sinatra at 159, slim_framework around 200) also get the + // 5-tool surface. The empirical 5-tool floor was set on <150 + // probes; iter3 measurement showed sinatra is structurally the + // SAME problem as cobra (single-file WITHOUT-arm Read wins), + // so it deserves the same gating. + const TINY_REPO_FILE_THRESHOLD = 500; + const TINY_REPO_CORE_TOOLS = new Set([ + 'codegraph_search', + 'codegraph_context', + 'codegraph_node', + 'codegraph_explore', + 'codegraph_trace', + ]); + if (stats.fileCount < TINY_REPO_FILE_THRESHOLD) { + visible = visible.filter(t => TINY_REPO_CORE_TOOLS.has(t.name)); + } + + return visible.map(tool => { if (tool.name === 'codegraph_explore') { return { ...tool, @@ -334,7 +767,7 @@ export class ToolHandler { return tool; }); } catch { - return tools; + return visible; } } @@ -350,7 +783,16 @@ export class ToolHandler { private getCodeGraph(projectPath?: string): CodeGraph { if (!projectPath) { if (!this.cg) { - throw new Error('CodeGraph not initialized for this project. Run \'codegraph init\' first.'); + const searched = this.defaultProjectHint ?? process.cwd(); + throw new Error( + 'No CodeGraph project is loaded for this session.\n' + + `Searched for a .codegraph/ directory starting from: ${searched}\n` + + 'The index is likely fine — this is a working-directory detection issue: ' + + "the MCP client launched the server outside your project and didn't report the " + + 'workspace root. Fix it either way:\n' + + ' • Pass projectPath to the tool call, e.g. projectPath: "/absolute/path/to/your/project"\n' + + ' • Or add --path to the server\'s MCP config args: ["serve", "--mcp", "--path", "/absolute/path/to/your/project"]' + ); } return this.cg; } @@ -360,6 +802,18 @@ export class ToolHandler { return this.projectCache.get(projectPath)!; } + // Reject sensitive system directories before opening. Only validate a + // path that actually exists — a nested or not-yet-created sub-path of a + // real project must still be allowed to resolve UP to its .codegraph/ + // root below (issue #238), so we don't run the existence-checking + // validator on paths that are meant to walk up. + if (existsSync(projectPath)) { + const pathError = validateProjectPath(projectPath); + if (pathError) { + throw new Error(pathError); + } + } + // Walk up parent directories to find nearest .codegraph/ const resolvedRoot = findNearestCodeGraphRoot(projectPath); @@ -367,6 +821,17 @@ export class ToolHandler { throw new Error(`CodeGraph not initialized in ${projectPath}. Run 'codegraph init' in that project first.`); } + // If the path resolves to the default project, reuse the already-open + // default instance rather than opening a SECOND connection to the same DB. + // A duplicate connection serializes reads against the watcher's auto-sync + // writes; on the wasm backend (no WAL) that surfaces as intermittent + // "database is locked" on concurrent tool calls. See issue #238. Deliberately + // not cached under projectPath — the server owns and closes the default + // instance, so routing it through projectCache.closeAll() would double-close it. + if (this.cg && this.cg.getProjectRoot() === resolvedRoot) { + return this.cg; + } + // Check if we already have this resolved root cached (different path, same project) if (this.projectCache.has(resolvedRoot)) { const cg = this.projectCache.get(resolvedRoot)!; @@ -392,45 +857,253 @@ export class ToolHandler { cg.close(); } this.projectCache.clear(); + this.worktreeMismatchCache.clear(); } /** - * Validate that a value is a non-empty string + * Validate that a value is a non-empty string within length bounds. + * + * The `maxLength` cap protects against MCP clients that ship huge + * payloads (10MB+ query strings either by accident or maliciously). + * Without this, a single oversized input can pin the FTS5 index or + * exhaust memory before any real work runs. */ - private validateString(value: unknown, name: string): string | ToolResult { + private validateString( + value: unknown, + name: string, + maxLength: number = MAX_INPUT_LENGTH + ): string | ToolResult { if (typeof value !== 'string' || value.length === 0) { return this.errorResult(`${name} must be a non-empty string`); } + if (value.length > maxLength) { + return this.errorResult( + `${name} exceeds maximum length of ${maxLength} characters (got ${value.length})` + ); + } + return value; + } + + /** + * Validate an optional path-like string input. Returns the value if + * valid (or undefined), or a ToolResult with the error. + */ + private validateOptionalPath( + value: unknown, + name: string + ): string | undefined | ToolResult { + if (value === undefined || value === null) return undefined; + if (typeof value !== 'string') { + return this.errorResult(`${name} must be a string`); + } + if (value.length > MAX_PATH_LENGTH) { + return this.errorResult( + `${name} exceeds maximum length of ${MAX_PATH_LENGTH} characters (got ${value.length})` + ); + } return value; } + /** + * Cached git worktree/index mismatch for a tool call's effective project. + * + * The "effective project" is what the request targets: an explicit + * `projectPath` arg, else the directory the server resolved its default + * project from (`defaultProjectHint`), else cwd. Memoized per start path — + * see `worktreeMismatchCache`. Best-effort: if the project can't be resolved + * (e.g. nothing initialized yet), it reports "no mismatch" so a tool is never + * broken by this check. + */ + private worktreeMismatchFor(projectPath?: string): WorktreeIndexMismatch | null { + const startPath = projectPath ?? this.defaultProjectHint ?? process.cwd(); + const cached = this.worktreeMismatchCache.get(startPath); + if (cached !== undefined) return cached; + + let mismatch: WorktreeIndexMismatch | null = null; + try { + mismatch = detectWorktreeIndexMismatch(startPath, this.getCodeGraph(projectPath).getProjectRoot()); + } catch { + // No resolvable project (or any other resolution error) → nothing to warn. + mismatch = null; + } + this.worktreeMismatchCache.set(startPath, mismatch); + return mismatch; + } + + /** + * Prefix a successful read-tool result with a compact worktree-mismatch + * notice when the resolved index belongs to a different git working tree than + * the caller's (issue #155). Without this, an agent in a nested worktree + * silently trusts main-branch results. No-op on error results and when there + * is no mismatch. `codegraph_status` is excluded — it embeds its own verbose + * warning — so it stays out of this path. + */ + private withWorktreeNotice(result: ToolResult, projectPath?: string): ToolResult { + if (result.isError) return result; + const mismatch = this.worktreeMismatchFor(projectPath); + if (!mismatch) return result; + + const notice = worktreeMismatchNotice(mismatch); + const [first, ...rest] = result.content; + if (first && first.type === 'text') { + return { ...result, content: [{ type: 'text', text: `${notice}\n\n${first.text}` }, ...rest] }; + } + return result; + } + + /** + * Annotate a successful read-tool result with per-file staleness — the + * non-blocking answer to issue #403. The file watcher tracks every event + * it sees per path; here we intersect "files referenced in this response" + * against that pending set and prepend a compact banner so the agent can + * fall back to Read for those *specific* files without waiting for the + * debounced sync to fire. Other pending files in the project (not + * referenced by this response) get a small footer so the agent has a + * complete picture without bloating the banner. + * + * Cost when nothing is pending — the common case — is one boolean check. + * No I/O, no parsing of markdown beyond a per-pending-file substring scan. + */ + private withStalenessNotice(result: ToolResult, projectPath?: string): ToolResult { + if (result.isError) return result; + + let cg: CodeGraph; + try { + cg = this.getCodeGraph(projectPath); + } catch { + return result; // no default project — leave as is + } + + // Cross-project `projectPath` calls open a cached CodeGraph WITHOUT a + // watcher (watchers are only attached to the default session project). + // When the cross-project path happens to be the same project as the + // default cg, the cached instance is the wrong one — its pendingFiles is + // permanently empty. Detect the equal-path case and prefer the default + // cg so the staleness signal still fires when an agent passes the + // explicit projectPath form of its own project. + if (this.cg && cg !== this.cg) { + try { + const sameProject = + resolvePath(this.cg.getProjectRoot()) === resolvePath(cg.getProjectRoot()); + if (sameProject) cg = this.cg; + } catch { + /* getProjectRoot may throw on a closed instance — leave cg as is */ + } + } + + // Defensive: some test fakes inject a partial CodeGraph stub without the + // newer pending-files API. Treat missing/throwing as "no pending files." + let pending: PendingFile[] = []; + try { + pending = cg.getPendingFiles?.() ?? []; + } catch { + return result; + } + if (pending.length === 0) return result; + + const [first, ...rest] = result.content; + if (!first || first.type !== 'text') return result; + + const text = first.text; + const inResponse: PendingFile[] = []; + const elsewhere: PendingFile[] = []; + for (const p of pending) { + // Substring match against the project-relative POSIX path — that's + // exactly the format both the watcher and every codegraph response + // emit, so a plain includes() is sufficient and avoids regex pitfalls. + if (text.includes(p.path)) inResponse.push(p); + else elsewhere.push(p); + } + + let banner = ''; + if (inResponse.length > 0) { + banner = formatStaleBanner(inResponse); + } + let footer = ''; + if (elsewhere.length > 0) { + footer = formatStaleFooter(elsewhere); + } + if (!banner && !footer) return result; + + const composed = [banner, text, footer].filter(Boolean).join('\n\n'); + return { ...result, content: [{ type: 'text', text: composed }, ...rest] }; + } + /** * Execute a tool by name */ async execute(toolName: string, args: Record): Promise { try { + // Block the first tool call on the engine's post-open reconcile so we + // never serve rows for files deleted/edited while no MCP server was + // running. The gate is cleared after first await — subsequent calls + // pay nothing. Catch-up failures are logged by the engine; we + // proceed regardless so a transient sync error never breaks tools. + if (this.catchUpGate) { + const gate = this.catchUpGate; + this.catchUpGate = null; + try { await gate; } catch { /* engine already logged */ } + } + // Honor the optional tool allowlist (CODEGRAPH_MCP_TOOLS): a trimmed + // surface rejects ablated tools defensively even if a client cached them. + if (!this.isToolAllowed(toolName)) { + return this.errorResult(`Tool ${toolName} is disabled via CODEGRAPH_MCP_TOOLS`); + } + // Cross-cutting input validation. All tools accept an optional + // `projectPath` and most accept either `query`, `task`, or + // `symbol` — bound their lengths centrally so individual handlers + // can stay focused on tool-specific logic. + const pathCheck = this.validateOptionalPath(args.projectPath, 'projectPath'); + if (typeof pathCheck === 'object' && pathCheck !== undefined) { + return pathCheck; + } + // The `path` and `pattern` properties used by codegraph_files are + // also path-shaped — apply the same cap. + if (args.path !== undefined) { + const check = this.validateOptionalPath(args.path, 'path'); + if (typeof check === 'object' && check !== undefined) return check; + } + if (args.pattern !== undefined) { + const check = this.validateOptionalPath(args.pattern, 'pattern'); + if (typeof check === 'object' && check !== undefined) return check; + } + + // Read tools resolve through a single result variable so cross-cutting + // notices — worktree-index mismatch (issue #155) and per-file + // staleness (issue #403) — can be applied in one place. status embeds + // its own verbose worktree warning but still flows through the + // staleness wrapper so its pending-files section stays consistent + // with what the read tools surface. + let result: ToolResult; switch (toolName) { case 'codegraph_search': - return await this.handleSearch(args); + result = await this.handleSearch(args); break; case 'codegraph_context': - return await this.handleContext(args); + result = await this.handleContext(args); break; case 'codegraph_callers': - return await this.handleCallers(args); + result = await this.handleCallers(args); break; case 'codegraph_callees': - return await this.handleCallees(args); + result = await this.handleCallees(args); break; case 'codegraph_impact': - return await this.handleImpact(args); + result = await this.handleImpact(args); break; case 'codegraph_explore': - return await this.handleExplore(args); + result = await this.handleExplore(args); break; case 'codegraph_node': - return await this.handleNode(args); + result = await this.handleNode(args); break; case 'codegraph_status': + // status embeds the pending-files list as a first-class section + // (see handleStatus), so we skip the auto-banner wrapper here to + // avoid duplicating the same info at the top of the response. return await this.handleStatus(args); case 'codegraph_files': - return await this.handleFiles(args); + result = await this.handleFiles(args); break; + case 'codegraph_trace': + result = await this.handleTrace(args); break; default: return this.errorResult(`Unknown tool: ${toolName}`); } + const withWorktree = this.withWorktreeNotice(result, args.projectPath as string | undefined); + return this.withStalenessNotice(withWorktree, args.projectPath as string | undefined); } catch (err) { return this.errorResult(`Tool execution failed: ${err instanceof Error ? err.message : String(err)}`); } @@ -457,7 +1130,16 @@ export class ToolHandler { return this.textResult(`No results found for "${query}"`); } - const formatted = this.formatSearchResults(results); + // Down-rank generated files within the FTS-returned set so a search + // for "Send" surfaces the hand-written keeper before .pb.go stubs + // that share the name. Stable: only reorders generated vs. not. + const ranked = [...results].sort((a, b) => { + const aGen = isGeneratedFile(a.node.filePath) ? 1 : 0; + const bGen = isGeneratedFile(b.node.filePath) ? 1 : 0; + return aGen - bGen; + }); + + const formatted = this.formatSearchResults(ranked); return this.textResult(this.truncateOutput(formatted)); } @@ -475,7 +1157,21 @@ export class ToolHandler { } const cg = this.getCodeGraph(args.projectPath as string | undefined); - const maxNodes = (args.maxNodes as number) || 20; + // On tiny repos (<150 files), trim maxNodes hard — the entire repo + // is grep-able in a turn so a 20-node context is wasted budget. + // 8 covers the typical 1-3 entry-point + their immediate neighbors + // without dragging in the rest of the small codebase. + let defaultMaxNodes = 20; + let isTinyRepo = false; + let isSmallRepo = false; + try { + const stats = cg.getStats(); + if (stats.fileCount < 150) { defaultMaxNodes = 8; isTinyRepo = true; } + else if (stats.fileCount < 500) { isSmallRepo = true; } + } catch { + // stats failure — fall back to the standard default + } + const maxNodes = (args.maxNodes as number) || defaultMaxNodes; const includeCode = args.includeCode !== false; const context = await cg.buildContext(task, { @@ -490,13 +1186,190 @@ export class ToolHandler { ? '\n\n⚠️ **Ask user:** UX preferences, edge cases, acceptance criteria' : ''; + // Auto-trace for flow queries: when the task is asking "how does X + // reach/flow/propagate from A to B", run the trace internally and + // append its body to the context response. Saves the agent the + // follow-up codegraph_trace call that was the #2 cost driver on + // multi-module flow questions (Q3 / etcd Q2 in the audit). + const flowTrace = await this.maybeInlineFlowTrace(task, cg); + + // Iter3 — sufficiency steering on small repos. + // + // Measured economics on tiny (<150) and small (<500) projects: every + // additional MCP tool call costs ~$0.02-0.05 in cache-write tokens + // (5K-15K per response at $3.75/1M). The agent reflexively follows + // codegraph_context with explore/node even when the context response + // is already sufficient — that pattern drove the cost gap that + // smaller bodies (iter2) failed to close (smaller bodies just shifted + // the agent to Read instead). Direct directive on small-repo + // responses: tell the agent the context call IS the comprehensive + // pass for a project of this size and that follow-ups should be + // narrow (trace from→to, node single-symbol) — not another broad + // explore that re-bundles the same content. + // ITER4: unified strong directive for both tiny (<150) and small + // (<500) tiers — measured iter3 result was that the soft <500 + // wording was IGNORED on sinatra (5 tool calls, +92% loss) while + // the strong <150 wording was followed on cobra/slim (3 calls, + // -21%/-22% wins). The single-file-framework problem (sinatra) + // is structurally the same as cobra's; both deserve the same + // sufficiency steering. + let smallRepoTail = ''; + let smallRepoRouteInline = ''; + if (isTinyRepo || isSmallRepo) { + // Iter12: backend-computed routing manifest for routing queries. + // Builds a URL → handler map directly from the graph (each route + // node has a `references` edge to its handler), then inlines the + // top handler file's source. The agent gets the canonical + // routing answer in one MCP call — no need to parse framework + // DSL or grep for handlers. + // + // Replaces iter10's raw route-file inline. The manifest is more + // information-dense (parsed URL→handler map vs raw config DSL) + // and we still inline the top handler file's source so the agent + // has the implementation bodies inline too. + const isRouteQuery = /\b(route|routes|routing|request|handler|endpoint|api|controller|middleware|dispatch|invok)/i.test(task); + if (isRouteQuery) { + try { + const manifest = cg.getRoutingManifest(40); + if (manifest) { + // 1) Compact URL→handler list (~30-60 lines, ~1-2KB). + const lines: string[] = [ + `\n\n## Routing manifest (${manifest.totalRoutes} routes, top handler file holds ${manifest.topHandlerFileCount})`, + '', + '| URL | Handler | Location |', + '|---|---|---|', + ]; + for (const e of manifest.entries) { + lines.push(`| \`${e.url}\` | \`${e.handler}\` | ${e.handlerFile}:${e.handlerLine} |`); + } + // 2) Inline the top handler file's source. + if (manifest.topHandlerFile && manifest.topHandlerFileCount >= 2) { + try { + const fullPath = pathModule.join(cg.getProjectRoot(), manifest.topHandlerFile); + const stat = statSync(fullPath); + if (stat.size > 0 && stat.size <= 16000) { + const source = readFileSync(fullPath, 'utf-8'); + const capped = source.length > 7000 ? source.slice(0, 7000) + '\n... (truncated)' : source; + const ext = (manifest.topHandlerFile.match(/\.([a-z]+)$/i)?.[1] || '').toLowerCase(); + const lang = + ext === 'rb' ? 'ruby' : ext === 'py' ? 'python' : + ext === 'go' ? 'go' : ext === 'rs' ? 'rust' : + ext === 'js' || ext === 'jsx' ? 'javascript' : + ext === 'ts' || ext === 'tsx' ? 'typescript' : + ext === 'java' ? 'java' : ext === 'kt' ? 'kotlin' : + ext === 'cs' ? 'csharp' : ext === 'php' ? 'php' : + ext === 'swift' ? 'swift' : ext === 'yml' || ext === 'yaml' ? 'yaml' : ''; + lines.push(''); + lines.push(`### Top handler file (\`${manifest.topHandlerFile}\` — ${manifest.topHandlerFileCount}/${manifest.totalRoutes} routes, full source inlined — do NOT Read)`); + lines.push(''); + lines.push('```' + lang); + lines.push(capped); + lines.push('```'); + } + } catch { /* file read failed, skip the source inline */ } + } + smallRepoRouteInline = lines.join('\n'); + } + } catch { + // Manifest build failed — drop silently + } + } + const sizeQualifier = isTinyRepo ? 'under 150' : 'under 500'; + const routingClause = smallRepoRouteInline + ? ' The URL→handler manifest and top handler file are also inlined above — answer routing questions from them.' + : ''; + smallRepoTail = `\n\n---\n> **This project is small** (${sizeQualifier} indexed files). The entry points and code above cover the relevant surface — **do NOT call codegraph_explore as a follow-up; its content will largely duplicate this response**. If you need a specific flow, call \`codegraph_trace from→to\`. If you need one specific symbol's body, call \`codegraph_node \`.${routingClause} Otherwise, answer from what is above.`; + } + // buildContext returns string when format is 'markdown' if (typeof context === 'string') { - return this.textResult(context + reminder); + return this.textResult(this.truncateOutput(context + flowTrace + reminder + smallRepoRouteInline + smallRepoTail)); } // If it returns TaskContext, format it - return this.textResult(this.formatTaskContext(context) + reminder); + return this.textResult(this.truncateOutput(this.formatTaskContext(context) + flowTrace + reminder + smallRepoRouteInline + smallRepoTail)); + } + + /** + * Detect a flow-style task ("how does X reach Y", "trace the path from A to B") + * and pre-run trace between the most likely endpoints, returning the trace + * body to splice into the context response. Returns '' for non-flow queries + * or when no plausible endpoint pair can be extracted. + * + * Conservative by design: only fires when the task has both a clear flow + * keyword AND at least two distinct PascalCase / camelCase identifiers. + * False positives waste a graph query; false negatives just fall back to + * the agent calling trace itself (existing path-proximity wiring handles + * disambiguation either way). + */ + private async maybeInlineFlowTrace(task: string, cg: CodeGraph): Promise { + const lower = task.toLowerCase(); + const FLOW_KEYWORDS = [ + 'trace ', + 'from ', + 'reach ', + 'flow ', + 'propagat', + 'how does ', + 'how do ', + ]; + if (!FLOW_KEYWORDS.some((k) => lower.includes(k))) return ''; + + // Extract candidate symbols — PascalCase or camelCase identifiers ≥3 chars. + // Filter out common non-symbol words and the flow keywords themselves. + const STOP_WORDS = new Set([ + 'how', 'does', 'the', 'and', 'from', 'through', 'reach', 'reaches', + 'flow', 'path', 'trace', 'cross', 'module', 'modules', 'where', + 'update', 'updates', 'updated', 'when', 'what', 'this', 'that', + ]); + const ids: string[] = []; + const seen = new Set(); + const re = /\b([A-Z][a-z]+(?:[A-Z][a-z]*)+|[a-z]+[A-Z][a-z]*(?:[A-Z][a-z]*)*)\b/g; + let m: RegExpExecArray | null; + while ((m = re.exec(task)) !== null) { + const sym = m[1]!; + if (sym.length < 3) continue; + const key = sym.toLowerCase(); + if (STOP_WORDS.has(key) || seen.has(key)) continue; + seen.add(key); + ids.push(sym); + } + + if (ids.length < 2) return ''; + + // The first two distinct symbols, in order of appearance, are the most + // likely from/to endpoints — "from X ... through to Y" naturally places + // them in that order in the prose. If the trace fails to connect, it + // still returns the inlined endpoint bodies (the trace-failure rewrite). + const fromSym = ids[0]!; + const toSym = ids[1]!; + + let traceResult: ToolResult; + try { + traceResult = await this.handleTrace({ + from: fromSym, + to: toSym, + projectPath: cg.getProjectRoot(), + } as Record); + } catch { + return ''; + } + // Extract the textual body. Defensive: handleTrace's contract is the + // standard tool-result shape used elsewhere in this file. + const body = traceResult.content + ?.map((c) => (c.type === 'text' ? c.text : '')) + .filter(Boolean) + .join('\n') + .trim(); + if (!body) return ''; + return [ + '', + '## Inline flow trace', + '', + `Auto-traced \`${fromSym}\` → \`${toSym}\` because the query looks like a flow question. No follow-up codegraph_trace is needed for this pair.`, + '', + body, + ].join('\n'); } /** @@ -641,8 +1514,511 @@ export class ToolHandler { return this.textResult(this.truncateOutput(formatted)); } - /** Maximum output for explore tool — sized to stay under MCP client token limits (~10k tokens) */ - private static readonly EXPLORE_MAX_OUTPUT = 35000; + /** + * Handle codegraph_trace — shortest CALL PATH between two symbols. + * + * Exposes GraphTraverser.findPath: the chain of functions from `from` to `to`, + * each hop annotated with file:line and the call-site line. This is the + * capability grep/Read structurally cannot provide. When no static path + * exists, the chain has almost certainly broken at dynamic dispatch + * (callbacks, descriptors, metaclasses) — we say so and surface the start + * symbol's outgoing calls so the agent bridges the one missing hop with + * codegraph_node rather than blindly reading. + */ + private async handleTrace(args: Record): Promise { + const from = this.validateString(args.from, 'from'); + if (typeof from !== 'string') return from; + const to = this.validateString(args.to, 'to'); + if (typeof to !== 'string') return to; + + const cg = this.getCodeGraph(args.projectPath as string | undefined); + const fromMatches = this.findAllSymbols(cg, from); + if (fromMatches.nodes.length === 0) return this.textResult(`Symbol "${from}" not found in the codebase`); + const toMatches = this.findAllSymbols(cg, to); + if (toMatches.nodes.length === 0) return this.textResult(`Symbol "${to}" not found in the codebase`); + + // Trace along call edges only — a true call path. Names can map to several + // nodes, so try a few from×to candidate pairs until a usable path turns up. + // + // MAX_HOPS guard: a BFS shortest path longer than this on a dense call graph + // is almost always a spurious wander through unrelated code (django's + // `_fetch_all → … → execute_sql` BFS detours through prefetch/filter), not + // the real execution flow — and a confident-but-wrong 15-hop trace is worse + // than none. Over-cap paths are rejected and reported as "no direct path" + // (which, on real code, means the flow breaks at dynamic dispatch). + const edgeKinds: Edge['kind'][] = ['calls']; + const MAX_HOPS = 7; + // Path-proximity pairing: in a multi-module repo a symbol name like + // `EndBlocker` exists in 20+ modules. FTS picks one almost arbitrarily; + // the WRONG pair (e.g. simapp's wrapper EndBlocker paired with gov's Tally) + // has no static path, falls through to the dynamic-dispatch failure branch, + // and surfaces unrelated bodies — exactly the cosmos-Q3 trace failure mode. + // Score every from×to combo by shared file-path prefix length; try the + // most-co-located pair first (e.g. `x/gov/abci.go::EndBlocker` × + // `x/gov/keeper/tally.go::Tally` share `x/gov/`). + // + // Consider the FULL candidate set, not just the FTS top-5: the right + // EndBlocker for a gov-module flow may rank 8th in FTS but share the + // entire `x/gov/` prefix with the destination. Path-proximity supersedes + // FTS for this disambiguation. Findpath trials are still capped by + // FINDPATH_PAIR_BUDGET below to bound graph traversal cost. + const sharedDirPrefixLen = (a: string, b: string): number => { + const aDir = a.replace(/[^/]+$/, ''); + const bDir = b.replace(/[^/]+$/, ''); + let i = 0; + while (i < aDir.length && i < bDir.length && aDir[i] === bDir[i]) i++; + return i; + }; + // Cosmos-Q3 surfaced a second-order failure: `enterprise/group/x/group/` + // SHARES MORE of its path with `enterprise/group/x/group/keeper/tally.go` + // (24 chars) than `x/gov/abci.go` shares with `x/gov/keeper/tally.go` + // (6 chars), so pure shared-prefix prefers the side-experiment module + // over the canonical one — even though the user's question is clearly + // about the main gov module. Penalize candidates living under prefixes + // that conventionally hold extensions / experiments / vendored code, so + // the canonical-path pair wins even when its shared prefix is short. + const isLessCanonicalPath = (p: string): boolean => + /^(enterprise|contrib|examples?|sample|playground|vendor|third[_-]?party|deprecated|legacy)\//i.test(p); + const LESS_CANONICAL_PENALTY = 100; // any canonical candidate beats any less-canonical one + const scorePair = (a: string, b: string): number => + sharedDirPrefixLen(a, b) + - (isLessCanonicalPath(a) ? LESS_CANONICAL_PENALTY : 0) + - (isLessCanonicalPath(b) ? LESS_CANONICAL_PENALTY : 0); + const fromCands = fromMatches.nodes; + const toCands = toMatches.nodes; + const pairs: Array<{ f: Node; t: Node; score: number }> = []; + for (const f of fromCands) { + for (const t of toCands) { + pairs.push({ f, t, score: scorePair(f.filePath, t.filePath) }); + } + } + // Sort by shared prefix desc, then by FTS order (already encoded in the + // pairs' insertion order — both for f and t). The tiebreaker preserves + // findAllSymbols' generated-file-last ranking. + pairs.sort((a, b) => b.score - a.score); + // Cap how many graph-path probes we attempt so a 50×50 cross-product + // doesn't blow up on a god-named symbol like `Get` (well-named flows have + // their good pair near the top of the sort anyway). + const FINDPATH_PAIR_BUDGET = 20; + const fromTry = fromCands; + const toTry = toCands; + let path: Array<{ node: Node; edge: Edge | null }> | null = null; + let overCap: Array<{ node: Node; edge: Edge | null }> | null = null; + let bestPair: { f: Node; t: Node } | null = null; + let triedPairs = 0; + for (const { f, t } of pairs) { + if (path) break; + if (triedPairs >= FINDPATH_PAIR_BUDGET) break; + triedPairs++; + const p = cg.findPath(f.id, t.id, edgeKinds); + if (p && p.length > 1) { + if (p.length <= MAX_HOPS) { path = p; bestPair = { f, t }; break; } + if (!overCap || p.length < overCap.length) { overCap = p; bestPair = { f, t }; } + } else if (!bestPair) { + // No path yet — remember the top-scored pair so the failure branch + // surfaces the most-co-located candidates' bodies, not whatever FTS + // happened to put first. + bestPair = { f, t }; + } + } + + if (!path) { + // No static path — almost always a dynamic-dispatch break. INSTEAD of + // telling the agent to chase the gap with codegraph_node/callers/callees + // (which fans out into 3-4 follow-up tool calls + a Read), inline the + // material those would have returned right here. Measured on cosmos-Q3: + // the failed-trace + subsequent fan-out used to cost ~2× a single + // sufficient trace call; this branch closes that gap. + // Prefer the path-proximity-best pair we identified above (e.g. gov's + // EndBlocker × gov's Tally) over the FTS top-pick (simapp's wrapper). + const start = bestPair?.f ?? fromTry[0]!; + const end = bestPair?.t ?? toTry[0]!; + const fileCache = new Map(); + const lines = [ + `No direct static call path from "${from}" to "${to}" — the chain almost certainly breaks at dynamic dispatch (a callback / interface dispatch / framework hook / metaclass). Both endpoint bodies + their immediate neighbors are inlined below; answer from them — a follow-up codegraph_node/callers/callees on these would just return what is already here.`, + '', + ]; + if (overCap) { + lines.push( + `> Indirect chain of ${overCap.length} hops exists but is over the ${MAX_HOPS}-hop cap (usually a BFS wander through unrelated code, not the real execution flow).`, + '', + ); + } + + // Track which node IDs we've already inlined a body for so we don't + // double-emit when a callee of FROM is also surfaced separately. + const inlinedBodies = new Set(); + const inlineBody = (n: Node, lineCap: number, charCap: number): boolean => { + if (inlinedBodies.has(n.id)) return false; + inlinedBodies.add(n.id); + const body = this.sourceRangeAt(cg, n.filePath, n.startLine, n.endLine, fileCache, lineCap, charCap); + if (body) { lines.push(body); return true; } + return false; + }; + + const inlineEndpoint = ( + label: 'FROM' | 'TO', + node: Node, + ) => { + lines.push(`### ${label}: \`${node.name}\` (${node.filePath}:${node.startLine}-${node.endLine})`); + inlineBody(node, 120, 3600); + const callers = cg.getCallers(node.id).slice(0, 6); + if (callers.length > 0) { + lines.push(`**Callers of \`${node.name}\`:** ` + + callers.map(c => `${c.node.name} (${c.node.filePath}:${c.node.startLine})`).join(', ')); + } + const callees = cg.getCallees(node.id).slice(0, 8); + if (callees.length > 0) { + lines.push(`**\`${node.name}\` calls:** ` + + callees.map(c => `${c.node.name} (${c.node.filePath}:${c.node.startLine})`).join(', ')); + } + lines.push(''); + }; + inlineEndpoint('FROM', start); + if (end.id !== start.id) inlineEndpoint('TO', end); + + // Inline the OTHER top-level functions/methods in TO's file — that's + // where the missing dynamic-dispatch flow usually lives. Concrete + // measurement from cosmos-Q1: `msgServer.Send` statically calls only + // utility functions (`StringToBytes`, `Wrapf`); its real next-hop + // `SendCoins` is invoked via an embedded-interface call (`k.Keeper.SendCoins`) + // that static parsing CAN'T see. The flow IS in the same file as the + // destination (`x/bank/keeper/send.go`: SendCoins → subUnlockedCoins → + // addCoins → setBalance). Pre-inlining those file-mates is what + // replaces the agent's "trace fail → search SendCoins → node SendCoins + // → trace again" fan-out. + const NEIGHBOR_LINES = 40; + const NEIGHBOR_CHARS = 1200; + const NEIGHBOR_K = 5; + const fileSiblings = (anchor: Node): Node[] => { + // Functions and methods in the same file as the anchor, excluding + // the anchor itself and anything we've already inlined. Sort by + // distance from the anchor's startLine so the closest symbols come + // first (the flow is usually adjacent in the file). + const sameFile = cg + .getNodesByKind('function') + .filter((n) => n.filePath === anchor.filePath) + .concat( + cg.getNodesByKind('method').filter((n) => n.filePath === anchor.filePath), + ); + return sameFile + .filter((n) => n.id !== anchor.id && !inlinedBodies.has(n.id)) + .sort((a, b) => + Math.abs(a.startLine - anchor.startLine) - Math.abs(b.startLine - anchor.startLine), + ) + .slice(0, NEIGHBOR_K); + }; + const renderSiblings = (label: string, siblings: Node[]) => { + if (siblings.length === 0) return; + lines.push(`### ${label}`); + for (const sib of siblings) { + lines.push(''); + lines.push(`- \`${sib.name}\` (${sib.filePath}:${sib.startLine}-${sib.endLine})`); + inlineBody(sib, NEIGHBOR_LINES, NEIGHBOR_CHARS); + } + lines.push(''); + }; + renderSiblings( + `Other functions in \`${end.filePath}\` (the flow that the dynamic-dispatch hop reaches — bodies inlined)`, + fileSiblings(end), + ); + + lines.push( + '> Endpoint bodies + the other functions in the destination\'s file are inlined above. Together they typically cover the missing dynamic-dispatch boundary (interface-method calls like `k.Keeper.SendCoins` that static parsing can\'t follow). **No further codegraph_node / codegraph_callers / codegraph_callees / Read / Grep is needed for any symbol already shown here** — call them again only if you need to walk DEEPER than what is inlined.', + ); + return this.textResult(this.truncateOutput(lines.join('\n') + fromMatches.note + toMatches.note)); + } + + const lines: string[] = [ + `## Trace: ${from} → ${to}`, + '', + `Full execution path below — ${path.length} hops, each with its body, plus what the destination calls. This is the complete flow; answer from it.`, + '', + `${path.length} hops:`, + '', + ]; + // Inline what each hop needs so the agent doesn't Read/Grep to get it: the + // call-site source line, the registration site for dynamic-dispatch hops, AND + // the hop's own body (capped per hop so the trace stays path-scoped). Earlier + // versions inlined only the call-site line, which left agents calling explore + // or Read for the bodies — the exact follow-up the ablation experiment measured. + const fileCache = new Map(); + for (let i = 0; i < path.length; i++) { + const step = path[i]!; + if (step.edge) { + const synth = this.synthEdgeNote(step.edge); + if (synth) { + lines.push(` ↓ ${synth.label}`); + if (synth.registeredAt) { + const regSrc = this.sourceLineAt(cg, synth.registeredAt, fileCache); + lines.push(` ↳ registered at ${synth.registeredAt}${regSrc ? ` ${regSrc}` : ''}`); + } + } else { + // The call happens in the PREVIOUS hop's file at edge.line. + const prev = path[i - 1]; + const ref = prev && step.edge.line ? `${prev.node.filePath}:${step.edge.line}` : undefined; + const callSrc = this.sourceLineAt(cg, ref, fileCache); + lines.push(` ↓ ${step.edge.kind}${step.edge.line ? `@${step.edge.line}` : ''}${callSrc ? ` ${callSrc}` : ''}`); + } + } + lines.push(`${i + 1}. ${step.node.name} (${step.node.filePath}:${step.node.startLine}-${step.node.endLine})`); + const body = this.sourceRangeAt(cg, step.node.filePath, step.node.startLine, step.node.endLine, fileCache, 60, 1800); + if (body) lines.push(body); + } + // The "last mile": what the destination does next. Agents otherwise explore/Read + // for exactly this (e.g. renderStaticScene → _renderStaticScene → the canvas draw), + // so inlining the destination's callees is what actually stops the investigation — + // sufficiency, not a "don't explore" instruction. + const dest = path[path.length - 1]!.node; + const destCallees = cg.getCallees(dest.id) + .filter(c => !path.some(p => p.node.id === c.node.id)) + .slice(0, 6); + if (destCallees.length > 0) { + lines.push('', `### \`${dest.name}\` then calls (the destination's immediate work):`); + for (const c of destCallees) { + lines.push('', `- ${c.node.name} (${c.node.filePath}:${c.node.startLine}-${c.node.endLine})`); + const body = this.sourceRangeAt(cg, c.node.filePath, c.node.startLine, c.node.endLine, fileCache, 16, 600); + if (body) lines.push(body); + } + } + lines.push('', '> Full path + every hop body + the destination\'s calls are inlined above — the complete flow. Answer from it; a Read is only needed to chase a specific local variable\'s data-flow.'); + return this.textResult(this.truncateOutput(lines.join('\n'))); + } + + /** + * Describe a synthesized (dynamic-dispatch) edge for human output: how the + * callback was wired up — the bridge static parsing can't see. Returns null + * for ordinary static edges. Used by trace + the node trail so a synthesized + * hop reads as "registered via onUpdate at App.tsx:3148", not a bare arrow. + */ + private synthEdgeNote(edge: Edge | null): { label: string; compact: string; registeredAt?: string } | null { + if (!edge || edge.provenance !== 'heuristic') return null; + const m = edge.metadata as Record | undefined; + const registeredAt = typeof m?.registeredAt === 'string' ? m.registeredAt : undefined; + const at = registeredAt ? ` @${registeredAt}` : ''; + if (m?.synthesizedBy === 'callback') { + const via = m.via ? `\`${String(m.via)}\`` : 'a registrar'; + const field = m.field ? ` on .${String(m.field)}` : ''; + return { + label: `callback — registered via ${via}${field} (dynamic dispatch)`, + compact: `dynamic: callback via ${via}${at}`, + registeredAt, + }; + } + if (m?.synthesizedBy === 'event-emitter') { + const ev = m.event ? `\`${String(m.event)}\`` : 'an event'; + return { + label: `event ${ev} — emit → handler (dynamic dispatch)`, + compact: `dynamic: event ${ev}${at}`, + registeredAt, + }; + } + if (m?.synthesizedBy === 'react-render') { + return { + label: `React re-render — \`setState\` re-runs render() (dynamic dispatch)`, + compact: `dynamic: React re-render via setState${at}`, + registeredAt, + }; + } + if (m?.synthesizedBy === 'jsx-render') { + const child = m.via ? `<${String(m.via)}>` : 'a child component'; + return { + label: `renders ${child} (JSX child — dynamic dispatch)`, + compact: `dynamic: renders ${child}`, + registeredAt, + }; + } + if (m?.synthesizedBy === 'vue-handler') { + const ev = m.event ? `@${String(m.event)}` : 'a template event'; + return { + label: `Vue template handler — bound to ${ev} (dynamic dispatch)`, + compact: `dynamic: Vue ${ev} handler`, + registeredAt, + }; + } + if (m?.synthesizedBy === 'interface-impl') { + return { + label: `interface/abstract dispatch — runs the implementation override (dynamic dispatch)`, + compact: `dynamic: interface → impl${at}`, + registeredAt, + }; + } + return null; + } + + /** + * Read one trimmed source line at "relpath:line" (relative to the project + * root). `cache` holds split file contents so a multi-hop trace reads each + * file at most once. Returns null if the file/line can't be resolved. + */ + private sourceLineAt(cg: CodeGraph, ref: string | undefined, cache: Map): string | null { + if (!ref) return null; + const i = ref.lastIndexOf(':'); + if (i < 0) return null; + const filePath = ref.slice(0, i); + const line = parseInt(ref.slice(i + 1), 10); + if (!Number.isFinite(line) || line < 1) return null; + let fileLines = cache.get(filePath); + if (!fileLines) { + const abs = validatePathWithinRoot(cg.getProjectRoot(), filePath); + if (!abs || !existsSync(abs)) return null; + try { fileLines = readFileSync(abs, 'utf-8').split('\n'); } catch { return null; } + cache.set(filePath, fileLines); + } + const raw = fileLines[line - 1]; + if (raw == null) return null; + const t = raw.trim(); + return t.length > 160 ? t.slice(0, 157) + '…' : t; + } + + /** + * Read a hop's body — filePath lines [startLine..endLine] — for inlining into + * a trace, capped (lines + chars) so the whole path stays path-scoped even on + * a 7-hop chain. Dedents to the body's own indentation and marks truncation. + * Shares `cache` with sourceLineAt so each file is read at most once per trace. + */ + private sourceRangeAt( + cg: CodeGraph, + filePath: string, + startLine: number, + endLine: number, + cache: Map, + maxLines = 28, + maxChars = 1200 + ): string | null { + if (!Number.isFinite(startLine) || startLine < 1) return null; + let fileLines = cache.get(filePath); + if (!fileLines) { + const abs = validatePathWithinRoot(cg.getProjectRoot(), filePath); + if (!abs || !existsSync(abs)) return null; + try { fileLines = readFileSync(abs, 'utf-8').split('\n'); } catch { return null; } + cache.set(filePath, fileLines); + } + const end = Number.isFinite(endLine) && endLine >= startLine ? endLine : startLine; + let slice = fileLines.slice(startLine - 1, end); + if (slice.length === 0) return null; + let omitted = 0; + if (slice.length > maxLines) { omitted = slice.length - maxLines; slice = slice.slice(0, maxLines); } + const nonBlank = slice.filter(l => l.trim().length > 0); + const dedent = nonBlank.length ? Math.min(...nonBlank.map(l => l.length - l.trimStart().length)) : 0; + let text = slice.map((l, i) => ` ${startLine + i}\t${l.slice(dedent)}`).join('\n'); + if (text.length > maxChars) { + text = text.slice(0, maxChars).replace(/\n[^\n]*$/, ''); + omitted = Math.max(omitted, 1); + } + if (omitted > 0) text += `\n … (+${omitted} more line${omitted === 1 ? '' : 's'})`; + return text; + } + + /** + * Flow-from-named-symbols: an agent's codegraph_explore query is a bag of + * symbol names that usually spans the flow it's investigating (e.g. + * "PmsProductController getList PmsProductService list PmsProductServiceImpl"). + * Surface the longest call chain AMONG those named symbols — scoped to what the + * agent explicitly named, so (unlike a fuzzy relevance set) there's no + * wrong-feature wandering. Rides synthesized edges, so controller→service- + * interface→impl shows up. Returns '' if no chain of >=3 nodes exists. + * + * Ambiguous tokens (Java `list` → dozens of nodes) are disambiguated by + * CO-NAMING: the agent names the class too, so we keep only `list` candidates + * whose qualifiedName contains another named token (`PmsProductServiceImpl::list`), + * dropping unrelated `OmsOrderService::list`. + */ + private buildFlowFromNamedSymbols(cg: CodeGraph, query: string): { text: string; pathNodeIds: Set; namedNodeIds: Set; uniqueNamedNodeIds: Set } { + const EMPTY = { text: '', pathNodeIds: new Set(), namedNodeIds: new Set(), uniqueNamedNodeIds: new Set() }; + try { + const CALLABLE = new Set(['method', 'function', 'component', 'constructor']); + // Strip only a REAL file extension (Create.cs → Create); KEEP qualified + // names (Class.method / Class::method) — the agent's most precise input, + // resolved exactly by findAllSymbols. (The old strip mangled Class.method + // into Class, throwing the method away.) + const FILE_EXT = /\.(?:java|kt|kts|ts|tsx|js|jsx|mjs|cjs|cs|py|go|rb|php|swift|rs|cpp|cc|cxx|c|h|hpp|scala|lua|dart|vue|svelte)$/i; + const tokens = [...new Set( + query.split(/[\s,()[\]]+/) + .map((t) => t.replace(FILE_EXT, '').trim()) + .filter((t) => t.length >= 3 && /^[A-Za-z_$][\w$]*(?:(?:::|\.)[\w$]+)*$/.test(t)) + )].slice(0, 16); + if (tokens.length < 2) return EMPTY; + // Pool of name SEGMENTS (Class + method from every token) used to + // disambiguate an ambiguous SIMPLE name: keep a candidate only if its + // CONTAINER class is itself named in the query. + const segPool = new Set(); + for (const t of tokens) for (const s of t.toLowerCase().split(/::|\./)) if (s) segPool.add(s); + const named = new Map(); + // Nodes whose token is SPECIFIC — a (near-)unique callable name (<=3 defs in + // the whole graph). These are safe to SPARE a file on: the agent named THIS + // method (`getResponseWithInterceptorChain`, 1 def). A hyper-polymorphic name + // (`as_sql`, 110 defs across every Expression/Compiler subclass) is NOT here, + // so naming it doesn't keep every backend variant full and flood the budget. + const uniqueNamedNodeIds = new Set(); + for (const t of tokens) { + const cands = this.findAllSymbols(cg, t).nodes.filter((n) => CALLABLE.has(n.kind)); + // A qualified or otherwise-specific name (<=3 hits) keeps all; an + // ambiguous simple name keeps only candidates whose container is named. + const specific = cands.length <= 3; + const pick = specific + ? cands + : cands.filter((n) => { + const segs = (n.qualifiedName || '').toLowerCase().split(/::|\./).filter(Boolean); + const container = segs.length >= 2 ? segs[segs.length - 2] : ''; + return !!container && segPool.has(container); + }); + for (const n of pick.slice(0, 6)) { + named.set(n.id, n); + if (specific) uniqueNamedNodeIds.add(n.id); + } + if (named.size > 40) break; + } + if (named.size < 2) return EMPTY; + const MAX_HOPS = 7; + let best: Array<{ node: Node; edge: Edge | null }> | null = null; + // BFS the full call graph (incl. synth edges) from each named seed, but + // only ACCEPT a sink that is also named — both ends anchored to symbols the + // agent named, so the chain stays on-topic while bridging intermediates + // (e.g. the exact interface overload) that the token resolution missed. + for (const seed of [...named.values()].slice(0, 8)) { + const parent = new Map(); + parent.set(seed.id, { prev: null, edge: null, node: seed }); + const q: Array<{ id: string; depth: number; streak: number }> = [{ id: seed.id, depth: 0, streak: 0 }]; + let deep: string | null = null, deepDepth = 0; + const MAX_BRIDGE = 1; // ≤1 consecutive UNNAMED hop: bridge one missing intermediate, never wander a god-function's fan-out + for (let h = 0; h < q.length && parent.size < 1500; h++) { + const { id, depth, streak } = q[h]!; + if (id !== seed.id && named.has(id) && depth > deepDepth) { deep = id; deepDepth = depth; } + if (depth >= MAX_HOPS - 1) continue; + for (const c of cg.getCallees(id)) { + if (c.edge.kind !== 'calls' || parent.has(c.node.id)) continue; + const newStreak = named.has(c.node.id) ? 0 : streak + 1; + if (newStreak > MAX_BRIDGE) continue; + parent.set(c.node.id, { prev: id, edge: c.edge, node: c.node }); + q.push({ id: c.node.id, depth: depth + 1, streak: newStreak }); + } + } + if (!deep) continue; + const chain: Array<{ node: Node; edge: Edge | null }> = []; + let cur: string | null = deep; + while (cur) { const p = parent.get(cur); if (!p) break; chain.push({ node: p.node, edge: p.edge }); cur = p.prev; } + chain.reverse(); + if (!best || chain.length > best.length) best = chain; + } + if (!best || best.length < 3) return EMPTY; + const out = ['## Flow (call path among the symbols you queried)', '']; + for (let i = 0; i < best.length; i++) { + const step = best[i]!; + if (step.edge) { const sy = this.synthEdgeNote(step.edge); out.push(` ↓ ${sy ? sy.compact : step.edge.kind}`); } + out.push(`${i + 1}. ${step.node.name} (${step.node.filePath}:${step.node.startLine})`); + } + out.push('', '> Full source for these symbols is below; codegraph_trace(from,to) for the exact path between two endpoints.', ''); + // namedNodeIds = every callable the agent explicitly named (a superset of + // the spine). A file holding one is something the agent asked to SEE, so it + // must keep full source even if it's an off-spine polymorphic sibling — the + // agent named `getResponseWithInterceptorChain` / `SQLCompiler.execute_sql` + // as the mechanism, not as an interchangeable leaf. See the skeleton gate. + return { text: out.join('\n'), pathNodeIds: new Set(best.map((s) => s.node.id)), namedNodeIds: new Set(named.keys()), uniqueNamedNodeIds }; + } catch { + return EMPTY; + } + } /** * Handle codegraph_explore — deep exploration in a single call @@ -650,15 +2026,29 @@ export class ToolHandler { * Strategy: find relevant symbols via graph traversal, group by file, * then read contiguous file sections covering all symbols per file. * This replaces multiple codegraph_node + Read calls. + * + * Output size is adaptive to project file count via + * `getExploreOutputBudget` — see #185 for why a fixed 35k cap was a + * tax on small projects while earning its keep on large ones. */ private async handleExplore(args: Record): Promise { const query = this.validateString(args.query, 'query'); if (typeof query !== 'string') return query; const cg = this.getCodeGraph(args.projectPath as string | undefined); - const maxFiles = clamp((args.maxFiles as number) || 12, 1, 20); const projectRoot = cg.getProjectRoot(); + // Resolve adaptive output budget from project size. Falls back to the + // largest-tier defaults if stats aren't available, which preserves + // pre-#185 behavior for callers that hit the rare stats failure. + let budget: ExploreOutputBudget; + try { + budget = getExploreOutputBudget(cg.getStats().fileCount); + } catch { + budget = getExploreOutputBudget(Infinity); + } + const maxFiles = clamp((args.maxFiles as number) || budget.defaultMaxFiles, 1, 20); + // Step 1: Find relevant context with generous parameters. // Use a large maxNodes budget — explore has its own 35k char output limit // that prevents context bloat, so more nodes just means better coverage @@ -674,6 +2064,38 @@ export class ToolHandler { return this.textResult(`No relevant code found for "${query}"`); } + // Graph-aware glue: findRelevantContext builds the subgraph from name/text + // search, so a method that BRIDGES named symbols — e.g. App.tsx's + // triggerRender, which calls the named triggerUpdate — is never a search hit + // and gets missed, forcing the agent to Read the file to trace it. Pull in + // the callers/callees of the entry (root) nodes, but ONLY those that live in + // files the subgraph already surfaces (where the agent reads to fill gaps), + // so we add wiring without dragging in unrelated files. These get an + // importance boost below so they survive the per-file cluster budget. + const glueNodeIds = new Set(); + const subgraphFiles = new Set(); + for (const n of subgraph.nodes.values()) subgraphFiles.add(n.filePath); + const GLUE_NODE_CAP = 60; + for (const rootId of subgraph.roots) { + if (glueNodeIds.size >= GLUE_NODE_CAP) break; + let neighbors: Node[] = []; + try { + neighbors = [ + ...cg.getCallers(rootId).map(c => c.node), + ...cg.getCallees(rootId).map(c => c.node), + ]; + } catch { + continue; + } + for (const nb of neighbors) { + if (glueNodeIds.size >= GLUE_NODE_CAP) break; + if (subgraph.nodes.has(nb.id)) continue; + if (!subgraphFiles.has(nb.filePath)) continue; + subgraph.nodes.set(nb.id, nb); + glueNodeIds.add(nb.id); + } + } + // Step 2: Group nodes by file, score by relevance const fileGroups = new Map(); const entryNodeIds = new Set(subgraph.roots); @@ -703,11 +2125,50 @@ export class ToolHandler { } // Only include files that have entry points or nodes directly connected to entry points - const relevantFiles = [...fileGroups.entries()].filter(([, group]) => group.score >= 3); + let relevantFiles = [...fileGroups.entries()].filter(([, group]) => group.score >= 3); // Extract query terms for relevance checking const queryTerms = query.toLowerCase().split(/\s+/).filter(t => t.length >= 3); + // Test/spec/icon/i18n file detector — used both for the pre-sort hard + // filter (tiny tier) and the comparator deprioritization (all tiers). + const isLowValue = (p: string) => { + const lp = p.toLowerCase(); + return ( + /\/(tests?|__tests?__|spec)\//.test(lp) || + /_test\.go$/.test(lp) || + /(?:^|\/)test_[^/]+\.py$/.test(lp) || + /_test\.py$/.test(lp) || + /_spec\.rb$/.test(lp) || + /_test\.rb$/.test(lp) || + /\.(test|spec)\.[jt]sx?$/.test(lp) || + /(test|spec|tests)\.(java|kt|scala)$/.test(lp) || + /(tests?|spec)\.cs$/.test(lp) || + /tests?\.swift$/.test(lp) || + /_test\.dart$/.test(lp) || + /\bicons?\b/.test(lp) || + /\bi18n\b/.test(lp) + ); + }; + + // Hard-exclude test/spec files (ALL tiers, not just tiny). One slipped test + // file dominates the per-file budget on small repos (cobra's `command_test.go` + // displaced `args.go`) AND wastes budget on large ones (Django's + // `custom_lookups/tests.py` ate ~2.3 KB of the 28 KB cap, crowding out the + // SQLCompiler mechanism the agent then Read). A test file almost never answers + // an architecture question. Skip when the query itself is about tests — the + // legitimate "explore the tests" case — and only cut if ≥2 non-test candidates + // remain (else tests are the only signal for this area). + { + const queryMentionsTests = /\b(test|tests|testing|spec|verify|verifies)\b/i.test(query); + if (!queryMentionsTests) { + const nonLow = relevantFiles.filter(([p]) => !isLowValue(p)); + if (nonLow.length >= 2) { + relevantFiles = nonLow; + } + } + } + // Sort files: highest relevance first, deprioritize low-value files const sortedFiles = relevantFiles.sort((a, b) => { const aPath = a[0].toLowerCase(); @@ -724,15 +2185,20 @@ export class ToolHandler { const bRelevant = hasQueryRelevance(bPath, b[1].nodes); if (aRelevant !== bRelevant) return aRelevant ? -1 : 1; - // Deprioritize test files, icon files, and i18n files - const isLowValue = (p: string) => - /\/(tests?|__tests?__|spec)\//i.test(p) || - /\bicons?\b/i.test(p) || - /\bi18n\b/i.test(p); const aLow = isLowValue(aPath); const bLow = isLowValue(bPath); if (aLow !== bLow) return aLow ? 1 : -1; + // Deprioritize generated source (.pb.go / .pulsar.go / _mocks.go / …) — + // the agent rarely needs to see the protobuf scaffold or gomock output + // when asking about the actual flow, and dumping their bodies inflates + // the response (the cosmos Q3 explore otherwise leads with + // `expected_keepers_mocks.go`, displacing the real `tally.go` content + // and forcing the agent to Read tally.go anyway). + const aGen = isGeneratedFile(a[0]); + const bGen = isGeneratedFile(b[0]); + if (aGen !== bGen) return aGen ? 1 : -1; + if (a[1].score !== b[1].score) return b[1].score - a[1].score; return b[1].nodes.length - a[1].nodes.length; }); @@ -750,7 +2216,7 @@ export class ToolHandler { e.kind !== 'contains' // skip contains — it's implied by file grouping ); - if (significantEdges.length > 0) { + if (budget.includeRelationships && significantEdges.length > 0) { lines.push('### Relationships'); lines.push(''); @@ -767,29 +2233,89 @@ export class ToolHandler { } for (const [kind, edges] of byKind) { - // Show up to 15 relationships per kind - const shown = edges.slice(0, 15); + const cap = budget.maxEdgesPerRelationshipKind; + const shown = edges.slice(0, cap); lines.push(`**${kind}:**`); for (const e of shown) { lines.push(`- ${e.source} → ${e.target}`); } - if (edges.length > 15) { - lines.push(`- ... and ${edges.length - 15} more`); + if (edges.length > cap) { + lines.push(`- ... and ${edges.length - cap} more`); } lines.push(''); } } // Step 4: Read contiguous file sections + // Compute the flow spine once — used both to prepend the Flow section (below) + // and to gate adaptive source sizing: files on the spine get full source, + // off-spine peers skeletonize. + const flow = this.buildFlowFromNamedSymbols(cg, query); + + // Polymorphic-sibling detector for adaptive sizing. A class that implements/ + // extends a supertype shared by >= MIN_SIBLINGS classes is one of many + // INTERCHANGEABLE implementations (OkHttp's 14 `: Interceptor` classes — + // showing one + the rest as signatures is enough), as opposed to a DISTINCT + // pipeline step (Excalidraw's `renderStaticScene`, which shares no supertype and + // must stay full or the agent loses real content). Only off-spine sibling files + // skeletonize; distinct steps and on-spine files keep full source. Cache + // supertype→(has ≥N implementers) so this stays a handful of edge queries. + const MIN_SIBLINGS = 3; + const siblingSuper = new Map(); + const isPolymorphicSibling = (nodes: Node[]): boolean => { + for (const n of nodes) { + for (const e of cg.getOutgoingEdges(n.id)) { + if (e.kind !== 'implements' && e.kind !== 'extends') continue; + let many = siblingSuper.get(e.target); + if (many === undefined) { + many = cg.getIncomingEdges(e.target) + .filter((x) => x.kind === 'implements' || x.kind === 'extends').length >= MIN_SIBLINGS; + siblingSuper.set(e.target, many); + } + if (many) return true; + } + } + return false; + }; + + // A file that DEFINES a polymorphic supertype (a class/interface with ≥ + // MIN_SIBLINGS implementers) AND co-locates its subclasses is a redundant + // "family" file — Django's compiler.py holds `SQLCompiler` + its 4 subclasses + // (SQLInsert/Update/Delete/AggregateCompiler) in 2,266 lines. Such files are + // huge and read-anyway, so they should STILL skeletonize even when the agent + // named a method in them: a full one eats ~6.5K of the explore budget (Django + // is pinned at the 28K cap, truncating), starving the sibling files the agent + // then Reads. This flag OVERRIDES the named-callable spare below — it does NOT + // by itself spare a file. (OkHttp's RealCall implements the `Lockable` mixin + // but defines no ≥3-impl supertype, so the named spare keeps it full.) + const superMany = new Map(); + const definesPolymorphicSupertype = (nodes: Node[]): boolean => { + for (const n of nodes) { + if (n.kind !== 'class' && n.kind !== 'interface' && n.kind !== 'struct' + && n.kind !== 'trait' && n.kind !== 'protocol' && n.kind !== 'type_alias') continue; + let many = superMany.get(n.id); + if (many === undefined) { + many = cg.getIncomingEdges(n.id) + .filter((x) => x.kind === 'implements' || x.kind === 'extends').length >= MIN_SIBLINGS; + superMany.set(n.id, many); + } + if (many) return true; + } + return false; + }; + lines.push('### Source Code'); lines.push(''); + lines.push('> The code below is the **verbatim, current on-disk source** of these files — re-read from disk on this call and line-numbered, byte-for-byte identical to what the Read tool returns. It is NOT a summary, outline, or stale cache. Treat each block as a Read you have already performed: do not Read a file shown here.'); + lines.push(''); let totalChars = lines.join('\n').length; let filesIncluded = 0; + let anyFileTrimmed = false; for (const [filePath, group] of sortedFiles) { if (filesIncluded >= maxFiles) break; - if (totalChars > ToolHandler.EXPLORE_MAX_OUTPUT * 0.9) break; + if (totalChars > budget.maxOutputChars * 0.9) break; const absPath = validatePathWithinRoot(projectRoot, filePath); if (!absPath || !existsSync(absPath)) continue; @@ -804,15 +2330,173 @@ export class ToolHandler { const fileLines = fileContent.split('\n'); const lang = group.nodes[0]?.language || ''; + // Adaptive sizing (CODEGRAPH_ADAPTIVE_EXPLORE, default on): collapse a file + // to a per-symbol view when it's a redundant member of a polymorphic family. + // Engages iff ALL hold: + // 1. a flow spine exists, + // 2. no symbol in the file is on that spine (it's not the mechanism path), + // 3. it IS a polymorphic sibling (≥ MIN_SIBLINGS impls of a shared supertype), + // 4. it is NOT SPARED, where a file is spared iff the agent named a + // (near-)UNIQUE callable in it (`getResponseWithInterceptorChain`, 1 def → + // keep RealCall.kt full) UNLESS the file DEFINES the family supertype (a + // base+subclasses "family" file like Django's compiler.py — collapse it). + // Uniqueness matters: `as_sql` has 110 defs across every Compiler/Expression + // subclass; naming it must NOT keep every backend variant + test file full + // and flood the budget. That's why the spare reads uniqueNamedNodeIds. + // Within a collapsed file the render is PER-SYMBOL (condition B): a method the + // agent NAMED or that's on the spine is shown with its FULL body (so the agent + // doesn't Read the file back for it — Django's SQLCompiler.execute_sql/as_sql); + // every other symbol is just its signature. So the base mechanism survives while + // the file's other ~80 symbols + the redundant subclasses collapse to one line each. + const spareNamed = group.nodes.some(n => flow.uniqueNamedNodeIds.has(n.id)); + const fileDefinesSuper = definesPolymorphicSupertype(group.nodes); + const spared = spareNamed && !fileDefinesSuper; + if (adaptiveExploreEnabled() && flow.pathNodeIds.size > 0 + && !group.nodes.some(n => flow.pathNodeIds.has(n.id)) + && isPolymorphicSibling(group.nodes) + && !spared) { + const CALLABLE_BODY = new Set(['method', 'function', 'constructor', 'component']); + const syms = group.nodes + .filter(n => n.kind !== 'import' && n.kind !== 'export' && n.startLine > 0) + .sort((a, b) => a.startLine - b.startLine); + // Pass 1: choose which symbols get a FULL body, by priority, greedily within + // a per-file body cap — so one huge family file can't body every named method + // and crowd out the other flow files (Django's query.py). A symbol earns a + // body if it's on-spine, or UNIQUELY named (`SQLCompiler.execute_sql`), or a + // co-named method WHEN this file DEFINES the family supertype (so the base + // `SQLCompiler.as_sql` body shows, but the 110 leaf `as_sql` overrides — and + // OkHttp's 5 `intercept`s if the agent names `intercept` — stay signatures). + const prio = (n: Node) => !CALLABLE_BODY.has(n.kind) ? 99 + : flow.pathNodeIds.has(n.id) ? 0 + : flow.uniqueNamedNodeIds.has(n.id) ? 1 + : (fileDefinesSuper && flow.namedNodeIds.has(n.id)) ? 2 : 99; + const bodyCap = budget.maxCharsPerFile * 2; + const bodyIds = new Set(); + let bodyChars = 0; + for (const n of syms.filter(n => prio(n) < 99 && n.endLine >= n.startLine).sort((a, b) => prio(a) - prio(b))) { + const sz = fileLines.slice(n.startLine - 1, Math.min(n.endLine, n.startLine + 220)).join('\n').length; + if (bodyChars + sz > bodyCap && bodyIds.size > 0) continue; + bodyIds.add(n.id); + bodyChars += sz; + } + // Pass 2: render in line order — full body for chosen symbols, else the + // signature line (capped, with a "+N more" tail so the structure map of a + // god-file doesn't itself bloat the budget). + const skel: string[] = []; + let coveredUntil = 0; // skip symbols already inside an emitted body + let sigCount = 0, sigDropped = 0; + const SIG_MAX = Math.max(12, budget.maxSymbolsInFileHeader * 2); + for (const n of syms) { + if (n.startLine <= coveredUntil) continue; + if (bodyIds.has(n.id)) { + const end = Math.min(n.endLine, n.startLine + 220); + const body = fileLines.slice(n.startLine - 1, end).join('\n'); + skel.push(exploreLineNumbersEnabled() ? numberSourceLines(body, n.startLine) : body); + coveredUntil = end; + } else { + // Elide the body, emit the signature. node.startLine can point at a + // decorator/annotation, so scan forward for the line that names the symbol. + let lineNo = n.startLine; + for (let k = 0; k < 4; k++) { + if ((fileLines[n.startLine - 1 + k] || '').includes(n.name)) { lineNo = n.startLine + k; break; } + } + if (lineNo <= coveredUntil) continue; + if (sigCount >= SIG_MAX) { sigDropped++; continue; } + const sig = (fileLines[lineNo - 1] || '').trim(); + if (sig) { skel.push(exploreLineNumbersEnabled() ? `${lineNo}\t${sig}` : sig); sigCount++; } + } + } + if (sigDropped > 0) skel.push(`… +${sigDropped} more (signatures elided)`); + if (skel.length > 0) { + const names = [...new Set(group.nodes.filter(n => n.kind !== 'import' && n.kind !== 'export').map(n => n.name))] + .slice(0, budget.maxSymbolsInFileHeader).join(', '); + const tag = bodyIds.size > 0 + ? 'focused (the methods you named in full, the rest as signatures; Read for more)' + : 'skeleton (signatures only; Read for a full body)'; + lines.push(`#### ${filePath} — ${names} · ${tag}`, '', '```' + lang, skel.join('\n'), '```', ''); + totalChars += skel.join('\n').length + 120; + filesIncluded++; + continue; + } + } + + // Whole-small-file rule: if a relevant file is small enough to afford, + // return it ENTIRELY instead of clustering. Clustering exists to tame + // god-files (App.tsx ~13k lines); on a ~134-line component a cluster is a + // lossy subset of a file the agent will just Read in full anyway — costing + // a round-trip and a re-read every later turn. Reserve clustering for files + // too big to ship whole. Still bounded by the total maxOutputChars check. + const WHOLE_FILE_MAX_LINES = 220; + const WHOLE_FILE_MAX_CHARS = budget.maxCharsPerFile * 3; + if (fileLines.length <= WHOLE_FILE_MAX_LINES && fileContent.length <= WHOLE_FILE_MAX_CHARS) { + const body = fileContent.replace(/\n+$/, ''); + let wholeSection = exploreLineNumbersEnabled() ? numberSourceLines(body, 1) : body; + const uniqSymbols = [...new Set( + group.nodes + .filter(n => n.kind !== 'import' && n.kind !== 'export') + .map(n => `${n.name}(${n.kind})`) + )]; + const headerNames = uniqSymbols.slice(0, budget.maxSymbolsInFileHeader); + const omitted = uniqSymbols.length - headerNames.length; + const wholeHeader = `#### ${filePath} — ${omitted > 0 ? `${headerNames.join(', ')}, +${omitted} more` : headerNames.join(', ')}`; + + if (totalChars + wholeSection.length + 200 > budget.maxOutputChars) { + const remaining = budget.maxOutputChars - totalChars - 200; + if (remaining < 500) break; + wholeSection = wholeSection.slice(0, remaining) + '\n... (trimmed) ...'; + anyFileTrimmed = true; + } + lines.push(wholeHeader, '', '```' + lang, wholeSection, '```', ''); + totalChars += wholeSection.length + 200; + filesIncluded++; + continue; + } + // Cluster nearby symbols to avoid reading huge gaps between distant symbols. - // Sort by start line, then merge overlapping/adjacent ranges (within 15 lines). - // Include both node ranges AND edge source locations so template sections - // with component usages/calls are covered (not just script block symbols). - const ranges: Array<{ start: number; end: number; name: string; kind: string }> = group.nodes - .filter(n => n.startLine > 0 && n.endLine > 0) - // Skip file/component nodes that span the entire file — they'd create one giant cluster - .filter(n => !(n.kind === 'component' && n.startLine === 1 && n.endLine >= fileLines.length - 1)) - .map(n => ({ start: n.startLine, end: n.endLine, name: n.name, kind: n.kind })); + // Sort by start line, then merge overlapping/adjacent ranges (within the + // adaptive gap threshold). Include both node ranges AND edge source + // locations so template sections with component usages/calls are + // covered (not just script block symbols). + // + // Each range carries an `importance` score so we can rank clusters + // when the per-file budget forces us to drop some: entry-point nodes + // are worth 10, directly-connected nodes 3, peripheral nodes 1, and + // bare edge-source lines 2 (less than a connected node but more than + // a peripheral one — they hint at a reference but aren't a definition). + // Container kinds whose body can span most/all of a file. When such a + // node covers most of the file we drop it from the ranges: keeping it + // would merge every method inside it into one giant cluster spanning + // the whole file, which then tail-trims down to just the container's + // opening lines (its header/declarations) and buries the methods the + // query actually asked about (#185 follow-up — Session.swift in + // Alamofire is the canonical case: the `Session` class spans ~1,400 + // lines). We want the granular symbols inside, not the envelope. + const ENVELOPE_KINDS = new Set(['file', 'module', 'class', 'struct', 'interface', 'enum', 'namespace', 'protocol', 'trait', 'component']); + // Cluster from this file's gathered nodes PLUS any callable the agent NAMED that + // lives here. Explore's relevance gather can miss a named method def in a huge + // non-sibling file — Django's query.py is 3,040 lines and `_fetch_all` (L2237) + // was gathered only as call-reference edges, never as a def, so it formed no + // cluster and the agent Read it back. Inject named defs directly and rank them + // ABOVE connected/glue nodes (importance 9) so their cluster wins the per-file + // budget — the agent explicitly asked for these symbols. + const rangeNodes = new Map(); + for (const n of group.nodes) if (n.startLine > 0 && n.endLine > 0) rangeNodes.set(n.id, n); + for (const id of flow.namedNodeIds) { + if (rangeNodes.has(id)) continue; + const n = cg.getNode(id); + if (n && n.filePath === filePath && n.startLine > 0 && n.endLine > 0) rangeNodes.set(id, n); + } + const ranges: Array<{ start: number; end: number; name: string; kind: string; importance: number }> = [...rangeNodes.values()] + // Drop whole-file envelope nodes (containers covering >50% of the file). + .filter(n => !(ENVELOPE_KINDS.has(n.kind) && (n.endLine - n.startLine + 1) > fileLines.length * 0.5)) + .map(n => { + let importance = 1; + if (entryNodeIds.has(n.id)) importance = 10; + else if (flow.namedNodeIds.has(n.id)) importance = 9; // agent named it → keep its cluster + else if (glueNodeIds.has(n.id)) importance = 6; // bridging caller/callee of an entry + else if (connectedToEntry.has(n.id)) importance = 3; + return { start: n.startLine, end: n.endLine, name: n.name, kind: n.kind, importance }; + }); // Add edge source locations in this file — captures template references // (component usages, event handlers) that aren't nodes themselves. @@ -829,7 +2513,7 @@ export class ToolHandler { // Look up target name from subgraph first, fall back to edge kind const targetNode = subgraph.nodes.get(edge.target); const targetName = targetNode?.name ?? edge.kind; - ranges.push({ start: edge.line, end: edge.line, name: targetName, kind: edge.kind }); + ranges.push({ start: edge.line, end: edge.line, name: targetName, kind: edge.kind, importance: 2 }); } } @@ -837,46 +2521,150 @@ export class ToolHandler { if (ranges.length === 0) continue; - const GAP_THRESHOLD = 15; // merge sections within 15 lines of each other - const clusters: Array<{ start: number; end: number; symbols: string[] }> = []; - let current = { start: ranges[0]!.start, end: ranges[0]!.end, symbols: [`${ranges[0]!.name}(${ranges[0]!.kind})`] }; + const gapThreshold = budget.gapThreshold; + const clusters: Array<{ start: number; end: number; symbols: string[]; score: number; maxImportance: number }> = []; + let current = { + start: ranges[0]!.start, + end: ranges[0]!.end, + symbols: [`${ranges[0]!.name}(${ranges[0]!.kind})`], + score: ranges[0]!.importance, + maxImportance: ranges[0]!.importance, + }; for (let i = 1; i < ranges.length; i++) { const r = ranges[i]!; - if (r.start <= current.end + GAP_THRESHOLD) { + if (r.start <= current.end + gapThreshold) { current.end = Math.max(current.end, r.end); current.symbols.push(`${r.name}(${r.kind})`); + current.score += r.importance; + current.maxImportance = Math.max(current.maxImportance, r.importance); } else { clusters.push(current); - current = { start: r.start, end: r.end, symbols: [`${r.name}(${r.kind})`] }; + current = { + start: r.start, + end: r.end, + symbols: [`${r.name}(${r.kind})`], + score: r.importance, + maxImportance: r.importance, + }; } } clusters.push(current); - // Build file section output from clusters + // Build file section output from clusters, capped by per-file budget. + // The pathological case (#185): a file like Session.swift where every + // method is adjacent collapses into one cluster spanning the whole + // file, and dumping that into the agent's context is most of the + // token cost on small projects. We pick clusters in priority order + // until the per-file char cap is hit. Truly enormous single clusters + // get tail-trimmed with a marker. const contextPadding = 3; + const withLineNumbers = exploreLineNumbersEnabled(); + const buildSection = (c: { start: number; end: number }): string => { + const startIdx = Math.max(0, c.start - 1 - contextPadding); + const endIdx = Math.min(fileLines.length, c.end + contextPadding); + const slice = fileLines.slice(startIdx, endIdx).join('\n'); + // startIdx is 0-based, so the slice's first line is line startIdx + 1. + return withLineNumbers ? numberSourceLines(slice, startIdx + 1) : slice; + }; + // Language-neutral separator (no `//` — not a comment in Python, Ruby, + // etc.). With line numbers on, the line-number jump also signals the gap. + const GAP_MARKER = '\n\n... (gap) ...\n\n'; + + // Rank clusters for inclusion under the per-file cap. Entry-point + // clusters come first: a cluster containing a query entry point + // (importance 10) must outrank a dense block of mere declarations, + // otherwise on a large file like Session.swift the top-of-file class + // header + property list (many adjacent low-importance nodes, high + // density) wins the budget and buries the actual methods the query + // asked about (perform/didCreateURLRequest/task live deep in the + // file). Within the same importance tier, prefer density (score per + // line) so we still favor focused clusters over sprawling ones, then + // smaller span as a cheap-to-include tiebreak. + const rankedClusters = clusters + .map((c, i) => ({ idx: i, span: c.end - c.start + 1, c })) + .sort((a, b) => { + if (b.c.maxImportance !== a.c.maxImportance) return b.c.maxImportance - a.c.maxImportance; + const densityA = a.c.score / a.span; + const densityB = b.c.score / b.span; + if (densityB !== densityA) return densityB - densityA; + if (b.c.score !== a.c.score) return b.c.score - a.c.score; + return a.span - b.span; + }); + + // Per-file budget is the SMALLER of the per-file cap and what's left of the + // total output cap — so selection (which ranks by importance) keeps the + // high-importance clusters and drops peripheral ones, instead of the + // downstream source-order trim slicing off whatever comes last in the file. + // That source-order slice is what cut Django's `_fetch_all` (L2237, importance + // 9 — agent-named) when query.py was the last of four big files to be emitted. + const fileBudget = Math.min(budget.maxCharsPerFile, Math.max(0, budget.maxOutputChars - totalChars - 200)); + const chosenIndices = new Set(); + let projectedChars = 0; + for (const rc of rankedClusters) { + const sectionLen = buildSection(rc.c).length + (chosenIndices.size > 0 ? GAP_MARKER.length : 0); + // Always take the top-ranked cluster, even if oversize, so we don't + // return an empty file section (agent would then re-Read the file, + // negating the savings). + if (chosenIndices.size === 0) { + chosenIndices.add(rc.idx); + projectedChars += sectionLen; + continue; + } + if (projectedChars + sectionLen > fileBudget) continue; + chosenIndices.add(rc.idx); + projectedChars += sectionLen; + } + + // Emit chosen clusters in source order so the file reads top-to-bottom. let fileSection = ''; const allSymbols: string[] = []; - - for (const cluster of clusters) { - const startIdx = Math.max(0, cluster.start - 1 - contextPadding); - const endIdx = Math.min(fileLines.length, cluster.end + contextPadding); - const section = fileLines.slice(startIdx, endIdx).join('\n'); - - if (fileSection.length > 0) { - fileSection += '\n\n// ... (gap) ...\n\n'; - } + let fileTrimmed = false; + for (let i = 0; i < clusters.length; i++) { + if (!chosenIndices.has(i)) continue; + const cluster = clusters[i]!; + const section = buildSection(cluster); + if (fileSection.length > 0) fileSection += GAP_MARKER; fileSection += section; allSymbols.push(...cluster.symbols); } - // Skip if this section would blow the output limit - if (totalChars + fileSection.length + 200 > ToolHandler.EXPLORE_MAX_OUTPUT) { - const budget = ToolHandler.EXPLORE_MAX_OUTPUT - totalChars - 200; - if (budget < 500) break; - const trimmed = fileSection.slice(0, budget) + '\n// ... trimmed ...'; + // If a single chosen cluster is still oversize (long monolithic + // function), tail-trim it. Better one trimmed view than nothing. + if (fileSection.length > budget.maxCharsPerFile) { + fileSection = fileSection.slice(0, budget.maxCharsPerFile) + '\n... (trimmed) ...'; + fileTrimmed = true; + } + if (chosenIndices.size < clusters.length || fileTrimmed) { + anyFileTrimmed = true; + } - lines.push(`#### ${filePath} — ${allSymbols.join(', ')}`); + // Dedupe + cap the symbols list shown in the per-file header. Some + // files (Session.swift in Alamofire) produced 3.4KB symbol lists + // from cluster scoring + edge-source lines, dwarfing the per-file + // body cap. Show top names by frequency, with a "+N more" tail. + const symbolCounts = new Map(); + for (const s of allSymbols) { + symbolCounts.set(s, (symbolCounts.get(s) ?? 0) + 1); + } + const sortedSymbols = [...symbolCounts.entries()] + .sort((a, b) => b[1] - a[1]) + .map(([name]) => name); + const headerCap = budget.maxSymbolsInFileHeader; + const headerSymbols = sortedSymbols.slice(0, headerCap); + const omittedCount = sortedSymbols.length - headerSymbols.length; + const headerSuffix = omittedCount > 0 + ? `${headerSymbols.join(', ')}, +${omittedCount} more` + : headerSymbols.join(', '); + const fileHeader = `#### ${filePath} — ${headerSuffix}`; + + // Respect the total output cap on a file-by-file basis. + if (totalChars + fileSection.length + 200 > budget.maxOutputChars) { + const remaining = budget.maxOutputChars - totalChars - 200; + if (remaining < 500) break; + const trimmed = fileSection.slice(0, remaining) + '\n... (trimmed) ...'; + + lines.push(fileHeader); lines.push(''); lines.push('```' + lang); lines.push(trimmed); @@ -884,10 +2672,11 @@ export class ToolHandler { lines.push(''); totalChars += trimmed.length + 200; filesIncluded++; + anyFileTrimmed = true; break; } - lines.push(`#### ${filePath} — ${allSymbols.join(', ')}`); + lines.push(fileHeader); lines.push(''); lines.push('```' + lang); lines.push(fileSection); @@ -898,40 +2687,67 @@ export class ToolHandler { filesIncluded++; } - // Add remaining files as references (from both relevant and peripheral files) - const remainingRelevant = sortedFiles.slice(filesIncluded); - const peripheralFiles = [...fileGroups.entries()] - .filter(([, group]) => group.score < 3) - .sort((a, b) => b[1].score - a[1].score); - const remainingFiles = [...remainingRelevant, ...peripheralFiles]; - if (remainingFiles.length > 0) { - lines.push('### Additional relevant files (not shown)'); - lines.push(''); - for (const [filePath, group] of remainingFiles.slice(0, 10)) { - const symbols = group.nodes.map(n => `${n.name}:${n.startLine}`).join(', '); - lines.push(`- ${filePath}: ${symbols}`); - } - if (remainingFiles.length > 10) { - lines.push(`- ... and ${remainingFiles.length - 10} more files`); + // Add remaining files as references (from both relevant and peripheral files). + // Small projects (per budget) skip this — the relevant story already fits + // in the source section, and a trailing pointer list is pure overhead. + if (budget.includeAdditionalFiles) { + const remainingRelevant = sortedFiles.slice(filesIncluded); + const peripheralFiles = [...fileGroups.entries()] + .filter(([, group]) => group.score < 3) + .sort((a, b) => b[1].score - a[1].score); + const remainingFiles = [...remainingRelevant, ...peripheralFiles]; + if (remainingFiles.length > 0) { + lines.push('### Not shown above — explore these names for their source'); + lines.push(''); + for (const [filePath, group] of remainingFiles.slice(0, 10)) { + const symbols = group.nodes.map(n => `${n.name}:${n.startLine}`).join(', '); + lines.push(`- ${filePath}: ${symbols}`); + } + if (remainingFiles.length > 10) { + lines.push(`- ... and ${remainingFiles.length - 10} more files`); + } } } - // Add completeness signal so agents know they don't need to re-read these files - lines.push(''); - lines.push('---'); - lines.push(`> **Complete source code is included above for ${filesIncluded} files.** You do NOT need to re-read these files — the relevant sections are already shown in full. Only use Read/Grep for files listed under "Additional relevant files" if you need more detail.`); + // Add completeness signal so agents know they don't need to re-read these files. + // On small projects the budget gates this off — but if we actually had to + // trim or drop clusters, surface a brief note so the agent knows it can + // still Read for more detail. + if (budget.includeCompletenessSignal) { + lines.push(''); + lines.push('---'); + lines.push(`> **Complete source for ${filesIncluded} files is included above — do NOT re-read them.** If your question also needs files/symbols listed under "Not shown above" (or any area this call didn't cover), make ANOTHER codegraph_explore targeting those names — it returns the same source with line numbers and is cheaper and more complete than reading. Reserve Read for a single specific line range explore can't surface.`); + } else if (anyFileTrimmed) { + lines.push(''); + lines.push(`> Some file sections were trimmed for size. For a specific symbol you still need, run another \`codegraph_explore\` (or \`codegraph_node\`) with its exact name — line-numbered source, cheaper and more complete than Read.`); + } // Add explore budget note based on project size - try { - const stats = cg.getStats(); - const budget = getExploreBudget(stats.fileCount); - lines.push(''); - lines.push(`> **Explore budget: ${budget} calls max for this project (${stats.fileCount.toLocaleString()} files indexed).** Stop exploring and synthesize your answer once you've used ${budget} calls — do NOT make additional explore calls beyond this budget.`); - } catch { - // Stats unavailable — skip budget note + if (budget.includeBudgetNote) { + try { + const stats = cg.getStats(); + const callBudget = getExploreBudget(stats.fileCount); + lines.push(''); + lines.push(`> **Explore budget: ${callBudget} calls for this project (${stats.fileCount.toLocaleString()} files indexed).** Each call covers ~6 files; if your question spans more, spend your remaining calls on the uncovered area BEFORE falling back to Read — another explore is cheaper and more complete than reading those files. Synthesize once you've used ${callBudget}.`); + } catch { + // Stats unavailable — skip budget note + } } - return this.textResult(lines.join('\n')); + // Hard-cap to the adaptive budget. The per-file loop bounds the source + // sections, but the relationship map, additional-files list, and + // completeness/budget notes can still push the assembled output past + // maxOutputChars (observed 30k against a 28k tier cap). A fat explore + // payload persists in the agent's context and is re-read as cache-input + // on every subsequent turn, so the overrun is paid many times over. + const output = flow.text + lines.join('\n'); + if (output.length > budget.maxOutputChars) { + const cut = output.slice(0, budget.maxOutputChars); + const lastNewline = cut.lastIndexOf('\n'); + const safe = lastNewline > budget.maxOutputChars * 0.8 ? cut.slice(0, lastNewline) : cut; + return this.textResult(safe + '\n\n... (output truncated to budget; the source above is complete and verbatim — treat it as already Read. For any area not covered, run another codegraph_explore with the specific names — do NOT Read these files.)'); + } + return this.textResult(output); } /** @@ -951,41 +2767,121 @@ export class ToolHandler { } let code: string | null = null; + let outline: string | null = null; if (includeCode) { - code = await cg.getCode(match.node.id); + // For container symbols (class/interface/struct/…), the full body is the + // sum of every method body — a wall of source (e.g. a 10k-char class) + // that bloats context and is rarely needed in full. Return a structural + // outline (members + signatures + line numbers) instead; the agent can + // Read or codegraph_node a specific method for its body. Leaf symbols + // (function/method/etc.) return their full body as before. + if (CONTAINER_NODE_KINDS.has(match.node.kind)) { + outline = this.buildContainerOutline(cg, match.node); + } + if (!outline) { + code = await cg.getCode(match.node.id); + } } - const formatted = this.formatNodeDetails(match.node, code) + match.note; + const trail = this.formatTrail(cg, match.node); + const formatted = this.formatNodeDetails(match.node, code, outline) + trail + match.note; return this.textResult(this.truncateOutput(formatted)); } + /** + * Build the "trail" for a symbol: its direct callees (what it calls) and + * callers (what calls it), each with file:line — so codegraph_node doubles as + * the structural Grep→Read→expand primitive: a spot PLUS where to go next. + * Capped to stay cheap. Walk the graph by calling codegraph_node on a trail + * entry; no Read needed for covered hops. Empty edges on a non-leaf often mean + * dynamic dispatch the static graph couldn't resolve — that absence is itself + * a signal (read that one hop) rather than a dead end. + */ + private formatTrail(cg: CodeGraph, node: Node): string { + const TRAIL_CAP = 12; + const fmt = (e: { node: Node; edge: Edge }) => { + const base = `${e.node.name} (${e.node.filePath}:${e.node.startLine})`; + const synth = this.synthEdgeNote(e.edge); + return synth ? `${base} [${synth.compact}]` : base; + }; + const collect = (edges: Array<{ node: Node; edge: Edge }>): Array<{ node: Node; edge: Edge }> => { + const seen = new Set([node.id]); + const out: Array<{ node: Node; edge: Edge }> = []; + for (const e of edges) { + if (seen.has(e.node.id)) continue; + seen.add(e.node.id); + out.push(e); + } + return out; + }; + const callees = collect(cg.getCallees(node.id)); + const callers = collect(cg.getCallers(node.id)); + if (callees.length === 0 && callers.length === 0) return ''; + const lines: string[] = ['', '### Trail — codegraph_node any of these to follow it (no Read needed)']; + if (callees.length > 0) { + lines.push(`**Calls →** ${callees.slice(0, TRAIL_CAP).map(fmt).join(', ')}${callees.length > TRAIL_CAP ? `, +${callees.length - TRAIL_CAP} more` : ''}`); + } + if (callers.length > 0) { + lines.push(`**Called by ←** ${callers.slice(0, TRAIL_CAP).map(fmt).join(', ')}${callers.length > TRAIL_CAP ? `, +${callers.length - TRAIL_CAP} more` : ''}`); + } + return lines.join('\n'); + } + /** * Handle codegraph_status */ private async handleStatus(args: Record): Promise { - const cg = this.getCodeGraph(args.projectPath as string | undefined); + let cg = this.getCodeGraph(args.projectPath as string | undefined); + // Same trick as withStalenessNotice — when an explicit projectPath + // resolves to the same project as the default session cg, prefer the + // default so getPendingFiles() (only populated by the default's watcher) + // is non-empty when there are pending edits. + if (this.cg && cg !== this.cg) { + try { + if (resolvePath(this.cg.getProjectRoot()) === resolvePath(cg.getProjectRoot())) { + cg = this.cg; + } + } catch { /* closed instance — leave as is */ } + } const stats = cg.getStats(); + // Warn when this index actually belongs to a different git working tree + // (e.g. the server resolved up from a nested worktree to the main checkout). + // Queries then reflect that tree's branch, not the worktree being edited. + // status shows the verbose, multi-line form; the read tools get the compact + // one-liner via withWorktreeNotice. Both share the cached detection. + const mismatch = this.worktreeMismatchFor(args.projectPath as string | undefined); + const lines: string[] = [ '## CodeGraph Status', '', + ]; + if (mismatch) { + lines.push(`> ⚠ ${worktreeMismatchWarning(mismatch).replace(/\n/g, '\n> ')}`, ''); + } + lines.push( `**Files indexed:** ${stats.fileCount}`, `**Total nodes:** ${stats.nodeCount}`, `**Total edges:** ${stats.edgeCount}`, `**Database size:** ${(stats.dbSizeBytes / 1024 / 1024).toFixed(2)} MB`, - ]; + ); - // Surface the active SQLite backend. Without this, users on the - // silent WASM fallback (better-sqlite3 install failed) see "slow" - // indexing and DB-lock errors with no signal of why. - const backend = cg.getBackend(); - if (backend === 'native') { - lines.push(`**Backend:** native (better-sqlite3)`); + // Surface the active SQLite backend (node:sqlite, Node's built-in real + // SQLite — full WAL + FTS5, no native build). + lines.push(`**Backend:** node:sqlite (Node built-in) — full WAL + FTS5`); + + // Effective journal mode. 'wal' ⇒ concurrent reads never block on a writer; + // anything else ⇒ they can ("database is locked"). node:sqlite supports WAL + // everywhere, so a non-wal mode means the filesystem can't (network/ + // virtualized mounts, WSL2 /mnt). See issue #238. + const journalMode = cg.getJournalMode(); + if (journalMode === 'wal') { + lines.push(`**Journal mode:** wal (concurrent reads safe)`); } else { lines.push( - `**Backend:** ⚠ wasm (better-sqlite3 unavailable) — ` + - `5-10x slower than native. Fix: ${WASM_FALLBACK_FIX_RECIPE}` + `**Journal mode:** ⚠ ${journalMode || 'unknown'} — WAL not active, so reads ` + + `can block on a concurrent write (WAL appears unsupported on this filesystem)` ); } @@ -1004,6 +2900,21 @@ export class ToolHandler { } } + // Per-file freshness — the inverse of the auto-prepended staleness banner + // (issue #403). Surfacing it inside `status` gives the agent a single + // place to ask "is the index caught up?" rather than inferring from + // banners on other tool calls. + const pending = cg.getPendingFiles(); + if (pending.length > 0) { + lines.push('', '### Pending sync:'); + const now = Date.now(); + for (const p of pending) { + const ageMs = Math.max(0, now - p.lastSeenMs); + const label = p.indexing ? 'indexing in progress' : 'pending sync'; + lines.push(`- ${p.path} (edited ${ageMs}ms ago, ${label})`); + } + } + return this.textResult(lines.join('\n')); } @@ -1025,9 +2936,20 @@ export class ToolHandler { return this.textResult('No files indexed. Run `codegraph index` first.'); } - // Filter by path prefix - let files = pathFilter - ? allFiles.filter(f => f.path.startsWith(pathFilter) || f.path.startsWith('./' + pathFilter)) + // Filter by path prefix. Stored paths are project-relative POSIX (e.g. + // "src/foo.ts"), but agents commonly pass project-root variants like "/", + // ".", "./", "" or Windows-style "src\foo" — and prefixes with leading + // "/", "./" or "\". Normalize all of those before matching so the agent + // gets results instead of falling back to Read/Glob (see #426). + const normalizedFilter = pathFilter + ? pathFilter + .replace(/\\/g, '/') + .replace(/^(?:\.?\/+)+/, '') + .replace(/^\.$/, '') + .replace(/\/+$/, '') + : ''; + let files = normalizedFilter + ? allFiles.filter(f => f.path === normalizedFilter || f.path.startsWith(normalizedFilter + '/')) : allFiles; // Filter by glob pattern @@ -1204,9 +3126,22 @@ export class ToolHandler { * Returns the best match and a note about alternatives if any. */ /** - * Check if a node matches a symbol query, supporting both simple names and - * qualified "Parent.child" notation (e.g., "Session.request" matches a method - * named "request" inside a class named "Session"). + * Check if a node matches a symbol query. + * + * Accepts simple names (`run`) and three flavors of qualifier: + * - dotted `Session.request` (TS/JS/Python) + * - colon-pair `stage_apply::run` (Rust, C++, Ruby) + * - slash `configurator/stage_apply` (path-ish) + * + * Multi-level qualifiers compose: `crate::configurator::stage_apply::run` + * works. Rust path prefixes (`crate`, `super`, `self`) are stripped so + * the canonical `crate::module::symbol` form resolves. + * + * Resolution order, last part must always equal `node.name`: + * 1. Suffix-match against `qualifiedName` (handles class-scoped methods + * where the extractor builds the qualified name from the AST stack) + * 2. File-path containment (handles file-derived modules in Rust/ + * Python — `stage_apply::run` matches a `run` in `stage_apply.rs`) */ private matchesSymbol(node: Node, symbol: string): boolean { // Simple name match @@ -1214,21 +3149,52 @@ export class ToolHandler { // File basename match (e.g., "product-card" matches "product-card.liquid") if (node.kind === 'file' && node.name.replace(/\.[^.]+$/, '') === symbol) return true; - // Qualified name match: "Parent.child" → look for "::Parent::child" in qualified_name - if (symbol.includes('.')) { - const parts = symbol.split('.'); - const qualifiedSuffix = parts.join('::'); - if (node.qualifiedName.includes(qualifiedSuffix)) return true; - } - - return false; + // Qualified-name lookups: split on any supported separator. `\w` keeps + // identifier chars (incl. `_`) intact; everything else is treated as + // a separator we tolerate. + if (!/[.\/]|::/.test(symbol)) return false; + const parts = symbol.split(/::|[./]/).filter((p) => p.length > 0); + if (parts.length < 2) return false; + + const lastPart = parts[parts.length - 1]!; + if (node.name !== lastPart) return false; + + // Stage 1: qualified-name suffix match. The extractor joins the + // semantic hierarchy with `::`, so `Session.request` and + // `Session::request` both become `Session::request` here. + const colonSuffix = parts.join('::'); + if (node.qualifiedName.includes(colonSuffix)) return true; + + // Stage 2: file-path containment. Rust modules and Python packages + // are not in `qualifiedName` — they're encoded in the file path. So + // `stage_apply::run` matches a `run` in any file whose path + // contains a `stage_apply` segment (with or without an extension). + // + // Filter out Rust path prefixes that have no file-system equivalent. + const containerHints = parts.slice(0, -1).filter((p) => !RUST_PATH_PREFIXES.has(p)); + if (containerHints.length === 0) return false; + + const segments = node.filePath.split('/').filter((s) => s.length > 0); + return containerHints.every((hint) => + segments.some((seg) => seg === hint || seg.replace(/\.[^.]+$/, '') === hint) + ); } private findSymbol(cg: CodeGraph, symbol: string): { node: Node; note: string } | null { - // Use higher limit for qualified lookups (e.g., "Session.request") since the - // target may rank lower in FTS when there are many partial matches - const limit = symbol.includes('.') ? 50 : 10; - const results = cg.searchNodes(symbol, { limit }); + // Use higher limit for qualified lookups (e.g., "Session.request", + // "stage_apply::run") since the target may rank lower in FTS when + // there are many partial matches across the qualifier parts. + const isQualified = /[.\/]|::/.test(symbol); + const limit = isQualified ? 50 : 10; + let results = cg.searchNodes(symbol, { limit }); + + // FTS strips colons as a special char, so `stage_apply::run` searches + // for the literal `stage_applyrun` and finds nothing. Re-search by + // the bare last part and let `matchesSymbol` filter by qualifier. + if (isQualified && results.length === 0) { + const tail = lastQualifierPart(symbol); + if (tail && tail !== symbol) results = cg.searchNodes(tail, { limit }); + } if (results.length === 0 || !results[0]) { return null; @@ -1241,16 +3207,29 @@ export class ToolHandler { } if (exactMatches.length > 1) { + // Down-rank generated files (.pb.go, .pulsar.go, _grpc.pb.go, …) + // so a query like "Send" prefers the keeper implementation over + // the protobuf-generated interface stub. Stable sort preserves + // FTS order within each group. See generated-detection.ts. + const ranked = [...exactMatches].sort((a, b) => { + const aGen = isGeneratedFile(a.node.filePath) ? 1 : 0; + const bGen = isGeneratedFile(b.node.filePath) ? 1 : 0; + return aGen - bGen; + }); // Multiple exact matches - pick first, note the others - const picked = exactMatches[0]!.node; - const others = exactMatches.slice(1).map(r => + const picked = ranked[0]!.node; + const others = ranked.slice(1).map(r => `${r.node.name} (${r.node.kind}) at ${r.node.filePath}:${r.node.startLine}` ); - const note = `\n\n> **Note:** ${exactMatches.length} symbols named "${symbol}". Showing results for \`${picked.filePath}:${picked.startLine}\`. Others: ${others.join(', ')}`; + const note = `\n\n> **Note:** ${ranked.length} symbols named "${symbol}". Showing results for \`${picked.filePath}:${picked.startLine}\`. Others: ${others.join(', ')}`; return { node: picked, note }; } - // No exact match, use best fuzzy match + // No exact match. For qualified lookups, don't silently fall back + // to a fuzzy result — the user typed a specific qualifier, and + // resolving `stage_apply::nonexistent_fn` to the unrelated + // `stage_apply.rs` file would be actively misleading (#173). + if (isQualified) return null; return { node: results[0]!.node, note: '' }; } @@ -1259,7 +3238,15 @@ export class ToolHandler { * results across all matching symbols (e.g., multiple classes with an `execute` method). */ private findAllSymbols(cg: CodeGraph, symbol: string): { nodes: Node[]; note: string } { - const results = cg.searchNodes(symbol, { limit: 50 }); + let results = cg.searchNodes(symbol, { limit: 50 }); + + // Mirror the fallback in `findSymbol` for qualified queries — FTS + // strips colons, so a module-qualified lookup needs a second pass + // by the bare last part. + if (results.length === 0 && /[.\/]|::/.test(symbol)) { + const tail = lastQualifierPart(symbol); + if (tail && tail !== symbol) results = cg.searchNodes(tail, { limit: 50 }); + } if (results.length === 0) { return { nodes: [], note: '' }; @@ -1272,11 +3259,20 @@ export class ToolHandler { return { nodes: [node], note: '' }; } - const locations = exactMatches.map(r => + // Same generated-file down-rank as findSymbol — keeps callers/callees + // /impact aggregation aligned (a query against "Send" returns the + // hand-written implementations before the protobuf scaffold). + const ranked = [...exactMatches].sort((a, b) => { + const aGen = isGeneratedFile(a.node.filePath) ? 1 : 0; + const bGen = isGeneratedFile(b.node.filePath) ? 1 : 0; + return aGen - bGen; + }); + + const locations = ranked.map(r => `${r.node.kind} at ${r.node.filePath}:${r.node.startLine}` ); - const note = `\n\n> **Note:** Aggregated results across ${exactMatches.length} symbols named "${symbol}": ${locations.join(', ')}`; - return { nodes: exactMatches.map(r => r.node), note }; + const note = `\n\n> **Note:** Aggregated results across ${ranked.length} symbols named "${symbol}": ${locations.join(', ')}`; + return { nodes: ranked.map(r => r.node), note }; } /** @@ -1350,7 +3346,29 @@ export class ToolHandler { return lines.join('\n'); } - private formatNodeDetails(node: Node, code: string | null): string { + /** + * Build a compact structural outline of a container symbol from its + * indexed children (methods, fields, properties, …) — name, kind, + * line number, and signature — so the agent gets the shape of a class + * without the full source of every method. Returns '' when the container + * has no indexed children, so the caller can fall back to full source. + */ + private buildContainerOutline(cg: CodeGraph, node: Node): string { + const children = cg.getChildren(node.id) + .filter(c => c.kind !== 'import' && c.kind !== 'export') + .sort((a, b) => (a.startLine ?? 0) - (b.startLine ?? 0)); + if (children.length === 0) return ''; + + const lines = [`**Members (${children.length}):**`, '']; + for (const c of children) { + const loc = c.startLine ? `:${c.startLine}` : ''; + const sig = c.signature ? ` — \`${c.signature}\`` : ''; + lines.push(`- ${c.name} (${c.kind})${loc}${sig}`); + } + return lines.join('\n'); + } + + private formatNodeDetails(node: Node, code: string | null, outline?: string | null): string { const location = node.startLine ? `:${node.startLine}` : ''; const lines: string[] = [ `## ${node.name} (${node.kind})`, @@ -1367,8 +3385,14 @@ export class ToolHandler { lines.push('', node.docstring); } - if (code) { - lines.push('', '```' + node.language, code, '```'); + if (outline) { + lines.push('', outline, '', + `> Structural outline only. Read \`${node.filePath}\` or call codegraph_node on a specific member for its body.`); + } else if (code) { + // Line-numbered (cat -n style, like codegraph_explore and Read) so the + // agent can cite/edit exact lines without re-Reading the file for them. + const numbered = node.startLine ? numberSourceLines(code, node.startLine) : code; + lines.push('', '```' + node.language, numbered, '```'); } return lines.join('\n'); diff --git a/src/mcp/transport.ts b/src/mcp/transport.ts index 440389189..aecc0368f 100644 --- a/src/mcp/transport.ts +++ b/src/mcp/transport.ts @@ -1,10 +1,21 @@ /** - * MCP Stdio Transport + * MCP JSON-RPC Transports * - * Handles JSON-RPC 2.0 communication over stdin/stdout for MCP protocol. + * Two flavors share the same wire format (newline-delimited JSON-RPC 2.0): + * + * - `StdioTransport` — original transport; reads/writes the process's + * stdin/stdout. Used by direct-mode MCP servers. + * - `SocketTransport` — wraps a single `net.Socket`. Used by the shared-daemon + * architecture (see {@link ./daemon}) to multiplex multiple MCP clients onto + * one CodeGraph instance via per-connection sessions. + * + * Both implement {@link JsonRpcTransport} so the session-level protocol logic + * (initialize / tools/list / tools/call, plus server-initiated `roots/list`) + * is identical regardless of where the bytes come from. */ import * as readline from 'readline'; +import type { Socket } from 'net'; /** * JSON-RPC 2.0 Request @@ -56,91 +67,99 @@ export const ErrorCodes = { export type MessageHandler = (message: JsonRpcRequest | JsonRpcNotification) => Promise; /** - * Stdio Transport for MCP - * - * Reads JSON-RPC messages from stdin and writes responses to stdout. + * Generic JSON-RPC transport interface — common surface for stdio and socket + * carriers. Anything below the session layer (initialize, tool dispatch, etc.) + * talks to this, not to a concrete transport class. */ -export class StdioTransport { - private rl: readline.Interface | null = null; - private messageHandler: MessageHandler | null = null; - - /** - * Start listening for messages on stdin - */ - start(handler: MessageHandler): void { - this.messageHandler = handler; - - this.rl = readline.createInterface({ - input: process.stdin, - output: process.stdout, - terminal: false, - }); +export interface JsonRpcTransport { + start(handler: MessageHandler): void; + stop(): void; + send(response: JsonRpcResponse): void; + notify(method: string, params?: unknown): void; + request(method: string, params?: unknown, timeoutMs?: number): Promise; + sendResult(id: string | number, result: unknown): void; + sendError(id: string | number | null, code: number, message: string, data?: unknown): void; +} - this.rl.on('line', async (line) => { - await this.handleLine(line); - }); +/** + * Shared implementation of newline-delimited JSON-RPC 2.0 over any + * `Readable`/`Writable` stream pair. Stdio and socket transports both wrap + * this — the only difference between them is which streams get plugged in + * and how a "close" propagates back to the owning code. + */ +abstract class LineBasedJsonRpcTransport implements JsonRpcTransport { + protected messageHandler: MessageHandler | null = null; + // Outstanding server-initiated requests (e.g. roots/list), keyed by the id + // we sent. Responses from the client are matched back here. + protected pending = new Map void; + reject: (error: Error) => void; + }>(); + protected nextRequestId = 1; + protected stopped = false; - this.rl.on('close', () => { - process.exit(0); - }); - } + abstract start(handler: MessageHandler): void; + protected abstract write(line: string): void; + protected abstract idPrefix(): string; + abstract stop(): void; /** - * Stop listening + * Send a server-initiated request to the client and await its response. + * + * MCP is bidirectional: the server can ask the client questions too. We use + * this for `roots/list` — the spec-blessed way to learn the workspace root + * when the client didn't pass one in `initialize` (see issue #196). Rejects + * on timeout so callers can fall back rather than hang forever. */ - stop(): void { - if (this.rl) { - this.rl.close(); - this.rl = null; - } + request(method: string, params?: unknown, timeoutMs = 5000): Promise { + const id = `${this.idPrefix()}-${this.nextRequestId++}`; + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + this.pending.delete(id); + reject(new Error(`Timed out after ${timeoutMs}ms waiting for "${method}" response`)); + }, timeoutMs); + // Don't let a pending request keep the process alive on shutdown. + timer.unref?.(); + this.pending.set(id, { + resolve: (value) => { clearTimeout(timer); resolve(value); }, + reject: (error) => { clearTimeout(timer); reject(error); }, + }); + this.write(JSON.stringify({ jsonrpc: '2.0', id, method, params })); + }); } - /** - * Send a response - */ send(response: JsonRpcResponse): void { - const json = JSON.stringify(response); - process.stdout.write(json + '\n'); + this.write(JSON.stringify(response)); } - /** - * Send a notification (no id) - */ notify(method: string, params?: unknown): void { - const notification: JsonRpcNotification = { - jsonrpc: '2.0', - method, - params, - }; - process.stdout.write(JSON.stringify(notification) + '\n'); + const notification: JsonRpcNotification = { jsonrpc: '2.0', method, params }; + this.write(JSON.stringify(notification)); } - /** - * Send a success response - */ sendResult(id: string | number, result: unknown): void { - this.send({ - jsonrpc: '2.0', - id, - result, - }); + this.send({ jsonrpc: '2.0', id, result }); + } + + sendError(id: string | number | null, code: number, message: string, data?: unknown): void { + this.send({ jsonrpc: '2.0', id, error: { code, message, data } }); } /** - * Send an error response + * Fail any in-flight server-initiated requests so their awaiters don't hang. + * Called from `stop()` in subclasses. */ - sendError(id: string | number | null, code: number, message: string, data?: unknown): void { - this.send({ - jsonrpc: '2.0', - id, - error: { code, message, data }, - }); + protected rejectPending(reason: string): void { + for (const { reject } of this.pending.values()) { + reject(new Error(reason)); + } + this.pending.clear(); } /** - * Handle an incoming line of JSON + * Handle an incoming line of JSON. Both transports feed lines here. */ - private async handleLine(line: string): Promise { + protected async handleLine(line: string): Promise { const trimmed = line.trim(); if (!trimmed) return; @@ -152,6 +171,20 @@ export class StdioTransport { return; } + // Response to a server-initiated request (has id + result/error, no method). + // Route it to the awaiting requester instead of the message handler — these + // used to be dropped as "Invalid Request" because they carry no method. + const obj = parsed as Record; + if ( + obj?.jsonrpc === '2.0' && + typeof obj.method !== 'string' && + 'id' in obj && + ('result' in obj || 'error' in obj) + ) { + this.handleResponse(obj); + return; + } + // Validate basic JSON-RPC structure if (!this.isValidMessage(parsed)) { this.sendError(null, ErrorCodes.InvalidRequest, 'Invalid Request: not a valid JSON-RPC 2.0 message'); @@ -174,6 +207,24 @@ export class StdioTransport { } } + /** + * Resolve (or reject) the pending server-initiated request matching this + * response's id. Unknown ids are ignored — the client may echo something we + * never sent, or a request may have already timed out. + */ + private handleResponse(msg: Record): void { + const id = msg.id as string | number; + const pending = this.pending.get(id); + if (!pending) return; + this.pending.delete(id); + if ('error' in msg && msg.error) { + const err = msg.error as { message?: string }; + pending.reject(new Error(err.message || 'Request failed')); + } else { + pending.resolve(msg.result); + } + } + /** * Check if message is a valid JSON-RPC 2.0 message */ @@ -185,3 +236,172 @@ export class StdioTransport { return true; } } + +export interface StdioTransportOptions { + /** + * If true, the transport calls `process.exit(0)` when stdin closes. Set to + * `false` in shared-daemon mode where the stdio "session" is just *one* of + * many clients — losing it shouldn't drag the daemon down. The default + * (true) matches the original single-process behavior callers rely on. + */ + exitOnClose?: boolean; + /** + * Optional callback fired when the stdin stream closes. The daemon uses + * this to decrement its connected-clients refcount. + */ + onClose?: () => void; +} + +/** + * Stdio Transport for MCP + * + * Reads JSON-RPC messages from stdin and writes responses to stdout. Used by + * the direct (single-process) MCP server path, where the MCP host launches + * one server per session and talks to it over the child's stdio. Also used by + * shared-daemon mode for the launcher's session (with `exitOnClose: false`) + * so the daemon outlives its launcher. + */ +export class StdioTransport extends LineBasedJsonRpcTransport { + private rl: readline.Interface | null = null; + private opts: Required; + + constructor(opts: StdioTransportOptions = {}) { + super(); + this.opts = { + exitOnClose: opts.exitOnClose ?? true, + onClose: opts.onClose ?? (() => { /* no-op */ }), + }; + } + + start(handler: MessageHandler): void { + this.messageHandler = handler; + + this.rl = readline.createInterface({ + input: process.stdin, + output: process.stdout, + terminal: false, + }); + + this.rl.on('line', async (line) => { + await this.handleLine(line); + }); + + this.rl.on('close', () => { + this.opts.onClose(); + if (this.opts.exitOnClose) { + process.exit(0); + } + }); + } + + stop(): void { + if (this.stopped) return; + this.stopped = true; + this.rejectPending('Transport stopped'); + if (this.rl) { + this.rl.close(); + this.rl = null; + } + } + + protected write(line: string): void { + process.stdout.write(line + '\n'); + } + + protected idPrefix(): string { + return 'cg-srv'; + } +} + +/** + * Socket Transport for MCP daemon sessions. + * + * Wraps a single `net.Socket` (Unix domain socket on POSIX, named pipe on + * Windows). One instance per connected MCP client. Unlike {@link StdioTransport}, + * `stop()` and stream-close *don't* call `process.exit` — a daemon-side session + * ending must not bring down the whole daemon. + */ +export class SocketTransport extends LineBasedJsonRpcTransport { + private buffer = ''; + private closeHandlers: Array<() => void> = []; + + constructor(private socket: Socket, private prefix: string = 'cg-sock') { + super(); + } + + /** + * Register a callback fired exactly once when the socket closes (from either + * side). Used by the daemon to decrement its connected-clients refcount. + */ + onClose(handler: () => void): void { + this.closeHandlers.push(handler); + } + + start(handler: MessageHandler): void { + this.messageHandler = handler; + + this.socket.setEncoding('utf8'); + this.socket.on('data', (chunk: string) => { + this.buffer += chunk; + let idx; + // Drain every complete line; tail-fragment stays in the buffer for the + // next chunk. The handler is async but we don't await it here — JSON-RPC + // permits out-of-order responses, and serializing here would deadlock if + // a handler issued a server-initiated request that needed a *later* line + // to arrive (e.g. roots/list mid-tools-call). + while ((idx = this.buffer.indexOf('\n')) !== -1) { + const line = this.buffer.slice(0, idx); + this.buffer = this.buffer.slice(idx + 1); + void this.handleLine(line); + } + }); + + this.socket.on('close', () => this.handleSocketClose()); + this.socket.on('error', (err) => { + // Don't crash the daemon over a broken pipe; just shut this connection. + process.stderr.write(`[CodeGraph daemon] socket error: ${err.message}\n`); + this.handleSocketClose(); + }); + } + + stop(): void { + if (this.stopped) return; + this.stopped = true; + this.rejectPending('Transport stopped'); + if (!this.socket.destroyed) { + this.socket.end(); + this.socket.destroy(); + } + } + + /** + * Write a one-shot line directly to the socket (no JSON-RPC framing applied + * by this class — caller produces the line). The daemon uses this for the + * hello/handshake line that precedes the JSON-RPC stream. + */ + writeRaw(line: string): void { + if (!this.socket.destroyed) { + this.socket.write(line.endsWith('\n') ? line : line + '\n'); + } + } + + protected write(line: string): void { + if (!this.socket.destroyed) { + this.socket.write(line + '\n'); + } + } + + protected idPrefix(): string { + return this.prefix; + } + + private handleSocketClose(): void { + if (this.stopped) return; + this.stopped = true; + this.rejectPending('Socket closed'); + for (const h of this.closeHandlers) { + try { h(); } catch { /* never let a close-handler take the daemon down */ } + } + this.closeHandlers = []; + } +} diff --git a/src/mcp/version.ts b/src/mcp/version.ts new file mode 100644 index 000000000..cef1b7834 --- /dev/null +++ b/src/mcp/version.ts @@ -0,0 +1,36 @@ +/** + * Resolved package version, computed once at module load. + * + * The version string is the rendezvous datum between cooperating daemon and + * proxy processes: the daemon advertises its version in the hello line, and + * the proxy refuses to share IPC across a mismatch (falls back to direct + * mode). Keeping the resolution in one place avoids drift between the CLI + * `--version` output (which reads `package.json` directly) and the daemon + * handshake. + * + * Resolution strategy: read the bundled `package.json` two levels up from + * this file — same relative position whether we're loaded from `src/mcp/` or + * the `dist/mcp/` output, since `tsc` preserves the layout. If reading fails + * (e.g. the package was unpacked oddly), fall back to "0.0.0-unknown" — a + * sentinel that will never match a real version, so the proxy harmlessly + * falls back to direct mode. + */ + +import * as fs from 'fs'; +import * as path from 'path'; + +function readPackageVersion(): string { + try { + const pkgPath = path.join(__dirname, '..', '..', 'package.json'); + const raw = fs.readFileSync(pkgPath, 'utf8'); + const parsed = JSON.parse(raw); + if (typeof parsed?.version === 'string' && parsed.version.length > 0) { + return parsed.version; + } + } catch { + // Fall through to sentinel. + } + return '0.0.0-unknown'; +} + +export const CodeGraphPackageVersion = readPackageVersion(); diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts new file mode 100644 index 000000000..43e7bf0ba --- /dev/null +++ b/src/resolution/callback-synthesizer.ts @@ -0,0 +1,1133 @@ +/** + * Callback / observer edge synthesis — Phase 1 + 2. + * + * Closes dynamic-dispatch holes where a dispatcher invokes callbacks registered + * elsewhere. Two channel shapes: + * + * (1) Field-backed observer (Phase 1): + * onUpdate(cb) { this.callbacks.add(cb); } // registrar + * triggerUpdate() { for (cb of this.callbacks) cb(); } // dispatcher + * scene.onUpdate(this.triggerRender) // registration + * → synthesize triggerUpdate → triggerRender + * + * (2) String-keyed EventEmitter (Phase 2): + * this.on('mount', function onmount(){...}) // registration + * fn.emit('mount', this) // dispatch + * → synthesize (method containing emit('mount')) → onmount + * + * Whole-graph pass after base resolution. High-precision/low-recall by design: + * named callbacks only; field channels paired by file+field; EventEmitter + * channels capped by event fan-out (generic names like 'error' skipped — they + * need receiver-type matching, deferred to Phase 3). All synthesized edges are + * tagged `provenance:'heuristic'`. See docs/design/callback-edge-synthesis.md. + */ +import type { Edge, Node } from '../types'; +import type { QueryBuilder } from '../db/queries'; +import type { ResolutionContext } from './types'; +import { isGeneratedFile } from '../extraction/generated-detection'; +import { stripCommentsForRegex } from './strip-comments'; + +const REGISTRAR_NAME = /^(on[A-Z]\w*|subscribe|addListener|addEventListener|register|watch|listen|addCallback)$/; +const DISPATCHER_NAME = /(emit|trigger|notify|dispatch|fire|publish|flush)/i; +const MAX_CALLBACKS_PER_CHANNEL = 40; +const EVENT_FANOUT_CAP = 6; // skip events with more handlers/dispatchers than this (too generic without type info) + +const ON_RE = /\.(?:on|once|addListener)\(\s*['"]([^'"]+)['"]\s*,\s*(?:function\s+(\w+)|(?:this\.)?(\w+))/g; +const EMIT_RE = /\.(?:emit|fire|dispatchEvent)\(\s*['"]([^'"]+)['"]/g; +const SETSTATE_RE = /this\.setState\s*\(/; +const FLUTTER_SETSTATE_RE = /\bsetState\s*\(/; // Flutter: setState((){…}) / this.setState +const JSX_TAG_RE = /<([A-Z][A-Za-z0-9_]*)[\s/>]/g; +const MAX_JSX_CHILDREN = 30; +// Vue SFC templates: kebab-case child components ( → ElButton) and +// event bindings (@click="fn" / v-on:click="fn"). PascalCase children () +// are already caught by JSX_TAG_RE via the SFC component node. +const VUE_KEBAB_RE = /<([a-z][a-z0-9]*(?:-[a-z0-9]+)+)[\s/>]/g; +const VUE_HANDLER_RE = /(?:@|v-on:)([a-zA-Z][\w-]*)(?:\.[\w]+)*\s*=\s*"([^"]+)"/g; +// Composable/hook destructure: `const { close: closeSidebar } = useSidebarControl()`. +// Captures the destructure body + the called composable; only `use*` calls qualify. +const VUE_DESTRUCTURE_RE = /(?:const|let|var)\s*\{([^}]+)\}\s*=\s*(\w+)\s*\(/g; + +function kebabToPascal(s: string): string { + return s.split('-').map((p) => p.charAt(0).toUpperCase() + p.slice(1)).join(''); +} + +function sliceLines(content: string, startLine?: number, endLine?: number): string | null { + if (!startLine || !endLine) return null; + return content.split('\n').slice(startLine - 1, endLine).join('\n'); +} + +function registrarField(src: string): string | null { + const m = src.match(/this\.(\w+)\.(?:add|push|set)\(/); + return m ? m[1]! : null; +} + +function dispatcherField(src: string): string | null { + const forOf = src.match(/\bof\s+(?:Array\.from\(\s*)?this\.(\w+)/); + if (forOf && /\b\w+\s*\(/.test(src)) return forOf[1]!; + const forEach = src.match(/this\.(\w+)\.forEach\(/); + if (forEach) return forEach[1]!; + return null; +} + +const FN_KINDS = new Set(['method', 'function', 'component']); + +/** Innermost function/method node whose line range contains `line`. */ +function enclosingFn(nodesInFile: Node[], line: number): Node | null { + let best: Node | null = null; + for (const n of nodesInFile) { + if (!FN_KINDS.has(n.kind)) continue; + const end = n.endLine ?? n.startLine; + if (n.startLine <= line && end >= line) { + if (!best || n.startLine >= best.startLine) best = n; // prefer the tightest (latest-starting) encloser + } + } + return best; +} + +/** Phase 1: field-backed observer channels (registrar/dispatcher share a store). */ +function fieldChannelEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] { + const candidates = [...queries.getNodesByKind('method'), ...queries.getNodesByKind('function')]; + const registrars: Array<{ node: Node; field: string }> = []; + const dispatchers: Array<{ node: Node; field: string }> = []; + + for (const m of candidates) { + const isReg = REGISTRAR_NAME.test(m.name); + const isDisp = DISPATCHER_NAME.test(m.name); + if (!isReg && !isDisp) continue; + const content = ctx.readFile(m.filePath); + const src = content && sliceLines(content, m.startLine, m.endLine); + if (!src) continue; + if (isReg) { const f = registrarField(src); if (f) registrars.push({ node: m, field: f }); } + if (isDisp) { const f = dispatcherField(src); if (f) dispatchers.push({ node: m, field: f }); } + } + + const edges: Edge[] = []; + const seen = new Set(); + for (const reg of registrars) { + const chDispatchers = dispatchers.filter( + (d) => d.node.filePath === reg.node.filePath && d.field === reg.field + ); + if (chDispatchers.length === 0) continue; + const argRe = new RegExp(`${reg.node.name}\\s*\\(\\s*(?:this\\.)?(\\w+)`); + let added = 0; + for (const e of queries.getIncomingEdges(reg.node.id, ['calls'])) { + if (added >= MAX_CALLBACKS_PER_CHANNEL) break; + if (!e.line) continue; + const caller = queries.getNodeById(e.source); + if (!caller) continue; + const line = ctx.readFile(caller.filePath)?.split('\n')[e.line - 1]; + const am = line?.match(argRe); + if (!am) continue; + const fn = ctx.getNodesByName(am[1]!).find((n) => n.kind === 'method' || n.kind === 'function'); + if (!fn) continue; + for (const disp of chDispatchers) { + if (disp.node.id === fn.id) continue; + const key = `${disp.node.id}>${fn.id}`; + if (seen.has(key)) continue; + seen.add(key); + edges.push({ + source: disp.node.id, target: fn.id, kind: 'calls', line: disp.node.startLine, + provenance: 'heuristic', + metadata: { + synthesizedBy: 'callback', via: reg.node.name, field: reg.field, + // Where the callback was wired up (`scene.onUpdate(this.triggerRender)`). + // This is the #1 thing an agent reads/greps to explain the flow — surface + // it so node/trace/context can show it without a callers() + Read round-trip. + registeredAt: `${caller.filePath}:${e.line}`, + }, + }); + added++; + } + } + } + return edges; +} + +/** Phase 2: string-keyed EventEmitter channels (on('e', fn) ↔ emit('e')). */ +function eventEmitterEdges(ctx: ResolutionContext): Edge[] { + const emitsByEvent = new Map>(); // event → dispatcher node ids + const handlersByEvent = new Map>(); // event → handler id → registration site (file:line) + + for (const file of ctx.getAllFiles()) { + const content = ctx.readFile(file); + if (!content) continue; + const hasEmit = content.includes('.emit(') || content.includes('.fire(') || content.includes('.dispatchEvent('); + const hasOn = content.includes('.on(') || content.includes('.once(') || content.includes('.addListener('); + if (!hasEmit && !hasOn) continue; + const nodesInFile = ctx.getNodesInFile(file); + const lineOf = (idx: number) => content.slice(0, idx).split('\n').length; + + if (hasEmit) { + EMIT_RE.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = EMIT_RE.exec(content))) { + const disp = enclosingFn(nodesInFile, lineOf(m.index)); + if (!disp) continue; + const set = emitsByEvent.get(m[1]!) ?? new Set(); + set.add(disp.id); emitsByEvent.set(m[1]!, set); + } + } + if (hasOn) { + ON_RE.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = ON_RE.exec(content))) { + const handlerName = m[2] || m[3]; + if (!handlerName) continue; + const handler = ctx.getNodesByName(handlerName).find((n) => n.kind === 'function' || n.kind === 'method'); + if (!handler) continue; + const map = handlersByEvent.get(m[1]!) ?? new Map(); + map.set(handler.id, `${file}:${lineOf(m.index)}`); handlersByEvent.set(m[1]!, map); + } + } + } + + const edges: Edge[] = []; + const seen = new Set(); + for (const [event, dispatchers] of emitsByEvent) { + const handlers = handlersByEvent.get(event); + if (!handlers) continue; + // Precision guard: a generic event name with many handlers/dispatchers can't + // be matched without receiver-type info (Phase 3) — skip rather than over-link. + if (dispatchers.size > EVENT_FANOUT_CAP || handlers.size > EVENT_FANOUT_CAP) continue; + for (const d of dispatchers) for (const [h, registeredAt] of handlers) { + if (d === h) continue; + const key = `${d}>${h}`; + if (seen.has(key)) continue; + seen.add(key); + edges.push({ source: d, target: h, kind: 'calls', provenance: 'heuristic', metadata: { synthesizedBy: 'event-emitter', event, registeredAt } }); + } + } + return edges; +} + +/** + * Phase 4: React class-component re-render. `this.setState(...)` re-runs the + * component's `render()`, but that hop is React-internal — no static edge — so a + * flow like "mutation → setState → canvas repaint" dead-ends at setState even + * though `render → getRenderableElements → …` is fully call-connected after it. + * Bridge it: for each class that has a `render` method, link every sibling method + * whose body calls `this.setState(` → `render`. The setState gate keeps this to + * React class components (a non-React class with a `render` method won't call + * `this.setState`). Over-approximation (all setState methods reach render) is + * accepted — it's reachability-correct, like the callback channels. + */ +function reactRenderEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] { + const edges: Edge[] = []; + const seen = new Set(); + for (const cls of queries.getNodesByKind('class')) { + const children = queries.getOutgoingEdges(cls.id, ['contains']) + .map((e) => queries.getNodeById(e.target)) + .filter((n): n is Node => !!n && n.kind === 'method'); + const render = children.find((n) => n.name === 'render'); + if (!render) continue; + let added = 0; + for (const m of children) { + if (added >= MAX_CALLBACKS_PER_CHANNEL) break; + if (m.id === render.id) continue; + const content = ctx.readFile(m.filePath); + const src = content && sliceLines(content, m.startLine, m.endLine); + if (!src || !SETSTATE_RE.test(src)) continue; + const key = `${m.id}>${render.id}`; + if (seen.has(key)) continue; + seen.add(key); + edges.push({ + source: m.id, target: render.id, kind: 'calls', line: m.startLine, + provenance: 'heuristic', + metadata: { synthesizedBy: 'react-render', via: 'setState', registeredAt: `${render.filePath}:${render.startLine}` }, + }); + added++; + } + } + return edges; +} + +/** + * Phase 4b: Flutter setState → build (the Dart analog of react-render). In a + * StatefulWidget's State class, `setState(() {…})` re-runs `build(context)`, but + * that hop is framework-internal (Flutter calls build), so a flow like + * "onPressed → _increment → setState → rebuilt UI" dead-ends at setState. Bridge + * it: for each Dart class with a `build` method, link every sibling method whose + * body calls `setState(` → `build`. The setState gate + `.dart` file keep this to + * Flutter State classes. Over-approximation accepted (reachability-correct). + */ +function flutterBuildEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] { + const edges: Edge[] = []; + const seen = new Set(); + for (const cls of queries.getNodesByKind('class')) { + const children = queries.getOutgoingEdges(cls.id, ['contains']) + .map((e) => queries.getNodeById(e.target)) + .filter((n): n is Node => !!n && n.kind === 'method'); + const build = children.find((n) => n.name === 'build'); + if (!build || !build.filePath.endsWith('.dart')) continue; + let added = 0; + for (const m of children) { + if (added >= MAX_CALLBACKS_PER_CHANNEL) break; + if (m.id === build.id) continue; + const content = ctx.readFile(m.filePath); + const src = content && sliceLines(content, m.startLine, m.endLine); + if (!src || !FLUTTER_SETSTATE_RE.test(src)) continue; + const key = `${m.id}>${build.id}`; + if (seen.has(key)) continue; + seen.add(key); + edges.push({ + source: m.id, target: build.id, kind: 'calls', line: m.startLine, + provenance: 'heuristic', + metadata: { synthesizedBy: 'flutter-build', via: 'setState', registeredAt: `${build.filePath}:${build.startLine}` }, + }); + added++; + } + } + return edges; +} + +/** + * Phase 4c: C++ virtual override. A call through a base/interface pointer + * (`db->Get(...)`, `iter->Next()`) dispatches at runtime to a subclass override, + * but that hop is a vtable indirection — no static call edge — so a flow stops at + * the abstract base method. Bridge it like react-render: for each C++ class that + * `extends` a base, link each base method → the subclass method of the same name + * (the override), so trace/callees from the interface method reach the + * implementation(s). Over-approximation accepted (reachability-correct); capped + * per class and gated to C++ to avoid touching other languages' dispatch. + */ +function cppOverrideEdges(queries: QueryBuilder): Edge[] { + const edges: Edge[] = []; + const seen = new Set(); + const methodsOf = (classId: string): Node[] => + queries + .getOutgoingEdges(classId, ['contains']) + .map((e) => queries.getNodeById(e.target)) + .filter((n): n is Node => !!n && n.kind === 'method'); + for (const cls of queries.getNodesByKind('class')) { + const subMethods = methodsOf(cls.id).filter((n) => n.language === 'cpp'); + if (subMethods.length === 0) continue; + for (const ext of queries.getOutgoingEdges(cls.id, ['extends'])) { + const base = queries.getNodeById(ext.target); + if (!base || base.language !== 'cpp' || base.id === cls.id) continue; + const baseMethods = new Map(methodsOf(base.id).map((m) => [m.name, m])); + let added = 0; + for (const m of subMethods) { + if (added >= MAX_CALLBACKS_PER_CHANNEL) break; + const bm = baseMethods.get(m.name); + if (!bm || bm.id === m.id) continue; + const key = `${bm.id}>${m.id}`; + if (seen.has(key)) continue; + seen.add(key); + edges.push({ + source: bm.id, + target: m.id, + kind: 'calls', + line: bm.startLine, + provenance: 'heuristic', + metadata: { synthesizedBy: 'cpp-override', via: m.name, registeredAt: `${m.filePath}:${m.startLine}` }, + }); + added++; + } + } + } + return edges; +} + +/** + * Phase 5.5: interface / abstract dispatch (Java, Kotlin). A call through an + * injected interface (`@Autowired FooService svc; svc.list()`) or an abstract + * base dispatches at runtime to the implementing class's override — a vtable + * indirection with no static call edge — so a request→service flow stops at the + * interface method. Bridge it like cpp-override: for each class that + * `implements` an interface (or `extends` an abstract base), link each + * base/interface method → the class's same-name method (the override) so + * trace/callees reach the implementation. Over-approximation accepted + * (reachability-correct); capped per class, gated to JVM languages. + */ +// Languages whose static `implements`/`extends` edges should bridge an +// interface (or abstract base) method to the matching concrete-class method. +// The set is "languages with explicit nominal subtyping and a single class +// kind that holds methods" — i.e. the shape this loop expects. Swift and +// Scala fit shape-wise (Swift `protocol`/`class`, Scala `trait`/`class`) +// and are added below; their concrete-side nodes can be a `struct` (Swift) +// or an `object` (Scala) so the loop also iterates those kinds. +const IFACE_OVERRIDE_LANGS = new Set([ + 'java', 'kotlin', 'csharp', 'typescript', 'javascript', 'swift', 'scala', +]); +function interfaceOverrideEdges(queries: QueryBuilder): Edge[] { + const edges: Edge[] = []; + const seen = new Set(); + const methodsOf = (classId: string): Node[] => + queries + .getOutgoingEdges(classId, ['contains']) + .map((e) => queries.getNodeById(e.target)) + .filter((n): n is Node => !!n && n.kind === 'method'); + // Concrete-side kinds vary by language: `class` covers Java / Kotlin / + // C# / TS / Swift-classes / Scala-classes; `struct` covers Swift value + // types that conform to protocols. Iterate both. + const concreteKinds = ['class', 'struct'] as const; + for (const kind of concreteKinds) { + for (const cls of queries.getNodesByKind(kind)) { + const implMethods = methodsOf(cls.id).filter((n) => IFACE_OVERRIDE_LANGS.has(n.language)); + if (implMethods.length === 0) continue; + for (const sup of queries.getOutgoingEdges(cls.id, ['implements', 'extends'])) { + const base = queries.getNodeById(sup.target); + if (!base || !IFACE_OVERRIDE_LANGS.has(base.language) || base.id === cls.id) continue; + // Group impl methods by name to handle OVERLOADS: an interface `list()` and + // `list(params)` are distinct nodes and a call may resolve to either, so + // link every base overload → every same-name impl overload (keying by name + // alone would drop all but one and miss the resolved overload). + const implByName = new Map(); + for (const m of implMethods) { + const arr = implByName.get(m.name); + if (arr) arr.push(m); else implByName.set(m.name, [m]); + } + let added = 0; + for (const bm of methodsOf(base.id)) { + if (added >= MAX_CALLBACKS_PER_CHANNEL) break; + for (const m of implByName.get(bm.name) ?? []) { + if (added >= MAX_CALLBACKS_PER_CHANNEL) break; + if (bm.id === m.id) continue; + const key = `${bm.id}>${m.id}`; + if (seen.has(key)) continue; + seen.add(key); + edges.push({ + source: bm.id, + target: m.id, + kind: 'calls', + line: bm.startLine, + provenance: 'heuristic', + metadata: { synthesizedBy: 'interface-impl', via: m.name, registeredAt: `${m.filePath}:${m.startLine}` }, + }); + added++; + } + } + } + } + } + return edges; +} + +/** + * Go gRPC stub → impl bridge. The protoc-gen-go-grpc codegen emits an + * `UnimplementedXxxServer` struct in `*_grpc.pb.go` carrying one method + * per service RPC; the real handler is a hand-written struct in another + * file (`x/bank/keeper/msg_server.go::msgServer.Send` in cosmos-sdk). + * Go's structural typing means no `implements` edge exists for our + * resolver to follow, so `trace("Send","SendCoins")` lands on the + * empty stub and reports "no path" (validated empirically — the cosmos + * Q1 r1 trace failure that drove this work). + * + * Bridge: for each `UnimplementedXxxServer` whose RPC-method names are + * a SUBSET of some other Go struct's method names, emit `calls` edges + * `stub.method → impl.method` (paired by name). Excludes the gRPC + * internal markers `mustEmbedUnimplementedXxxServer` and + * `testEmbeddedByValue`, and skips candidate impls that themselves + * live in a generated file (their `xxxClient` / sibling stubs would + * otherwise look like impls). + * + * Multiple candidates is allowed and capped at MAX_CALLBACKS_PER_CHANNEL — + * a service often has both a production impl and one or more test + * mocks; linking to all preserves trace utility without false-favoring. + * + * Provenance: `heuristic`, `synthesizedBy: 'go-grpc-stub-impl'`. The + * stub's source line is the wiring site shown in the trace trail. + */ +function goGrpcStubImplEdges(queries: QueryBuilder): Edge[] { + const edges: Edge[] = []; + const seen = new Set(); + + const STUB_RE = /^Unimplemented.*Server$/; + // gRPC internal-helper methods that appear on every Unimplemented*Server; + // not part of the service contract, so exclude when computing the RPC-method + // signature used to match impls. + const isInternalMarker = (n: string) => n.startsWith('mustEmbed') || n === 'testEmbeddedByValue'; + + // Methods directly contained by each Go struct, name-only. Built once. + const methodNamesByStruct = new Map>(); + const methodNodesByStruct = new Map(); + const goStructs: Node[] = []; + for (const s of queries.getNodesByKind('struct')) { + if (s.language !== 'go') continue; + goStructs.push(s); + const ms = queries + .getOutgoingEdges(s.id, ['contains']) + .map((e) => queries.getNodeById(e.target)) + .filter((n): n is Node => !!n && n.kind === 'method'); + methodNodesByStruct.set(s.id, ms); + methodNamesByStruct.set(s.id, new Set(ms.map((m) => m.name))); + } + + for (const stub of goStructs) { + if (!STUB_RE.test(stub.name)) continue; + // The stub MUST live in a generated file — that's what tells us this is + // a protoc-emitted scaffold rather than someone naming a struct + // `UnimplementedXxxServer` by hand. Without this gate we'd also bridge + // such hand-written structs and create misleading edges. + if (!isGeneratedFile(stub.filePath)) continue; + + const stubMethods = (methodNodesByStruct.get(stub.id) ?? []).filter( + (m) => !isInternalMarker(m.name), + ); + if (stubMethods.length === 0) continue; + const stubMethodNames = stubMethods.map((m) => m.name); + + for (const cand of goStructs) { + if (cand.id === stub.id) continue; + // Skip generated-file candidates — they're siblings (msgClient, + // UnsafeMsgServer, …) whose method sets coincidentally match. + if (isGeneratedFile(cand.filePath)) continue; + + const candNames = methodNamesByStruct.get(cand.id); + if (!candNames) continue; + // Subset: every RPC method must exist on the candidate by name. + // Signature-level match would tighten this further, but name-match + // alone already gives one-to-one pairing in real codebases because + // gRPC method-name sets are highly distinctive (Send + MultiSend + + // UpdateParams + SetSendEnabled is unique to bank's MsgServer). + if (!stubMethodNames.every((n) => candNames.has(n))) continue; + + const candMethods = methodNodesByStruct.get(cand.id) ?? []; + let added = 0; + for (const sm of stubMethods) { + if (added >= MAX_CALLBACKS_PER_CHANNEL) break; + for (const cm of candMethods) { + if (added >= MAX_CALLBACKS_PER_CHANNEL) break; + if (cm.name !== sm.name) continue; + const key = `${sm.id}>${cm.id}`; + if (seen.has(key)) continue; + seen.add(key); + edges.push({ + source: sm.id, + target: cm.id, + kind: 'calls', + line: sm.startLine, + provenance: 'heuristic', + metadata: { + synthesizedBy: 'go-grpc-stub-impl', + via: cm.name, + registeredAt: `${cm.filePath}:${cm.startLine}`, + }, + }); + added++; + } + } + } + } + return edges; +} + +/** + * Phase 5: React JSX child rendering. A component that returns `` + * mounts Child — React calls it — but JSX instantiation isn't a static call edge, + * so a render tree (App.render → StaticCanvas → renderStaticScene) breaks at the + * JSX hop. Link parent → each capitalized JSX child it renders. File-oriented + * (read each JSX file once). Precision gate: the child name must resolve to a + * component/function/class node — TS generics like `Array` resolve to a type + * (or nothing) and are dropped. + */ +function reactJsxChildEdges(ctx: ResolutionContext): Edge[] { + const edges: Edge[] = []; + const seen = new Set(); + const PARENT_KINDS = new Set(['method', 'function', 'component']); + for (const file of ctx.getAllFiles()) { + const content = ctx.readFile(file); + if (!content || (!content.includes(''))) continue; // JSX-file gate + const parents = ctx.getNodesInFile(file).filter((n) => PARENT_KINDS.has(n.kind)); + for (const parent of parents) { + const src = sliceLines(content, parent.startLine, parent.endLine); + if (!src || (!src.includes(''))) continue; + const names = new Set(); + JSX_TAG_RE.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = JSX_TAG_RE.exec(src))) names.add(m[1]!); + let added = 0; + for (const name of names) { + if (added >= MAX_JSX_CHILDREN) break; + const child = ctx.getNodesByName(name).find( + (n) => n.kind === 'component' || n.kind === 'function' || n.kind === 'class' + ); + if (!child || child.id === parent.id) continue; + const key = `${parent.id}>${child.id}`; + if (seen.has(key)) continue; + seen.add(key); + edges.push({ + source: parent.id, target: child.id, kind: 'calls', line: parent.startLine, + provenance: 'heuristic', + metadata: { synthesizedBy: 'jsx-render', via: name }, + }); + added++; + } + } + } + return edges; +} + +/** + * Phase 6: Vue SFC templates. The `.vue` extractor only parses `