Skip to content

Commit 2d72891

Browse files
colbymchenryclaude
andauthored
feat(kernel): R7a C/C++ walker — dual-lang ccpp module, preParse hoist, 7 new blanks, c/cpp default-routed (colbymchenry#1346)
Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps byte-identical on all five + linux at kernel scale (10.4M dump lines, same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x). Deferral guard corrected by measurement (C/C++ error incidence 9-42%; --max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse cost deferred files paid. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 44561b6 commit 2d72891

20 files changed

Lines changed: 3212 additions & 81 deletions

CHANGELOG.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
1111

1212
### New Features
1313

14-
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, and Go projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, and django-scale codebases (Lombok-generated members included). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
14+
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, and C++ projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, django-, git-, and protobuf-scale codebases (Lombok-generated members, C function-pointer tables, and Unreal-Engine-style macro-heavy headers included; CUDA and Metal sources ride the C++ path). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
1515
- Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=<count>` to tune when it engages.
1616
- Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle.
1717
- The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical.
@@ -34,6 +34,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
3434
- Deleting a whole directory is now picked up by watch mode: the files inside it are removed from the index on the next auto-sync instead of lingering as stale records until an unrelated edit happened to trigger one. Operating systems often report a directory deletion as a single event on the directory itself (with no per-file events for its contents), which the watcher previously discarded. (#1285)
3535
- `codegraph sync` now gets the same slow-disk fix that made full indexing fast in 1.4.0: database checkpointing is deferred for the whole incremental run instead of firing every few megabytes of writes. On mechanical drives and other high-latency storage, a small sync on a large index no longer stalls for minutes at near-zero CPU — the cost of a sync scales with what changed, not with the size of the existing index. The same `CODEGRAPH_NO_WAL_DEFER=1` switch turns it off. (#1248)
3636
- C functions declared with a project-specific attribute macro in front of a typedef'd return type (`SEC_ATTR UINT32 MyFunc(VOID)` — common in embedded and kernel code) are now indexed under their real names. Previously the parser tripped over the unknown macro and stored the parameter list as the function name, leaving entries like `"(VOID)"` in the graph and making the real function unfindable. (#1211)
37+
- Macro-heavy C and C++ code indexes much more completely. Six ubiquitous idioms that previously tripped the parser into error recovery — dropping or garbling the surrounding symbols — now parse cleanly: the `#ifdef __cplusplus` / `extern "C" {` compatibility guard in C headers, iterator macros in statement position (`list_for_each_entry(pos, head, member) { … }` and the whole Linux-kernel/git/jemalloc family, braced or single-statement), the Linux/sparse declaration annotations (`static int __init foo(void)`, `void __user *buf`, `container_of(p, struct T, m)`), trailing parameter annotations (`int argc UNUSED`, git's house style), namespace-management macros alone on a line (`FMT_BEGIN_NAMESPACE`, Qt's `Q_OBJECT`), and function attribute macros in front of C++ return types. On git's own repository this nearly doubles the number of indexed symbols, and on the Linux kernel's `kernel/` and `mm/` directories it triples them; blast radius and callers get correspondingly more complete. A related fix stops the existing macro handling from corrupting `#define` lines that mention the same macro names, which removed a class of phantom parse errors in fmt-style headers.
3738
- C++ methods defined out-of-line inside a namespace (`namespace sim { Output MyClass::Apply(...) { ... } }`) now carry the namespace in their qualified name, matching their class. Fully-qualified call sites from other files (`sim::MyClass::Apply(...)`) resolve to the definition again, so `codegraph callers` and file impact no longer come up empty for this pattern. (#1291)
3839
- C++ methods defined out-of-line on a template class (`template <typename T> T Box<T>::get() { ... }`) no longer keep the template parameter list in their qualified name. They now index as `Box::get` — identical to an inline definition of the same method — so they link to their class and resolve from call sites again, and pathological multi-line template parameter lists can no longer blow the qualified name past filesystem name limits. (#1286)
3940
- Go route detection no longer misidentifies ordinary method calls that share HTTP verb names — `cache.Put("key", value)`, `store.Get("config", out)`, `bus.Handle("user.created", handler)` and the like were being indexed as HTTP routes, polluting route listings in cache-heavy codebases. A registration now has to look like one: its first argument must be a `/`-prefixed path (all routers) or a Go 1.22 `"METHOD /path"` pattern on `Handle`/`HandleFunc`, which now also extracts the method instead of listing the route as `ANY`. (#1259)

__tests__/extraction.test.ts

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11162,3 +11162,139 @@ import DataStore from '../data/DataStore';
1116211162
});
1116311163
});
1116411164
});
11165+
11166+
// R7a preParse additions — the blanking passes added so macro-heavy C/C++
11167+
// parses clean enough for the kernel route (each also improves the wasm
11168+
// path's own graphs). Offset preservation is load-bearing everywhere.
11169+
describe('C/C++ kernel-port preParse blanks (R7a)', () => {
11170+
it('blankCCplusplusGuardBodies blanks extern-C guard bodies, keeps directives', async () => {
11171+
const { blankCCplusplusGuardBodies } = await import('../src/extraction/languages/c-cpp');
11172+
const src = [
11173+
'#ifdef __cplusplus',
11174+
'extern "C" {',
11175+
'#endif',
11176+
'int real_decl(void);',
11177+
'#ifdef __cplusplus',
11178+
'}',
11179+
'#endif',
11180+
'',
11181+
].join('\n');
11182+
const out = blankCCplusplusGuardBodies(src);
11183+
expect(out.length).toBe(src.length);
11184+
expect(out).not.toContain('extern "C"');
11185+
expect(out).toContain('#ifdef __cplusplus'); // directives stay
11186+
expect(out).toContain('int real_decl(void);');
11187+
// A guard with a nested directive bails (needs real preprocessing).
11188+
const nested = [
11189+
'#ifdef __cplusplus',
11190+
'#define EXTERNC extern "C"',
11191+
'#endif',
11192+
'',
11193+
].join('\n');
11194+
expect(blankCCplusplusGuardBodies(nested)).toBe(nested);
11195+
// The `#ifndef` inverse guard is C-visible and must be untouched.
11196+
const inverse = ['#ifndef __cplusplus', 'int c_only(void);', '#endif', ''].join('\n');
11197+
expect(blankCCplusplusGuardBodies(inverse)).toBe(inverse);
11198+
});
11199+
11200+
it('blankLoneMacroLines blanks namespace-management macros, spares expression operands', async () => {
11201+
const { blankLoneMacroLines } = await import('../src/extraction/languages/c-cpp');
11202+
const src = ['FMT_BEGIN_NAMESPACE', 'struct S { int x; };', 'FMT_END_NAMESPACE', ''].join('\n');
11203+
const out = blankLoneMacroLines(src);
11204+
expect(out.length).toBe(src.length);
11205+
expect(out).not.toContain('FMT_BEGIN_NAMESPACE');
11206+
expect(out).toContain('struct S { int x; };');
11207+
// An ALL-CAPS operand alone on a line inside a multi-line expression is
11208+
// NOT a lone macro — the next line starts with an operator.
11209+
const expr = ['int x = 0', ' | FLAG_ONE', ' | FLAG_TWO;', ''].join('\n');
11210+
expect(blankLoneMacroLines(expr)).toBe(expr);
11211+
const cont = ['int y =', 'SOME_FLAG', '| OTHER;', ''].join('\n');
11212+
expect(blankLoneMacroLines(cont)).toBe(cont);
11213+
// Underscore-free solid words are too risky and stay.
11214+
const bare = ['NDEBUG', 'int z;', ''].join('\n');
11215+
expect(blankLoneMacroLines(bare)).toBe(bare);
11216+
});
11217+
11218+
it('blankCStatementMacroCalls blanks indented iterator macros, keeps the block', async () => {
11219+
const { blankCStatementMacroCalls } = await import('../src/extraction/languages/c-cpp');
11220+
const src = [
11221+
'static void walk(struct list *head) {',
11222+
'\tlist_for_each_entry(pos, head, member) {',
11223+
'\t\tuse(pos);',
11224+
'\t}',
11225+
'}',
11226+
'',
11227+
].join('\n');
11228+
const out = blankCStatementMacroCalls(src);
11229+
expect(out.length).toBe(src.length);
11230+
expect(out).not.toContain('list_for_each_entry');
11231+
expect(out).toContain('use(pos);');
11232+
// A real call statement ends with `;` — untouched.
11233+
expect(out).toContain('use(pos);');
11234+
const call = ['void f(void) {', '\tdo_thing(a, b);', '}', ''].join('\n');
11235+
expect(blankCStatementMacroCalls(call)).toBe(call);
11236+
// Column-0 `name(args) {` is an implicit-int function definition — untouched.
11237+
const kandr = ['main(argc, argv)', '{', '\treturn 0;', '}', ''].join('\n');
11238+
expect(blankCStatementMacroCalls(kandr)).toBe(kandr);
11239+
// Control-flow keywords are never macros.
11240+
const ctrl = ['void g(int x) {', '\twhile (x) {', '\t\tx--;', '\t}', '}', ''].join('\n');
11241+
expect(blankCStatementMacroCalls(ctrl)).toBe(ctrl);
11242+
});
11243+
11244+
it('blankCTrailingParamAttrMacros blanks `name UNUSED` params, spares call args', async () => {
11245+
const { blankCTrailingParamAttrMacros } = await import('../src/extraction/languages/c-cpp');
11246+
const src = 'static int run(int argc UNUSED, const char **argv UNUSED)\n{\n\treturn 0;\n}\n';
11247+
const out = blankCTrailingParamAttrMacros(src);
11248+
expect(out.length).toBe(src.length);
11249+
expect(out).not.toContain('UNUSED');
11250+
expect(out).toContain('int argc ');
11251+
// A macro CONSTANT as a call argument is preceded by `,`/`(`, never by a
11252+
// bare identifier — untouched.
11253+
const call = 'void f(void) {\n\tconnect(sock, DEFAULT_TIMEOUT);\n}\n';
11254+
expect(blankCTrailingParamAttrMacros(call)).toBe(call);
11255+
});
11256+
11257+
it('blankCKernelAnnotations blanks sparse/section dunders, spares parameterized ones and real types', async () => {
11258+
const { blankCKernelAnnotations } = await import('../src/extraction/languages/c-cpp');
11259+
const src = [
11260+
'static int __init audit_init(void) { return 0; }',
11261+
'void copy(void __user *dst, const char *src);',
11262+
'__bpf_kfunc void bpf_iter_destroy(struct bpf_iter_num *it);',
11263+
'__printf(1, 2) void log_fmt(const char *fmt, ...);',
11264+
'struct e *entry = container_of(r, struct audit_entry, rule);',
11265+
'__u32 count = 0;',
11266+
'',
11267+
].join('\n');
11268+
const out = blankCKernelAnnotations(src);
11269+
expect(out.length).toBe(src.length);
11270+
expect(out).not.toContain('__init');
11271+
expect(out).not.toContain('__user');
11272+
expect(out).not.toContain('__bpf_kfunc');
11273+
// Parameterized annotations keep their name — blanking it would strand
11274+
// the argument list as a floating parenthesis.
11275+
expect(out).toContain('__printf(1, 2)');
11276+
// container_of's type-keyword argument blanks; other `struct` keywords stay.
11277+
expect(out).toContain('container_of(r, audit_entry, rule)');
11278+
expect(out).toContain('struct e *entry');
11279+
// Real dunder TYPES are not annotations.
11280+
expect(out).toContain('__u32 count');
11281+
});
11282+
11283+
it('restoreDirectiveLines keeps #define lines out of the blanking blast radius', async () => {
11284+
const { extractFromSource } = await import('../src/extraction');
11285+
// FMT_API matches the _API-suffix member blank; without the directive
11286+
// restore the #define loses its NAME and the file gains a parse error.
11287+
const src = [
11288+
'#define FMT_API FMT_VISIBILITY("default")',
11289+
'class Widget {',
11290+
' public:',
11291+
' int size() const { return 1; }',
11292+
'};',
11293+
'',
11294+
].join('\n');
11295+
const result = extractFromSource('lib.hpp', src, 'cpp');
11296+
expect(result.errors).toEqual([]);
11297+
expect(result.nodes.some((n) => n.kind === 'class' && n.name === 'Widget')).toBe(true);
11298+
expect(result.nodes.some((n) => n.kind === 'method' && n.name === 'size')).toBe(true);
11299+
});
11300+
});
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
/* Torture fixture for the C kernel walker (R7a) — exercises every c-path
2+
* branch of the checklist: fn-ptr tables, typedef enum/struct, file-scope
3+
* consts incl. multi-declarator, the macro-prototype misparse skip,
4+
* value-refs (+ shadow prune), the leading-attr-macro preParse blank, and
5+
* the C call shapes. Must parse ERROR-FREE post-preParse or the kernel arm
6+
* defers. */
7+
#include <stdio.h>
8+
#include <sys/socket.h>
9+
#include "local_ops.h"
10+
11+
/** Retry budget for the poller. */
12+
static const int MAX_RETRIES = 3;
13+
static const int LOW_WATER = 2, HIGH_WATER = 8;
14+
static int counter = 0;
15+
static const char *BANNER = "torture";
16+
17+
/* Bare identifier declarators are the macro-prototype misparse shape and are
18+
* skipped by design (uninit scalars are the accepted loss). */
19+
int bare_global;
20+
MYLIB_API config_handle;
21+
22+
/* Leading attribute macro — blanked by preParseCSource (#1211), so the real
23+
* name survives on both arms. */
24+
SEC_ATTR UINT32 masked_entry(VOID) { return 0; }
25+
26+
typedef enum { STATE_IDLE, STATE_RUNNING, STATE_DONE } run_state_t;
27+
28+
typedef struct {
29+
int fd;
30+
void (*on_recv)(int);
31+
} conn_t;
32+
33+
typedef struct conn_pool conn_pool_t;
34+
35+
typedef int (*cb_t)(int);
36+
37+
enum wire_flags { WIRE_A = 1, WIRE_B = 2 };
38+
39+
struct packet {
40+
int len;
41+
unsigned char body[64];
42+
struct packet *next;
43+
cb_t *async_cb;
44+
};
45+
46+
/** Sum helper (docstring). */
47+
static int add(int a, int b) { return a + b; }
48+
49+
static int cb_a(int x) { return add(x, 1); }
50+
static int cb_b(int x) { return add(x, 2); }
51+
52+
/* fn-ptr table at file scope — the ungated 'list' capture positions. */
53+
static cb_t DISPATCH_TABLE[] = { cb_a, cb_b };
54+
55+
static void handle_recv(int fd);
56+
57+
/* struct initializer — the ungated 'value' capture positions. */
58+
static const struct handler_ops OPS = { .recv = handle_recv, .flags = WIRE_A };
59+
60+
static void handle_recv(int fd) {
61+
struct packet pkt;
62+
pkt.len = fd;
63+
printf("fd=%d retries=%d\n", fd, MAX_RETRIES);
64+
}
65+
66+
/* Local shadow of a file-scope const — the shadow prune drops HIGH_WATER as a
67+
* value-ref target while LOW_WATER stays live. */
68+
static int shadowed_reader(void) {
69+
int HIGH_WATER = 99;
70+
return HIGH_WATER + LOW_WATER;
71+
}
72+
73+
static int use_table(int idx, int v) {
74+
cb_t fn = DISPATCH_TABLE[idx];
75+
int r = (*fn)(v);
76+
conn_t c = { 1, 0 };
77+
c.on_recv(r);
78+
return counter + r;
79+
}
80+
81+
static void spawn_workers(void) {
82+
register_handler(cb_a);
83+
signal_connect(&cb_b);
84+
}
Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
/// Torture fixture for the C++ kernel walker (R7a) — namespaces (incl. C++17
2+
/// nested + anonymous), out-of-line Cls::method defs, templates + template
3+
/// bases, operator definitions, stack construction, local fn-ptrs, UE-macro
4+
/// shapes THROUGH the hoisted preParse, using-aliases, access specifiers,
5+
/// static-member value reads, and the cpp call shapes. Must parse ERROR-FREE
6+
/// post-preParse or the kernel arm defers (spaced operator CALL SITES live in
7+
/// torture-defer.cpp — they produce ERROR nodes by design).
8+
#include <vector>
9+
#include "widget_base.hpp"
10+
11+
namespace app {
12+
13+
/** Engine config (docstring). */
14+
class Config {
15+
public:
16+
int retries;
17+
void apply();
18+
int helper_count() const { return 2; }
19+
20+
private:
21+
int secret;
22+
};
23+
24+
void Config::apply() { retries = helper_count(); }
25+
26+
namespace detail {
27+
struct Counter {
28+
int value;
29+
Counter *next;
30+
};
31+
} // namespace detail
32+
33+
int detail_probe() { return 1; }
34+
35+
} // namespace app
36+
37+
namespace app::net {
38+
class Session {
39+
public:
40+
void open();
41+
virtual ~Session() {}
42+
};
43+
void Session::open() {}
44+
} // namespace app::net
45+
46+
namespace {
47+
int hidden_helper() { return 3; }
48+
} // namespace
49+
50+
template <typename T>
51+
class Base {
52+
public:
53+
T item;
54+
};
55+
56+
template <typename T>
57+
class Box : public Base<T> {
58+
public:
59+
T get() const { return value_; }
60+
T unwrap();
61+
62+
private:
63+
T value_;
64+
};
65+
66+
template <typename T>
67+
T Box<T>::unwrap() {
68+
return value_;
69+
}
70+
71+
class Derived : public Base<int>, private app::Config {
72+
public:
73+
Derived() : total_(0) {}
74+
int total() const { return total_; }
75+
76+
private:
77+
int total_;
78+
};
79+
80+
struct Vec2 {
81+
float x, y;
82+
Vec2 operator+(const Vec2 &o) const { return {x + o.x, y + o.y}; }
83+
explicit operator bool() const { return x != 0 || y != 0; }
84+
Vec2 origin();
85+
};
86+
87+
enum class Mode : unsigned char { Off, On };
88+
enum Legacy { LEGACY_A, LEGACY_B };
89+
typedef struct {
90+
int id;
91+
} packet_t;
92+
using Handle = app::Config;
93+
94+
// UE-macro shapes — every one below is recovered by the hoisted preParse
95+
// (export macro, reflection markup, inline specifier, API member prefix).
96+
class MYMODULE_API Widget : public app::Config {
97+
public:
98+
UPROPERTY(EditAnywhere, Category = "State")
99+
float Health;
100+
FORCEINLINE float GetHealth() const { return Health; }
101+
ENGINE_API virtual void Tick(float Delta);
102+
};
103+
104+
void Widget::Tick(float Delta) { Health += Delta; }
105+
106+
Config GlobalConfig;
107+
int build_number = 7;
108+
109+
template <typename T>
110+
T compute_seed(T v) {
111+
return v + 1;
112+
}
113+
114+
float drive_helper(float v) { return v; }
115+
116+
Widget *make_widget() { return new Widget(); }
117+
118+
float drive() {
119+
Widget local;
120+
app::Config cfg;
121+
Vec2 a{1, 2};
122+
Vec2 b(a);
123+
Vec2 c2(1.5f, 2.5f);
124+
float f = a.x + b.y + c2.x;
125+
make_widget()->Tick(0.5f);
126+
auto kernel = &compute_seed<float>;
127+
if (f > 1) {
128+
kernel = &drive_helper;
129+
}
130+
float r = kernel(f);
131+
int flags = GlobalConfig.retries;
132+
Mode m = Mode::Off;
133+
int leg = LEGACY_A;
134+
app::detail_probe();
135+
compute_seed<int>(2);
136+
auto mp = &app::Config::apply;
137+
(void)mp;
138+
(void)m;
139+
return r + f + flags + leg;
140+
}

0 commit comments

Comments
 (0)