Skip to content

[codex] Parallelize incremental parse step - #1052

Draft
Extrodox wants to merge 1 commit into
CodeGraphContext:mainfrom
Extrodox:codex/incremental-index-parallel-parse
Draft

[codex] Parallelize incremental parse step#1052
Extrodox wants to merge 1 commit into
CodeGraphContext:mainfrom
Extrodox:codex/incremental-index-parallel-parse

Conversation

@Extrodox

@Extrodox Extrodox commented Jun 2, 2026

Copy link
Copy Markdown

Summary

  • Reuse the changed file parse result from update_file_in_graph() instead of parsing the same file again during incremental relinking.
  • Parse remaining affected files with a bounded ThreadPoolExecutor driven by PARALLEL_WORKERS.
  • Keep graph writes and relationship relinking serialized to avoid backend lock contention.
  • Add regression coverage for the worker-count config and changed-file parse reuse.

Why

Incremental indexing was effectively serial for affected-file parsing, and the changed file could be parsed twice in one update pass. This makes large affected sets slower than necessary even when parsing work is CPU-bound and independent per file.

Validation

  • PYTHONPATH=src /tmp/cgc-pr-venv/bin/python -m pytest tests/unit/tools/test_graph_builder_perf_fixes.py::TestWatcherIncrementalHandleModification::test_incremental_parse_workers_uses_parallel_workers_config tests/unit/tools/test_graph_builder_perf_fixes.py::TestWatcherIncrementalHandleModification::test_handle_modification_reuses_changed_file_parse_result -q
  • PYTHONPATH=src /tmp/cgc-pr-venv/bin/python -m compileall -q src/codegraphcontext/core/watcher.py

Note: the default Homebrew Python in this shell is externally managed and lacked project dependencies, so validation used a temporary venv under /tmp/cgc-pr-venv.

@vercel

vercel Bot commented Jun 2, 2026

Copy link
Copy Markdown

@Extrodox is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

@Shashankss1205

Copy link
Copy Markdown
Collaborator

Overlaps merged #1071 watcher sync and conflicts with current watcher.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog tasks

Development

Successfully merging this pull request may close these issues.

2 participants