Add a knowledge indexing generator for an LLM-friendly docs index by deruyter92 · Pull Request #3453 · DeepLabCut/DeepLabCut

deruyter92 · 2026-08-18T14:47:07Z

Motivation
Recently the DeepLabCut documentation got an overhaul. This includes the recent launch of the api-docs. However for LLM agents, searching our documentation can still be a difficult task that easily fills the context window with irrelevant information.

The current PR adds a module tools/knowledge_indexing for generating a LLM-friendly index of the DeepLabCut API and user docs, and deploys it alongside the existing docs. This helps to point out AI agents where to retrieve relevant information.

Implementation

All indexing is deployed on our gh-pages website. This makes it discoverable for LLMs (without adding the contents in the TOC).
A single discovery point, llms.txt is generated at the site root following the the llmstxt.org convention and points at the structured index at knowledge/<version>/.. .
The index itself uses JSONL format: knowledge/<version>/{api,docs}.jsonl. It contains one JSON object per record (with a content_hash)
A per-version manifest.json and a top-level knowledge/manifest.json enumerates every version
Versioning is asymmetric: api.jsonl is versioned per dev-docs release, mirroring mike. The main user docs knowledge index always indexes the currently deployed user docs in docs.jsonl (the user docs have no versioning).
Dependencies: A dedicated knowledge-index extra (docutils, griffelib, markdown-it-py) is added to make the CI workflow lean (no other installs needed)

How to use
More information is added in tools/knowledge_indexing/README.md.
To locally build the full knowledge-index and llms.txt, run:

python -m tools.knowledge_indexing

The main knowledge-index is always deployed to gh-pages together with the user docs and dev-docs, triggered by updates on the main branch. To deploy a specific version label, just run the pre-existing manual dispatch workflow for versioned dev-docs API docs, this now automatically adds / removes the corresponding llm-index as well.

Status

Add generation tool for llm-friendly indexing of the docs + api-docs
Add CI workflow to auto-generate index for each new version / each deployed api-docs

C-Achard

Great work so far!
A few questions and some hopefully useful suggestions below, but nothing major besides the revision granularity

C-Achard

A few CI-related comments, really liking the new version!
I also started on the write.py side a bit.

Were you thinking of adding tests as for the other tools as well?

deruyter92 · 2026-09-04T13:24:53Z

@C-Achard, I'm implementing tests now and addressing your other review comments (most of it is implemented but I hadn't replied yet). I'll tag you for re-review when done

deruyter92 · 2026-09-04T14:21:11Z

@C-Achard, let me know what you think. I've addressed all your comments. I am quite happy with the current version. When we actually start using it ourselves with Amadeus, we might come up with further improvements (e.g. better content summarizing from docs).

C-Achard

Quite a few new finds; let me know how severe/actually import you think these are for our use

C-Achard · 2026-09-04T14:39:09Z

+    return Path(__file__).resolve().parents[3]
+
+
+@pytest.fixture(scope="session", autouse=True)


This is imported by pytest on collection, but the body only runs once it starts setting up tests. So the imports have already executed by then.

Adding

[tool.pytest.ini_options] pythonpath = ["."]

in the pyproject would allow to skip this fixture.

Or moving to the general tools conftest.py

(Note that in CI the pythonpath is already handled by the pytest -m call)

good point. This never ran before collection. I realized now that the same problem actually pre-existed for the test-selector, and was only working 'by coincidence' since python -m pytest adds cwd to path.

See #3484 for a fix to the pre-existing problem (namely, adding the root dir via pyproject.toml). For now, I've cherry-picked the fix from there, but I'd suggest we just merge that PR before this one.

(also removed the dead code that I had written in the current PR) thanks for spotting

see 48812d9 and e564394

C-Achard · 2026-09-04T14:49:55Z

+
+        anchor = _anchor(text)
+        seen[anchor] += 1
+        # Sphinx registers only the first occurrence of a repeated heading, so


This assumption is worth a regression check against the real docs build, since we have a lot of duplicate headings, so if it's wrong it mislinks a lot of sections silently.
The solution below may be complex so feel free to research alternatives or ignore + document it.

Concretely, docutils' set_id uses make_id(name) for the first occurrence and falls back to auto-generated id1, id2, … on collision. Maybe write a regression check and update based on the real behavior:

Build one minimal page with two identical headings using the real _config.yml

Read the generated HTML ids

Encode that observed behaviour in _anchor() / _read_structure

For the exact assertion, anchor and docs_url should ideally carry the observed published anchor. id needs to stay unique for _check_unique_ids, so derive it from the resolved anchor rather than suffixing the first occurrence's, and if the build does assign unique ids per heading, the seen Counter and the suffix are not needed anymore, since they only exist to work around the shared assumption.

One more thing that was flagged: _config.yml sets no myst_heading_anchors, which is why make_id is the right function, but that's an undocumented dependency on a jupyter-book default, and if anyone sets it later every anchor in docs.jsonl goes wrong at once. Same test covers it; a one-line comment in _anchor() naming that config would pin it.

This is an important comment. Indeed I checked and we mis-linked the duplicate headings to the first occurrence.. (and I can confirm we have a lot of duplicate headings).

I've tried a few approaches but it is difficult to reliably render the correct links from the markdown source (modelling docutils and sphinx behavior). So instead I now switched to building from the built HTML. The site is built anyway in the same workflow, so passing it as an artifact is not a big issue.

The code for extracting anchors etc is now changed a lot. And I added a regression test. See 1934cad

C-Achard · 2026-09-04T14:53:58Z

+    return records
+
+
+def _check_unique_ids(filename: str, ids: Iterable[str]) -> None:


Symbol and submodule IDs cannot collid, and _check_unique_ids safely detects that, but a technically valid package containing both a symbol and a same-named submodule (which is arguably bad practice but may occur) aborts the build. Should we disambiguate module and symbol IDs by type, or accept and explicitly document this as a limitation?

Good suggestion. Symbol IDs are now disambiguated using #

see 15ebb8d

C-Achard · 2026-09-04T14:59:43Z

+    has_docs = (knowledge_dir / docs_version_label / DOCS_FILE).is_file()
+    manifest = TopManifest(
+        docs_path=f"{docs_version_label}/{DOCS_FILE}" if has_docs else "",
+        api_latest=docs_version_label,


api_latest is always "main" here, so knowledge/manifest.json tells an agent that the latest API index is the rolling dev build, but manage-dev-docs.yml already has a notion of "latest stable": mark_latest_release passes latest-release to mike as an alias and as the /dev/ default. That alias never reaches the knowledge index as it's keyed on version_label alone, so an agent asking "what's the current release API?" gets main, whose api.jsonl URLs point at unreleased signatures.

"Known gaps" frames this as version-numbering work left for later, but the ordering problem is already solved upstream: mike is told which version is the release. Could deploy-knowledge-index.yml take aliases (as deploy-dev-docs-mike.yml already does) and record them per version, so api.latest resolves to whatever carries latest-release and falls back to main? That also makes knowledge/latest-release/... addressable without knowing the version number.

good suggestion. Implemented as you proposed! see 34fde37

C-Achard · 2026-09-04T15:01:45Z

+            continue
+
+        target, kind = member, _own_kind(member)
+        if kind not in ("function", "class"):


Does this retains methods signatures for classes?
Can we record one level into class members with kind: "method" (skipping private and inherited methods)?

Added. A class's own public methods are recorded as kind: "method" rows named Class.method, so the agent does not have to fetch our API docs to read the methods now. b3b3b74

C-Achard · 2026-09-04T15:22:34Z

Quite a few new finds; let me know how severe/actually import you think these are for our use

Some extras:

docs_index.py:361 — _split_frontmatter splits on the first two ---.
A --- inside the frontmatter, or a page whose body opens with a horizontal
rule, silently truncates everything before it and we index a partial page with
no error. Matching a closing delimiter at line start would be safer.
write.py:106 — shutil.rmtree(..., ignore_errors=True) while _delete
already checks is_dir() first. The flag can no longer mean "absent", only
"removal failed", and we then print "Deleted the ... API index".
test_write.py builds nodes with SimpleNamespace, so a field rename in
ApiNode / DocsPageNode won't be caught. Importing the real dataclasses only
costs the griffe/docutils import at module load.
The delete path in deploy-knowledge-index.yml installs the whole
knowledge-index group, including griffelib, which it never imports.

deruyter92 · 2026-09-07T15:27:15Z

Some extras:

docs_index.py:361 — _split_frontmatter splits on the first two ---.
A --- inside the frontmatter, or a page whose body opens with a horizontal
rule, silently truncates everything before it and we index a partial page with
no error. Matching a closing delimiter at line start would be safer.

write.py:106 — shutil.rmtree(..., ignore_errors=True) while _delete
already checks is_dir() first. The flag can no longer mean "absent", only
"removal failed", and we then print "Deleted the ... API index".

test_write.py builds nodes with SimpleNamespace, so a field rename in
ApiNode / DocsPageNode won't be caught. Importing the real dataclasses only
costs the griffe/docutils import at module load.

The delete path in deploy-knowledge-index.yml installs the whole
knowledge-index group, including griffelib, which it never imports.

Addressed now!

regex matching in _split_frontmatter is improved. 30cfa07, tests in e3ae6b9
shutil.rmtree -> ignore_errors=True is now dropped. Anything reaching there is a real failure and raises now. e7355d7
Using real dataclasses now instead of a fake replacement. They are moved to schemas.py alongside the published shapes to make importing cheap. f06cfd6, c27ecac
This is guarded by a conditional now. The delete branch installs pyyaml only, not the entire group. 9b4f223

C-Achard · 2026-09-08T08:32:10Z

@deruyter92 Are the CI failures related to rebasing?

C-Achard · 2026-09-08T08:32:25Z

Let me know if you'd like me to investigate

deruyter92 · 2026-09-08T11:31:31Z

@C-Achard, the problem was that I was trying too hard to avoid installing the new knowledge-index dependencies during test time. They are actually required now after the latest additions in tests/tools.

Meanwhile, I think I addressed all your comments. Let me know if anything else comes up!

unsafe free-text inputs are now read from the environment (as text).

the knowledge-index deletion logic did not work correctly with peaceiris when `keep_files` is True. Also: the pre-existing workflow allowed the deletion of "main" which always contains the most recent documentaion and is reset after every push to main. The current commit fixes this and prevents deleting "main" in the first place. Only published releases can be deleted, and "main" is always tracking the most recent documentation version. For the knowledge-index, only the API docs index can be removed, as the main user docs are always kept up to date with the latest "main" version.

For API-docs deployment, a checkout from a specific tag is used, but this was done AFTER the latest deeplabcut was already cloned. Since checkout only WRITES the new state of files, stale paths (non-existent in the new checkout) would still remain on disk, potentially causing mixed API to be reported.

…g checkout For API-docs deployment, a checkout from a specific tag is used, but this was done AFTER the latest deeplabcut was already cloned. Since checkout only WRITES the new state of files, stale paths (non-existent in the new checkout) would still remain on disk, potentially causing mixed API to be reported.

…ther than extra)

…s absent or cannot be removed

…add test delete version raises

…nly, not delete

Required for the docs indexer tests. It was already implicitly installed via huggingface-hub, so this installs nothing new. But added explicitly now so we don't rely on that chain.

Every H1 was consumed as a title candidate, so pages using H1 throughout lost those headings from the index entirely. Only the first claims the title; the rest become sections (701 -> 711 rows).

A non-mapping `deeplabcut:` block crashed on .get(); it now raises a message naming the file. A heading that slugs to nothing gets a digest id instead of being named `docs:page#`.

Anchors were derived from markdown with make_id(), but this did not reliably handle repeats. e.g. many repeated "Overview" headings were deep-linked to the wrong section, silently. --docs-html points at a built site and anchors are derived from its <section id> attributes, keyed by (heading text, occurrence). If no built site is available, it falls back to link the page instead of the section anchor. CI passes the artifact (which is already published anyway). A page missing from --docs-html fails the build. test_docs_anchors_regression.py builds a page with the repo's _config.yml and asserts every anchor lands on its own heading. It is skipped when no jupyter-book is available so it does not run in CI

manage-dev-docs already tells mike which version is latest-release, but that never reached the knowledge index. this commit threads the aliases through to the indexer.

Symbol ids were `api:<module>.<name>`, the same shape as a submodule's id, so a package publishing both a submodule `foo` and a symbol `foo` would abort the build on the duplicate-id check.

api.jsonl recorded modules, functions and classes, so an agent looking for a method signature had to open the class's reference page. Now we record a class's own public methods as `kind: "method"` rows named `Class.method`.

separate knowledge-index and dev dependency groups were designed to keep CI lean, but after expanding tests, running tests without knowledge-index is not feasible anyway so we're adding it - add `--group knowledge-index` in `intelligent-testing.yml` workflow (required to run all tests in tests/tools) - revert adding markdown-it-py in dev dependencies, it belongs in the knowledge-index group, which is installed now anyway during testing - revert the docutils lazy importing (which did not buy us anything)

C-Achard reviewed Aug 19, 2026

View reviewed changes

Comment thread tools/knowledge_indexing/__main__.py Resolved

C-Achard reviewed Aug 19, 2026

View reviewed changes

deruyter92 added this to the v3.1.0 milestone Aug 25, 2026

deruyter92 requested a review from C-Achard August 26, 2026 13:07

deruyter92 marked this pull request as ready for review August 26, 2026 13:07

C-Achard reviewed Aug 26, 2026

View reviewed changes

deruyter92 added the 3.1 label Aug 28, 2026

deruyter92 changed the base branch from main to dev August 31, 2026 09:43

deruyter92 modified the milestones: Bug fixes v3.1.0, Improve agentic interaction Aug 31, 2026

deruyter92 requested a review from C-Achard September 4, 2026 14:19

C-Achard reviewed Sep 4, 2026

View reviewed changes

deruyter92 added enhancement New feature or request 3.0.2 and removed 3.1 labels Sep 7, 2026

deruyter92 self-assigned this Sep 7, 2026

deruyter92 marked this pull request as draft September 7, 2026 07:12

deruyter92 requested a review from C-Achard September 7, 2026 15:27

deruyter92 marked this pull request as ready for review September 7, 2026 15:27

deruyter92 changed the base branch from dev to jaap/fix-tool-imports-pytest September 7, 2026 15:30

deruyter92 force-pushed the jaap/fix-tool-imports-pytest branch from 1c57813 to cc37cd0 Compare September 8, 2026 11:31

deruyter92 force-pushed the jaap/add-llm-knowledge-index branch from 6270402 to cc4ae4a Compare September 8, 2026 11:32

Base automatically changed from jaap/fix-tool-imports-pytest to dev September 8, 2026 13:52

deruyter92 added 25 commits September 8, 2026 15:52

validate version labels + tighten manifest reading

21d68b9

CI: harden reusable docs workflows against script injection

c2db3dd

unsafe free-text inputs are now read from the environment (as text).

check if dev-docs version is deployed before attempting a delete

8b23cf9

mark dev-docs and knowledge-index dependencies as dependecy-group (ra…

908c6dc

…ther than extra)

update uv.lock

67111b8

add tests for tools/knowledge-indexing

e7ab837

set ignore_errors=False on delete_version -> raise if directory i…

ff047dd

…s absent or cannot be removed

fix docs_index frontmatter matching via regex

0630f83

update test_write: use real API dataclasses instead of NameSpace + …

6b618fb

…add test delete version raises

add tests for frontmatter matching

e10d279

conditional guard dependency install knowledge-indexing: deployment-o…

76cb4ae

…nly, not delete

Move intermediate nodes to schemas (importable without dependencies)

58461ed

remove tests/tools/knowledge_indexing/conftest.py

8ce5334

update pyproject.toml and uv lock - add markdow-it-py as dev dependency

fcad299

Required for the docs indexer tests. It was already implicitly installed via huggingface-hub, so this installs nothing new. But added explicitly now so we don't rely on that chain.

lazy import docutils (so it is not required at test-time)

e5bcc48

Only the first H1 titles a docs page

57a9f93

Every H1 was consumed as a title candidate, so pages using H1 throughout lost those headings from the index entirely. Only the first claims the title; the rest become sections (701 -> 711 rows).

Harden docs frontmatter and record ids

2c884cb

A non-mapping `deeplabcut:` block crashed on .get(); it now raises a message naming the file. A heading that slugs to nothing gets a digest id instead of being named `docs:page#`.

Resolve api.latest through the latest-release alias

b5fb2a0

manage-dev-docs already tells mike which version is latest-release, but that never reached the knowledge index. this commit threads the aliases through to the indexer.

Separate symbol ids from submodule ids with #

4453c30

Symbol ids were `api:<module>.<name>`, the same shape as a submodule's id, so a package publishing both a submodule `foo` and a symbol `foo` would abort the build on the duplicate-id check.

Index class methods as their own api records

02b4f95

api.jsonl recorded modules, functions and classes, so an agent looking for a method signature had to open the class's reference page. Now we record a class's own public methods as `kind: "method"` rows named `Class.method`.

deruyter92 force-pushed the jaap/add-llm-knowledge-index branch from cc4ae4a to 3eaa5e1 Compare September 8, 2026 13:52

deruyter92 merged commit 1ff346f into dev Sep 8, 2026
0 of 6 checks passed

deruyter92 deleted the jaap/add-llm-knowledge-index branch September 8, 2026 13:52

This was referenced Sep 8, 2026

Follow-up hardening for the LLM knowledge index #3494

Merged

[Sync] merge to main for 3.0.2 release - bug fixes; knowledge indexing; future compatibility & dependencies #3495

Merged

		return Path(__file__).resolve().parents[3]


		@pytest.fixture(scope="session", autouse=True)

		return records


		def _check_unique_ids(filename: str, ids: Iterable[str]) -> None:

Conversation

deruyter92 commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

C-Achard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

C-Achard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deruyter92 commented Sep 4, 2026

Uh oh!

deruyter92 commented Sep 4, 2026

Uh oh!

C-Achard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

C-Achard commented Sep 4, 2026

Uh oh!

deruyter92 commented Sep 7, 2026

Uh oh!

C-Achard commented Sep 8, 2026

Uh oh!

C-Achard commented Sep 8, 2026

Uh oh!

deruyter92 commented Sep 8, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

deruyter92 commented Aug 18, 2026 •

edited

Loading