Add a knowledge indexing generator for an LLM-friendly docs index by deruyter92 · Pull Request #3453 · DeepLabCut/DeepLabCut

deruyter92 · 2026-08-18T14:47:07Z

Motivation
Recently the DeepLabCut documentation got an overhaul. This includes the recent launch of the api-docs. However for LLM agents, searching our documentation can still be a difficult task that easily fills the context window with irrelevant information.

The current PR adds a module tools/knowledge_indexing for generating a LLM-friendly index of the DeepLabCut API and user docs, and deploys it alongside the existing docs. This helps to point out AI agents where to retrieve relevant information.

Implementation

All indexing is deployed on our gh-pages website. This makes it discoverable for LLMs (without adding the contents in the TOC).
A single discovery point, llms.txt is generated at the site root following the the llmstxt.org convention and points at the structured index at knowledge/<version>/.. .
The index itself uses JSONL format: knowledge/<version>/{api,docs}.jsonl. It contains one JSON object per record (with a content_hash)
A per-version manifest.json and a top-level knowledge/manifest.json enumerates every version
Versioning is asymmetric: api.jsonl is versioned per dev-docs release, mirroring mike. The main user docs knowledge index always indexes the currently deployed user docs in docs.jsonl (the user docs have no versioning).
Dependencies: A dedicated knowledge-index extra (docutils, griffelib, markdown-it-py) is added to make the CI workflow lean (no other installs needed)

How to use
More information is added in tools/knowledge_indexing/README.md.
To locally build the full knowledge-index and llms.txt, run:

python -m tools.knowledge_indexing

The main knowledge-index is always deployed to gh-pages together with the user docs and dev-docs, triggered by updates on the main branch. To deploy a specific version label, just run the pre-existing manual dispatch workflow for versioned dev-docs API docs, this now automatically adds / removes the corresponding llm-index as well.

Status

Add generation tool for llm-friendly indexing of the docs + api-docs
Add CI workflow to auto-generate index for each new version / each deployed api-docs

Adds tools/knowledge_indexing, which reads the repository and writes a directory of small YAML files describing the codebase and user docs, one node per file, so an agent can load them lazily by id. Everything comes from the checkout, not from built or deployed docs: griffe reads the source statically for API signatures and docstrings, markdown-it-py parses the Markdown, and _toc.yml defines which pages are published along with their part/parent/child hierarchy. On the current checkout that is 270 nodes: 207 modules with 685 documented symbols, and 63 pages with 656 sections. Docs pages carry sections rather than concepts. A section is a retrievable part of one document, with an anchor, an absolute docs_url and the first paragraph of prose under the heading; anchors come from docutils.nodes.make_id, so they match the ids Sphinx emits. Concepts are reusable entities discussed across documents, which slugged headings are not, so none are generated. symbols.yaml maps every symbol id to the module node documenting it, for lookups that start from a bare name. index.yaml records the schema version, git revision and the two site base URLs the

C-Achard

Great work so far!
A few questions and some hopefully useful suggestions below, but nothing major besides the revision granularity

C-Achard · 2026-08-19T08:21:34Z

A general comment is that library version versus docs could use a little more granularity and separation to ensure the source is perfectly clear always. E.g. which API revision is used vs which user docs revision

Should we consider file content hashes for Nodes ? For correctness checks and maybe more optimized building (mainly the former)

Good suggestion, added hashes now.

A general comment is that library version versus docs could use a little more granularity and separation to ensure the source is perfectly clear always. E.g. which API revision is used vs which user docs revision

Dev-docs api knowledge index can now be build separately from the main docs index using --version-label, to build separate granular knowledge indices. For the latest main docs and the api-index tagged "main": these are build together, with a shared revision timestamp. This matches how the CI currently really works: deploy latest (all) or add/remove a specific version (dev-docs only).

hooking it up in CI now. Please let me know what you think when I'm done (if it's granular enough in practice).

C-Achard · 2026-08-19T08:24:47Z

Curiosity, does Amadeus know about the current local commit and is it compared to the knowledge system's commit? How should it handle mismatches?
(This is mostly a concern for testing with local DLC versions, and likely only affects us, but I wonder)

Good question. I don't have a definite answer yet.

Per your suggestion I've added content hashes and the revision field allows to compare the specific source. When it mismatches, that means that no specific docs are available for this version (as would be the case when a human would go and look at our versioned documentation, not finding the specific version they were looking for).

The user can either change their DLC version to one of the documented versions, or Amadeus could assume the latest available version and reason from there.

Maybe a warning and a fallback is nicer indeed?

C-Achard · 2026-08-19T08:26:49Z

+- **The user docs are not versioned** upstream, so `docs-pages/` is identical
+  across index versions and only stamped with the revision it came from.


Could we make use of the frontmatter metadata for this?

the problem is: we could perfectly make a versioned index for user docs (we know the commit when the index is generated), but the ACTUAL docs on gh-pages are always just the latest.

We could choose to index historical references (the doc source files on git) but I think it would make more sense to point to the rendered live docs (which can also be opened by the user). These live docs are unfortunately not versioned, but I don't think it is a huge problem: the main user guidance doesn't change that much. The API docs are versioned anyway.

BTW, the status, last_verified and other metadata fields are still very useful. They are included in the indexing description so downstream LLMs can make use of it. Super helpful product of the docs audit.

Meanwhile I've restructured the output a bit (see 83c8587) such that dev API-docs are listed per version, and the main docs are explicitly marked as unversioned (only latest: "main")

- replace the per-node YAML with knowledge/<version>/{api,docs}.jsonl plus a manifest.json per version and a top-level knowledge/manifest.json - Add llms.txt - restructure schemas: separate build-requirement schemas (docs_index.py, api_index.py) and output schemas (schemas.py)

…`version_label`)

…ge indexing tool

- add new deploy-knowledge-index workflow for publishing _build/knowledge-index via peaceiris gh-pages - hook new workflow in deploy docs and deploy dev-docs-mike

…rence to the knowledge-index

C-Achard

A few CI-related comments, really liking the new version!
I also started on the write.py side a bit.

Were you thinking of adding tests as for the other tools as well?

C-Achard · 2026-08-26T14:48:35Z

+
+    output: Path = args.output or repo / DEFAULT_OUTPUT_ROOT
+    knowledge_dir = output / KNOWLEDGE_DIR
+    api_base_url = API_BASE_URL.format(version=args.version_label)


Should the label be checked a bit more thoroughly if it's going to end up in the URL?

C-Achard · 2026-08-26T14:50:12Z

+def _git_revision(repo: Path) -> str:
+    """Full commit hash of `repo`, or "" if it is not a git checkout."""
+    try:
+        result = subprocess.run(
+            ["git", "-C", str(repo), "rev-parse", "HEAD"],
+            capture_output=True,
+            text=True,
+            timeout=10,
+            check=True,
+        )
+    except (OSError, subprocess.SubprocessError):
+        return ""
+    return result.stdout.strip()


Is this stricter in CI?

C-Achard · 2026-08-26T14:52:00Z

+      - name: Remove deleted version
+        if: inputs.action == 'delete'
+        shell: bash
+        run: rm -rf "_build/knowledge-index/knowledge/${{ inputs.version_label }}"


It seems that since the deploy runs with keep_files this may not reliably remove the published version

Also deletion deltes both api and docs JSONs, which affects both at once, whereas building seems to be independent. Maybe worth separating the two if possible

C-Achard · 2026-08-26T14:53:23Z

            --deploy-prefix "${{ inputs.deploy-prefix }}" \
            "${{ inputs.version_label }}"
+
+  knowledge-index:


This may need a concurrency guard shared with other workflows operating on gh-pages, as having several workflows modifying concurrently may be a problem

C-Achard · 2026-08-26T14:53:27Z

      deploy-prefix: dev
    secrets: inherit
+
+  deploy-user-docs-knowledge-index:


This may need a concurrency guard shared with other workflows operating on gh-pages, as having several workflows modifying concurrently may be a problem

C-Achard · 2026-08-26T14:53:37Z

This may need a concurrency guard shared with other workflows operating on gh-pages, as having several workflows modifying concurrently may be a problem

C-Achard · 2026-08-26T14:56:28Z

+
+      - name: Check out existing knowledge index from gh-pages
+        uses: actions/checkout@v6
+        continue-on-error: true


Should this be more error-specific? Since checkout failing != does not exist

C-Achard · 2026-08-26T15:00:32Z

+    """Read a JSON object, or None if `path` doesn't exist or isn't valid JSON."""
+    try:
+        return json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):


This might be a tad permissive if there is a malformed output, since None means "keep that side of the index", being unable to read an existing file makes it as if it should be ignored safely

C-Achard · 2026-08-26T15:11:33Z

+    api_provenance = ApiProvenance.from_dict(existing["api"]) if existing and existing.get("api") else None
+    docs_provenance = DocsProvenance.from_dict(existing["docs"]) if existing and existing.get("docs") else None


Is there a check that the files exist rather than only being listed by the manifest?

unsafe free-text inputs are now read from the environment (as text).

the knowledge-index deletion logic did not work correctly with peaceiris when `keep_files` is True. Also: the pre-existing workflow allowed the deletion of "main" which always contains the most recent documentaion and is reset after every push to main. The current commit fixes this and prevents deleting "main" in the first place. Only published releases can be deleted, and "main" is always tracking the most recent documentation version. For the knowledge-index, only the API docs index can be removed, as the main user docs are always kept up to date with the latest "main" version.

For API-docs deployment, a checkout from a specific tag is used, but this was done AFTER the latest deeplabcut was already cloned. Since checkout only WRITES the new state of files, stale paths (non-existent in the new checkout) would still remain on disk, potentially causing mixed API to be reported.

…g checkout For API-docs deployment, a checkout from a specific tag is used, but this was done AFTER the latest deeplabcut was already cloned. Since checkout only WRITES the new state of files, stale paths (non-existent in the new checkout) would still remain on disk, potentially causing mixed API to be reported.

…ther than extra)

C-Achard reviewed Aug 19, 2026

View reviewed changes

Comment thread tools/knowledge_indexing/__main__.py Outdated

C-Achard reviewed Aug 19, 2026

View reviewed changes

deruyter92 added this to the v3.1.0 milestone Aug 25, 2026

deruyter92 and others added 13 commits August 26, 2026 09:41

fix default output dir: _build

053d694

use full commit hash of repo

ef5bf73

add duplicate check

af711db

add content hash and distinct package_version (in addition to docs …

be6a384

…`version_label`)

update pyproject.toml: fix missing griffelib dependency for knowled…

fce810d

…ge indexing tool

allow separate adjustment of doc index and dev-docs api index

d41019e

add CI workflows for knowlede-index deployment

ee8e213

- add new deploy-knowledge-index workflow for publishing _build/knowledge-index via peaceiris gh-pages - hook new workflow in deploy docs and deploy dev-docs-mike

update docstrings and readme

58ac50b

separate dependencies for leaner CI build&deployment of knowledge-index

25d5594

simplify llms.txt no per-page summary of doc sections. Prominent refe…

f2295f4

…rence to the knowledge-index

Merge branch 'main' into jaap/add-llm-knowledge-index

08080bd

update uv.lock

8785d2d

deruyter92 requested a review from C-Achard August 26, 2026 13:07

deruyter92 marked this pull request as ready for review August 26, 2026 13:07

C-Achard reviewed Aug 26, 2026

View reviewed changes

deruyter92 added 7 commits August 27, 2026 12:50

validate version labels + tighten manifest reading

8badfd6

CI: harden reusable docs workflows against script injection

922fa03

unsafe free-text inputs are now read from the environment (as text).

check if dev-docs version is deployed before attempting a delete

baead09

mark dev-docs and knowledge-index dependencies as dependecy-group (ra…

24f1bc1

…ther than extra)

deruyter92 added the 3.1 label Aug 28, 2026

		- The user docs are not versioned upstream, so `docs-pages/` is identical
		across index versions and only stamped with the revision it came from.

		api_provenance = ApiProvenance.from_dict(existing["api"]) if existing and existing.get("api") else None
		docs_provenance = DocsProvenance.from_dict(existing["docs"]) if existing and existing.get("docs") else None

Uh oh!

Uh oh!

Conversation

deruyter92 commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

C-Achard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

C-Achard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

deruyter92 commented Aug 18, 2026 •

edited

Loading