Skip to content
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
891e190
Add `__doc__` crate
ShaharNaveh Nov 1, 2025
e74c26f
Base auto-generate ci
ShaharNaveh Nov 3, 2025
d6e00ba
Add dummy files
ShaharNaveh Nov 3, 2025
b309e90
Update docs
ShaharNaveh Nov 3, 2025
d35a6bd
Set codegen-units to 1 for doc db
ShaharNaveh Nov 3, 2025
232fe6d
Mark `*.inc.rs` as auto generated
ShaharNaveh Nov 3, 2025
231318a
Disable doctest
ShaharNaveh Nov 7, 2025
aaa2006
Reset docs
ShaharNaveh Nov 7, 2025
c2b2249
Simplify `generate.py`
ShaharNaveh Nov 7, 2025
2db1227
No need for optimizatins in dev
ShaharNaveh Nov 7, 2025
c3b191e
simplify `lib.rs`
ShaharNaveh Nov 7, 2025
34d4f32
Regenerate docs
ShaharNaveh Nov 7, 2025
83ab568
ignore cspell
ShaharNaveh Nov 7, 2025
efd4de9
Merge remote-tracking branch 'upstream/main' into move-doc
ShaharNaveh Nov 9, 2025
6140114
Configurable python-version
ShaharNaveh Nov 9, 2025
7a432ec
Fix logic error
ShaharNaveh Nov 9, 2025
a590b8d
Output as json
ShaharNaveh Nov 9, 2025
315ef1d
GHA now uploads `data.inc.rs`
ShaharNaveh Nov 9, 2025
100a803
Generate `data.inc.rs`
ShaharNaveh Nov 9, 2025
833c786
Add info comments
ShaharNaveh Nov 9, 2025
25cf146
Fix typo of missing type
ShaharNaveh Nov 9, 2025
83c03a5
Fix typo
ShaharNaveh Nov 9, 2025
4e21b6f
Assert raw_doc is not None
ShaharNaveh Nov 9, 2025
174cb87
Check if attr is callable
ShaharNaveh Nov 9, 2025
1542895
Merge remote-tracking branch 'upstream/main' into move-doc
ShaharNaveh Nov 10, 2025
4544be3
Rename to `rustpython-doc`
ShaharNaveh Nov 10, 2025
e089740
Merge remote-tracking branch 'upstream/main' into move-doc
ShaharNaveh Nov 10, 2025
15546ab
Rename to `crates/doc`
ShaharNaveh Nov 14, 2025
4bee896
Merge remote-tracking branch 'upstream/main' into move-doc
ShaharNaveh Nov 14, 2025
9e5ed29
Increase unix CI timeout to 35 minutes (was 30)
ShaharNaveh Nov 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ vm/src/stdlib/ast/gen.rs linguist-generated -merge
Lib/*.py text working-tree-encoding=UTF-8 eol=LF
**/*.rs text working-tree-encoding=UTF-8 eol=LF
*.pck binary
crates/rustpython_doc_db/src/*.inc.rs linguist-generated=true
51 changes: 51 additions & 0 deletions .github/workflows/update-doc-db.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Update doc DB

permissions:
contents: read

on:
workflow_dispatch:
inputs:
python-version:
description: Target python version to generate doc db for
type: string
default: "3.13.9"

defaults:
run:
shell: bash

jobs:
generate:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os:
- ubuntu-latest
- windows-latest
- macos-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
persist-credentials: false
sparse-checkout: |
crates/rustpython_doc_db

- uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
with:
python-version: ${{ inputs.python-version }}

- name: Generate (${{ matrix.os }})
run: python crates/rustpython_doc_db/generate.py

- name: Detect output file
id: diff
run: echo "diff=$(git diff --name-only)" >> $GITHUB_OUTPUT

- uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
with:
name: doc-db-${{ env.PYTHON_VERSION }}-${{ matrix.os }}
path: ${{ steps.diff.outputs.diff }}
if-no-files-found: error
retention-days: 7
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Major: Workflow fails if generate script produces no changes.

The workflow detects output via git diff --name-only (line 43) and uploads artifacts with if-no-files-found: error (line 49). If the generated output is already up-to-date (no changes), the job fails, which contradicts the use case where this workflow may be idempotent.

Additionally, line 39 runs the generate script with no explicit error handling—if it fails, the workflow continues to the artifact upload step.

Consider:

  1. For idempotent behavior: Use if-no-files-found: warn instead of error to allow the workflow to complete gracefully when no changes are detected.
  2. For explicit error handling: Add set -e or explicit checks after the generate step.

Apply this diff:

      - name: Generate (${{ matrix.os }})
        run: python crates/rustpython_doc_db/generate.py

      - name: Detect output file
        id: diff
        run: echo "diff=$(git diff --name-only)" >> $GITHUB_OUTPUT

      - uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
        with:
          name: doc-db-${{ env.PYTHON_VERSION }}-${{ matrix.os }}
          path: ${{ steps.diff.outputs.diff }}
-         if-no-files-found: error
+         if-no-files-found: warn

Based on learnings, the maintainers prefer simplicity and accept workflows that may fail gracefully in edge cases; however, the intent here is unclear.

🤖 Prompt for AI Agents
.github/workflows/update-doc-db.yml lines 38-50: the workflow will fail when the
generate script produces no git-diff (upload-artifact uses if-no-files-found:
error) and also doesn't stop early if the generate script itself fails; change
the upload step to use if-no-files-found: warn to make idempotent runs succeed
with no changes, and make the generate step fail fast by running the script
under a strict shell (e.g., add set -e before running the python command or
check the script exit status and exit non-zero on failure) so the job correctly
surfaces generation errors.

Comment thread
coderabbitai[bot] marked this conversation as resolved.
overwrite: true
85 changes: 66 additions & 19 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@ opt-level = 3
# https://github.com/rust-lang/rust/issues/92869
# lto = "thin"

# Doesn't change often
[profile.release.package.rustpython_doc_db]
codegen-units = 1

[profile.bench]
lto = "thin"
codegen-units = 1
Expand Down Expand Up @@ -132,6 +136,7 @@ members = [
"derive-impl",
"wtf8",
"wasm/lib",
"crates/*",
]

[workspace.package]
Expand All @@ -156,13 +161,14 @@ rustpython-pylib = { path = "pylib", version = "0.4.0" }
rustpython-stdlib = { path = "stdlib", default-features = false, version = "0.4.0" }
rustpython-sre_engine = { path = "vm/sre_engine", version = "0.4.0" }
rustpython-wtf8 = { path = "wtf8", version = "0.4.0" }
rustpython-doc = { git = "https://github.com/RustPython/__doc__", tag = "0.3.0", version = "0.3.0" }
rustpython_doc_db = { path = "crates/rustpython_doc_db", version = "0.4.0" }
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
rustpython_doc_db = { path = "crates/rustpython_doc_db", version = "0.4.0" }
rustpython-doc_db = { path = "crates/rustpython_doc_db", version = "0.4.0" }

- is a namespace split, _ is a (sort of) whitespace split

Copy link
Copy Markdown
Contributor Author

@ShaharNaveh ShaharNaveh Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is worth discussing more. I haven't seen a single project that has both - and _ in it's name. I think we should stick to one separator and use it (I'm in favor of _)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have that names for every crates.
I don't think mixing rustpython- and rustpython_ for different crates gives any value.

Actually I was missing the fundamental question about this change.
We'd better to keep using the previous crate name rustpython-doc. Moving crate doesn't mean renaming(creating a new) crate.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have that names for every crates. I don't think mixing rustpython- and rustpython_ for different crates gives any value.

think we should choose one sperator and stick to it, as long as it's consistent.

Actually I was missing the fundamental question about this change. We'd better to keep using the previous crate name rustpython-doc. Moving crate doesn't mean renaming(creating a new) crate.

Yeah, ofc

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think we should choose one sperator and stick to it, as long as it's consistent.

I understand that there’s some validity to that point.
However, it’s not fair to justify it simply by calling it “consistent,” because the fact that all RustPython crates share the rustpython- prefix is already a form of consistency in itself.
It’s just a different kind of consistency.
If these two forms of consistency conflict, I would reject introducing a new crate with a different prefix.


ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", tag = "0.14.1" }
ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", tag = "0.14.1" }
ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", tag = "0.14.1" }
ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", tag = "0.14.1" }

phf = { version = "0.13.1", default-features = false, features = ["macros"]}
ahash = "0.8.12"
ascii = "1.1"
bitflags = "2.9.4"
Expand Down
17 changes: 17 additions & 0 deletions crates/rustpython_doc_db/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[package]
name = "rustpython_doc_db"
version.workspace = true
authors.workspace = true
edition.workspace = true
rust-version.workspace = true
repository.workspace = true
license-file = "LICENSE"

[dependencies]
phf = { workspace = true }

[lib]
doctest = false # Crashes when true

[lints]
workspace = true
Loading
Loading