Skip to content

Commit 6ca2f58

Browse files
rickeylevrickeylev
authored
docs: generate Starlark domain markup instead of regular markdown (bazel-contrib#1919)
This switches our doc generation over to using the Starlark domain markup from the sphinx_stardoc plugin instead of using regular markdown. This allows the docs generated from code to better integrate with each other and other parts of the doc site. Overview of changes: * Makes the doc paths under the API directory more directly mirror their actual location. e.g. moves "defs.md" -> "python/defs.md". This is so the //tools doc entries have a more natural location, but can also be used for our other top-level directories. * Adds API docs for some of the well known targets we have. These aren't automatically generated, but use the Starlark domain markup, so integrate nicely with everything. * Ensures default values are parsable as Python expressions. Stardoc returns values like "<function foo>" or 'Label(*, "//bar")' in some cases for the default value of args/attrs. * Ensures function signatures don't crash doc rendering. Stardoc gives bad/incomplete information, so reconstructing the original signature of a function is tricky. * Allows references flags using leading slashes and a value, e.g. `--foo=bar`. This makes it more natural to write while cross referencing to the flag. * Implements `{any}` xref resolution. It was just totally broken before. * Adds some additional bzl files that get documented. * Adds some more Bazel external references. * Fixes some missing bzl_library dependencies. * A few minor QoL improvements to the docs dev server: * Print the serving directory when CTRL+C is received. This makes it easier to find the raw files that are being generated. * Fix an error during shutdown about an unterminated generator. * The `sphinx_stardocs.footer` arg is removed. This was always just a hack to get extra link targets into the generated bzl docs. It's no longer needed when the bzl domain is used. * Using `@repo//pkg:file.bzl%Name` syntax is supported in type expressions (e.g. `:type:` option or `{type}` role) by quoting the label. The quoting is necessary because, under the hood, the expressions are parsed as Python. * Objects directives support an `:origin-key` directive. This records the label identity that Bazel sees for an object (as from the Stardoc origin_key field). The markdown generate doesn't generate this for everything yet because some things are documented twice (e.g. py_binary in defs.bzl and py_binary.bzl), which would cause a crash (type things trying to define the same id). * Add `*` and `**` to var-args and var-kwargs in signatures. * Allow providers to be refered to using the `type` role. This allows providers to be referenced in `:type:` directives (e.g. in a provider field).
1 parent c7defbc commit 6ca2f58

24 files changed

Lines changed: 792 additions & 403 deletions

File tree

‎docs/sphinx/BUILD.bazel‎

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ _TARGET_COMPATIBLE_WITH = select({
3030
"@platforms//os:linux": [],
3131
"@platforms//os:macos": [],
3232
"//conditions:default": ["@platforms//:incompatible"],
33-
})
33+
}) if IS_BAZEL_7_OR_HIGHER else ["@platforms//:incompatible"]
3434

3535
# See README.md for instructions. Short version:
3636
# * `bazel run //docs/sphinx:docs.serve` in a separate terminal
@@ -76,24 +76,34 @@ sphinx_inventory(
7676
sphinx_stardocs(
7777
name = "bzl_api_docs",
7878
docs = {
79-
"api/cc/py_cc_toolchain.md": dict(
79+
"api/python/cc/py_cc_toolchain.md": dict(
8080
dep = "//python/private:py_cc_toolchain_bzl",
8181
input = "//python/private:py_cc_toolchain_rule.bzl",
8282
public_load_path = "//python/cc:py_cc_toolchain.bzl",
8383
),
84-
"api/cc/py_cc_toolchain_info.md": "//python/cc:py_cc_toolchain_info_bzl",
85-
"api/defs.md": "//python:defs_bzl",
86-
"api/entry_points/py_console_script_binary.md": "//python/entry_points:py_console_script_binary_bzl",
87-
"api/packaging.md": "//python:packaging_bzl",
88-
"api/pip.md": "//python:pip_bzl",
84+
"api/python/cc/py_cc_toolchain_info.md": "//python/cc:py_cc_toolchain_info_bzl",
85+
"api/python/defs.md": "//python:defs_bzl",
86+
"api/python/entry_points/py_console_script_binary.md": "//python/entry_points:py_console_script_binary_bzl",
87+
"api/python/packaging.md": "//python:packaging_bzl",
88+
"api/python/pip.md": "//python:pip_bzl",
89+
"api/python/py_binary.md": "//python:py_binary_bzl",
90+
"api/python/py_cc_link_params_info.md": "//python:py_cc_link_params_info_bzl",
91+
"api/python/py_library.md": "//python:py_library_bzl",
92+
"api/python/py_runtime.md": "//python:py_runtime_bzl",
93+
"api/python/py_runtime_info.md": "//python:py_runtime_info_bzl",
94+
"api/python/py_runtime_pair.md": dict(
95+
dep = "//python/private:py_runtime_pair_rule_bzl",
96+
input = "//python/private:py_runtime_pair_rule.bzl",
97+
public_load_path = "//python:py_runtime_pair.bzl",
98+
),
99+
"api/python/py_test.md": "//python:py_test_bzl",
89100
} | ({
90101
# Bazel 6 + Stardoc isn't able to parse something about the python bzlmod extension
91-
"api/extensions/python.md": "//python/extensions:python_bzl",
102+
"api/python/extensions/python.md": "//python/extensions:python_bzl",
92103
} if IS_BAZEL_7_OR_HIGHER else {}) | ({
93104
# This depends on @pythons_hub, which is only created under bzlmod,
94-
"api/extensions/pip.md": "//python/extensions:pip_bzl",
105+
"api/python/extensions/pip.md": "//python/extensions:pip_bzl",
95106
} if IS_BAZEL_7_OR_HIGHER and BZLMOD_ENABLED else {}),
96-
footer = "_stardoc_footer.md",
97107
tags = ["docs"],
98108
target_compatible_with = _TARGET_COMPATIBLE_WITH,
99109
)
@@ -112,6 +122,8 @@ sphinx_build_binary(
112122
requirement("sphinx_rtd_theme"),
113123
requirement("myst_parser"),
114124
requirement("readthedocs_sphinx_ext"),
125+
requirement("typing_extensions"),
126+
"//sphinxdocs/src/sphinx_stardoc",
115127
],
116128
)
117129

‎docs/sphinx/api/python/config_settings/index.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
:::{bzl:currentfile} //python/config_settings:BUILD.bazel
2+
:::
3+
4+
# //python/config_settings
5+
6+
:::{bzl:flag} precompile
7+
Determines if Python source files should be compiled at build time.
8+
9+
NOTE: The flag value is overridden by the target level `precompile` attribute,
10+
except for the case of `force_enabled` and `forced_disabled`.
11+
12+
Values:
13+
14+
* `auto`: Automatically decide the effective value based on environment,
15+
target platform, etc.
16+
* `enabled`: Compile Python source files at build time. Note that
17+
{bzl:obj}`--precompile_add_to_runfiles` affects how the compiled files are included into
18+
a downstream binary.
19+
* `disabled`: Don't compile Python source files at build time.
20+
* `if_generated_source`: Compile Python source files, but only if they're a
21+
generated file.
22+
* `force_enabled`: Like `enabled`, except overrides target-level setting. This
23+
is mostly useful for development, testing enabling precompilation more
24+
broadly, or as an escape hatch if build-time compiling is not available.
25+
* `force_disabled`: Like `disabled`, except overrides target-level setting. This
26+
is useful useful for development, testing enabling precompilation more
27+
broadly, or as an escape hatch if build-time compiling is not available.
28+
:::
29+
30+
:::{bzl:flag} precompile_source_retention
31+
Determines, when a source file is compiled, if the source file is kept
32+
in the resulting output or not.
33+
34+
NOTE: This flag is overridden by the target level `precompile_source_retention`
35+
attribute.
36+
37+
Values:
38+
39+
* `keep_source`: Include the original Python source.
40+
* `omit_source`: Don't include the orignal py source.
41+
* `omit_if_generated_source`: Keep the original source if it's a regular source
42+
file, but omit it if it's a generated file.
43+
:::
44+
45+
:::{bzl:flag} precompile_add_to_runfiles
46+
Determines if a target adds its compiled files to its runfiles.
47+
48+
When a target compiles its files, but doesn't add them to its own runfiles, it
49+
relies on a downstream target to retrieve them from
50+
{bzl:obj}`PyInfo.transitive_pyc_files`
51+
52+
Values:
53+
* `always`: Always include the compiled files in the target's runfiles.
54+
* `decided_elsewhere`: Don't include the compiled files in the target's
55+
runfiles; they are still added to {bzl:obj}`PyInfo.transitive_pyc_files`. See
56+
also: {bzl:obj}`py_binary.pyc_collection` attribute. This is useful for allowing
57+
incrementally enabling precompilation on a per-binary basis.
58+
:::
59+
60+
:::{bzl:flag} pyc_collection
61+
Determine if `py_binary` collects transitive pyc files.
62+
63+
NOTE: This flag is overridden by the target level `pyc_collection` attribute.
64+
65+
Values:
66+
* `include_pyc`: Include `PyInfo.transitive_pyc_files` as part of the binary.
67+
* `disabled`: Don't include `PyInfo.transitive_pyc_files` as part of the binary.
68+
:::

‎docs/sphinx/api/python/index.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
:::{bzl:currentfile} //python:BUILD.bazel
2+
:::
3+
4+
# //python
5+
6+
:::{bzl:target} toolchain_type
7+
8+
Identifier for the toolchain type for the target platform.
9+
:::
10+
11+
:::{bzl:target} exec_tools_toolchain_type
12+
13+
Identifier for the toolchain type for exec tools used to build Python targets.
14+
:::
15+
16+
:::{bzl:target} current_py_toolchain
17+
18+
Helper target to resolve to the consumer's current Python toolchain. This target
19+
provides:
20+
21+
* `PyRuntimeInfo`: The consuming target's target toolchain information
22+
23+
:::

‎docs/sphinx/api/tools/precompiler/index.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
:::{bzl:currentfile} //tools/precompiler:BUILD.bazel
2+
:::
3+
4+
# //tools/precompiler
5+
6+
:::{bzl:flag} execution_requirements
7+
Determines the execution requirements `//tools/precompiler:precompiler` uses.
8+
9+
This is a repeatable string_list flag. The values are `key=value` entries, each
10+
of which are added to the execution requirements for the `PyCompile` action to
11+
generate pyc files.
12+
13+
Customizing this flag mostly allows controlling whether Bazel runs the
14+
precompiler as a regular worker, persistent worker, or regular action.
15+
:::

‎docs/sphinx/bazel_inventory.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,5 @@ python bzl:doc 1 reference/be/python -
2222
str bzl:type 1 rules/lib/string -
2323
struct bzl:type 1 rules/lib/builtins/struct -
2424
target-name bzl:doc 1 concepts/labels#target-names -
25+
CcInfo bzl:provider 1 rules/lib/providers/CcInfo -
26+
CcInfo.linking_context bzl:provider-field 1 rules/lib/providers/CcInfo#linking_context -

‎docs/sphinx/conf.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
"sphinx.ext.intersphinx",
2828
"myst_parser",
2929
"sphinx_rtd_theme", # Necessary to get jquery to make flyout work
30+
"sphinx_stardoc.stardoc",
3031
]
3132

3233
# Adapted from the template code:
@@ -89,6 +90,10 @@
8990

9091
myst_substitutions = {}
9192

93+
# --- sphinx_stardoc configuration
94+
95+
bzl_default_repository_name = "@rules_python"
96+
9297
# -- Options for HTML output
9398
# See https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
9499
# For additional html settings

‎docs/sphinx/precompiling.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,14 +29,15 @@ incrementally control precompiling on a per-binry basis.
2929

3030
To use this approach, the two basic steps are:
3131
1. Disable pyc files from being automatically added to runfiles:
32-
`--@rules_python//python/config_settings:precompile_add_to_runfiles=decided_elsewhere`,
32+
{bzl:obj}`--@rules_python//python/config_settings:precompile_add_to_runfiles=decided_elsewhere`,
3333
2. Set the `pyc_collection` attribute on the binaries/tests that should or should
3434
not use precompiling.
3535

36-
The default for the `pyc_collection` attribute is controlled by a flag, so you
37-
can use an opt-in or opt-out approach by setting the flag:
38-
* targets must opt-out: `--@rules_python//python/config_settings:pyc_collection=include_pyc`,
39-
* targets must opt-in: `--@rules_python//python/config_settings:pyc_collection=disabled`,
36+
The default for the `pyc_collection` attribute is controlled by the flag
37+
{bzl:obj}`--@rules_python//python/config_settings:pyc_collection`, so you
38+
can use an opt-in or opt-out approach by setting its value:
39+
* targets must opt-out: `--@rules_python//python/config_settings:pyc_collection=include_pyc`
40+
* targets must opt-in: `--@rules_python//python/config_settings:pyc_collection=disabled`
4041

4142
## Advanced precompiler customization
4243

‎python/BUILD.bazel‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,10 @@ bzl_library(
130130
bzl_library(
131131
name = "py_cc_link_params_info_bzl",
132132
srcs = ["py_cc_link_params_info.bzl"],
133+
deps = [
134+
"//python/private/common:providers_bzl",
135+
"@rules_python_internal//:rules_python_config_bzl",
136+
],
133137
)
134138

135139
bzl_library(
@@ -185,6 +189,7 @@ bzl_library(
185189
"//python/private:reexports_bzl",
186190
"//python/private:util_bzl",
187191
"//python/private/common:providers_bzl",
192+
"@rules_python_internal//:rules_python_config_bzl",
188193
],
189194
)
190195

0 commit comments

Comments
 (0)