Skip to content

Commit a6cb620

Browse files
aignasaignas
authored
feat(pip): support specifying requirements per (os, arch) (bazel-contrib#1885)
This PR implements a better way of specifying the requirements files for different (os, cpu) tuples. It allows for more granular specification than what is available today and allows for future extension to have all of the sources in the select statements in the hub repository. This is replacing the previous selection of the requirements and there are a few differences in behaviour that should not be visible to the external user. Instead of selecting the right file which we should then use to create `whl_library` instances we parse all of the provided requirements files and merge them based on the contents. The merging is done based on the blocks within the requirement file and this allows the starlark code to understand if we are working with different versions of the same package on different target platforms. Fixes bazel-contrib#1868 Work towards bazel-contrib#1643, bazel-contrib#735
1 parent 8d31c5f commit a6cb620

27 files changed

Lines changed: 1071 additions & 382 deletions

‎CHANGELOG.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,14 +25,12 @@ A brief description of the categories of changes:
2525
* (toolchains) Optional toolchain dependency: `py_binary`, `py_test`, and
2626
`py_library` now depend on the `//python:exec_tools_toolchain_type` for build
2727
tools.
28-
2928
* (deps): Bumped `bazel_skylib` to 1.6.1.
3029
* (bzlmod): The `python` and internal `rules_python` extensions have been
3130
marked as `reproducible` and will not include any lock file entries from now
3231
on.
3332

3433
### Fixed
35-
3634
* (gazelle) Remove `visibility` from `NonEmptyAttr`.
3735
Now empty(have no `deps/main/srcs/imports` attr) `py_library/test/binary` rules will
3836
be automatically deleted correctly. For example, if `python_generation_mode`
@@ -66,9 +64,16 @@ A brief description of the categories of changes:
6664
`transitive_pyc_files`, which tell the pyc files a target makes available
6765
directly and transitively, respectively.
6866
* `//python:features.bzl` added to allow easy feature-detection in the future.
67+
* (pip) Allow specifying the requirements by (os, arch) and add extra
68+
validations when parsing the inputs. This is a non-breaking change for most
69+
users unless they have been passing multiple `requirements_*` files together
70+
with `extra_pip_args = ["--platform=manylinux_2_4_x86_64"]`, that was an
71+
invalid usage previously but we were not failing the build. From now on this
72+
is explicitly disallowed.
6973

7074
[precompile-docs]: /precompiling
7175

76+
7277
## [0.32.2] - 2024-05-14
7378

7479
[0.32.2]: https://github.com/bazelbuild/rules_python/releases/tag/0.32.2

‎MODULE.bazel‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -61,9 +61,11 @@ pip.parse(
6161
experimental_index_url = "https://pypi.org/simple",
6262
hub_name = "rules_python_publish_deps",
6363
python_version = "3.11",
64-
requirements_darwin = "//tools/publish:requirements_darwin.txt",
65-
requirements_lock = "//tools/publish:requirements.txt",
66-
requirements_windows = "//tools/publish:requirements_windows.txt",
64+
requirements_by_platform = {
65+
"//tools/publish:requirements.txt": "linux_*",
66+
"//tools/publish:requirements_darwin.txt": "osx_*",
67+
"//tools/publish:requirements_windows.txt": "windows_*",
68+
},
6769
)
6870
use_repo(pip, "rules_python_publish_deps")
6971

‎docs/sphinx/pip.md‎

Lines changed: 55 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,41 @@ load("@pip_deps//:requirements.bzl", "install_deps")
1919
install_deps()
2020
```
2121

22+
For `bzlmod` an equivalent `MODULE.bazel` would look like:
23+
```starlark
24+
pip = use_extension("//python/extensions:pip.bzl", "pip")
25+
pip.parse(
26+
hub_name = "pip_deps",
27+
requirements_lock = ":requirements.txt",
28+
)
29+
use_repo(pip, "pip_deps")
30+
```
31+
2232
You can then reference installed dependencies from a `BUILD` file with:
2333

2434
```starlark
2535
load("@pip_deps//:requirements.bzl", "requirement")
2636

37+
py_library(
38+
name = "bar",
39+
...
40+
deps = [
41+
"//my/other:dep",
42+
"@pip_deps//requests",
43+
"@pip_deps//numpy",
44+
],
45+
)
46+
```
47+
48+
The rules also provide a convenience macro for translating the entries in the
49+
`requirements.txt` file (e.g. `opencv-python`) to the right bazel label (e.g.
50+
`@pip_deps//opencv_python`). The convention of bazel labels is lowercase
51+
`snake_case`, but you can use the helper to avoid depending on this convention
52+
as follows:
53+
54+
```starlark
55+
load("@pip_deps//:requirements.bzl", "requirement")
56+
2757
py_library(
2858
name = "bar",
2959
...
@@ -35,33 +65,39 @@ py_library(
3565
)
3666
```
3767

38-
In addition to the `requirement` macro, which is used to access the generated `py_library`
39-
target generated from a package's wheel, The generated `requirements.bzl` file contains
40-
functionality for exposing [entry points][whl_ep] as `py_binary` targets as well.
68+
If you would like to access [entry points][whl_ep], see the `py_console_script_binary` rule documentation.
4169

4270
[whl_ep]: https://packaging.python.org/specifications/entry-points/
4371

72+
(per-os-arch-requirements)=
73+
## Requirements for a specific OS/Architecture
74+
75+
In some cases you may need to use different requirements files for different OS, Arch combinations. This is enabled via the `requirements_by_platform` attribute in `pip.parse` extension and the `pip_parse` repository rule. The keys of the dictionary are labels to the file and the values are a list of comma separated target (os, arch) tuples.
76+
77+
For example:
4478
```starlark
45-
load("@pip_deps//:requirements.bzl", "entry_point")
46-
47-
alias(
48-
name = "pip-compile",
49-
actual = entry_point(
50-
pkg = "pip-tools",
51-
script = "pip-compile",
52-
),
53-
)
79+
# ...
80+
requirements_by_platform = {
81+
"requirements_linux_x86_64.txt": "linux_x86_64",
82+
"requirements_osx.txt": "osx_*",
83+
"requirements_linux_exotic.txt": "linux_exotic",
84+
"requirements_some_platforms.txt": "linux_aarch64,windows_*",
85+
},
86+
# For the list of standard platforms that the rules_python has toolchains for, default to
87+
# the following requirements file.
88+
requirements_lock = "requirements_lock.txt",
5489
```
5590

56-
Note that for packages whose name and script are the same, only the name of the package
57-
is needed when calling the `entry_point` macro.
91+
In case of duplicate platforms, `rules_python` will raise an error as there has
92+
to be unambiguous mapping of the requirement files to the (os, arch) tuples.
5893

94+
An alternative way is to use per-OS requirement attributes.
5995
```starlark
60-
load("@pip_deps//:requirements.bzl", "entry_point")
61-
62-
alias(
63-
name = "flake8",
64-
actual = entry_point("flake8"),
96+
# ...
97+
requirements_windows = "requirements_windows.txt",
98+
requirements_darwin = "requirements_darwin.txt",
99+
# For the remaining platforms (which is basically only linux OS), use this file.
100+
requirements_lock = "requirements_lock.txt",
65101
)
66102
```
67103

‎examples/bzlmod/MODULE.bazel‎

Lines changed: 11 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -128,8 +128,17 @@ pip.parse(
128128
],
129129
hub_name = "pip",
130130
python_version = "3.9",
131-
requirements_lock = "//:requirements_lock_3_9.txt",
132-
requirements_windows = "//:requirements_windows_3_9.txt",
131+
# The requirements files for each platform that we want to support.
132+
requirements_by_platform = {
133+
# Default requirements file for needs to explicitly provide the platforms
134+
"//:requirements_lock_3_9.txt": "linux_*,osx_*",
135+
# This API allows one to specify additional platforms that the users
136+
# configure the toolchains for themselves. In this example we add
137+
# `windows_aarch64` to illustrate that `rules_python` won't fail to
138+
# process the value, but it does not mean that this example will work
139+
# on Windows ARM.
140+
"//:requirements_windows_3_9.txt": "windows_x86_64,windows_aarch64",
141+
},
133142
# These modifications were created above and we
134143
# are providing pip.parse with the label of the mod
135144
# and the name of the wheel.
@@ -193,14 +202,3 @@ local_path_override(
193202
module_name = "other_module",
194203
path = "other_module",
195204
)
196-
197-
# =====
198-
# Config for testing duplicate packages in requirements
199-
# =====
200-
#
201-
pip.parse(
202-
hub_name = "dupe_requirements",
203-
python_version = "3.9", # Must match whatever is marked is_default=True
204-
requirements_lock = "//tests/dupe_requirements:requirements.txt",
205-
)
206-
use_repo(pip, "dupe_requirements")

‎examples/bzlmod/tests/dupe_requirements/BUILD.bazel‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

‎examples/bzlmod/tests/dupe_requirements/dupe_requirements_test.py‎

Lines changed: 0 additions & 4 deletions
This file was deleted.

‎examples/bzlmod/tests/dupe_requirements/requirements.in‎

Lines changed: 0 additions & 2 deletions
This file was deleted.

‎examples/bzlmod/tests/dupe_requirements/requirements.txt‎

Lines changed: 0 additions & 97 deletions
This file was deleted.

‎examples/pip_parse/MODULE.bazel‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ use_repo(
2121

2222
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
2323
pip.parse(
24+
download_only = True,
2425
experimental_requirement_cycles = {
2526
"sphinx": [
2627
"sphinx",

‎examples/pip_parse_vendored/requirements.bzl‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
"""Starlark representation of locked requirements.
22
3-
@generated by rules_python pip_parse repository rule
4-
from @//:requirements.txt
3+
@generated by rules_python pip_parse repository rule.
54
"""
65

76
load("@rules_python//python:pip.bzl", "pip_utils")

0 commit comments

Comments
 (0)