Skip to content

Commit d66e55c

Browse files
aignasrickeylev
aignas
and
rickeylev
authored
feat(bzlmod): cross-platform builds without experimental_index_url (bazel-contrib#2325)
With this change we finally generate the same lock file within the legacy code `pip.parse` code path and it allows to slowly transition to using the new code path as much as possible without user doing anything. This moves the selection of the host-compatible lock file from the extension evaluation to the build phase - note, we will generate extra repositories here which might not work on the host platform, however, if the users are consuming the `whl_library` repos through the hub repo only, then everything should work. A known issue is that it may break `bazel query` and in these usecases it is advisable to use `cquery` until we have `sdist` cross-building from source fully working. Summary: - feat: reuse the `render_pkg_aliases` for when filename is not known but platform is known - feat: support generating the extra config settings required - feat: `get_whl_flag_versions` now generates extra args for the rules - feat: make lock file generation the same irrespective of the host platform - test: add an extra test with multiple requirements files - feat: support cross-platform builds using `download_only = True` in legacy setups Note, that users depending on the naming of the whl libraries will need to start using `extra_hub_aliases` attribute instead to keep their setups not relying on this implementation detail. Fixes bazel-contrib#2268 Work towards bazel-contrib#260 --------- Co-authored-by: Richard Levasseur <richardlev@gmail.com>
1 parent 2038975 commit d66e55c

13 files changed

Lines changed: 630 additions & 174 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,12 @@ A brief description of the categories of changes:
3939
[`pip.parse#extra_pip_args`](https://rules-python.readthedocs.io/en/latest/api/rules_python/python/extensions/pip.html#pip.parse.extra_pip_args)
4040
* (pip.parse) {attr}`pip.parse.whl_modifications` now normalizes the given whl names
4141
and now `pyyaml` and `PyYAML` will both work.
42+
* (bzlmod) `pip.parse` spoke repository naming will be changed in an upcoming
43+
release in places where the users specify different package versions per
44+
platform in the same hub repository. The naming of the spoke repos is considered
45+
an implementation detail and we advise the users to use the `hub` repository
46+
directly to avoid such breakage in the future. If `rules_python` is missing
47+
features to allow one to do that, please raise tickets.
4248

4349
{#v0-0-0-fixed}
4450
### Fixed
@@ -51,6 +57,12 @@ A brief description of the categories of changes:
5157
pass the `extra_pip_args` value when building an `sdist`.
5258
* (pypi) The patched wheel filenames from now on are using local version specifiers
5359
which fixes usage of the said wheels using standard package managers.
60+
* (bzlmod) The extension evaluation has been adjusted to always generate the
61+
same lock file irrespective if `experimental_index_url` is set by any module
62+
or not. Fixes
63+
[#2268](https://github.com/bazelbuild/rules_python/issues/2268). A known
64+
issue is that it may break `bazel query` and in these use cases it is
65+
advisable to use `cquery` or switch to `download_only = True`
5466

5567
{#v0-0-0-added}
5668
### Added
@@ -63,6 +75,11 @@ A brief description of the categories of changes:
6375
* (pip.parse) {attr}`pip.parse.extra_hub_aliases` can now be used to expose extra
6476
targets created by annotations in whl repositories.
6577
Fixes [#2187](https://github.com/bazelbuild/rules_python/issues/2187).
78+
* (bzlmod) `pip.parse` now supports `whl-only` setup using
79+
`download_only = True` where users can specify multiple requirements files
80+
and use the `pip` backend to do the downloading. This was only available for
81+
users setting {bzl:obj}`pip.parse.experimental_index_url`, but now users have
82+
more options whilst we continue to work on stabilizing the experimental feature.
6683

6784
{#v0-0-0-removed}
6885
### Removed

‎MODULE.bazel‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,10 @@ register_toolchains("@pythons_hub//:all")
5656

5757
pip = use_extension("//python/private/pypi:pip.bzl", "pip_internal")
5858
pip.parse(
59+
# NOTE @aignas 2024-10-26: We have an integration test that depends on us
60+
# being able to build sdists for this hub, so explicitly set this to False.
61+
download_only = False,
62+
experimental_index_url = "https://pypi.org/simple",
5963
hub_name = "rules_python_publish_deps",
6064
python_version = "3.11",
6165
requirements_by_platform = {
@@ -90,17 +94,20 @@ dev_python.override(
9094
)
9195

9296
dev_pip = use_extension(
93-
"//python/private/pypi:pip.bzl",
94-
"pip_internal",
97+
"//python/extensions:pip.bzl",
98+
"pip",
9599
dev_dependency = True,
96100
)
97101
dev_pip.parse(
98-
download_only = True, # this will not add the `sdist` values to the transitive closures at all.
102+
download_only = True,
103+
experimental_index_url = "https://pypi.org/simple",
99104
hub_name = "dev_pip",
100105
python_version = "3.11",
101106
requirements_lock = "//docs:requirements.txt",
102107
)
103108
dev_pip.parse(
109+
download_only = True,
110+
experimental_index_url = "https://pypi.org/simple",
104111
hub_name = "pypiserver",
105112
python_version = "3.11",
106113
requirements_lock = "//examples/wheel:requirements_server.txt",

‎docs/pypi-dependencies.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,59 @@ leg of the dependency manually. For instance by making
307307
perhaps `apache-airflow-providers-common-sql`.
308308

309309

310+
(bazel-downloader)=
311+
### Multi-platform support
312+
313+
Multi-platform support of cross-building the wheels can be done in two ways - either
314+
using {bzl:attr}`experimental_index_url` for the {bzl:obj}`pip.parse` bzlmod tag class
315+
or by using the {bzl:attr}`pip.parse.download_only` setting. In this section we
316+
are going to outline quickly how one can use the latter option.
317+
318+
Let's say you have 2 requirements files:
319+
```
320+
# requirements.linux_x86_64.txt
321+
--platform=manylinux_2_17_x86_64
322+
--python-version=39
323+
--implementation=cp
324+
--abi=cp39
325+
326+
foo==0.0.1 --hash=sha256:deadbeef
327+
bar==0.0.1 --hash=sha256:deadb00f
328+
```
329+
330+
```
331+
# requirements.osx_aarch64.txt contents
332+
--platform=macosx_10_9_arm64
333+
--python-version=39
334+
--implementation=cp
335+
--abi=cp39
336+
337+
foo==0.0.3 --hash=sha256:deadbaaf
338+
```
339+
340+
With these 2 files your {bzl:obj}`pip.parse` could look like:
341+
```
342+
pip.parse(
343+
hub_name = "pip",
344+
python_version = "3.9",
345+
# Tell `pip` to ignore sdists
346+
download_only = True,
347+
requirements_by_platform = {
348+
"requirements.linux_x86_64.txt": "linux_x86_64",
349+
"requirements.osx_aarch64.txt": "osx_aarch64",
350+
},
351+
)
352+
```
353+
354+
With this, the `pip.parse` will create a hub repository that is going to
355+
support only two platforms - `cp39_osx_aarch64` and `cp39_linux_x86_64` and it
356+
will only use `wheels` and ignore any sdists that it may find on the PyPI
357+
compatible indexes.
358+
359+
```{note}
360+
This is only supported on `bzlmd`.
361+
```
362+
310363
(bazel-downloader)=
311364
### Bazel downloader and multi-platform wheel hub repository.
312365

‎examples/bzlmod/MODULE.bazel‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,9 @@ pip.parse(
221221
"host",
222222
],
223223
hub_name = "pip",
224+
# Parse all requirements files for the same lock file on all OSes, this will
225+
# become the default with 1.0 release
226+
parse_all_requirements_files = True,
224227
python_version = "3.10",
225228
# The requirements files for each platform that we want to support.
226229
requirements_by_platform = {

‎examples/bzlmod/MODULE.bazel.lock‎

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎python/private/pypi/config_settings.bzl‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,13 @@ def config_settings(
148148
)
149149

150150
def _dist_config_settings(*, suffix, plat_flag_values, **kwargs):
151+
if kwargs.get("constraint_values"):
152+
# Add python version + platform config settings
153+
_dist_config_setting(
154+
name = suffix.strip("_"),
155+
**kwargs
156+
)
157+
151158
flag_values = {_flags.dist: ""}
152159

153160
# First create an sdist, we will be building upon the flag values, which
@@ -277,7 +284,7 @@ def _plat_flag_values(os, cpu, osx_versions, glibc_versions, muslc_versions):
277284

278285
return ret
279286

280-
def _dist_config_setting(*, name, is_pip_whl, is_python, python_version, native = native, **kwargs):
287+
def _dist_config_setting(*, name, is_python, python_version, is_pip_whl = None, native = native, **kwargs):
281288
"""A macro to create a target that matches is_pip_whl_auto and one more value.
282289
283290
Args:
@@ -310,6 +317,10 @@ def _dist_config_setting(*, name, is_pip_whl, is_python, python_version, native
310317
# `python_version` setting.
311318
return
312319

320+
if not is_pip_whl:
321+
native.config_setting(name = _name, **kwargs)
322+
return
323+
313324
config_setting_name = _name + "_setting"
314325
native.config_setting(name = config_setting_name, **kwargs)
315326

0 commit comments

Comments
 (0)