Add ppc64le support by clnperez · Pull Request #165 · astral-sh/python-build-standalone

clnperez · 2023-03-07T23:24:47Z

Adding support for ppc64le. As with s390x support (added 7a0c31d), this only needed additions for the build triples and a new build target defined. I'd like to get this added to use with the envoy builds. (See envoyproxy/envoy#25090).

Signed-off-by: Christy Norman <christy@linux.vnet.ibm.com>

indygreg · 2023-03-08T00:12:35Z

I haven’t looked at the code yet, but missing platform additions are very much welcomed and appreciated! So let me know when this is ready to review and we’ll get it merged.

ninehills · 2023-03-08T05:32:09Z

I am also working on the porting of the envoy to the ppc64le architecture. Next, I will test this PR.~

I find some bug in this pr, and fix it in:

clnperez#1

Then trigger the github action:

https://github.com/ninehills/python-build-standalone/actions/runs/4361856285

ninehills · 2023-03-08T08:29:57Z

Local build successfully. @clnperez you can merge my pr.

clnperez · 2023-03-08T14:44:34Z

thanks @ninehills!

fix ppc64le support

clnperez · 2023-03-08T18:13:47Z

@indygreg happy to hear that! i merged @ninehills's changes into this PR. Would you rather me squash and re-push? Or will you just do a squash & merge?

ninehills · 2023-03-09T06:14:39Z

I found that some of the validation and publishing logic still needs to be modified. I will continue testing.

ninehills · 2023-03-09T07:44:22Z

github workflow run successfully: https://github.com/ninehills/python-build-standalone/actions/runs/4371696198

pr already pushed.

fix validation and release

clnperez · 2023-03-10T17:38:47Z

removing from WIP status, as @ninehills has tested and the workflow ran successfully on my fork. PTAL @indygreg and lmk how you want me to handle the multiple commits. thanks!

clnperez · 2023-03-16T17:42:01Z

@indygreg can you take a look at this one?

indygreg

This PR looks great!

I think the only needed change is the platform name is being advertised as linux-powerpc64le. But I think it should be linux-ppc64le. (I'm not 100% sure about this.)

CPython's configure picks up the platform name from the detected target triple. From build logs we see: checking host system type... powerpc64le-unknown-linux-gnu. So it goes with powerpc64le.

In order to change / fix this, we'll need to patch configure.ac to set an appropriate _PYTHON_HOST_PLATFORM value when the target triple is powerpc64le-unknown-linux-*. The existing patch-apple-cross.patch does this. I'd write a similar patch or rename/extend this existing patch to cover ppc64le. Make sure PYTHON_ARCH in build-cpython.sh matches the new platform name.

If you'd like some help with this, just let me know and I can do it. I'm also open to contrasting opinions that the platform name of powerpc64le is actually correct.

This change looks great otherwise! Thanks for your hard work.

indygreg · 2023-03-18T22:15:08Z

    PYTHON_ARCH="mips64el-linux-gnuabi64"
    ;;
+ppc64le-unknown-linux-gnu)
+    PYTHON_ARCH="powerpc64le-linux-gnu"


According to https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#manylinux / PEP 599, I think this should be ppc64le.

i think it should be as well, and i see NixOS/nixpkgs#168083 referencing PEP600 and coming to the same conclusion (setting pythonHostPlatform and therefore _PYTHON_HOST_PLATFORM to ppc64le : https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/cpython/default.nix#L209) . But... the cpython configure.ac that @ninehills linked does have powerpc64le- instead

I see that you're using PYTHON_ARCH to create a symlink. If we change this will this impact that at all or be fine? I was trying to poke around and see where things land as far as directory structure on my dev VM running ppc64le -- but it's currently unavailable so I just have to bother you with questions.

I changed PYTHON_ARCH from powerpc64le-linux-gnu to ppc64le-linux-gnu, then built failed:

Github action: powerpc64le-linux-gnu

It seems that CPython has hard-coded this value. Unless we patch the CPython source code, it's best to keep the value of PYTHON_ARCH as powerpc64le-linux-gnu.

build error message:

Traceback (most recent call last): File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1088, in <module> sys.exit(main()) ^^^^^^ File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1071, in main build_cpython( File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 803, in build_cpython fh.write(build_env.get_output_archive("python")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/buildenv.py", line 116, in get_output_archive data = normalize_tar_archive(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/utils.py", line 338, in normalize_tar_archive filedata = tf.extractfile(ti) ^^^^^^^^^^^^^^^^^^ File "/ opt /hostedtoolcache/Python/3.11.2/x64/lib/python3.11/tarfile.py", line 2143, in extractfile return self.extractfile(self._find_link_target(tarinfo)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/ opt /hostedtoolcache/Python/3.11.2/x64/lib/python3.11/tarfile.py", line 2452, in _find_link_target raise KeyError("linkname %r not found" % linkname) KeyError: "linkname 'python/install/lib/python3.9/config-3.9d-ppc64le-linux-gnu/libpython3.9d.a' not found" make: *** [Makefile:278: /home/runner/work/python-build-standalone/python-build-standalone/build/cpython-3.9.16-ppc64le-unknown-linux-gnu-debug.tar] Error 1 Error: Process completed with exit code 2.

ninehills · 2023-03-20T03:31:57Z

It seems that CPython can only accept powerpc64le-linux-gnu during compilation (possibly referencing the architecture naming of gcc), but the standard architecture name we built should be ppc64le according to the standard.

https://github.com/python/cpython/blob/5e6661bce968173fa45b74fa2111098645ff609c/configure.ac#L1011

Other discussions:
pypa/manylinux#687

clnperez · 2023-03-28T22:32:33Z

@ninehills how did you package & test this with envoy when you built it? i can cross-compile just fine, but am having no luck figuring out what was built and how it is turned into a consumable release archive.

ninehills · 2023-03-29T04:54:40Z

@ninehills how did you package & test this with envoy when you built it? i can cross-compile just fine, but am having no luck figuring out what was built and how it is turned into a consumable release archive.

https://github.com/ninehills/python-build-standalone/actions/runs/4524288535

Now build is success.

clnperez · 2023-03-29T19:25:34Z

@indygreg can you weigh in?

i can see from the github workflow that the archives are built -- but when I run just the power cross-build (./build-linux.py --target-triple ppc64le-unknown-linux-gnu) -- i don't get one. so i tried ./build-linux.py --target-triple ppc64le-unknown-linux-gnu --python cpython-3.11 --optimizations noopt and then I get:
compressing Python archive to /home/christy/go/src/github.com/indygreg/python-build-standalone/dist/cpython-3.11.1-ppc64le-unknown-linux-gnu-noopt-20230309T0934.tar.zst

It doesn't seem that the type of build via build-flags would be the deciding factor on whether or not something shows up in /dist.

indygreg · 2023-04-08T23:33:57Z

It seems like ppc64le is what's in PEP 599 / 600 and was the historically accurate target name. But powerpc64le seems to be what people want to transition to. And CPython's configure.ac appears to insist on powerpc64le.

So I think it makes sense to keep go with powerpc64le since it will be future compatible. But I wouldn't be surprised if we run into issues installing binary wheels when using this platform name since ppc64le was historically used. 🤷‍♂️

clnperez · 2023-04-13T21:13:03Z

thanks @indygreg! it is a messy space for sure 🤞🏻

clnperez · 2023-04-13T21:14:02Z

oh -- do i need to squash these commits? anything else to merge?

indygreg · 2023-04-23T23:55:13Z

I have this cherry picked into my local repo. Will push this to main once CI passes.

Thanks for your hard work and patience!

clnperez · 2023-04-24T19:04:01Z

Thanks @indygreg. Does this project have a release schedule we should wait for, or can we just request a new one be made so we can add this to the envoy builds?

indygreg · 2023-04-24T20:28:03Z

There's no formal release schedule. I try to cut releases after new Python versions, important security vulns, or other major changes. We're several weeks overdue for a release. Please create a new issue to track the next release. Hopefully I'll find time in the next 1-2 weeks.

clnperez · 2023-05-19T19:27:57Z

adding to rules_python via bazel-contrib/rules_python#1234. thanks for the release @indygreg 🎉

Add ppc64le support

a792964

Signed-off-by: Christy Norman <christy@linux.vnet.ibm.com>

clnperez mentioned this pull request Mar 7, 2023

ppc64le CI build always fails envoyproxy/envoy#25090

Open

fix ppc64le support

e87f5da

Merge pull request #1 from ninehills/ppc64le-support

26ab732

fix ppc64le support

ninehills added 3 commits March 9, 2023 14:17

fix ppc64le validation

8fe40af

add ppc64le release

9d5bb23

Merge branch 'clnperez:ppc64le-support' into ppc64le-support

14f038f

Merge pull request #2 from ninehills/ppc64le-support

eeff0c1

fix validation and release

clnperez changed the title ~~WIP: Add ppc64le support~~ Add ppc64le support Mar 10, 2023

indygreg reviewed Mar 18, 2023

View reviewed changes

indygreg approved these changes Apr 8, 2023

View reviewed changes

indygreg closed this in e3ae98b Apr 24, 2023

clnperez mentioned this pull request Apr 25, 2023

New Release Tracker (to pick up ppc64le support) #171

Closed

Uh oh!

Uh oh!

Conversation

clnperez commented Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

indygreg commented Mar 8, 2023 via email

Uh oh!

ninehills commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ninehills commented Mar 8, 2023

Uh oh!

clnperez commented Mar 8, 2023

Uh oh!

clnperez commented Mar 8, 2023

Uh oh!

ninehills commented Mar 9, 2023

Uh oh!

ninehills commented Mar 9, 2023

Uh oh!

clnperez commented Mar 10, 2023

Uh oh!

clnperez commented Mar 16, 2023

Uh oh!

indygreg left a comment

Choose a reason for hiding this comment

Uh oh!

indygreg Mar 18, 2023

Choose a reason for hiding this comment

Uh oh!

clnperez Mar 22, 2023

Choose a reason for hiding this comment

Uh oh!

ninehills Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

ninehills commented Mar 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clnperez commented Mar 28, 2023

Uh oh!

ninehills commented Mar 29, 2023

Uh oh!

clnperez commented Mar 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

indygreg commented Apr 8, 2023

Uh oh!

clnperez commented Apr 13, 2023

Uh oh!

clnperez commented Apr 13, 2023

Uh oh!

indygreg commented Apr 23, 2023

Uh oh!

clnperez commented Apr 24, 2023

Uh oh!

indygreg commented Apr 24, 2023

Uh oh!

clnperez commented May 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clnperez commented Mar 7, 2023 •

edited

Loading

ninehills commented Mar 8, 2023 •

edited

Loading

ninehills commented Mar 20, 2023 •

edited

Loading

clnperez commented Mar 29, 2023 •

edited

Loading