Repository navigation
ARM builds? #84
Description
This is a very speculative issue. I'm aware that PEP 513 says manylinux1 is only defined for i686 and x86_64. But it would be nice if instructions telling people to pip install compiled modules worked easily and quickly on things like the Raspberry Pi too. It might also be valuable to the few brave souls trying to make Python work on mobile platforms.
So this issue is a place to work out what would be needed.
- Architecture requirements: there are a lot of different versions of ARM, and there are confusingly similar looking version schemes for the architectures (e.g.
ARMv7) and cores (e.g.ARM11). There may also be optional modules. Debian supports three buckets of ARM architectures:armhfrequires ARMv7 and a floating point unit (hf = hardware float).armelsupports older architectures, and cores without an FPU.arm64is for the newer 64-bit processors. In terms of raspis:
- Raspi 0 and 1 have ARMv6 architecture, so Debian armhf does not support them. They do have FPUs, though, so Raspian is compiled to make use of this.
- Raspi 2 has ARMv7
- Raspi 3 has a 64-bit ARMv8 architecture.
-
Minimum library versions: These probably don't need to be all that old, because the systems I think this would target are likely to be running a relatively recent distro. But we'd need to work out what they are.
-
Distro for building: What distro would be the equivalent of the Centos 5.11 we use for building x86 and x64 manylinux wheels?
-
Build platform: Are there any CI services which offer ARM builds? Is virtualisation fast enough? Can we make it easy for people to build wheels on their own Raspberry Pi or similar device.
Activity
rmcgibbo commentedon Dec 12, 2016
I am not personally very knowledgeable about ARM, so I don't know the direct answers to any of those questions. It would be really excellent if manylinux could support ARM at some point in the future though -- I have the impression that it's a growing platform.
For the manylinux1 spec, @njsmith and I leveraged a lot of Continuum's work to define the base distro and minimum library versions. Since the manylinux1 release, Continuum has added some support for ARM to conda and their packages, so perhaps this would be a good place to begin to identify best practices.
takluyver commentedon Dec 12, 2016
Thanks, that's good to know. It looks like they're targeting ARMv7 and using Raspbian to run the builds.
njsmith commentedon Dec 12, 2016
KellenSunderland commentedon Aug 17, 2017
Using dockcross works quite well for ARM builds. I've created a few packages that use cross-compilation rather than virtualization. Builds that take hours on a Pi typically take a minute or two. This Dockerfile for example builds packages that run well for me with a Pi3 + Raspbian jessi. https://github.com/dockcross/dockcross/blob/master/linux-armv7/Dockerfile
takluyver commentedon Aug 20, 2017
Since my motivation for this was getting Python packages on a Raspberry Pi, I'll drop a link to piwheels, @bennuttall's project to compile wheels for that platform.
bennuttall commentedon Aug 21, 2017
An overview:
Although the hardware spec of Pi 1/Zero is ARMv6, Pi 2 is ARMv7 and Pi 3 is ARMv8, the reality is that Raspbian runs in ARMv6 user space regardless on your hardware. When you build a wheel on a Pi it's tagged armv6l on Pi 1/Zero and armv7l on Pi 2/3, but really it's always ARMv6.
So my project piwheels provides wheels for Raspberry Pi, but I wouldn't bank on the platform wheels working on other ARM boards. There's no particular need for maintainers to upload their wheels to PyPI as Pi users can use piwheels as their repo instead.
Soon, in Raspbian, pip will be configured to use piwheels. (update: this is now true)QJKX commentedon Feb 9, 2019
Python on Termux (an android Linux environment) would also benefit from arm wheels.
NotAFile commentedon Feb 9, 2019
We're having this problem with testing ARM in CI. The lack of wheels forces us to compile all dependencies for a fresh build, which takes around 20 minutes.
vielmetti commentedon Feb 15, 2019
There is a CI system available, Drone Cloud, that can build natively for 32-bit and 64-bit Arm systems. There's also bare metal resources in the @WorksOnArm project to tackle this. Happy to help tackle this, especially for arm64 (aarch64).
vielmetti commentedon Feb 15, 2019
@bennuttall - does piwheels support 32-bit Arm only, or do you also have 64-bit arm64 support?
bennuttall commentedon Feb 15, 2019
We only provide wheels for 32-bit systems.
vielmetti commentedon Feb 23, 2019
Referencing https://marcin.juszkiewicz.com.pl/2019/02/23/we-need-arm64-systems-for-developers-again/ which describes the resulting pain that
pip install scipy, which takes a few seconds on x86, takes hours on arm64 because of lack of wheels.hrw commentedon Feb 23, 2019
I discussed that with Pypi devs some time ago on irc.
Whole service depends on whatever authors upload. So if we want non-x86 wheels then we need to convince authors to build and upload arm wheels.
hrw commentedon Feb 23, 2019
And when it comes to what to use as a minimal...
I think that Debian jessie is sensible minimum. Eventually CentOS 7.
Older releases did not supported arm64.
ralphie-boy commentedon Apr 16, 2019
I need to do python development that can include the python SpeechRecognition package. Samsung recently released the Galaxy Book 2 with the latest 64 bit Qualcomm Snapdragon processor (850). It comes with Windows 10 S which can be upgraded to Windows Pro. Windows Pro will support WSL (Windows Subsystem for Linux). WSL will allow me seamlessly run Ubuntu. Ubuntu will have built-in Python. The question is what happens when I try to install the SpeechRecognition package? I know for example this will not work on iPad. Will the Arm 64 architecture support this package?
31 remaining items
takluyver commentedon Jul 2, 2020
To be clear, you don't need manylinux support to build and use Python extensions for a particular ARM platform. https://www.piwheels.org/ already does this for Raspberry Pi, for example, and you could do something similar for ARM on AWS. So long as you know a particular target platform, you can compile on the same platform.
Manylinux is meant to make it possible to build a wheel on Fedora with one ARM chip and then install it on Ubuntu with a different one - which is obviously useful, but also much harder.
geoffreyblake commentedon Dec 18, 2020
This issue is pretty old and hasn't seen updates for ~6 months, but is there still debate on whether manylinux2014_aarch64 is stable and abiding by PEP 599?
I've set up a tester to automatically try some of the top wheels to see how Arm64 support is coming along using Graviton systems and the wheels built with manylinux2014_aarch64 appear to work well on both armv8.2 and armv8.0 systems running a cross-section of distros (Ubuntu 20.04, AL2, Centos8): https://github.com/geoffreyblake/arm64-python-wheel-tester
The failures seen by the tester for the packages tested are due to lack of wheels.
Don't build aarch64 wheels for now
Don't build aarch64 wheels for now
mayeut commentedon Mar 9, 2021
Original post from @webknjaz in #1029
mayeut commentedon Mar 9, 2021
@webknjaz,
Just to go a bit further, PEP600 which supersedes PEP599 doesn't restrict architectures at all.
The main reasons not to add those are:
ppc64)QEMU does not solve everything. It's painfully slow and GHA builds are timing out now and then on
ppc64lesince I could not get them running back on travis-ci yet.I've seen in the past and I'm still seeing crashes, only when running QEMU which prohibits me from debugging at all locally. I unfortunately do not have time to go through debugging/reporting/contributing/waiting for fixes in QEMU.
webknjaz commentedon Mar 9, 2021
Do you know you could increase the timeout per-job and per-step? https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepstimeout-minutes
mayeut commentedon Mar 9, 2021
@webknjaz,
there's a hard limit per job: 6 hours. that's the one we're hitting.
we'd need to rewrite again all scripts to bypass this limit and anyway, debugging in CI is not very sustainable, debugging in CI with QEMU even less no ?
MrNaif2018 commentedon Sep 20, 2021
@mayeut What about using qemu on aarch64 machines? CircleCI has added arm64 machines support and it's pretty stable, and qemu emulation runs almost at the same speed as native builds (because I guess arm32 is supported by kernel or the spec), and they might add arm32 machines soon.
As for testing locally: I agree, testing locally on amd64 with qemu is painful, but what about testing on aarch64 machines?
Oracle cloud (as part with partnership with arm I think) provides always free 24 gb ram and 4 cpu arm machine:
https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm
So it's not fully local debugging, but pretty convenient (vscode can run via ssh too)
Would that help moving forward the armv7 wheels creation?
Red-M commentedon Oct 10, 2021
I've just added such support for my own packages on arm64/aarch64 via using a full KVM to run the aarch64 docker containers for
manylinuxand its super slow. If you really want these, buy a bunch of RPi4s and run CI VMs on those via libvirt or KVM (hint, pimox). Otherwise piwheels exists, so make sure you provide ansdist.Edit: To make this painfully clear, virtulized (x86 emulating arm) is incredibility slow (in my scenario, up to ~3 times slower for releasing a single build on the same hardware)
What would be nice is providing buildchains in the
manylinuxcontainers for other cpu architectures to allow for easy building without the need for actual hardware and to provide already cross-compiled python versions that already exist for the x86 binaries (eg, drop/supporting a version of python brings it to all supported architectures). This will make the containers larger but at least building wheels (eg, for arm) won't suck as much to do so.No, cloud services do not count even if they are free. People need to be able to do this on any machine they own for reproduce-able builds (so the barrier to entry to confirmation is only running a docker container)
mayeut commentedon Oct 29, 2022
These do exists in some form outside of manylinux #1125 (comment) however, while it might be suitable for maturin / pyO3 where there is probably a decent/recent rust cross-compiler, it's not the case for the gcc toolchain as noted in that comment.
Moving this to discussion.