prereq-build: clean up python check and narrow down its version whitelist - #11747

I don't like the first commit, I've been trying to work on reducing the presence of makefile rules in prereq-build

I don't see a particular problem with letting the regex be a catch-all, it removes the need to edit this when testing an unreleased version

in my opinion, issues with python should be fixed by means other than version control, hopefully before the version in question gets released. and if someone has a beta version installed they should already know to not expect perfect behavior from it.

the real question is, how much in Openwrt relies on having distutils as a separate module if at all, or is it just the presence of a function that was needed

I don't remember the situation. @neheb I would like to know your opinion. do you think this strengthens the idea of building our own python or is it an easy fix?

distutils are used by meson. Not needed with newer python versions though. I think @eli-schwartz known more.

Meson still relies on distutils without exception, even on Python 3.12 -- this is not the end of the world since that is still in alpha, and a third of a year away from merely reaching the first beta. Work is in progress to have Meson stop needing that, but it's not merged yet.

My expectation is that, probably, Meson will require distutils on python 3.10.2 and under, mostly because we need to check if the python is a Debian version of python and this is only knowable using distutils. If not for that, we would only need distutils on python 3.7 and under. (Also, if not for Debian I would have been motivated to solve this a year ago and we would not be having this discussion today.)

Yay, Debian...

Personally, I am not worried about people running an alpha release of python. They can pip install setuptools which provides distutils regardless of whether there is a copy of distutils in the stdlib, which means any software that relies on distutils for whatever reason will probably work.

... probably. Because setuptools has an, uhm, "evolving" API. (Meson does not do anything complex enough to ever hit that. numpy.distutils, among others, is so unstable that their official recommendation is to pin setuptools 59 and never upgrade.)

I don't like the first commit, I've been trying to work on reducing the presence of makefile rules in prereq-build

Okay, but running the very same thing twice just for another shortcut doesn't seem optimal either. Any suggestions? Maybe we can even drop the python one because python3 is now used everywhere anyway, but I didn't want to risk regressing anything.

I don't see a particular problem with letting the regex be a catch-all, it removes the need to edit this when testing an unreleased version

True, I don't particularly like to limit a user's option either. But what I don't like even more are reoccurring reports that something doesn't work, when it's known to not work. That's just a waste of everybody's time and it gets really annoying. And that's where 60af8d7 came from, where we got reports like every other week ("x fails to build"). And most of the time that was because of debian's weird python packaging. With that commit those reports stopped.

I can vaguely see that happening again with 3.12

They can pip install setuptools which provides distutils regardless of whether there is a copy of distutils in the stdlib, which means any software that relies on distutils for whatever reason will probably work.

But you'd have to change import distutils to from setuptools import distutils, right? As simple as that may be, every user of that package has to be adapted to it. Yay

But you'd have to change import distutils to from setuptools import distutils, right? As simple as that may be, every user of that package has to be adapted to it. Yay

Nope! Setuptools installs a global hook that runs as soon as you start the python binary. It causes all attempts to import distutils to instead import setuptools._distutils (it has an underscore BTW, it's an internal implementation detail) and return that as the imported "distutils" module. You cannot (trivially) import the standard library one at all.

Sneaky!

But alright, as per your expectation, I'll keep the distutils patch here as-is then. Which is to limit the OpenWrt's distutils check to <3.12 and don't check for any substitute for >=3.12, yet.

And that in return means we should probably start checking for setuputils on >=3.12. Because we use meson v0.61.x, which uses distutils, which then is only available via setuputils. Unless a future meson point release backports mesonbuild/meson#11133

But I'm not blindly adding that here, someone on 3.12 needs to verify and check that of course

@dhewg hi... this came up #12610

in short... package manager update the python version and the link still point to the old version that now it's not there anymore...
I was proposing a solution where we link for python3 directly instead of the version... and i remember there was already a version for this...

We have 2 solution...

• one build python ourself... (we also have a python
• link to python3 instead of the direct version and fix bugs as new version of python gets released...

Linking to python3 without the .10 at the end, is unlikely to actually make things work. You'll just get errors that modules aren't installed, because the module directory is versioned and so are .so compiled extensions.

Actually rebuilding against the new version is the best approach.

Linking to python3 without the .10 at the end, is unlikely to actually make things work. You'll just get errors that modules aren't installed, because the module directory is versioned and so are .so compiled extensions.

i see good point... yep i think just compiling python might be a good idea... but it's a real PITA...

I have in my RFC pull request some commits to handle the general stale symlink situation

I'll split it into a new PR

also I have #10800

I was going to update it to use a config option but I didn't do so yet, so it's a little stale

#12617 for the general symlink case

Rebased, and reworked the first commit, in case this is still of interest

This lacks a check for python3.12

Because this issue is 15 month old.
By now it's safe to assume that this doesn't go any further, so let's just close this (although I still consider the idea behind the patches a good idea)

Because this issue is 15 month old.

Oops. Sorry for the noise, that wasn't intentional. 🙈

I recommend you reopen and then have only the last commit presented. that's the one that we need and I think everyone would agree with