Repository navigation
Opt-in macro for a new clean API? Subset of functions with no known issues #54
Description
The Python C API has a long history. Many functions known to be ackwards are kept because of "backward compatibility". The cost of removing them is higher than the benefits.
My use case is: I would like to write a new C extension with only APIs with no known issues. For example, I would like to avoid borrowed references, ambiguous return values, etc.
Can we define a subset of the Python C API for that?
It can be a practical implementation of issue #53: make the C API smaller. Such subset would be smaller by design.
In such subset, we can consider to have different a different deprecation policy: more aggressive. Developers who opt-in for such API are aware of the ongoing effort to fix the C API and so are more eagger to update their code than unmaintained projects which continue to work only because the C API didn't break so far.
In my first C API hack in 2017, I added multiple macros to opt-out from APIs known to have issues. Examples:
Py_NEWCAPI: future-proof name, right? :-)Py_NEWCAPI_NO_STRUCTPy_NEWCAPI_NO_MACROPy_NEWCAPI_BORROWED_REF
I'm not sure if such design is future-proof, C extensions may not be aware that next opt-out options would be added. Maybe it's better to have a single big button: "drop all legacy, give me only good API".
Question: if we design such subset, would it be ok to override names of the regular API? For example, can we rename PyDict_GetItemRef() (regular API) as PyDict_GetItem()? How would it work at the ABI level? Would it be confusing when exchanging code written with the two APIs?
In issue #52, @markshannon proposed using different name prefixes. Rather than using suffixes.
So maybe PyDict_GetItemRef() could become PyNew_Dict_GetItem(), PyNewFinal2CorrectThisThime_Dict_GetItem()? :-)
Activity
markshannon commentedon Jun 23, 2023
Please don't use
New. It won't always be new.E.g. New College is over 700 years old.
vstinner commentedon Jun 26, 2023
Oh sorry, I wasn't clear. The "Py_NEWCAPI" macro was a temporary name that I chose in 2017 in my experimental project until I could come up with a better name. It was just to show the option of having many macros. I don't know if we need 1 or many of them.
vstinner commentedon Jun 26, 2023
Windows <windows.h> has something like that:
#define WIN32_LEAN_AND_MEAN:https://devblogs.microsoft.com/oldnewthing/20091130-00/?p=15863
encukou commentedon Jun 26, 2023
This seems like a solution. AFAIK, the point of this repo is to problems, so we can design solutions that address multiple related problems at once.
IMO, it would be good to keep the actual problems separate: in that case you could take a proposed solution, and check which problems it solves, keeps, or makes worse.
encukou commentedon Jul 11, 2023
Py_LIMITED_APIwas already used for this, back when the set of "known problems" was much smaller. But nowadays, deprecating things from it is difficult. It “upper” bound:Py_LIMITED_API=Xbasically means “nothing newer than X”. The “lower” bound -- “avoid API considered problematic in X” -- is stuck on Python 3.2.How about adding such a “lower” bound, reusing
Py_LIMITED_API's versioning scheme?Something like
Py_NO_LEGACY=0x30c0000?vstinner commentedon Jul 18, 2023
If a single opt-in macro causes too many changes, maybe it would be better to have a smoother migration plan by having multiple macros to only exclude a small set of functions. Examples:
PyObject*is annoying :-)And maybe we can also have a "superset" which combines all other opt-in macros.
Yeah, I think that it's important that a C extension can do the work once for a specific API version and let expect to have no new error/warning until the maintainers decide to upgrade to the next Py_NO_LEGACY version.
API versionning sounds like a good idea.
The C API is big and it's hard to keep track of each C API changes.
vstinner commentedon Jul 19, 2023
See also issue #62: How can an user access old removed functions? Can a 3rd party project provide them?
davidhewitt commentedon Jul 19, 2023
I just wanted to note that with these proposed macros to remove functions, it might be easier for other language bindings if the macros do indeed only remove functions (and not redefine them).
A recent example which we had to deal with in PyO3 was the
PY_SSIZE_T_CLEANmacro, which had the effect of redefiningPyObject_CallFunctionas_PyObject_CallFunction_SizeT. Given thePyObject_CallFunctionsymbol continued to exist in the ABI, it took us some time to notice that we were calling an outdated symbol when functionality changed on Python 3.10.vstinner commentedon Sep 1, 2023
When reviewing python/cpython#108768, I would like to have a macro to opt-out for C API known to ignore exceptions silently: PyDict_GetItem(), PyMapping_HasKey(), PyObject_HasAttrString(), etc.
Even if I'm following closely the Python development, I know the C internals of Python, but it's still really hard to me to remind which functions have a clean error management, and which ones have a crappy error management (ignore silently errors).
scoder commentedon Oct 18, 2023
Does it have to be? If someone sets
Py_LIMITED_APIto, say, Python 3.15, and Python 3.15 has decided to remove a "stable" C-API function after a deprecation period, then that function can be excluded from the 3.15+ API and only left in#if Py_LIMITED_API < 3.15.That doesn't solve the problem of having to keep the function available in the ABI, but then, if there's a reason to remove the function, there's probably also a reason to break the ABI for it. But that's the general ABI problem, not a problem with
Py_LIMITED_APIor any other versioning macro.encukou commentedon Oct 24, 2023
Proposed solution issue: capi-workgroup/api-evolution#24
vstinner commentedon Mar 11, 2024
I close this issue, let's continue the discussion there.