Skip to content

mark extension types as immutable#5023

Merged
da-woods merged 4 commits intocython:masterfrom
maxbachmann:immutable
Jan 1, 2023
Merged

mark extension types as immutable#5023
da-woods merged 4 commits intocython:masterfrom
maxbachmann:immutable

Conversation

@maxbachmann
Copy link
Copy Markdown
Contributor

@maxbachmann maxbachmann commented Sep 12, 2022

Since python/cpython#25520 types are automatically marked as immutable if they are static.
While we still have the Py_TPFLAGS_HEAPTYPE hack in place we need to manually mark our types as immutable.

@maxbachmann
Copy link
Copy Markdown
Contributor Author

This leads e.g. to the following issue:

Python 3.10.6 (main, Aug  2 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from rapidfuzz.distance import Editop
>>> Editop.__name__ = 'test'
Segmentation fault (core dumped)

I will add a test for this later today.

@maxbachmann
Copy link
Copy Markdown
Contributor Author

maxbachmann commented Sep 12, 2022

@da-woods apparently we do rely on the mutability, so I am unsure how we want to fix this.

edit: as far as I can see this is a new test which was introduced as part of: #4955. Is this postinit usage something we actually support, or was this just added this way, since it did not fail?

@da-woods
Copy link
Copy Markdown
Contributor

Ah I probably didn't mean to rely on the mutability. I basically just enabled all the CPython dataclasses tests that seemed to pass. However, Cython dataclasses aren't Python dataclasses so they're allowed to differ in places. I'll revert the specific tests that rely on it.

I wasn't sure if immutability should be considered a feature or a side effect (i.e. whether there was a harm in leave things mutable). However, if you can cause segfaults with it so easily then we should definitely keep it.

@da-woods
Copy link
Copy Markdown
Contributor

The dataclass test is easily disabled. The second failing test is to do with setting Py_TPFLAGS_SEQUENCE on the memoryview wrapper class. I think that's well worth doing since it makes it behave like a sequence for pattern matching. It needs a bit of thought about how we do it with immutable classes though since it can't be done after the fact.

I think a @cython.tp_flags decorator might be useful (taking either "sequence" or "mapping") - I could imagine users might want to register their own classes like this so it shouldn't just be a utility code hack for memoryview.

I think proceed with this PR because we will want to make these immutable again, but fixing up the cases where we(/I) accidentally used mutability won't be instant.

@da-woods
Copy link
Copy Markdown
Contributor

Link 1 #5026

da-woods added a commit to da-woods/cython that referenced this pull request Sep 13, 2022
via an additional decorator for cdef classes.

Closes cython#5027 and removes blocker for cython#5023
@da-woods
Copy link
Copy Markdown
Contributor

#5030 solves the remaining blocker for this. It is adding a feature though so I'll be patient and let it be reviewed properly.

@da-woods
Copy link
Copy Markdown
Contributor

da-woods commented Dec 6, 2022

@scoder I'd quite like to get this in the next release. It's been a bug for a while but it seems like something to fix before people start relying on the wrong behaviour.

It could do with a test - I'll have a go at writing one later today.

It's blocked by #5030 (but there's other options if you don't like that - in the short-term just to disable the tests)

da-woods added a commit that referenced this pull request Dec 31, 2022
Registering cdef classes with collections.abs.sequence only works to set the type flag if cdef classes are mutable. Unfortunately we want them to be immutable since it's possible to corrupt memory with mutable cdef classes (#5023).

Therefore, add a Cython directive for marking cdef classes as a sequence, but only allow it in utility code for now. This is a cutdown version of #5030 - I don't want to add new public features without consensus, but I do want to merge the fix for mutable classes. Therefore I'm adding the feature privately for now.
@da-woods
Copy link
Copy Markdown
Contributor

Closing/reopening to rerun tests

@da-woods da-woods closed this Dec 31, 2022
@da-woods da-woods reopened this Dec 31, 2022
@da-woods da-woods added this to the 3.0 milestone Jan 1, 2023
@da-woods da-woods merged commit 5e5b242 into cython:master Jan 1, 2023
@da-woods
Copy link
Copy Markdown
Contributor

da-woods commented Jan 1, 2023

Thanks @maxbachmann

da-woods added a commit that referenced this pull request Dec 23, 2025
via an additional decorator for cdef classes.

Closes #5027 and removes blocker for #5023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants