Repository navigation
The function unicodedata.normalize() should always return an instance of the built-in str type. #129569
Description
Bug report
Bug description:
The current implementation of unicodedata.normalize() returns a new reference to the input string when the data is already normalized. It is fine for instances of the built-in str type. However, if the function receives an instance of a subclass of str, the return type becomes inconsistent.
import unicodedata
class MyStr(str):
pass
s1 = unicodedata.normalize('NFKC', MyStr('Å')) # U+00C5 (already normalized)
s2 = unicodedata.normalize('NFKC', MyStr('Å')) # U+0041 U+030A (not normalized)
print(type(s1), type(s2)) # <class '__main__.MyStr'> <class 'str'>
In addition, passing instances of user-defined str subclasses can lead to unexpected sharing of modifiable attributes:
import unicodedata
class MyStr(str):
pass
original = MyStr('ascii string')
original.is_original = True
verified = unicodedata.normalize('NFKC', original)
verified.is_original = False
print(original.is_original) # False
The solution would be to use the PyUnicode_FromObject() API for early returns in the normalize() function implementation instead of Py_NewRef() to make sure that the function always returns an instance of the built-in str type.
CPython versions tested on:
3.11, 3.13
Operating systems tested on:
Windows
Activity
picnixz commentedon Feb 2, 2025
Is it really an issue that we return the object as is? Can't we document that behaviour instead?
Some existing code may rely on this feature so I think we should have a proper depreciation period where we would warn in the future that non-str objects will not be accepted and that their underlying str() representation will be used.We also need to document that change. I'm not against making a true string for the normalized value but this needs to be communicated properly to the end user.
serhiy-storchaka commentedon Feb 2, 2025
I consider this a bug. An optimization should not affect the type of the result, and returning the original object is merely an optimization.
picnixz commentedon Feb 2, 2025
I guess it also makes sense in that sense. Ok for categorizing it as a bug then.
picnixz commentedon Feb 2, 2025
Note that @corona10 removed the backports labels on the PR so maybe we could discuss whether to back port this here as well?
corona10 commentedon Feb 2, 2025
If @serhiy-storchaka considers this a bug, then we can backport it. I was confused between this as behavior changes or a bug :)
gh-129569: The function unicodedata.normalize() always returns built-…
pythongh-129569: The function unicodedata.normalize() always returns …
pythongh-129569: The function unicodedata.normalize() always returns …
malemburg commentedon Feb 21, 2025
Since this is a breaking change, shouldn't there be a deprecation period applied ? Also note that this behavior has been in existence for 20+ years.
Also see https://discuss.python.org/t/the-function-unicodedata-normalize-should-always-return-an-instance-of-the-built-in-str-type/79090/10 for the discussion.
vstinner commentedon Feb 21, 2025
Oh, I didn't notice that there was a discussion on discuss.python.org about this change.