gh-62434: Add unicodedata.aliases() - #155737
Open
Jayman2000 wants to merge 4 commits into
Open
Conversation
Here’s one of the changes that 71f660e made: diff --git a/Tools/unicode/makeunicodedata.py b/Tools/unicode/makeunicodedata.py index 50d1acace81..db0f8ec 100644 --- a/Tools/unicode/makeunicodedata.py +++ b/Tools/unicode/makeunicodedata.py @@ -58,7 +58,7 @@ PUA_16 = range(0x100000, 0x10FFFE) # we use this ranges of PUA_15 to store name aliases and named sequences NAME_ALIASES_START = 0xF0000 -NAMED_SEQUENCES_START = 0xF0100 +NAMED_SEQUENCES_START = 0xF0200 old_versions = ["3.2.0"] Unfortuantely, there was a few different places where the old value of NAMED_SEQUENCES_START (0xF0100) was still being used even though it had been changed over 14 years ago. This change updates those places so that they use the current value of NAMED_SEQUENCES_START.
Before this change, there was a for loop that iterated over each entry in NameAliases.txt [1]. Each entry in NameAliases.txt has three fields. Before this change, the for loop called the first field “char”, the second field “name” and the third field “abbrev”. “char” was not a good name for the first field. There is two different ways that the word “char” can be interpreted. First, you could interpret “char” as being short for “character”. Second, you could interpret “char” as meaning the C data type char. Both interpretations would cause confusion here. The first field does not contain a character, and it does not contain a C char value. Instead, the first field contains a code point written as a hexadecimal number. “name” was not a good name for the second field. Unicode code points can have both names and aliases. The second field contains an alias, not the code point’s name. “abbrev” was not a good name for the third field. “abbrev” is (presumably) short for “abbreviation”. The third field does not contain an abbreviation. Instead, the third field contains the alias’s type. This change renames those variables to “aliased_codepoint”, “alias” and “alias_type”. These names are more accurate, and they follow the names used by NameAliases.txt itself: > # FORMAT > # > # Each line has three fields, as described here: > # > # First field: Code point > # Second field: Alias > # Third field: Type I had considered calling the first variable just “codepoint” instead of “aliased_codepoint”. I chose the name “alised_codepoint” in order to prepare for a future change. In that future change, the relevant code will be modified so that there is an additional variable that contains a code point. The name “aliased_codepoint” will make it easier for readers to understand the difference between the two variables. I had considered calling the last variable “type” in order to match the name in that above quotation. I decided to use the name “alias_type” in order to avoid overwriting the built-in “type” variable. --- Similarly, there was a for loop that iterated over UnicodeData.aliases. Before this change, that for loop called the data inside each element of UnicodeData.aliases “name” and “codepoint”. In order to keep things consistent, this change also updates that for loop so that it uses the term “alias” instead of “name” and the term “aliased_codepoint” instead of “codepoint”. [1]: <https://www.unicode.org/Public/17.0.0/ucd/NameAliases.txt>
In some situations, you have a character and you want a human-readable
description of that character. For example, I was writing some code that
parses a certain file format. In that file format, you are not allowed
to use certain characters. I wanted to be able to show the user an error
that looked something like this if the user used one of the banned
characters:
ERROR: You’re not allowed to use U+0000 NULL characters.
For most characters, you can use unicodedata.name() in order to get a
human-readable description of that character. Unfortunately,
unicodedata.name() does not work for U+0000 because U+0000 and many
other characters do not have names. In order to get a human-readable
description of U+0000, you need to look up its name aliases.
Before this change, you could turn an alias into a character (via
"\N{…}" and unicodedata.lookup()), but there was no way to turn a
character into its aliases. This change adds a new unicodedata.aliases()
function that allows you to turn a character into its aliases.
This commit implements pythongh-62434. In that thread, there were many
different proposals for potential ways to expose this data to Python
programs. This commit most closely matches flying-sheep’s proposal [1].
Here are the differences between what’s implemented in this commit, and
what flying-sheep proposed:
• flying-sheep proposed that unicodedata.aliases() would have an
optional type argument that could be used to filter the output by
alias type. The version of unicodedata.aliases() added by this commit
does not include an optional type argument. You can already filter the
output to your liking by simply using the returned dictionary. In
other words, unicodedata.aliases("\n", type="control") would have been
the same as unicodedata.aliases("\n")["control"]. Based on that fact,
having an optional type argument didn’t seem very useful.
• flying-sheep proposed that the return value would use the dict, str
and list data types. Two of those data types are mutable and one is
immutable. This commit uses the frozendict, str and tuple data types.
All three of those data types are immutable which is more consistent.
Plus, the purpose unicodedata.aliases() is to lookup data that can be
used later, not to create something that will later be modified. (I
mean, you could theoretically modify the dict or the lists that
flying-sheep’s proposal would have returned, but why would you?)
[1]: <python#62434 (comment)>
Documentation build overview
|
It turns out that it’s not valid to use a const variable in a static_assert [1]. We could get around this by making alias_type_labels_start a constexpr variable instead of a const variable, but the constexpr keyword wasn’t added to C until C23 [2], and we aren’t allowed to use features from C23 yet [3]. This change fixes building Python on Windows. [1]: <https://stackoverflow.com/a/78222714/7593853> [2]: <https://cppreference.com/c/language/constexpr> [3]: <https://peps.python.org/pep-0007/#c-standards>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In some situations, you have a character and you want a human-readable description of that character. For example, I was writing some code that parses a certain file format. In that file format, you are not allowed to use certain characters. I wanted to be able to show the user an error that looked something like this if the user used one of the banned characters:
For most characters, you can use
unicodedata.name()in order to get a human-readable description of that character. Unfortunately,unicodedata.name()does not work for U+0000 because U+0000 and many other characters do not have names. In order to get a human-readable description of U+0000, you need to look up its name aliases.At the moment, you can turn an alias into a character (via
"\N{…}"andunicodedata.lookup()), but there’s no way to turn a character into its aliases. This pull request adds a newunicodedata.aliases()function that allows you to turn a character into its aliases.See the commit messages for additional details.
Closes #62434.