Skip to content

gh-62434: Add unicodedata.aliases() - #155737

Open
Jayman2000 wants to merge 4 commits into
python:mainfrom
Jayman2000:implement-gh-62434
Open

gh-62434: Add unicodedata.aliases()#155737
Jayman2000 wants to merge 4 commits into
python:mainfrom
Jayman2000:implement-gh-62434

Conversation

@Jayman2000

@Jayman2000 Jayman2000 commented Aug 13, 2026

Copy link
Copy Markdown

In some situations, you have a character and you want a human-readable description of that character. For example, I was writing some code that parses a certain file format. In that file format, you are not allowed to use certain characters. I wanted to be able to show the user an error that looked something like this if the user used one of the banned characters:

ERROR: You’re not allowed to use U+0000 NULL characters.

For most characters, you can use unicodedata.name() in order to get a human-readable description of that character. Unfortunately, unicodedata.name() does not work for U+0000 because U+0000 and many other characters do not have names. In order to get a human-readable description of U+0000, you need to look up its name aliases.

At the moment, you can turn an alias into a character (via "\N{…}" and unicodedata.lookup()), but there’s no way to turn a character into its aliases. This pull request adds a new unicodedata.aliases() function that allows you to turn a character into its aliases.

See the commit messages for additional details.

Closes #62434.

Here’s one of the changes that 71f660e
made:

diff --git a/Tools/unicode/makeunicodedata.py b/Tools/unicode/makeunicodedata.py
index 50d1acace81..db0f8ec 100644
--- a/Tools/unicode/makeunicodedata.py
+++ b/Tools/unicode/makeunicodedata.py
@@ -58,7 +58,7 @@ PUA_16 = range(0x100000, 0x10FFFE)

 # we use this ranges of PUA_15 to store name aliases and named sequences
 NAME_ALIASES_START = 0xF0000
-NAMED_SEQUENCES_START = 0xF0100
+NAMED_SEQUENCES_START = 0xF0200

 old_versions = ["3.2.0"]

Unfortuantely, there was a few different places where the old value of
NAMED_SEQUENCES_START (0xF0100) was still being used even though it had
been changed over 14 years ago. This change updates those places so that
they use the current value of NAMED_SEQUENCES_START.
Before this change, there was a for loop that iterated over each entry
in NameAliases.txt [1]. Each entry in NameAliases.txt has three fields.
Before this change, the for loop called the first field “char”, the
second field “name” and the third field “abbrev”.

“char” was not a good name for the first field. There is two different
ways that the word “char” can be interpreted. First, you could interpret
“char” as being short for “character”. Second, you could interpret
“char” as meaning the C data type char. Both interpretations would cause
confusion here. The first field does not contain a character, and it
does not contain a C char value. Instead, the first field contains a
code point written as a hexadecimal number.

“name” was not a good name for the second field. Unicode code points can
have both names and aliases. The second field contains an alias, not the
code point’s name.

“abbrev” was not a good name for the third field. “abbrev” is
(presumably) short for “abbreviation”. The third field does not contain
an abbreviation. Instead, the third field contains the alias’s type.

This change renames those variables to “aliased_codepoint”, “alias” and
“alias_type”. These names are more accurate, and they follow the names
used by NameAliases.txt itself:

> # FORMAT
> #
> # Each line has three fields, as described here:
> #
> # First field:  Code point
> # Second field: Alias
> # Third field:  Type

I had considered calling the first variable just “codepoint” instead of
“aliased_codepoint”. I chose the name “alised_codepoint” in order to
prepare for a future change. In that future change, the relevant code
will be modified so that there is an additional variable that contains a
code point. The name “aliased_codepoint” will make it easier for readers
to understand the difference between the two variables.

I had considered calling the last variable “type” in order to match the
name in that above quotation. I decided to use the name “alias_type” in
order to avoid overwriting the built-in “type” variable.

---

Similarly, there was a for loop that iterated over UnicodeData.aliases.
Before this change, that for loop called the data inside each element of
UnicodeData.aliases “name” and “codepoint”. In order to keep things
consistent, this change also updates that for loop so that it uses the
term “alias” instead of “name” and the term “aliased_codepoint” instead
of “codepoint”.

[1]: <https://www.unicode.org/Public/17.0.0/ucd/NameAliases.txt>
In some situations, you have a character and you want a human-readable
description of that character. For example, I was writing some code that
parses a certain file format. In that file format, you are not allowed
to use certain characters. I wanted to be able to show the user an error
that looked something like this if the user used one of the banned
characters:

    ERROR: You’re not allowed to use U+0000 NULL characters.

For most characters, you can use unicodedata.name() in order to get a
human-readable description of that character. Unfortunately,
unicodedata.name() does not work for U+0000 because U+0000 and many
other characters do not have names. In order to get a human-readable
description of U+0000, you need to look up its name aliases.

Before this change, you could turn an alias into a character (via
"\N{…}" and unicodedata.lookup()), but there was no way to turn a
character into its aliases. This change adds a new unicodedata.aliases()
function that allows you to turn a character into its aliases.

This commit implements pythongh-62434. In that thread, there were many
different proposals for potential ways to expose this data to Python
programs. This commit most closely matches flying-sheep’s proposal [1].
Here are the differences between what’s implemented in this commit, and
what flying-sheep proposed:

• flying-sheep proposed that unicodedata.aliases() would have an
  optional type argument that could be used to filter the output by
  alias type. The version of unicodedata.aliases() added by this commit
  does not include an optional type argument. You can already filter the
  output to your liking by simply using the returned dictionary. In
  other words, unicodedata.aliases("\n", type="control") would have been
  the same as unicodedata.aliases("\n")["control"]. Based on that fact,
  having an optional type argument didn’t seem very useful.

• flying-sheep proposed that the return value would use the dict, str
  and list data types. Two of those data types are mutable and one is
  immutable. This commit uses the frozendict, str and tuple data types.
  All three of those data types are immutable which is more consistent.
  Plus, the purpose unicodedata.aliases() is to lookup data that can be
  used later, not to create something that will later be modified. (I
  mean, you could theoretically modify the dict or the lists that
  flying-sheep’s proposal would have returned, but why would you?)

[1]: <python#62434 (comment)>
@python-cla-bot

python-cla-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@read-the-docs-community

read-the-docs-community Bot commented Aug 13, 2026

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #34054291 | 📁 Comparing 77b5e02 against main (dd1194e)

  🔍 Preview build  

2 files changed
± library/unicodedata.html
± whatsnew/changelog.html

It turns out that it’s not valid to use a const variable in a
static_assert [1]. We could get around this by making
alias_type_labels_start a constexpr variable instead of a const
variable, but the constexpr keyword wasn’t added to C until C23 [2], and
we aren’t allowed to use features from C23 yet [3].

This change fixes building Python on Windows.

[1]: <https://stackoverflow.com/a/78222714/7593853>
[2]: <https://cppreference.com/c/language/constexpr>
[3]: <https://peps.python.org/pep-0007/#c-standards>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unicodedata module should provide access to codepoint aliases

1 participant