Skip to content

test_alt_digits_nl_langinfo fails for locale uk_UA #133740

Closed
Closed
test_alt_digits_nl_langinfo fails for locale uk_UA#133740
Labels
testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Description

@pablogsal
pablogsal
opened on May 9, 2025

Bug report

Bug description:

test_alt_digits_nl_langinfo (test.test__locale._LocaleTests.test_alt_digits_nl_langinfo) ...
  test_alt_digits_nl_langinfo (test.test__locale._LocaleTests.test_alt_digits_nl_langinfo) (locale='sr_YU') ... skipped "no locale 'sr_YU'"
  test_alt_digits_nl_langinfo (test.test__locale._LocaleTests.test_alt_digits_nl_langinfo) (locale='uk_UA') ... ERROR
  test_alt_digits_nl_langinfo (test.test__locale._LocaleTests.test_alt_digits_nl_langinfo) (locale='lzh_TW.UTF-8') ... skipped "no locale 'lzh_TW.UTF-8'"
  test_alt_digits_nl_langinfo (test.test__locale._LocaleTests.test_alt_digits_nl_langinfo) (locale='shn_MM.UTF-8') ... skipped "no locale 'shn_MM.UTF-8'"
  test_alt_digits_nl_langinfo (test.test__locale._LocaleTests.test_alt_digits_nl_langinfo) (locale='ar_AE.UTF-8') ... skipped "ALT_DIGITS is not set for locale 'ar_AE.UTF-8' on this platform"
  test_alt_digits_nl_langinfo (test.test__locale._LocaleTests.test_alt_digits_nl_langinfo) (locale='bn_IN.UTF-8') ... skipped "ALT_DIGITS is not set for locale 'bn_IN.UTF-8' on this platform"
test_era_nl_langinfo (test.test__locale._LocaleTests.test_era_nl_langinfo) ...
  test_era_nl_langinfo (test.test__locale._LocaleTests.test_era_nl_langinfo) (locale='sr_YU') ... skipped "no locale 'sr_YU'"
  test_era_nl_langinfo (test.test__locale._LocaleTests.test_era_nl_langinfo) (locale='lzh_TW.UTF-8') ... skipped "no locale 'lzh_TW.UTF-8'"
  test_era_nl_langinfo (test.test__locale._LocaleTests.test_era_nl_langinfo) (locale='shn_MM.UTF-8') ... skipped "no locale 'shn_MM.UTF-8'"
test_float_parsing (test.test__locale._LocaleTests.test_float_parsing) ... ok
test_lc_numeric_basic (test.test__locale._LocaleTests.test_lc_numeric_basic) ... ok
test_lc_numeric_localeconv (test.test__locale._LocaleTests.test_lc_numeric_localeconv) ... ok
test_lc_numeric_nl_langinfo (test.test__locale._LocaleTests.test_lc_numeric_nl_langinfo) ... ok

======================================================================
ERROR: test_alt_digits_nl_langinfo (test.test__locale._LocaleTests.test_alt_digits_nl_langinfo) (locale='uk_UA')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bb/data/tmp/314/refroot/amd64/opt/bb/lib64/python3.14/test/test__locale.py", line 220, in test_alt_digits_nl_langinfo
    alt_digits = nl_langinfo(locale.ALT_DIGITS)
UnicodeDecodeError: 'locale' codec can't decode byte 0xde in position 4: decoding error

----------------------------------------------------------------------
Ran 6 tests in 0.012s

This is in a Red Hat Enterprise Linux Server release 7.6 (Maipo) machine

CPython versions tested on:

CPython main branch, 3.14

Operating systems tested on:

Linux

Linked PRs

Activity

  1. pablogsal
    added
    type-bugAn unexpected behavior, bug, or error
    on May 9, 2025

  2. pablogsal commented on May 9, 2025

    @pablogsal
    pablogsal
    on May 9, 2025
    MemberAuthor

    CC @vstinner @serhiy-storchaka

  3. serhiy-storchaka commented on May 9, 2025

    @serhiy-storchaka
    serhiy-storchaka
    on May 9, 2025
    Member

    This is weird. ALT_DIGITS should be empty for uk_UA, as for most locales. Most likely this is a platform bug.

    How old is RHELS 7.6? Is it new or ancient? Does the test pass on other versions? What version of libc?

  4. pablogsal commented on May 9, 2025

    @pablogsal
    pablogsal
    on May 9, 2025
    MemberAuthor

    Sort of ancient (from 2014) :)

    This is the version of glib:

    glibc-2.17-260.el7_6.6.x86_64
glibc-2.17-260.el7_6.6.i686

  5. pablogsal commented on May 9, 2025

    @pablogsal
    pablogsal
    on May 9, 2025
    MemberAuthor

    Interestingly when I test with this:

    #include <stdio.h>
#include <locale.h>
#include <langinfo.h>

int main() {
    // Set the locale to Ukrainian (Ukraine)
    if (setlocale(LC_ALL, "uk_UA") == NULL) {
        perror("Unable to set uk_UA locale");
        return 1;
    }

    // Get ALT_DIGITS using nl_langinfo
#ifdef ALT_DIGITS
    const char *alt_digits = nl_langinfo(ALT_DIGITS);
    printf("ALT_DIGITS for uk_UA: %s\n", alt_digits);
#else
    printf("ALT_DIGITS is not supported on this system.\n");
#endif

    return 0;

    I get:

    ALT_DIGITS for uk_UA: 0

  6. pablogsal commented on May 9, 2025

    @pablogsal
    pablogsal
    on May 9, 2025
    MemberAuthor

    Also tried this:

    (refroot)$ cat lol.py
import locale

def get_alt_digits(locale_name):
    try:
        # Set the locale
        locale.setlocale(locale.LC_ALL, locale_name)
        # Get ALT_DIGITS
        alt_digits = locale.nl_langinfo(locale.ALT_DIGITS)
        print(f"ALT_DIGITS for {locale_name}: {alt_digits}")
    except AttributeError:
        print("ALT_DIGITS is not available in this Python implementation.")
    except locale.Error as e:
        print(f"Locale error: {e}")

# Test with Ukrainian locale
get_alt_digits('uk_UA.UTF-8')

    This prints:

    ALT_DIGITS for uk_UA.UTF-8: 0;січня;лютого;березня;квітня;травня;червня;липня;серпня;вересня;жовтня;листопада;грудня

    and with this:

    get_alt_digits('uk_UA')

    prints

    Traceback (most recent call last):
  File "/bb/data/tmp/314/lol.py", line 16, in <module>
    get_alt_digits('uk_UA')
    ~~~~~~~~~~~~~~^^^^^^^^^
  File "/bb/data/tmp/314/lol.py", line 8, in get_alt_digits
    alt_digits = locale.nl_langinfo(locale.ALT_DIGITS)
UnicodeDecodeError: 'locale' codec can't decode byte 0xde in position 4: decoding error

  7. pablogsal commented on May 9, 2025

    @pablogsal
    pablogsal
    on May 9, 2025
    MemberAuthor

    Some investigation with gdb:

    #0  _locale_nl_langinfo_impl (module=0x7fffefc810d0, item=131119) at /opt/bb/src/python/python3.14/Modules/_localemodule.c:692
692	            result = result != NULL ? result : "";
(gdb) l
687	    for (i = 0; langinfo_constants[i].name; i++) {
688	        if (langinfo_constants[i].value == item) {
689	            /* Check NULL as a workaround for GNU libc's returning NULL
690	               instead of an empty string for nl_langinfo(ERA).  */
691	            const char *result = nl_langinfo(item);
692	            result = result != NULL ? result : "";
693	            char *oldloc = NULL;
694	            if (langinfo_constants[i].category != LC_CTYPE
695	                && !is_all_ascii(result)
696	                && change_locale(langinfo_constants[i].category, &oldloc) < 0)
(gdb) p result
$8 = 0x7ffff59dc244 "0"
(gdb) p item
$9 = 131119
(gdb) n
693	            char *oldloc = NULL;
(gdb)
694	            if (langinfo_constants[i].category != LC_CTYPE
(gdb)
695	                && !is_all_ascii(result)
(gdb)
696	                && change_locale(langinfo_constants[i].category, &oldloc) < 0)
(gdb)
706	            if (item == ALT_DIGITS && *result) {
(gdb) p item == ALT_DIGITS && *result
$10 = 1
(gdb) n
707	                pyresult = decode_strings(result, 100);
(gdb) p result
$11 = 0x7ffff59dc244 "0"
(gdb) p decode_strings(result, 100)
$12 = (PyObject *) 0x0
  8. picnixz
    added
    testsTests in the Lib/test dir
    on May 9, 2025
  9. picnixz
    added this to Locale issues 🗺on May 9, 2025
  10. github-project-automation
    moved this to Todo in Locale issues 🗺on May 9, 2025

  11. pablogsal commented on May 9, 2025

    @pablogsal
    pablogsal
    on May 9, 2025
    MemberAuthor

    Looks like this is the problem:

    (gdb) p (char*)(nl_langinfo(item))
$6 = 0x7ffff59dc244 "0"

  12. vstinner commented on May 9, 2025

    @vstinner
    vstinner
    on May 9, 2025
    Member

    Related changes:

    commit dcc4fb2c9068f60353f0c0978948b7681f7745e6
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Mon Oct 21 21:54:12 2024 +0300

    gh-124969: Make locale.nl_langinfo(locale.ALT_DIGITS) returning a string again (GH-125774)
    
    This is a follow up of GH-124974. Only Glibc needed a fix.
    Now the returned value is a string consisting of semicolon-separated
    symbols on all Posix platforms.

commit 21c04e1a972bd1b6285e0ea41fa107d635bbe43a
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Wed Oct 9 11:42:08 2024 +0300

    gh-124969: Fix locale.nl_langinfo(locale.ALT_DIGITS) (GH-124974)
    
    Now it returns a tuple of up to 100 strings (an empty tuple on most locales).
    Previously it returned the first item of that tuple or an empty string.

  13. vstinner commented on May 9, 2025

    @vstinner
    vstinner
    on May 9, 2025
    Member

    Looks like this is the problem:
    (gdb) p (char*)(nl_langinfo(item))
    $6 = 0x7ffff59dc244 "0"

    You should look further in the string since glibc uses NUL separator in ALT_DIGITS value.

    $ gdb ./python
(gdb) run
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'ja_JP')
'ja_JP'
>>> locale.nl_langinfo(locale.ALT_DIGITS)
^C

(gdb) b decode_strings
(gdb) continue

>>> locale.nl_langinfo(locale.ALT_DIGITS)

Breakpoint 1, decode_strings (result=result@entry=0x7ffff2ae0caa "\241\273", max_count=max_count@entry=100)
    at ./Modules/_localemodule.c:643
643	{

# First
(gdb) p result
$19 = 0x7ffff2ae0caa "\241\273"

# Second
(gdb) p /x result[3]
$26 = 0xb0
(gdb) p /x result[4]
$27 = 0xec
(gdb) p /x result[5]
$28 = 0x0

# Third
(gdb) p /x result[6]
$30 = 0xc6
(gdb) p /x result[7]
$31 = 0xf3
(gdb) p /x result[8]
$32 = 0x0

# ...

(gdb) continue

# nl_langinfo() result in Python:
'〇;一;二;三;四;五;六;七;八;九;十;十一;十二;十三;十四;十五;十六;十七;十八;十九;二十;二十一;二十二;二十三;二十四;二十五;二十六;二十七;二十八;二十九;三十;三十一;三十二;三十三;三十四;三十五;三十六;三十七;三十八;三十九;四十;四十一;四十二;四十三;四十四;四十五;四十六;四十七;四十八;四十九;五十;五十一;五十二;五十三;五十四;五十五;五十六;五十七;五十八;五十九;六十;六十一;六十二;六十三;六十四;六十五;六十六;六十七;六十八;六十九;七十;七十一;七十二;七十三;七十四;七十五;七十六;七十七;七十八;七十九;八十;八十一;八十二;八十三;八十四;八十五;八十六;八十七;八十八;八十九;九十;九十一;九十二;九十三;九十四;九十五;九十六;九十七;九十八;九十九'

  14. pablogsal commented on May 9, 2025

    @pablogsal
    pablogsal
    on May 9, 2025
    MemberAuthor

    You should look further in the string since glibc uses NUL separator in ALT_DIGITS value.

    Yeah but that's the problem I think: we are passing the 0 directly to decode string, no? Or who you mean by "you" here?

  15. serhiy-storchaka commented on May 9, 2025

    @serhiy-storchaka
    serhiy-storchaka
    on May 9, 2025
    Member

    There was a hack that used ALT_DIGITS for month names in a genitive case. It was removed in https://sourceware.org/git/?p=glibc.git;a=commit;h=86530b9fed4466a7c05e20ec4d5fd89b4dc41fa6 .

    Glibc uses null separator instead of ";" for ALT_DIGITS (in violation of Posix). This is why you get "0". Python has a work around this, so it sees the whole thing, including non-ASCII month names.

    Binary locale files for the "uk_UA" locale are created using the "KOI8-U" encoding, as specified in localedata/SUPPORTED. But it seems that UTF-8 was used for decoding. Could you please print the results of setlocale(LC_CTYPE, NULL) and setlocale(LC_TIME, NULL)? And the result of nl_langinfo(CODESET) after calling setlocale(LC_CTYPE, setlocale(LC_TIME, NULL))?

  16. 21 remaining items

  17. serhiy-storchaka commented on Jul 3, 2025

    @serhiy-storchaka
    serhiy-storchaka
    on Jul 3, 2025
    Member

    There was a small regression -- there is no need to switch locale if the result is empty.

  18. serhiy-storchaka
    added 2 commits that reference this issue on Jul 3, 2025

    pythongh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS)

    7160573

    gh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS) (GH-136237)

    5c984ae
  19. miss-islington
    added a commit that references this issue on Jul 3, 2025

    pythongh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS) (py…

    22d2bd5
  20. serhiy-storchaka
    closed this as completedon Jul 3, 2025
  21. github-project-automation
    moved this from In Progress to Done in Locale issues 🗺on Jul 3, 2025
  22. serhiy-storchaka
    added a commit that references this issue on Jul 3, 2025

    [3.14] gh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS) (G…

    150262d
  23. AndPuQing
    added a commit that references this issue on Jul 11, 2025

    pythongh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS) (py…

    531ed83
  24. Pranjal095
    added 2 commits that reference this issue on Jul 12, 2025

    pythongh-133740: Fix locale.nl_langinfo(ALT_DIGITS) (python#134468)

    bb3ba12

    pythongh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS) (py…

    9f1c83e
  25. picnixz
    added a commit that references this issue on Jul 13, 2025

    pythongh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS) (py…

    528223c
  26. taegyunkim
    added 2 commits that reference this issue on Aug 4, 2025

    pythongh-133740: Fix locale.nl_langinfo(ALT_DIGITS) (python#134468)

    7b40de4

    pythongh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS) (py…

    fbc68ec
  27. Agent-Hellboy
    added a commit that references this issue on Aug 19, 2025

    pythongh-133740: Fix regression in locale.nl_langinfo(ALT_DIGITS) (py…

    a0917c7
  28. gostak-dd
    added a commit that references this issue on Jun 2, 2026

    pythongh-133740: Fix locale.nl_langinfo(ALT_DIGITS) (python#134468)

    6bdb12d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions