test_codecs fails with RuntimeError on NetBSD

Bug report

Bug description:

home$ ./python -m test test_codecs -m test_decode_surrogateescape
Using random seed: 2297497288
0:00:00 load avg: 0.07 Run 1 test sequentially in a single process
0:00:00 load avg: 0.07 [1/1] test_codecs
test test_codecs failed -- Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_codecs.py", line 3698, in check_decode_strings
    decoded = self.decode(encoded, errors)
  File "/home/blue/cpython/Lib/test/test_codecs.py", line 3665, in decode
    return _testinternalcapi.DecodeLocaleEx(encoded, 0, errors)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
RuntimeError: decode error: pos=9, reason=decoding error

test_codecs failed (1 error)

== Tests result: FAILURE ==

1 test failed:
    test_codecs

Total duration: 235 ms
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE

OS: NetBSD 10.0 amd64

CPython versions tested on:

CPython main branch

Operating systems tested on:

Other

Linked PRs

Merge branch 'main' into pythongh-124476

home$ locale
LANG="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="C"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=""

USE_FORCE_ASCII is defined, but the check_force_ascii function returns 0.

To reproduce:

home$ cat fail.py

import _testinternalcapi

def decode(encoded, errors="strict"):
    return _testinternalcapi.DecodeLocaleEx(encoded, 0, errors)

encoded, errors =  b'blatin1:\xa7\xe9', 'surrogateescape'
decoded = decode(encoded, errors)

home$ ./python fail.py

Traceback (most recent call last):
  File "/home/blue/cpython/fail.py", line 7, in <module>
    decoded = decode(encoded, errors)
  File "/home/blue/cpython/fail.py", line 4, in decode
    return _testinternalcapi.DecodeLocaleEx(encoded, 0, errors)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
RuntimeError: decode error: pos=9, reason=decoding error
home$

Example code that mimics the behavior for the decode_current_locale function.

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <wchar.h>
#include <limits.h>
#include <locale.h>

#define PY_SSIZE_T_MAX 		SSIZE_MAX
#define MAX_UNICODE 		0x10ffff

static const size_t DECODE_ERROR = ((size_t) - 1);
static const size_t INCOMPLETE_CHARACTER = (size_t) - 2;

static int is_valid_wide_char(wchar_t ch) {
    return ch <= MAX_UNICODE;
}

static size_t _Py_mbrtowc(wchar_t *pwc, const char *str, size_t len, mbstate_t *pmbs) {
    size_t count = mbrtowc(pwc, str, len, pmbs);
    if (count != 0 && count != DECODE_ERROR && count != INCOMPLETE_CHARACTER) {
        if (!is_valid_wide_char(*pwc)) {
            return DECODE_ERROR;
        }
    }
    return count;
}

static int decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen) {
    wchar_t *res;
    size_t argsize;
    unsigned char *in;
    wchar_t *out;
    mbstate_t mbs;

    argsize = strlen(arg) + 1;
    if (argsize > PY_SSIZE_T_MAX / sizeof(wchar_t)) {
        return -1;
    }
    res = (wchar_t *)malloc(argsize * sizeof(wchar_t));
    if (!res) {
        return -1;
    }

    in = (unsigned char*)arg;
    out = res;
    memset(&mbs, 0, sizeof mbs);
    while (argsize) {
        size_t converted = _Py_mbrtowc(out, (char*)in, argsize, &mbs);
        if (converted == 0) {
            break;
        }

        if (converted == INCOMPLETE_CHARACTER) {
	    goto decode_error;
        }

        if (converted == DECODE_ERROR) {
            *out++ = 0xdc00 + *in++;
            argsize--;
            memset(&mbs, 0, sizeof mbs);
            continue;
        }

        in += converted;
        argsize -= converted;
        out++;
    }

    *out = L'\0';
    if (wlen != NULL) {
        *wlen = out - res;
    }
    *wstr = res;
    return 0;

decode_error:
    free(res);
    return -1;
}

int main() {
    setlocale(LC_CTYPE, "en_US.UTF-8");
    const char *latin1_str = "blatin1:\xa7\xe9"; // "blatin1:§é"
    wchar_t *wstr = NULL;
    size_t wlen = 0;

    int result = decode_current_locale(latin1_str, &wstr, &wlen);
    if (result == -1) {
        fprintf(stderr, "Error: Decoding failed\n");
        return EXIT_FAILURE;
    }

    wprintf(L"Decoded wide string (length %zu): %ls\n", wlen, wstr);
    free(wstr);
    return EXIT_SUCCESS;
}

Output on NetBSD 10.0 amd64:

Error: Decoding failed

Output on Arch Linux x86_64:

Decoded wide string (length 10): blatin1:??

Merge branch 'main' into pythongh-124476

Dit it fail on 3.13 or was it only in 3.14?

Dit it fail on 3.13 or was it only in 3.14?

The issue is still observed in 3.12, 3.13 and 3.14

By the way, test_decode_surrogateescape passes when run sequentially, but fails when run in parallel. This makes me think that there is a race condition or environment-specific problem in the locale handling code.

╭─blue@home ~/cpython
╰─./python -m test test_codecs -m test_decode_surrogateescape                                 
Using random seed: 4146321672
0:00:00 load avg: 0.27 Run 1 test sequentially
0:00:00 load avg: 0.27 [1/1] test_codecs

== Tests result: SUCCESS ==

1 test OK.

Total duration: 186 ms
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered)
Result: SUCCESS
╭─blue@home ~/cpython
╰─$ ./python -m test test_codecs -m test_decode_surrogateescape -j0
Using random seed: 2781658390
0:00:00 load avg: 0.25 Run 1 test in parallel using 1 worker process
0:00:00 load avg: 0.25 [1/1/1] test_codecs failed (1 error)
test test_codecs failed -- Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_codecs.py", line 3539, in check_decode_strings
    decoded = self.decode(encoded, errors)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/blue/cpython/Lib/test/test_codecs.py", line 3506, in decode
    return _testinternalcapi.DecodeLocaleEx(encoded, 0, errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: decode error: pos=9, reason=decoding error

== Tests result: FAILURE ==

1 test failed:
    test_codecs

Total duration: 527 ms
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE
╭─blue@home ~/cpython
╰─./python -m test test_codecs -m test_decode_surrogateescape -j1                             
Using random seed: 1488421871
0:00:00 load avg: 0.25 Run 1 test in parallel using 1 worker process
0:00:00 load avg: 0.39 [1/1/1] test_codecs failed (1 error)
test test_codecs failed -- Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_codecs.py", line 3539, in check_decode_strings
    decoded = self.decode(encoded, errors)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/blue/cpython/Lib/test/test_codecs.py", line 3506, in decode
    return _testinternalcapi.DecodeLocaleEx(encoded, 0, errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: decode error: pos=9, reason=decoding error

== Tests result: FAILURE ==

1 test failed:
    test_codecs

Total duration: 514 ms
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE
╭─blue@home ~/cpython

pythongh-124476: Fix decoding from the locale encoding on C.UTF-8 loc…

@vstinner

When run in the same process, the locale is C. When run in a subprocess, the locale is C.UTF-8. There must be a bug that causes such difference.

But other bug is that decoding with the surrogateescape error handler fails on C.UTF-8. #132477 makes INCOMPLETE_CHARACTER to be interpreted in the same way as DECODE_ERROR.

cc @vstinner, the original author of that code.

gh-124476: Fix decoding from the locale encoding in the C.UTF-8 locale (

[3.13] pythongh-124476: Fix decoding from the locale encoding in the …

[3.13] gh-124476: Fix decoding from the locale encoding in the C.UTF-…

Navigation Menu

Repository navigation

test_codecs fails with RuntimeError on NetBSD #124476

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_codecs fails with RuntimeError on NetBSD #124476

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

furkanonder commented on Sep 24, 2024

furkanonder commented on Sep 24, 2024

picnixz commented on Mar 23, 2025

furkanonder commented on Mar 23, 2025

furkanonder commented on Mar 23, 2025

serhiy-storchaka commented on Apr 13, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions