Python cannot start in locale zh_TW.euctw

@encukou

$ LC_ALL=zh_TW.euctw ./python
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  is in build tree = 1
  stdlib dir = '/home/serhiy/py/cpython/Lib'
  sys._base_executable = '/home/serhiy/py/cpython/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/home/serhiy/py/cpython/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python311.zip',
    '/home/serhiy/py/cpython/Lib',
    '/home/serhiy/py/cpython/build/lib.linux-x86_64-3.11-pydebug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: unknown encoding: EUC-TW

Current thread 0x00007f1f24b4b280 (most recent call first):
  <no Python frame>

It is one of standard glibc locales. Other unsupported locales are hy_AM.armscii8 and ka_GE.georgianps, but I suppose they are less used, and there is no much official documentation available for that encodings.

[edit @StanFromIreland/@encukou] Another unsupported locale is IBM-943 (Japanese shift_jis) (#130228)

Linked PRs

gh-91992: Fall back to UTF-8 if an unsupported locale is provided #139429

Maybe, enable UTF-8 Mode when locale encoding is not found?

@methane

https://peps.python.org/pep-0686/ may fix the issue :-) cc @methane

Another instance of this #130228 with the IBM-943 (Japanese shift_jis) locale

@methane

I can get it to work forcing it to utf-8 in:

cpython/Python/initconfig.c

Lines 2276 to 2280 in 7473c60

    
           // See also config_get_fs_encoding() 
        
           static PyStatus 
        
           config_get_locale_encoding(PyConfig *config, const PyPreConfig *preconfig, 
        
                                      wchar_t **locale_encoding) 
        
           {

I could implement a fallback, but the question is, is there even a point? PEP 686 will force it to UTF8 if I am reading it correctly.

So, do we want fallback, or could we just implement the PEP now?

cc @methane

And do we also want a SC decision?

@StanFromIreland

@StanFromIreland PEP 686 makes UTF-8 Mode default since Python 3.15.
Until then, user can enable UTF-8 Mode by export PYTHONUTF8=1 in .bashrc.

Duplicate: #63658

PEP 686 makes UTF-8 Mode default since Python 3.15.
Until then, user can enable UTF-8 Mode by export PYTHONUTF8=1 in .bashrc.

I know, that is why I cc'ed you. This issue will be closed when that is implemented, but we are on 3.14 and not 3.15, do we still want a workaround here or do we want to just wait or implement it already?

It depends on how 'zh_TW.euctw' is used. If there are some users in the wild, we should add it to alias.
For example, #131370 (comment)

@malemburg

I propose a more general solution to this problem in #139429 as I discussed with @malemburg some time ago.

It is still a crash on 3.13 and 3.14?

I believe that the correct fix of this issue is to support iconv as a codecs provider by default. Locales on Posix systems can only have encodings supported by iconv. Similar issue was already solved on Windows -- all cpXXX codecs are now supported. I plan to do this, but it's difficult due to the difference in handling stateful encoding in iconv and Python.

It is still a crash on 3.13 and 3.14?

Python fails to start on 3.14:

$ LC_ALL=zh_TW.euctw python3.14
Python path configuration:
  (...)
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: unknown encoding: EUC-TW

Current thread 0x00007fef38d0ef00 [python3.14] (most recent call first):
  <no Python frame>

On Python 3.15, it just works:

$ LC_ALL=zh_TW.euctw ./python
Python 3.15.0a0 (heads/split_unicode_format:1352259b1e9, Oct  2 2025, 12:46:51) [GCC 15.2.1 20250808 (Red Hat 15.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale; locale.getencoding()
'EUC-TW'
>>> import sys; sys.getfilesystemencoding()
'utf-8'

Well, "just works" can be interpreted in different ways 😄

It's well possible that you get mojibake filenames when writing files to the FS with Python 3.15, or you fail to open a file, because the UTF-8 does not match the actual FS encoding.

Still, after all these years with Unicode, I think we can expect file systems to use UTF-8, just as much as we can expect terminals to run in UTF-8 mode, so I wouldn't worry too much.

It may be helpful to issue a warning, if there's a mismatch between the detected FS encoding and UTF-8, just to highlight the problem.

It's well possible that you get mojibake filenames when writing files to the FS with Python 3.15, or you fail to open a file, because the UTF-8 does not match the actual FS encoding.

Well, that's the PEP 686 – Make UTF-8 mode default. The PEP was accepted while we were well aware of the mojibake issue. Let's see how Python 3.15 goes :-)

Yes, I know and as I have mentioned elsewhere, I have been using UTF-8 mode for many years without any problems. It actually helped a lot in some situations, since the FS encoding is not something you can easily determine. It depends on the FS mount you are using and its configuration.

@serhiy-storchaka

UTF-8 mode is default now, so this will only affect users that explicitly turn UTF-8 mode off and use an encoding that Python has never supported.

I agree that correct fix of this issue is to support iconv, but honestly, it doesn't seem to be worth the effort. @serhiy-storchaka, do you still plan to do this?

Also, this is not a crash; it's a clean failure. It just doesn't have a regular traceback since it occurs too early.

Yes, I planned to do this in any case. Support of iconv is needed to support of all locales.

This is a crash, even if it was caused by the call of Py_FatalError() instead of segmentation fault.

BTW, I cannot run the debugger in non-UTF-8 locale:

$ LC_ALL=zh_TW.euctw gdb
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/usr/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/lib/python3.12'
  sys._base_executable = '/usr/bin/python'
  sys.base_prefix = '/usr'
  sys.base_exec_prefix = '/usr'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/bin/python'
  sys.prefix = '/usr'
  sys.exec_prefix = '/usr'
  sys.path = [
    '/usr/lib/python312.zip',
    '/usr/lib/python3.12',
    '/usr/lib/python3.12/lib-dynload',
  ]
Error occurred computing Python error message.
Python not initialized

BTW, I cannot run the debugger in non-UTF-8 locale

You might try to set the environment inside gdb, not outside, using (gdb) set environ command.

But you cannot run it in an existing environment at first place if it is not UTF-8.

Python no longer fails to start in the zh_TW.euctw locale since 3.15, even though it ships no EUC-TW codec.

On top of that, the iconv-based codec engine added in #152997 (PR #153001) provides actual EUC-TW support in 3.16 on platforms whose iconv() has it.

Closing as resolved.

Navigation Menu

Repository navigation

Python cannot start in locale zh_TW.euctw #91992

Linked PRs

2 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	// See also config_get_fs_encoding()
	static PyStatus
	config_get_locale_encoding(PyConfig config, const PyPreConfig preconfig,
	wchar_t **locale_encoding)
	{

Python cannot start in locale zh_TW.euctw #91992

Description

Linked PRs

Activity

methane commented on May 2, 2022

vstinner commented on May 25, 2022

StanFromIreland commented on Feb 17, 2025

StanFromIreland commented on Apr 4, 2025

methane commented on Apr 5, 2025

StanFromIreland commented on Apr 5, 2025

StanFromIreland commented on Apr 5, 2025

methane commented on Apr 5, 2025

StanFromIreland commented on Sep 29, 2025

2 remaining items

StanFromIreland commented on Sep 29, 2025

serhiy-storchaka commented on Sep 30, 2025

vstinner commented on Oct 2, 2025

malemburg commented on Oct 2, 2025

vstinner commented on Oct 3, 2025

malemburg commented on Oct 3, 2025

encukou commented on Oct 6, 2025

serhiy-storchaka commented on Oct 6, 2025

vstinner commented on Oct 6, 2025

serhiy-storchaka commented on Oct 6, 2025

serhiy-storchaka commented on Jul 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions