Repository navigation
Add an iconv-based codec engine to support all system locale encodings #152997
Description
Feature or enhancement
Python ships a fixed set of built-in codecs.
When the system locale uses an encoding none of them covers, codecs.lookup() / str.encode() / bytes.decode() raise LookupError: unknown encoding, and Python may fail to even start, aborting in init_fs_encoding when it cannot find a codec for the filesystem encoding.
These are ordinary, standard locales — zh_TW.euctw, hy_AM.armscii8, ka_GE.georgianps, and others ship with glibc and work in every other program.
Every such encoding is already known to the C library's iconv().
Proposal
Add an iconv-based codec engine, so Python can use any encoding iconv() knows, even without a dedicated built-in codec.
- A last-resort search function, so it never shadows a built-in codec — it only catches otherwise-unknown names.
- An explicit
iconv:prefix (e.g."iconv:latin1") forces the engine even when a built-in exists. - Full codec API: stateless encode/decode with all error handlers, incremental codecs, and stream reader/writer.
Design
- C:
_PyUnicode_DecodeIconv/_PyUnicode_EncodeIconvinObjects/unicodeobject.c, exposed as_codecs.iconv_decode/_codecs.iconv_encode.
The pivot is native-endian UTF-32 (a raw array ofPy_UCS4), so the conversion is independent ofwchar_t's representation (correct even wherewchar_tis not Unicode, e.g. Solaris/AIX) and keeps a 1:1 code-point/position mapping for error handlers. - Python:
Lib/encodings/_iconv_codecs.pyplus a search function inLib/encodings/__init__.py, mirroring_win_cp_codecs.py. - Build:
configuregains aniconvcheck definingHAVE_ICONV.
Prior art
gh-37800 ("Universal Unicode Codec for POSIX iconv", 2003) added an iconvcodec module, but it was removed after causing crashes (gh-37835, gh-37840, gh-37869) and build failures across Unixes (gh-38012 IRIX, gh-38019/38020 Solaris, gh-38033 Tru64, gh-38068 HP-UX, gh-38240 OpenBSD, gh-37865 Cygwin); follow-ups gh-38224/gh-38255/gh-38297 stalled.
This proposal avoids those pitfalls: no separate extension module (the conversion lives in the core unicodeobject.c), gated on a configure check, and pivoting through UTF-32 rather than wchar_t.
It is the POSIX counterpart of gh-123803, which made arbitrary Windows code pages usable as cpXXX encodings by delegating to the OS API.
Availability
Which encodings iconv provides is platform-dependent (glibc, musl, the citrus *BSDs and macOS/GNU libiconv all differ); OpenBSD, whose iconv has only UTF-8, gains nothing.
The feature is active only where iconv() is present, and transparent when absent.
Issues this resolves
- Python cannot start in locale zh_TW.euctw #91992 — Python cannot start in an unsupported locale (EUC-TW)
- Python does not support the GEORGIAN-PS charset #63658 — GEORGIAN-PS charset (a crash)
- missing vietnamese codec TCVN 5712:1993 in Python #65280 — Vietnamese TCVN 5712:1993
- Encode to EBCDIC doesn't take into account conversion table irregularities #74771 — EBCDIC cp1047 conversion table irregularities
- Add KOI8-RU as a known encoding #49464 (KOI8-RU) and Adding new CNS11643, a *huge* charset, support in cjkcodecs #46342 (CNS 11643) — requested charsets iconv provides
- relates to When Python is started on AIX in the IBM-943 (Japanese shift_jis) locale, it aborts. #130228 — abort in the AIX IBM-943 locale
Activity
BHUVANSH855 commentedon Jul 4, 2026
Hi @serhiy-storchaka,
I'm interested in working on this feature and have been studying the current implementation before starting.
From exploring the existing code, my understanding of the intended architecture is:
HAVE_ICONVdetection during configuration._PyUnicode_EncodeIconv()and_PyUnicode_DecodeIconv()inObjects/unicodeobject.c._codecs.iconv_encode()and_codecs.iconv_decode().Lib/encodings/_iconv_codecs.py, similar to_win_cp_codecs.py.Lib/encodings/__init__.py, while allowing theiconv:prefix to explicitly select the iconv implementation.I also noticed that CPython already uses
iconv()inPython/fileutils.cfor non-Unicodewchar_tconversion, so I wanted to check whether any of that implementation is expected to be reused or whether the new codec engine should be implemented independently.Before I start implementing, does this align with the intended design, or are there any architectural considerations or preferred implementation details that I should keep in mind?
Thanks!
serhiy-storchaka commentedon Jul 4, 2026
Caveats and limitations
By default the engine is a last-resort fallback that never shadows a built-in codec; the behavioral differences below apply only when it is forced over a built-in name via the
iconv:prefix.Availability (platform-dependent).
latin1, notlatin-1); not portable.Less strict than the built-ins.
iconv:utf-8etc. accept sequences CPython rejects (e.g. overlong forms) and report error positions differently.iconv:utf-16/utf-32add and expect a BOM.UCS-2aliasesUTF-16on some platforms (citrus *BSD), soiconv:ucs-2there adds a BOM and combines surrogate pairs instead of keeping each 16-bit unit independent.Interop.
xml.parsers.expatcleanly rejects them (never mis-parses). Tables for known multibyte encodings could be added later.Performance. Each stateless call opens and closes an
iconvdescriptor (like the Windows code-page codec); no caching.serhiy-storchaka commentedon Jul 4, 2026
@BHUVANSH855, thanks, and sorry for the overlap — this is already implemented in PR #153001.
The
iconv()inPython/fileutils.cis unrelated: it is theHAVE_NON_UNICODE_WCHAR_T_REPRESENTATIONfallback for platforms wherewchar_tis not Unicode. The new codec deliberately avoidswchar_tand pivots through UTF-32 instead, so none of it is reused.pythongh-152997: Freeze encodings._iconv_codecs and fix surrogate-pai…
pythongh-152997: Do not use iconv on Emscripten and WASI
pythongh-152997: Treat a nonreversible iconv() conversion as unencodable
gh-152997: Support system locale encodings via an iconv-based codec (G…
pythongh-153494: Use a charset unknown to iconv in the LookupError test
ownerjandel2-ship-it commentedon Jul 12, 2026
very good lessdon
ownerjandel2-ship-it commentedon Jul 12, 2026
very good lesson