iso-2022-cn-ext encoding silently returns b'' for non-ASCII text #154863
Description
Bug description:
Encoding non-ASCII text to ISO-2022-CN-EXT returns b'' with no error. The ASCII part is
lost with it, and the codec reports the whole input as consumed:
>>> 'ABC中DEF'.encode('iso-2022-cn-ext')
b''
>>> codecs.iconv_encode('ISO-2022-CN-EXT', '中') # (output, consumed)
(b'', 1)
>>> codecs.iconv_encode('ISO-2022-CN', '中') # without EXT
(b'\x1b$)A\x0eVP\x0f', 1)
No error handler changes it: strict, replace, ignore and xmlcharrefreplace all give
b''. Only strings stored as 2-byte characters are affected; a pure-ASCII string encodes
normally, and an unencodable character in a 1-byte or 4-byte string raises
UnicodeEncodeError as expected.
This is not an iconv limitation. iconv -f UCS-2LE -t ISO-2022-CN-EXT, which is the
conversion the codec asks for on a 2-byte string, produces the expected
\x1b$)A\x0eVP\x0f.
The iconv codecs are new in 3.16 (gh-152997), so no released version is affected.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Activity
tekinertekin commentedon Jul 29, 2026
gh-154863: Fix iconv encoding of ISO-2022-CN-EXT returning empty bytes (
encukou commentedon Aug 3, 2026
@serhiy-storchaka, this broke some buildbots, see e.g. here
gh-154863: Check the encoded bytes in the shift state flush test (GH-…
serhiy-storchaka commentedon Aug 4, 2026
The test only failed on macOS 15, not 13 or 26, and not other platform. Seems a platform bug. I cannot check details, so I just made the test more lenient.
encukou commentedon Aug 4, 2026
For the record, it also failed on FreeBSD.
serhiy-storchaka commentedon Aug 4, 2026
Oh, the ISO-2022-CN encoder crashes on two buildbots. Those workers are running libc with an unpatched security fix. freebsd-update on the 14.x worker should clear it, and the 15.0-CURRENT worker needs a rebuild.
pythongh-154863: Fix iconv encoding of ISO-2022-CN-EXT returning empt…
pythongh-154863: Check the encoded bytes in the shift state flush test (