iconv codec: encoding names that glibc strips to nothing resolve to the locale encoding #161924

Bug report

Bug description

Since the iconv-based codec (gh-152997, GH-153001) landed, codecs.lookup() on Linux/glibc accepts encoding names whose charset part glibc's iconv_open() reduces to nothing, and returns a codec for the current locale's encoding. Before 3.16 these names raise LookupError.

>>> 'é'.encode('!!!')
b'\xc3\xa9'                    # 3.16.0a2, LC_ALL=C.UTF-8;  3.15: LookupError: unknown encoding: !!!
>>> bytes('é', '한글')
b'\xc3\xa9'
>>> import codecs; codecs.lookup('@').name
'@'

The result depends on the environment: with LC_ALL=C the same call fails with UnicodeEncodeError: '!!!' codec can't encode character '\xe9', because the codec is now ASCII.

Cause

iconv_search_function in Lib/encodings/__init__.py rejects only the empty name and then probes the name with _iconv_encode(name, ''). glibc's iconv_open() first removes trailing whitespace, / and , and any //-suffix such as //TRANSLIT (gconv_parse_code() in iconv/gconv_charset.c), then drops every remaining character that is not an ASCII alphanumeric, / or one of _-.,: (strip() in iconv/gconv_charset.h), and treats the resulting empty name as the charset of the current locale (__gconv_open in iconv/gconv_open.c: "If the string is empty define this to mean the charset of the currently selected locale"). So the probe succeeds for any name made only of dropped characters, and also for names whose charset part is empty once the suffix is removed, such as ',', '/', '//TRANSLIT' and '//abc'.

    def iconv_search_function(encoding):
        ...
        if not name:
            return None
        # Test if the encoding is supported by iconv.
        try:
            _iconv_encode(name, '')
        except (LookupError, OSError):
            return None

Possible fix

Before probing, reject a name unless the part before its first / or , contains an ASCII alphanumeric. Requiring an alphanumeric anywhere in the name is not enough: '//TRANSLIT' and '//abc' contain one and still resolve to the locale encoding.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux