Skip to content

[3.8] bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466)#22705

Merged
miss-islington merged 1 commit into
python:3.8from
miss-islington:backport-2d2af32-3.8
Oct 15, 2020
Merged

[3.8] bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466)#22705
miss-islington merged 1 commit into
python:3.8from
miss-islington:backport-2d2af32-3.8

Conversation

@miss-islington

@miss-islington miss-islington commented Oct 15, 2020

Copy link
Copy Markdown
Contributor

When running in a non-UTF-8 locale, if an error occurs while importing a
native Python module (say because a dependent share library is missing),
the error message string returned may contain non-ASCII code points
causing a UnicodeDecodeError.

PyUnicode_DecodeFSDefault is used for buffers which may contain
filesystem paths. For consistency with os.strerror(),
PyUnicode_DecodeLocale is used for buffers which contain system error
messages. While the shortname parameter is always encoded in ASCII
according to PEP 489, it is left decoded using PyUnicode_FromString to
minimize the changes and since it should not affect the decoding (albeit
potentially slower).

In dynload_hpux, since the error buffer contains a message generated
from a static ASCII string and the module filesystem path,
PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as
is used elsewhere.

  • bpo-41894: Fix bugs in dynload error msg handling

For both dynload_aix and dynload_hpux, properly handle the possibility
that decoding strings may return NULL and when such an error happens,
properly decrement any previously decoded strings and return early.

In addition, in dynload_aix, ensure that we pass the decoded string
object pathname_ob to PyErr_SetImportError instead of the original
pathname buffer.

Co-authored-by: Serhiy Storchaka storchaka@gmail.com
(cherry picked from commit 2d2af32)

Co-authored-by: Kevin Adler kadler@us.ibm.com

https://bugs.python.org/issue41894

…GH-22466)

When running in a non-UTF-8 locale, if an error occurs while importing a
native Python module (say because a dependent share library is missing),
the error message string returned may contain non-ASCII code points
causing a UnicodeDecodeError.

PyUnicode_DecodeFSDefault is used for buffers which may contain
filesystem  paths. For consistency with os.strerror(),
PyUnicode_DecodeLocale is used for buffers which contain system error
messages. While the shortname parameter is always encoded in ASCII
according to PEP 489, it is left decoded using PyUnicode_FromString to
minimize the changes and since it should not affect the decoding (albeit
_potentially_ slower).

In dynload_hpux, since the error buffer contains a message generated
from a static ASCII string and the module filesystem path,
PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as
is used elsewhere.

* bpo-41894: Fix bugs in dynload error msg handling

For both dynload_aix and dynload_hpux, properly handle the possibility
that decoding strings may return NULL and when such an error happens,
properly decrement any previously decoded strings and return early.

In addition, in dynload_aix, ensure that we pass the decoded string
*object* pathname_ob to PyErr_SetImportError instead of the original
pathname buffer.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
(cherry picked from commit 2d2af32)

Co-authored-by: Kevin Adler <kadler@us.ibm.com>
@miss-islington

Copy link
Copy Markdown
Contributor Author

@kadler and @methane: Status check is done, and it's a success ✅ .

@miss-islington

Copy link
Copy Markdown
Contributor Author

@kadler and @methane: Status check is done, and it's a success ✅ .

@miss-islington miss-islington merged commit 47ca679 into python:3.8 Oct 15, 2020
@miss-islington miss-islington deleted the backport-2d2af32-3.8 branch October 15, 2020 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type-bug An unexpected behavior, bug, or error

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants