Skip to content

gh-94526: getpath_dirname() no longer encodes the path #97645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vstinner merged 1 commit into from
Sep 30, 2022
Merged

gh-94526: getpath_dirname() no longer encodes the path #97645

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
gh-94526: getpath_dirname() no longer encodes the path 
Fix the Python path configuration used to initialized sys.path at
Python startup. Paths are no longer encoded to UTF-8/strict to avoid
encoding errors if it contains surrogate characters (bytes paths are
decoded with the surrogateescape error handler).

getpath_basename() and getpath_dirname() functions no longer encode
the path to UTF-8/strict, but work directly on Unicode strings. These
functions now use PyUnicode_FindChar() and PyUnicode_Substring() on
the Unicode path, rather than strrchr() on the encoded bytes string.
  • Loading branch information
@vstinner
vstinner committed Sep 29, 2022
commit ac8255b36c37c06f6db0d3fd13c42016fac3cc2a
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Core and Builtins/2022-09-29-15-19-29.gh-issue-94526.wq5m6T.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Fix the Python path configuration used to initialized :data:`sys.path` at
Python startup. Paths are no longer encoded to UTF-8/strict to avoid encoding
errors if it contains surrogate characters (bytes paths are decoded with the
surrogateescape error handler). Patch by Victor Stinner.
23 changes: 14 additions & 9 deletions Modules/getpath.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,27 +82,32 @@ getpath_abspath(PyObject *Py_UNUSED(self), PyObject *args)
static PyObject *
getpath_basename(PyObject *Py_UNUSED(self), PyObject *args)
{
const char *path;
if (!PyArg_ParseTuple(args, "s", &path)) {
PyObject *path;
if (!PyArg_ParseTuple(args, "U", &path)) {
Copy link
Copy Markdown
Member

@serhiy-storchaka serhiy-storchaka Sep 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I would use METH_O and PyArg_Parse() in these functions, but this is another issue.

Why cannot they be implemented in Python?

Copy link
Copy Markdown
Member Author

@vstinner vstinner Sep 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I would use METH_O and PyArg_Parse() in these functions, but this is another issue.

I tried to minimize the changes.

Why cannot they be implemented in Python?

Ask @zooba who designed this. Maybe it can be changed?

Copy link
Copy Markdown
Member

@zooba zooba Sep 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perf, mostly. These trivial ones probably could be, but don't fall into the trap of trying to port the full ntpath/posixpath implementations into getpath - we don't have a lot of the functionality needed to handle those at this stage (e.g. no codecs, no os module).

return NULL;
}
const char *name = strrchr(path, SEP);
return PyUnicode_FromString(name ? name + 1 : path);
Py_ssize_t end = PyUnicode_GET_LENGTH(path);
Py_ssize_t pos = PyUnicode_FindChar(path, SEP, 0, end, -1);
if (pos < 0) {
return Py_NewRef(path);
}
return PyUnicode_Substring(path, pos + 1, end);
}


static PyObject *
getpath_dirname(PyObject *Py_UNUSED(self), PyObject *args)
{
const char *path;
if (!PyArg_ParseTuple(args, "s", &path)) {
PyObject *path;
if (!PyArg_ParseTuple(args, "U", &path)) {
return NULL;
}
const char *name = strrchr(path, SEP);
if (!name) {
Py_ssize_t end = PyUnicode_GET_LENGTH(path);
Py_ssize_t pos = PyUnicode_FindChar(path, SEP, 0, end, -1);
if (pos < 0) {
return PyUnicode_FromStringAndSize(NULL, 0);
}
return PyUnicode_FromStringAndSize(path, (name - path));
return PyUnicode_Substring(path, 0, pos);
}


Expand Down