Skip to content

gh-88500: Reduce memory use of urllib.unquote #96763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gpshead merged 5 commits into from
Dec 11, 2022
Merged

gh-88500: Reduce memory use of urllib.unquote #96763

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c02c357
gh-88500: Reduce memory use of urllib.unquote
gpshead Sep 12, 2022
142a7e2
Rename the internal implementation function.
gpshead Sep 16, 2022
a81248d
Merge branch 'main' into gh/88500/unquote_mem_use
gpshead Sep 16, 2022
4ffac18
NEWS entry, keep old var name for a smaller diff.
gpshead Sep 16, 2022
b87595b
Merge branch 'main' into gh/88500/unquote_mem_use
gpshead Nov 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
NEWS entry, keep old var name for a smaller diff.
  • Loading branch information
@gpshead
gpshead committed Sep 16, 2022
commit 4ffac18d4e8ce373ad052aa36c412cc6dda172d0
11 changes: 6 additions & 5 deletions Lib/urllib/parse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -615,7 +615,7 @@ def _unquote_impl(string: bytes | bytearray | str) -> bytes | bytearray:
if len(bits) == 1:
return string
res = bytearray(bits[0])
add_data = res.extend
append = res.extend
# Delay the initialization of the table to not waste memory
# if the function is never called
global _hextobyte
Expand All @@ -624,11 +624,11 @@ def _unquote_impl(string: bytes | bytearray | str) -> bytes | bytearray:
for a in _hexdig for b in _hexdig}
for item in bits[1:]:
try:
add_data(_hextobyte[item[:2]])
add_data(item[2:])
append(_hextobyte[item[:2]])
append(item[2:])
except KeyError:
add_data(b'%')
add_data(item)
append(b'%')
append(item)
return res

_asciire = re.compile('([\x00-\x7f]+)')
Expand Down Expand Up @@ -665,6 +665,7 @@ def unquote(string, encoding='utf-8', errors='replace'):
errors = 'replace'
return ''.join(_generate_unquoted_parts(string, encoding, errors))


def parse_qs(qs, keep_blank_values=False, strict_parsing=False,
encoding='utf-8', errors='replace', max_num_fields=None, separator='&'):
"""Parse a query given as a string argument.
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2022-09-16-08-21-46.gh-issue-88500.jQ0pCc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Reduced the memory usage of :func:`urllib.parse.unquote` and
:func:`urllib.parse.unquote_to_bytes` on large values.