Skip to content

gh-145261, gh-106939: multiprocessing.shared_memory: fix ShareableList corruption for multi-byte strings and null bytes #145266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zetzschest wants to merge 8 commits into
base: main
Choose a base branch
from
+96 −26
Open

gh-145261, gh-106939: multiprocessing.shared_memory: fix ShareableList corruption for multi-byte strings and null bytes #145266

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6261c86
gh-145261: Fix ShareableList corruption for multi-byte strings and by…
stefanzetzsche Feb 26, 2026
34003c6
📜🤖 Added by blurb_it.
blurb-it[bot] Feb 26, 2026
3f874bc
docs: Update ShareableList documentation for fixed trailing null bug
stefanzetzsche Feb 27, 2026
254230f
fix: Also preserve trailing nulls in strings, not just bytes
stefanzetzsche Feb 27, 2026
2c0935c
test: Update ShareableList tests to reflect actual byte lengths
stefanzetzsche Feb 27, 2026
eb4ce8a
docs: Update ShareableList documentation to reflect trailing null pre…
stefanzetzsche Feb 27, 2026
e3b5fd3
Merge remote-tracking branch 'origin/main' into fix/multiprocessing_s…
stefanzetzsche Mar 11, 2026
feba9b5
test: Add tests for trailing null preservation and multi-byte UTF-8 i…
stefanzetzsche Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: Also preserve trailing nulls in strings, not just bytes 
Extended the fix to remove rstrip from strings as well and store actual
byte lengths for both strings and bytes in format metadata.
  • Loading branch information
@stefanzetzsche
stefanzetzsche committed Feb 27, 2026
commit 254230f65f5cc673d6f619cbe65116a07428fd73
13 changes: 8 additions & 5 deletions Lib/multiprocessing/shared_memory.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ class ShareableList:
_alignment = 8
_back_transforms_mapping = {
0: lambda value: value, # int, float, bool
1: lambda value: value.rstrip(b'\x00').decode(_encoding), # str
1: lambda value: value.decode(_encoding), # str
2: lambda value: value, # bytes
3: lambda _value: None, # None
}
Expand Down Expand Up @@ -355,9 +355,11 @@ def __init__(self, sequence=None, *, name=None):
self._offset_data_start,
*(v.encode(_enc) if isinstance(v, str) else v for v in sequence)
)
# For bytes, store actual length so retrieval is exact
# For bytes and str, store actual byte length so retrieval is exact
_stored_formats = [
self._types_mapping[bytes] % (len(v),) if isinstance(v, bytes) else f
(self._types_mapping[str] % (len(v.encode(_enc)),) if isinstance(v, str)
else self._types_mapping[bytes] % (len(v),) if isinstance(v, bytes)
else f)
for v, f in zip(sequence, _formats)
]
struct.pack_into(
Expand Down Expand Up @@ -481,8 +483,9 @@ def __setitem__(self, position, value):

self._set_packing_format_and_transform(
position,
self._types_mapping[bytes] % (len(encoded_value),)
if isinstance(value, bytes) else new_format,
(self._types_mapping[bytes] % (len(encoded_value),) if isinstance(value, bytes)
else self._types_mapping[str] % (len(encoded_value),) if isinstance(value, str)
else new_format),
value
)
struct.pack_into(new_format, self.shm.buf, offset, encoded_value)
Expand Down