Implement PEP 688 and rework the buffer protocol around managed exports - #8523
Draft
youknowone wants to merge 2 commits into
Draft
Implement PEP 688 and rework the buffer protocol around managed exports#8523youknowone wants to merge 2 commits into
youknowone wants to merge 2 commits into
Conversation
A Python class could not export a buffer: the slot machinery had no bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as PyBUF_FULL_RO through a module of PyBUF_* constants. Add both slots. PyBuffer::release now runs a Python __release_buffer__ before the exporter's own release, once per acquisition, which PyBuffer tracks with an `acquired` flag that clones do not inherit. An export made by a Python __buffer__ is held by a _buffer_wrapper payload that counts its exports and drops the returned memoryview with the last one, and the view handed to __release_buffer__ is a _buffer_window that owns no export, so releasing it inside the hook is inert instead of re-entering it. Replace the PyBUF_* constants with a BufferFlags bitflags type whose composite requests are supersets of the simpler ones, so `contains` answers the REQ_* questions, and pass the request to PyBuffer::from_object. Each consumer now asks for what its counterpart asks for: y* arguments for SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray() and memoryview() for FULL_RO. memoryview checks the request in memory_getbuf, and array.array and mmap.mmap expose __release_buffer__. Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer) instead of attempting an acquisition, so an exception raised by __buffer__ is no longer reported as the object not being bytes-like, and a __buffer__ with side effects runs once. PyBytesInner becomes a y* conversion as a result: bytes and bytearray methods no longer accept iterables of ints, and find, index, count and __contains__ take the arguments parse_args_finds_byte and bytes_contains describe. A view exports its start offset in the descriptor rather than in its window, which fixes a panic when collecting from a negative-stride view. BytesIO.write rechecks closed after acquiring its buffer, which __buffer__ can close in between. Assisted-by: Claude Code:claude-opus-5
Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer` acquisition is shared by every handle taken from it, cloning takes another share instead of re-acquiring, and the exporter's release runs once when the last share goes away. Remove `retain`, the unsafe `drop_without_release`, the three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add `abort_acquisition` so a failed request does not run `bf_releasebuffer`. Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf` analogue, and drop the separate `start` fields on `PyMemoryView` and `PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`, which reproduces `PySlice_AdjustIndices` and keeps the adjusted start. Fix `zip_eq` to take its contiguous fast path only when both last dimensions are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and zero-dimensional views. Add `BufferDescriptor::projected` so a request without `PyBUF_ND`, `PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter that has suboffsets. Copy the source first in `memoryview` slice assignment when both sides reach the same root exporter. Hold the export across the resize in `bytearray.extend`, take `y*` in `marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord` over the concrete string types, fold `array`'s buffer slot into one `slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset` field through the `_ctypes` descriptors. Assisted-by: Claude
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
📦 Library DependenciesThe following Lib/ modules were modified. Here are their dependencies: [x] lib: cpython/Lib/struct.py dependencies:
dependent tests: (179 tests)
[ ] test: cpython/Lib/test/test_memoryview.py (TODO: 7) dependencies: dependent tests: (no tests depend on memoryview) [x] test: cpython/Lib/test/test_buffer.py dependencies: dependent tests: (no tests depend on buffer) [x] lib: cpython/Lib/io.py dependencies:
dependent tests: (108 tests)
[ ] test: cpython/Lib/test/test_marshal.py (TODO: 8) dependencies: dependent tests: (25 tests)
[x] lib: cpython/Lib/code.py dependencies:
dependent tests: (2 tests) [ ] lib: cpython/Lib/collections dependencies:
dependent tests: (331 tests)
[x] test: cpython/Lib/test/test_structseq.py (TODO: 7) dependencies: dependent tests: (no tests depend on structseq) [x] test: cpython/Lib/test/test_itertools.py (TODO: 4) dependencies: dependent tests: (56 tests)
Legend:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements PEP 688 (
__buffer__/__release_buffer__) and reworks the bufferprotocol underneath it so the two commits together match the CPython 3.14
semantics.
__buffer__/__release_buffer__A Python-level
__buffer__is exposed throughbf_getbufferand__release_buffer__throughbf_releasebuffer, mirroringslot_bf_getbufferand
slot_bf_releasebuffer.memoryview.__buffer__(flags)andmemoryview.__release_buffer__(view)are added.Managed exports
PyBuffertakes the_PyManagedBufferObjectshape: onebf_getbufferacquisition is shared by every handle taken from it, cloning takes another share
instead of re-acquiring, and the exporter's release runs exactly once when the
last share goes away. This removes
retain, the unsafedrop_without_release,three
impl Drops and theManuallyDropthat previously stood in for therefcount.
abort_acquisitionkeepsbf_releasebufferfrom running whenbf_getbufferitself failed.View offsets
The view start moves into
BufferDescriptor::offset, thePy_buffer.bufanalogue, replacing the separate
startfields onPyMemoryViewandPyBufferWrapperthat let an exported buffer disagree with the view it camefrom. Slicing goes through
SaturatedSlice::adjust_indices_start, reproducingPySlice_AdjustIndices.Other fixes found along the way
zip_eqtook its contiguous fast path when only one side's last dimension wascontiguous (
last_dim_is_contiguous).for_each_segmentandzip_eqmishandled zero-length and zero-dimensionalviews.
BufferDescriptor::projectednow reducesthe descriptor for a request without
PyBUF_ND/PyBUF_STRIDES/PyBUF_FORMAT, and a request withoutPyBUF_INDIRECTagainst an exporter withsuboffsets is rejected.
memoryviewslice assignment copies the source first when both sides reach thesame root exporter.
bytearray.extendnow holds the export across the resize.marshal.loadstakesy*,_overlappedtakesw*/y*,FsPathno longerprobes the buffer protocol,
ordis rewritten over the concrete string types,and
array's buffer slot is folded into a singleslot_as_buffer.Verification
406 tests OK, run=40,778 failures=2, against a404 tests OK, run=40,740 failures=2baseline. The one failure,test_future_stmt.test_future, is pre-existing and unrelated.extra_tests/snippets/builtin_memoryview.pygains 11 test functions, all ofwhich pass on CPython 3.14 as well.
debug_assertslive.
crates/stdlib/src/overlapped.rsis Windows-only and could not be compiledlocally, so it rests on CI.
🤖 Generated with Claude Code
https://claude.ai/code/session_01P9HewXGX8qcGSccUxGdSPV