Skip to content

Commit 01458dd

Browse files
jeff5jeff5
committed
Buffer API: much detailed rework of get-release strategy and request flags.
Inspired by difficulties encountered in advancing memoryview, there are changes here to the interfaces and classes that use them, together with documentation improvements. PyMemoryView now supports the BufferProtocol formally, but without new functionality. Regression tests pass (or fail where the previously did) for str and bytearray.
1 parent 4ae311c commit 01458dd

11 files changed

Lines changed: 804 additions & 437 deletions

File tree

‎src/org/python/core/BufferProtocol.java‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,15 @@
66
public interface BufferProtocol {
77

88
/**
9-
* Method by which the consumer requests the buffer from the exporter. The consumer
10-
* provides information on its intended method of navigation and the optional
11-
* features the buffer object must provide.
9+
* Method by which the consumer requests the buffer from the exporter. The consumer provides
10+
* information on its intended method of navigation and the features the buffer object is asked
11+
* (or assumed) to provide. Each consumer requesting a buffer in this way, when it has finished
12+
* using it, should make a corresponding call to {@link PyBuffer#release()} on the buffer it
13+
* obtained, since some objects alter their behaviour while buffers are exported.
1214
*
13-
* @param flags specification of options and the navigational capabilities of the consumer
15+
* @param flags specifying features demanded and the navigational capabilities of the consumer
1416
* @return exported buffer
17+
* @throws PyException (BufferError) when expectations do not correspond with the buffer
1518
*/
16-
PyBuffer getBuffer(int flags);
19+
PyBuffer getBuffer(int flags) throws PyException;
1720
}

‎src/org/python/core/PyBUF.java‎

Lines changed: 74 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -3,20 +3,21 @@
33
/**
44
* This interface provides a base for the key interface of the buffer API, {@link PyBuffer},
55
* including symbolic constants used by the consumer of a <code>PyBuffer</code> to specify its
6-
* requirements. The Jython buffer API emulates the CPython buffer API closely.
6+
* requirements and assumptions. The Jython buffer API emulates the CPython buffer API.
77
* <ul>
88
* <li>There are two reasons for separating parts of <code>PyBuffer</code> into this interface: The
99
* constants defined in CPython have the names <code>PyBUF_SIMPLE</code>,
1010
* <code>PyBUF_WRITABLE</code>, etc., and the trick of defining ours here means we can write
1111
* {@link PyBUF#SIMPLE}, {@link PyBUF#WRITABLE}, etc. so source code looks similar.</li>
1212
* <li>It is not so easy in Java as it is in C to treat a <code>byte</code> array as storing
1313
* anything other than <code>byte</code>, and we prepare for the possibility of buffers with a
14-
* series of different primitive types by defining here, those methods that would be in common
14+
* series of different primitive types by defining here those methods that would be in common
1515
* between <code>(Byte)Buffer</code> and an assumed future <code>FloatBuffer</code> or
1616
* <code>TypedBuffer&lt;T&gt;</code>. (Compare <code>java.nio.Buffer</code>.)</li>
1717
* </ul>
1818
* Except for other interfaces, it is unlikely any classes would implement <code>PyBUF</code>
19-
* directly.
19+
* directly. Users of the Jython buffer API can mostly overlook the distinction and just use
20+
* <code>PyBuffer</code>.
2021
*/
2122
public interface PyBUF {
2223

@@ -29,7 +30,8 @@ public interface PyBUF {
2930

3031
/**
3132
* The number of dimensions to the buffer. This number is the length of the <code>shape</code>
32-
* array.
33+
* array. The actual storage may be a linear array, but this is the number of dimensions in the
34+
* interpretation that the exporting object gives the data.
3335
*
3436
* @return number of dimensions
3537
*/
@@ -41,7 +43,8 @@ public interface PyBUF {
4143
* is the amount of buffer content addressed by one index or set of indices. In the simplest
4244
* case an item is a single unit (byte), and there is one dimension. In complex cases, the array
4345
* is multi-dimensional, and the item at each location is multi-unit (multi-byte). The consumer
44-
* must not modify this array.
46+
* must not modify this array. A valid <code>shape</code> array is always returned (difference
47+
* from CPython).
4548
*
4649
* @return the dimensions of the buffer as an array
4750
*/
@@ -56,40 +59,35 @@ public interface PyBUF {
5659

5760
/**
5861
* The total number of units (bytes) stored, which will be the product of the elements of the
59-
* shape, and the item size.
62+
* <code>shape</code> array, and the item size.
6063
*
6164
* @return the total number of units stored.
6265
*/
6366
int getLen();
6467

6568
/**
66-
* A buffer is (usually) coupled to the internal state of an exporting Python object, and that
67-
* object may have to restrict its behaviour while the buffer exists. The consumer must
68-
* therefore say when it has finished.
69-
*/
70-
void release();
71-
72-
/**
73-
* The "strides" array gives the distance in the storage array between adjacent items (in each
74-
* dimension). If the rawest parts of the buffer API, the consumer of the buffer is able to
75-
* navigate the exported storage. The "strides" array is part of the support for interpreting
76-
* the buffer as an n-dimensional array of items. In the one-dimensional case, the "strides"
77-
* array is In more dimensions, it provides the coefficients of the "addressing polynomial".
78-
* (More on this in the CPython documentation.) The consumer must not modify this array.
69+
* The <code>strides</code> array gives the distance in the storage array between adjacent items
70+
* (in each dimension). In the rawest parts of the buffer API, the consumer of the buffer is
71+
* able to navigate the exported storage. The "strides" array is part of the support for
72+
* interpreting the buffer as an n-dimensional array of items. It provides the coefficients of
73+
* the "addressing polynomial". (More on this in the CPython documentation.) The consumer must
74+
* not modify this array. A valid <code>strides</code> array is always returned (difference from
75+
* CPython).
7976
*
8077
* @return the distance in the storage array between adjacent items (in each dimension)
8178
*/
8279
int[] getStrides();
8380

8481
/**
85-
* The "suboffsets" array is a further part of the support for interpreting the buffer as an
86-
* n-dimensional array of items, where the array potentially uses indirect addressing (like a
87-
* real Java array of arrays, in fact). This is only applicable when there are more than 1
88-
* dimension and works in conjunction with the <code>strides</code> array. (More on this in the
89-
* CPython documentation.) When used, <code>suboffsets[k]</code> is an integer index, bit a byte
90-
* offset as in CPython. The consumer must not modify this array.
82+
* The <code>suboffsets</code> array is a further part of the support for interpreting the
83+
* buffer as an n-dimensional array of items, where the array potentially uses indirect
84+
* addressing (like a real Java array of arrays, in fact). This is only applicable when there
85+
* are more than 1 dimension and works in conjunction with the <code>strides</code> array. (More
86+
* on this in the CPython documentation.) When used, <code>suboffsets[k]</code> is an integer
87+
* index, bit a byte offset as in CPython. The consumer must not modify this array. When not
88+
* needed for navigation <code>null</code> is returned (as in CPython).
9189
*
92-
* @return
90+
* @return suboffsets array or null in not necessary for navigation
9391
*/
9492
int[] getSuboffsets();
9593

@@ -102,10 +100,10 @@ public interface PyBUF {
102100
*/
103101
boolean isContiguous(char order);
104102

105-
/* Constants taken from CPython object.h in v3.3.0a */
103+
/* Constants taken from CPython object.h in v3.3 */
106104

107105
/**
108-
* The maximum allowed number of dimensions (NumPy restriction?).
106+
* The maximum allowed number of dimensions (CPython restriction).
109107
*/
110108
static final int MAX_NDIM = 64;
111109
/**
@@ -123,53 +121,58 @@ public interface PyBUF {
123121
static final int SIMPLE = 0;
124122
/**
125123
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
126-
* specify that it requires {@link PyBuffer#getFormat()} to return the type of the unit (rather
127-
* than return <code>null</code>).
124+
* specify that it requires {@link PyBuffer#getFormat()} to return a <code>String</code>
125+
* indicating the type of the unit. This exists for compatibility with CPython, as Jython as the
126+
* format is always provided by <code>getFormat()</code>.
128127
*/
129-
// I don't understand why we need this, or why format MUST be null of this is not set.
130128
static final int FORMAT = 0x0004;
131129
/**
132130
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
133-
* specify that it it is prepared to navigate the buffer as multi-dimensional.
134-
* <code>getBuffer</code> will raise an exception if consumer does not specify the flag but the
135-
* exporter's buffer cannot be navigated without taking into account its multiple dimensions.
131+
* specify that it is prepared to navigate the buffer as multi-dimensional using the
132+
* <code>shape</code> array. <code>getBuffer</code> will raise an exception if consumer does not
133+
* specify the flag but the exporter's buffer cannot be navigated without taking into account
134+
* its multiple dimensions.
136135
*/
137-
static final int ND = 0x0008 | SIMPLE; // Differs from CPython by or'ing in SIMPLE
136+
static final int ND = 0x0008;
138137
/**
139138
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
140-
* specify that it it expects to use the "strides" array. <code>getBuffer</code> will raise an
141-
* exception if consumer does not specify the flag but the exporter's buffer cannot be navigated
142-
* without using the "strides" array.
139+
* specify that it expects to use the <code>strides</code> array. <code>getBuffer</code> will
140+
* raise an exception if consumer does not specify the flag but the exporter's buffer cannot be
141+
* navigated without using the <code>strides</code> array.
143142
*/
144143
static final int STRIDES = 0x0010 | ND;
145144
/**
146145
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
147-
* specify that it will assume C-order organisation of the units. <code>getBuffer</code> will raise an
148-
* exception if the exporter's buffer is not C-ordered. <code>C_CONTIGUOUS</code> implies
149-
* <code>STRIDES</code>.
146+
* specify that it will assume C-order organisation of the units. <code>getBuffer</code> will
147+
* raise an exception if the exporter's buffer is not C-ordered. <code>C_CONTIGUOUS</code>
148+
* implies <code>STRIDES</code>.
150149
*/
150+
// It is possible this should have been (0x20|ND) expressing the idea that C-order addressing
151+
// will be assumed *instead of* using a strides array.
151152
static final int C_CONTIGUOUS = 0x0020 | STRIDES;
152153
/**
153154
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
154-
* specify that it will assume Fortran-order organisation of the units. <code>getBuffer</code> will raise an
155-
* exception if the exporter's buffer is not Fortran-ordered. <code>F_CONTIGUOUS</code> implies
156-
* <code>STRIDES</code>.
155+
* specify that it will assume Fortran-order organisation of the units. <code>getBuffer</code>
156+
* will raise an exception if the exporter's buffer is not Fortran-ordered.
157+
* <code>F_CONTIGUOUS</code> implies <code>STRIDES</code>.
157158
*/
158159
static final int F_CONTIGUOUS = 0x0040 | STRIDES;
159160
/**
160161
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
161-
* specify that it
162+
* specify that it will assume a contiguous organisation of the units, but will enquire which
163+
* organisation it actually is.
162164
*
163165
* getBuffer will raise an exception if the exporter's buffer is not contiguous.
164166
* <code>ANY_CONTIGUOUS</code> implies <code>STRIDES</code>.
165167
*/
168+
// Further CPython strangeness since it uses the strides array to answer the enquiry.
166169
static final int ANY_CONTIGUOUS = 0x0080 | STRIDES;
167170
/**
168171
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
169-
* specify that it understands the "suboffsets" array. <code>getBuffer</code> will raise an
170-
* exception if consumer does not specify the flag but the exporter's buffer cannot be navigated
171-
* without understanding the "suboffsets" array. <code>INDIRECT</code> implies
172-
* <code>STRIDES</code>.
172+
* specify that it understands the <code>suboffsets</code> array. <code>getBuffer</code> will
173+
* raise an exception if consumer does not specify the flag but the exporter's buffer cannot be
174+
* navigated without understanding the <code>suboffsets</code> array. <code>INDIRECT</code>
175+
* implies <code>STRIDES</code>.
173176
*/
174177
static final int INDIRECT = 0x0100 | STRIDES;
175178
/**
@@ -208,31 +211,34 @@ public interface PyBUF {
208211
/* Constants for readability, not standard for CPython */
209212

210213
/**
211-
* Field mask, use as in <code>if ((capabilityFlags&ORGANISATION) == STRIDES) ...</code>.
212-
*/
213-
static final int ORGANISATION = SIMPLE | ND | STRIDES | INDIRECT;
214-
/**
215-
* Field mask, use as in if <code>((capabilityFlags&ORGANIZATION) == STRIDES) ...</code>.
216-
*
217-
* @see #ORGANISATION
218-
*/
219-
static final int ORGANIZATION = ORGANISATION;
220-
/**
221-
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
222-
* specify that it will assume C-order organisation of the units, irrespective of whether
223-
* the strides array is to be provided. <code>getBuffer</code> will raise an
224-
* exception if the exporter's buffer is not C-ordered. <code>C_CONTIGUOUS = IS_C_CONTIGUOUS | STRIDES</code>.
214+
* Field mask, use as in <code>if ((flags&NAVIGATION) == STRIDES) ...</code>. The importance of
215+
* the subset of flags defined by this mask is not so much in their "navigational" character as
216+
* in the way they are treated in a buffer request.
217+
* <p>
218+
* The <code>NAVIGATION</code> set are used to specify which navigation arrays the consumer will
219+
* use, and therefore the consumer must ask for all those necessary to use the buffer
220+
* successfully (which is a function of the buffer's actual type). Asking for extra ones is not
221+
* an error, since all are supplied (in Jython): asking for too few is an error.
222+
* <p>
223+
* Flags outside the <code>NAVIGATION</code> set, work the other way round. Asking for one the
224+
* buffer cannot match is an error: not asking for a feature the buffer does not have is an
225+
* error.
226+
*/
227+
static final int NAVIGATION = SIMPLE | ND | STRIDES | INDIRECT;
228+
/**
229+
* A constant used by the exporter in processing {@link BufferProtocol#getBuffer(int)} to check
230+
* for assumed C-order organisation of the units.
231+
* <code>C_CONTIGUOUS = IS_C_CONTIGUOUS | STRIDES</code>.
225232
*/
226233
static final int IS_C_CONTIGUOUS = C_CONTIGUOUS & ~STRIDES;
227234
/**
228-
* A constant used by the consumer in its call to {@link BufferProtocol#getBuffer(int)} to
229-
* specify that it will assume Fortran-order organisation of the units, irrespective of whether
230-
* the strides array is to be provided. <code>getBuffer</code> will raise an
231-
* exception if the exporter's buffer is not Fortran-ordered. <code>F_CONTIGUOUS = IS_F_CONTIGUOUS | STRIDES</code>.
235+
* A constant used by the exporter in processing {@link BufferProtocol#getBuffer(int)} to check
236+
* for assumed C-order Fortran-order organisation of the units.
237+
* <code>F_CONTIGUOUS = IS_F_CONTIGUOUS | STRIDES</code>.
232238
*/
233239
static final int IS_F_CONTIGUOUS = F_CONTIGUOUS & ~STRIDES;
234240
/**
235-
* Field mask, use as in <code>if (capabilityFlags&CONTIGUITY== ... ) ...</code>.
241+
* Field mask, use as in <code>if (flags&CONTIGUITY== ... ) ...</code>.
236242
*/
237243
static final int CONTIGUITY = (C_CONTIGUOUS | F_CONTIGUOUS | ANY_CONTIGUOUS) & ~STRIDES;
238244
}

0 commit comments

Comments
 (0)