Skip to content

Commit c249221

Browse files
vadmiumvadmium
committed
Issue python#20699: Merge io bytes-like fixes from 3.5
2 parents 0472217 + 6bb91f3 commit c249221

File tree

10 files changed

+128
-67
lines changed

10 files changed

+128
-67
lines changed

‎Doc/library/io.rst‎

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,8 @@ The text stream API is described in detail in the documentation of
6666
Binary I/O
6767
^^^^^^^^^^
6868

69-
Binary I/O (also called *buffered I/O*) expects and produces :class:`bytes`
69+
Binary I/O (also called *buffered I/O*) expects
70+
:term:`bytes-like objects <bytes-like object>` and produces :class:`bytes`
7071
objects. No encoding, decoding, or newline translation is performed. This
7172
category of streams can be used for all kinds of non-text data, and also when
7273
manual control over the handling of text data is desired.
@@ -227,9 +228,10 @@ I/O Base Classes
227228
when operations they do not support are called.
228229

229230
The basic type used for binary data read from or written to a file is
230-
:class:`bytes`. :class:`bytearray`\s are accepted too, and in some cases
231-
(such as :meth:`readinto`) required. Text I/O classes work with
232-
:class:`str` data.
231+
:class:`bytes`. Other :term:`bytes-like objects <bytes-like object>` are
232+
accepted as method arguments too. In some cases, such as
233+
:meth:`~RawIOBase.readinto`, a writable object such as :class:`bytearray`
234+
is required. Text I/O classes work with :class:`str` data.
233235

234236
Note that calling any method (even inquiries) on a closed stream is
235237
undefined. Implementations may raise :exc:`ValueError` in this case.
@@ -393,18 +395,22 @@ I/O Base Classes
393395

394396
.. method:: readinto(b)
395397

396-
Read up to ``len(b)`` bytes into :class:`bytearray` *b* and return the
398+
Read bytes into a pre-allocated, writable
399+
:term:`bytes-like object` *b*, and return the
397400
number of bytes read. If the object is in non-blocking mode and no bytes
398401
are available, ``None`` is returned.
399402

400403
.. method:: write(b)
401404

402-
Write the given :class:`bytes` or :class:`bytearray` object, *b*, to the
403-
underlying raw stream and return the number of bytes written. This can
404-
be less than ``len(b)``, depending on specifics of the underlying raw
405+
Write the given :term:`bytes-like object`, *b*, to the
406+
underlying raw stream, and return the number of
407+
bytes written. This can be less than the length of *b* in
408+
bytes, depending on specifics of the underlying raw
405409
stream, and especially if it is in non-blocking mode. ``None`` is
406410
returned if the raw stream is set not to block and no single byte could
407-
be readily written to it.
411+
be readily written to it. The caller may release or mutate *b* after
412+
this method returns, so the implementation should only access *b*
413+
during the method call.
408414

409415

410416
.. class:: BufferedIOBase
@@ -476,8 +482,8 @@ I/O Base Classes
476482

477483
.. method:: readinto(b)
478484

479-
Read up to ``len(b)`` bytes into bytearray *b* and return the number of
480-
bytes read.
485+
Read bytes into a pre-allocated, writable
486+
:term:`bytes-like object` *b* and return the number of bytes read.
481487

482488
Like :meth:`read`, multiple reads may be issued to the underlying raw
483489
stream, unless the latter is interactive.
@@ -487,7 +493,8 @@ I/O Base Classes
487493

488494
.. method:: readinto1(b)
489495

490-
Read up to ``len(b)`` bytes into bytearray *b*, using at most one call to
496+
Read bytes into a pre-allocated, writable
497+
:term:`bytes-like object` *b*, using at most one call to
491498
the underlying raw stream's :meth:`~RawIOBase.read` (or
492499
:meth:`~RawIOBase.readinto`) method. Return the number of bytes read.
493500

@@ -498,8 +505,8 @@ I/O Base Classes
498505

499506
.. method:: write(b)
500507

501-
Write the given :class:`bytes` or :class:`bytearray` object, *b* and
502-
return the number of bytes written (never less than ``len(b)``, since if
508+
Write the given :term:`bytes-like object`, *b*, and return the number
509+
of bytes written (always equal to the length of *b* in bytes, since if
503510
the write fails an :exc:`OSError` will be raised). Depending on the
504511
actual implementation, these bytes may be readily written to the
505512
underlying stream, or held in a buffer for performance and latency
@@ -509,6 +516,9 @@ I/O Base Classes
509516
data needed to be written to the raw stream but it couldn't accept
510517
all the data without blocking.
511518

519+
The caller may release or mutate *b* after this method returns,
520+
so the implementation should only access *b* during the method call.
521+
512522

513523
Raw File I/O
514524
^^^^^^^^^^^^
@@ -584,7 +594,8 @@ than raw I/O does.
584594
:class:`BufferedIOBase`. The buffer is discarded when the
585595
:meth:`~IOBase.close` method is called.
586596

587-
The argument *initial_bytes* contains optional initial :class:`bytes` data.
597+
The optional argument *initial_bytes* is a :term:`bytes-like object` that
598+
contains initial data.
588599

589600
:class:`BytesIO` provides or overrides these methods in addition to those
590601
from :class:`BufferedIOBase` and :class:`IOBase`:
@@ -682,7 +693,7 @@ than raw I/O does.
682693

683694
.. method:: write(b)
684695

685-
Write the :class:`bytes` or :class:`bytearray` object, *b* and return the
696+
Write the :term:`bytes-like object`, *b*, and return the
686697
number of bytes written. When in non-blocking mode, a
687698
:exc:`BlockingIOError` is raised if the buffer needs to be written out but
688699
the raw stream blocks.

‎Lib/_pyio.py‎

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -296,8 +296,9 @@ class IOBase(metaclass=abc.ABCMeta):
296296
called.
297297
298298
The basic type used for binary data read from or written to a file is
299-
bytes. bytearrays are accepted too, and in some cases (such as
300-
readinto) needed. Text I/O classes work with str data.
299+
bytes. Other bytes-like objects are accepted as method arguments too. In
300+
some cases (such as readinto), a writable object is required. Text I/O
301+
classes work with str data.
301302
302303
Note that calling any method (even inquiries) on a closed stream is
303304
undefined. Implementations may raise OSError in this case.
@@ -596,7 +597,7 @@ def readall(self):
596597
return data
597598

598599
def readinto(self, b):
599-
"""Read up to len(b) bytes into bytearray b.
600+
"""Read bytes into a pre-allocated bytes-like object b.
600601
601602
Returns an int representing the number of bytes read (0 for EOF), or
602603
None if the object is set not to block and has no data to read.
@@ -606,7 +607,8 @@ def readinto(self, b):
606607
def write(self, b):
607608
"""Write the given buffer to the IO stream.
608609
609-
Returns the number of bytes written, which may be less than len(b).
610+
Returns the number of bytes written, which may be less than the
611+
length of b in bytes.
610612
"""
611613
self._unsupported("write")
612614

@@ -659,7 +661,7 @@ def read1(self, size=None):
659661
self._unsupported("read1")
660662

661663
def readinto(self, b):
662-
"""Read up to len(b) bytes into bytearray b.
664+
"""Read bytes into a pre-allocated bytes-like object b.
663665
664666
Like read(), this may issue multiple reads to the underlying raw
665667
stream, unless the latter is 'interactive'.
@@ -673,7 +675,7 @@ def readinto(self, b):
673675
return self._readinto(b, read1=False)
674676

675677
def readinto1(self, b):
676-
"""Read up to len(b) bytes into *b*, using at most one system call
678+
"""Read bytes into buffer *b*, using at most one system call
677679
678680
Returns an int representing the number of bytes read (0 for EOF).
679681
@@ -701,8 +703,8 @@ def _readinto(self, b, read1):
701703
def write(self, b):
702704
"""Write the given bytes buffer to the IO stream.
703705
704-
Return the number of bytes written, which is never less than
705-
len(b).
706+
Return the number of bytes written, which is always the length of b
707+
in bytes.
706708
707709
Raises BlockingIOError if the buffer is full and the
708710
underlying raw stream cannot accept more data at the moment.
@@ -884,7 +886,8 @@ def write(self, b):
884886
raise ValueError("write to closed file")
885887
if isinstance(b, str):
886888
raise TypeError("can't write str to binary stream")
887-
n = len(b)
889+
with memoryview(b) as view:
890+
n = view.nbytes # Size of any bytes-like object
888891
if n == 0:
889892
return 0
890893
pos = self._pos
@@ -1090,14 +1093,13 @@ def read1(self, size):
10901093
def _readinto(self, buf, read1):
10911094
"""Read data into *buf* with at most one system call."""
10921095

1093-
if len(buf) == 0:
1094-
return 0
1095-
10961096
# Need to create a memoryview object of type 'b', otherwise
10971097
# we may not be able to assign bytes to it, and slicing it
10981098
# would create a new object.
10991099
if not isinstance(buf, memoryview):
11001100
buf = memoryview(buf)
1101+
if buf.nbytes == 0:
1102+
return 0
11011103
buf = buf.cast('B')
11021104

11031105
written = 0

‎Lib/test/test_io.py‎

Lines changed: 64 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,22 @@
4545
except ImportError:
4646
threading = None
4747

48+
try:
49+
import ctypes
50+
except ImportError:
51+
def byteslike(*pos, **kw):
52+
return array.array("b", bytes(*pos, **kw))
53+
else:
54+
def byteslike(*pos, **kw):
55+
"""Create a bytes-like object having no string or sequence methods"""
56+
data = bytes(*pos, **kw)
57+
obj = EmptyStruct()
58+
ctypes.resize(obj, len(data))
59+
memoryview(obj).cast("B")[:] = data
60+
return obj
61+
class EmptyStruct(ctypes.Structure):
62+
pass
63+
4864
def _default_chunk_size():
4965
"""Get the default TextIOWrapper chunk size"""
5066
with open(__file__, "r", encoding="latin-1") as f:
@@ -284,7 +300,9 @@ def write_ops(self, f):
284300
self.assertEqual(f.tell(), 6)
285301
self.assertEqual(f.seek(-1, 1), 5)
286302
self.assertEqual(f.tell(), 5)
287-
self.assertEqual(f.write(bytearray(b" world\n\n\n")), 9)
303+
buffer = bytearray(b" world\n\n\n")
304+
self.assertEqual(f.write(buffer), 9)
305+
buffer[:] = b"*" * 9 # Overwrite our copy of the data
288306
self.assertEqual(f.seek(0), 0)
289307
self.assertEqual(f.write(b"h"), 1)
290308
self.assertEqual(f.seek(-1, 2), 13)
@@ -297,20 +315,21 @@ def write_ops(self, f):
297315
def read_ops(self, f, buffered=False):
298316
data = f.read(5)
299317
self.assertEqual(data, b"hello")
300-
data = bytearray(data)
318+
data = byteslike(data)
301319
self.assertEqual(f.readinto(data), 5)
302-
self.assertEqual(data, b" worl")
320+
self.assertEqual(bytes(data), b" worl")
321+
data = bytearray(5)
303322
self.assertEqual(f.readinto(data), 2)
304323
self.assertEqual(len(data), 5)
305324
self.assertEqual(data[:2], b"d\n")
306325
self.assertEqual(f.seek(0), 0)
307326
self.assertEqual(f.read(20), b"hello world\n")
308327
self.assertEqual(f.read(1), b"")
309-
self.assertEqual(f.readinto(bytearray(b"x")), 0)
328+
self.assertEqual(f.readinto(byteslike(b"x")), 0)
310329
self.assertEqual(f.seek(-6, 2), 6)
311330
self.assertEqual(f.read(5), b"world")
312331
self.assertEqual(f.read(0), b"")
313-
self.assertEqual(f.readinto(bytearray()), 0)
332+
self.assertEqual(f.readinto(byteslike()), 0)
314333
self.assertEqual(f.seek(-6, 1), 5)
315334
self.assertEqual(f.read(5), b" worl")
316335
self.assertEqual(f.tell(), 10)
@@ -321,6 +340,10 @@ def read_ops(self, f, buffered=False):
321340
f.seek(6)
322341
self.assertEqual(f.read(), b"world\n")
323342
self.assertEqual(f.read(), b"")
343+
f.seek(0)
344+
data = byteslike(5)
345+
self.assertEqual(f.readinto1(data), 5)
346+
self.assertEqual(bytes(data), b"hello")
324347

325348
LARGE = 2**31
326349

@@ -641,10 +664,15 @@ def test_close_flushes(self):
641664
def test_array_writes(self):
642665
a = array.array('i', range(10))
643666
n = len(a.tobytes())
644-
with self.open(support.TESTFN, "wb", 0) as f:
645-
self.assertEqual(f.write(a), n)
646-
with self.open(support.TESTFN, "wb") as f:
647-
self.assertEqual(f.write(a), n)
667+
def check(f):
668+
with f:
669+
self.assertEqual(f.write(a), n)
670+
f.writelines((a,))
671+
check(self.BytesIO())
672+
check(self.FileIO(support.TESTFN, "w"))
673+
check(self.BufferedWriter(self.MockRawIO()))
674+
check(self.BufferedRandom(self.MockRawIO()))
675+
check(self.BufferedRWPair(self.MockRawIO(), self.MockRawIO()))
648676

649677
def test_closefd(self):
650678
self.assertRaises(ValueError, self.open, support.TESTFN, 'w',
@@ -803,6 +831,19 @@ def test_invalid_newline(self):
803831
with self.assertRaises(ValueError):
804832
self.open(support.TESTFN, 'w', newline='invalid')
805833

834+
def test_buffered_readinto_mixin(self):
835+
# Test the implementation provided by BufferedIOBase
836+
class Stream(self.BufferedIOBase):
837+
def read(self, size):
838+
return b"12345"
839+
read1 = read
840+
stream = Stream()
841+
for method in ("readinto", "readinto1"):
842+
with self.subTest(method):
843+
buffer = byteslike(5)
844+
self.assertEqual(getattr(stream, method)(buffer), 5)
845+
self.assertEqual(bytes(buffer), b"12345")
846+
806847

807848
class CIOTest(IOTest):
808849

@@ -1394,6 +1435,11 @@ def test_write(self):
13941435
bufio = self.tp(writer, 8)
13951436
bufio.write(b"abc")
13961437
self.assertFalse(writer._write_stack)
1438+
buffer = bytearray(b"def")
1439+
bufio.write(buffer)
1440+
buffer[:] = b"***" # Overwrite our copy of the data
1441+
bufio.flush()
1442+
self.assertEqual(b"".join(writer._write_stack), b"abcdef")
13971443

13981444
def test_write_overflow(self):
13991445
writer = self.MockRawIO()
@@ -1720,19 +1766,23 @@ def test_read1(self):
17201766
self.assertEqual(pair.read1(3), b"abc")
17211767

17221768
def test_readinto(self):
1723-
pair = self.tp(self.BytesIO(b"abcdef"), self.MockRawIO())
1769+
for method in ("readinto", "readinto1"):
1770+
with self.subTest(method):
1771+
pair = self.tp(self.BytesIO(b"abcdef"), self.MockRawIO())
17241772

1725-
data = bytearray(5)
1726-
self.assertEqual(pair.readinto(data), 5)
1727-
self.assertEqual(data, b"abcde")
1773+
data = byteslike(5)
1774+
self.assertEqual(getattr(pair, method)(data), 5)
1775+
self.assertEqual(bytes(data), b"abcde")
17281776

17291777
def test_write(self):
17301778
w = self.MockRawIO()
17311779
pair = self.tp(self.MockRawIO(), w)
17321780

17331781
pair.write(b"abc")
17341782
pair.flush()
1735-
pair.write(b"def")
1783+
buffer = bytearray(b"def")
1784+
pair.write(buffer)
1785+
buffer[:] = b"***" # Overwrite our copy of the data
17361786
pair.flush()
17371787
self.assertEqual(w._write_stack, [b"abc", b"def"])
17381788

‎Lib/test/test_memoryio.py‎

Lines changed: 10 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,16 @@ def __init__(me, initvalue, foo):
399399
del __main__.PickleTestMemIO
400400

401401

402-
class BytesIOMixin:
402+
class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
403+
# Test _pyio.BytesIO; class also inherited for testing C implementation
404+
405+
UnsupportedOperation = pyio.UnsupportedOperation
406+
407+
@staticmethod
408+
def buftype(s):
409+
return s.encode("ascii")
410+
ioclass = pyio.BytesIO
411+
EOF = b""
403412

404413
def test_getbuffer(self):
405414
memio = self.ioclass(b"1234567890")
@@ -426,18 +435,6 @@ def test_getbuffer(self):
426435
memio.close()
427436
self.assertRaises(ValueError, memio.getbuffer)
428437

429-
430-
class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin,
431-
BytesIOMixin, unittest.TestCase):
432-
433-
UnsupportedOperation = pyio.UnsupportedOperation
434-
435-
@staticmethod
436-
def buftype(s):
437-
return s.encode("ascii")
438-
ioclass = pyio.BytesIO
439-
EOF = b""
440-
441438
def test_read1(self):
442439
buf = self.buftype("1234567890")
443440
memio = self.ioclass(buf)

‎Modules/_io/bufferedio.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -190,8 +190,8 @@ bufferediobase_read1(PyObject *self, PyObject *args)
190190
PyDoc_STRVAR(bufferediobase_write_doc,
191191
"Write the given buffer to the IO stream.\n"
192192
"\n"
193-
"Returns the number of bytes written, which is never less than\n"
194-
"len(b).\n"
193+
"Returns the number of bytes written, which is always the length of b\n"
194+
"in bytes.\n"
195195
"\n"
196196
"Raises BlockingIOError if the buffer is full and the\n"
197197
"underlying raw stream cannot accept more data at the moment.\n");

0 commit comments

Comments
 (0)