Skip to content

gh-129205: Experiment BytesIO._readfrom() #130098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cmaloney wants to merge 8 commits into from
Closed

gh-129205: Experiment BytesIO._readfrom() #130098

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e350291
WIP: Pyio readfrom
cmaloney Feb 9, 2025
fd457a8
Working
cmaloney Feb 10, 2025
64abc84
WIP: Moving towards FileIO.readall using BytesIO.readfrom
cmaloney Feb 10, 2025
5f6bee9
WIP: FileIO.readall -> BytesIO._readfrom
cmaloney Feb 12, 2025
258025d
io.FileIO.readall working via io.BytesIO.readall
cmaloney Feb 13, 2025
4b3664f
Non-blocking readfrom working
cmaloney Feb 13, 2025
b3ca823
WIP: readfrom for general file objects with readinto, read
cmaloney Feb 13, 2025
deb12cc
tweak comments
cmaloney Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
WIP: readfrom for general file objects with readinto, read
  • Loading branch information
@cmaloney
cmaloney committed Feb 13, 2025
commit b3ca823715b3bd8e3b50ea9d93d94b4ccc5ebe28
12 changes: 4 additions & 8 deletions Lib/_compression.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,14 +111,10 @@ def read(self, size=-1):
return data

def readall(self):
chunks = []
# sys.maxsize means the max length of output buffer is unlimited,
# so that the whole input buffer can be decompressed within one
# .decompress() call.
while data := self.read(sys.maxsize):
chunks.append(data)

return b"".join(chunks)
# FIXME(cmaloney): non blocking support?
bio = io.BytesIO()
bio.readfrom(self)
return bio.getvalue()

# Rewind the file to the beginning of the data stream.
def _rewind(self):
Expand Down
12 changes: 11 additions & 1 deletion Lib/_pyio.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -962,10 +962,20 @@ def readfrom(self, file, /, *, estimate=None, limit=None):
if len(self._buffer) < target_read + self._pos:
self._buffer.resize(self._pos + target_read)

# File descriptor
if isinstance(file, int):
read_fn = lambda: os.readinto(file, memoryview(self._buffer)[self._pos:])
elif file_readinto := getattr(file, "readinto", None):
read_fn = lambda: file_readinto(memoryview(self._buffer)[self._pos:])
elif file_read := getattr(file, "read", None):
def read_fn():
data = file_read(len(self._buffer) - self._pos)
self._buffer[self._pos:self._pos + len(data)] = data

found_eof = False
start_pos = self._pos
try:
while n := os.readinto(file, memoryview(self._buffer)[self._pos:]):
while n := read_fn():
self._pos += n
# Expand buffer if needed.
if len(self._buffer) - self._pos <= 0:
Expand Down