Skip to content

IteratorByteStream: split large iterator chunks into 64 KB pieces #3752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Treadgold wants to merge 2 commits into
base: master
Choose a base branch
from
+32 −2
Open

IteratorByteStream: split large iterator chunks into 64 KB pieces #3752

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a48f2ab
IteratorByteStream: split large iterator chunks into 64 KB pieces
Treadgold Jan 31, 2026
fc2d503
IteratorByteStream: split large iterator chunks into 64KB pieces
Treadgold Jan 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
IteratorByteStream: split large iterator chunks into 64KB pieces
  • Loading branch information
@Treadgold
Treadgold committed Jan 31, 2026
commit fc2d503b4aa2dbda9180046fef63e9e8e7d04516
1 change: 1 addition & 0 deletions httpx/_content.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ def __iter__(self) -> Iterator[bytes]:
yield part[offset : offset + chunk_size]
offset += chunk_size


class AsyncIteratorByteStream(AsyncByteStream):
CHUNK_SIZE = 65_536

Expand Down
25 changes: 25 additions & 0 deletions tests/test_content.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -516,3 +516,28 @@ def test_allow_nan_false():
ValueError, match="Out of range float values are not JSON compliant"
):
httpx.Response(200, json=data_with_inf)


@pytest.mark.anyio
async def test_iterator_content_splits_large_chunks():
# Generator yielding a single large chunk (100 KB)
large_chunk = b"a" * 102_400 # 100 KB

def gen() -> typing.Iterator[bytes]:
yield large_chunk

# Pass generator to Request (internally uses IteratorByteStream)
request = httpx.Request(method, url, content=gen())

# Cast to Iterable[bytes] to make mypy happy
sync_stream: typing.Iterable[bytes] = request.stream # type: ignore

# Collect chunks
chunks = list(sync_stream)

# Each chunk must be <= 64 KB
for chunk in chunks:
assert len(chunk) <= 64 * 1024

# Total content matches original
assert b"".join(chunks) == large_chunk