requests: HTTP/1.1 response bodies and max_body limit#1119
Open
pablogventura wants to merge 7 commits into
Open
requests: HTTP/1.1 response bodies and max_body limit#1119pablogventura wants to merge 7 commits into
pablogventura wants to merge 7 commits into
Conversation
Introduce read_headers(), read_body(), and chunked decoding without changing the public request() API yet. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Parse Content-Length and chunked Transfer-Encoding; store body on Response.content. Default request line to HTTP/1.1. Add max_body limit (32 KiB default). Fix relative redirect Location URLs. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Cover Content-Length, chunked encoding, max_body, and relative redirects using the socket mock. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Document HTTP/1.1 response support, max_body, and remaining limits. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Order-independent mock asserts, live TCP tests, unix/ESP32 runners, prefer /lib over frozen requests on ESP32, and HEAD/auth/encoding fixes. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
I use
requestson small boards fairly often, and the current package left me hanging onchunked responses (
raw.read()never finished) and still spoke HTTP/1.0 on the wire. This PRis the first slice of the HTTP work I outlined in #1118: enough client behaviour for real
servers without pulling in TLS or streaming yet.
What I changed:
requests/_http.pyto parse status/headers and read bodies (Content-Length andchunked). The full body lands in
Response.contentso callers do not have to fight thesocket after the headers.
Connection: close(no keep-alive in this PR).resp.headersis a smallHeadersmapping with case-insensitive keys.Locationvalues on redirects are resolved against the original URL (fixes "requests" does not handle redirection with relative path correctly #931).max_bodyargument, default 32 KiB; passmax_body=Noneif you really want no cap.Closes #1118.
Testing
I started with the existing socket-mock tests and extended them for bodies, chunked encoding,
redirects, and
max_body. On my machine:All 14 mock tests pass on CPython 3.12.
I also ran the same file on the unix port (
build-standard) and addedtest_requests_live.py(local TCP server in a thread) plus
run-unix-tests.sh— 14 + 4 tests, all green.On hardware I flashed ESP32_GENERIC (ESP32-D0WD, 4 MiB flash), copied this branch’s package
to
/lib/requests(the board still ships an older frozenrequests, so tests prefer/lib),and ran
run-esp32-tests.sh(mock + live). Same counts, all pass.Finally I connected the board to my home WiFi and hit real HTTP endpoints (
example.com,httpbin.orgGET + redirect) — status 200, bodies as expected. I have not added HTTPS here;that stays for a follow-up (#838).
CI in this repo should pick up
python-ecosys/requests/test_requests.pywith the project’sMicroPython build; the live/ESP32 scripts are mainly for local regression.
Trade-offs and alternatives
documented it and added
max_bodyso a runaway response does not silently exhaust memory.(SECURITY: Requests module HTTPS - no server certificate verification. #838, requests: stream parameter is noop #777).
headers for diff stability).
Generative AI
I used generative AI tools while working on this PR (including help polishing this
description). I reviewed all of the code and take responsibility for the changes and the
text above.