Repository navigation
fix!: consistent client error paths (timeout=None, RemoteProtocolError, invalid URL, closed client, streaming bodies, async middleware) - #169
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Seven fixes to
ClientandAsyncClient, each one test-first. Every change is mirrored in both clients except the last two, which only apply to one of them.timeout=Nonenow means no timeout (breaking)._select_httpx2_optionstreated everyNoneoption as "not passed", soClient(timeout=None)silently kept httpx2's 5 second default. An explicittimeout=Noneis now forwarded to the owned httpx2 client. The other options still treatNoneas unset.httpx2.RemoteProtocolErrormaps toNetworkError. It used to become a bareTransportError, so neitherRetrynor the circuit breaker treated it as transient. The most common cause is a server dropping a keep-alive connection mid-response, which is exactly the case retry exists for.TransportErrorfrom every method.stream()already mappedhttpx2.InvalidURL, butget(),post(),request(), the*_with_responsemethods and the rest raised it raw, because the request is built outside the mapper._prepare_requestnow maps it.build_request()still raiseshttpx2.InvalidURL, since it returns an httpx2 object and sends nothing.stream()on a closed client raisesTransportError. It raised httpx2's rawRuntimeError, whileget()raisedTransportError. The closed-client check moved out of each_terminaland into the shared_httpx2_exception_mapper/_httpx2_exception_mapper_sync, which now take the httpx2 client, so the terminal andstream()share it. ARuntimeErroron an open client still propagates unchanged.send(). Only the verb methods setSTREAMING_BODY_MARKER, sosend(client.build_request(..., content=gen))andsend(httpx2.Request(..., content=gen))were retried with a consumed generator. With a real transport that sends an empty or truncated body;MockTransporthides it becauserequest.read()caches the body.send()andsend_with_response()now call_mark_streaming_body, which sets the marker whenrequest.streamis not anhttpx2.ByteStreamand the content type is notmultipart/form-data. The verb methods keep their existing argument-based check and go through new private_send/_send_with_response, so they are not checked twice.Clientrejects async middleware at construction. PassingAsyncRetry()or an@async_before_requestfunction toClientused to fail later in confusing ways.Client.__init__now raisesTypeErrorwhen a middleware, or its class's__call__, is a coroutine function.AsyncClientgets no reverse check.AsyncClientreads an unread response before decoding it. A middleware that returns an unread response with an async-only stream, such ashttpx2.Response(200, content=<async generator>), reached the decoder unread and failed withDecodeError.send()andsend_with_response()(and the verbs, through_send/_send_with_response) nowawait response.aread()first whenresponse.is_closedis false. Every buffered response is already closed, so the normal path only pays for the property read. The syncClientis unchanged here; the decoder-side sync read lands in a separate PR.Docs:
docs/index.md(timeout option),docs/errors.md(new table mapping httpx2 exceptions to httpware ones, plus invalid URL and closed client),docs/resilience.md(streaming-body refusal forsend()),docs/middleware.md(theTypeError).Release notes (0.22.0)
Breaking:
Client(timeout=None)andAsyncClient(timeout=None)now disable the timeout instead of keeping httpx2's 5 second default. To keep the default, leavetimeoutout. Passingtimeout=Nonetogether withhttpx2_client=now raises the sameTypeErroras any other option combined with a caller-owned client.Other behaviour changes:
httpx2.RemoteProtocolErrorraisesNetworkError(still aTransportError), soRetry/AsyncRetryretry it and circuit breakers count it.request()raisesTransportErrorinstead ofhttpx2.InvalidURL.stream()on a closed client raisesTransportErrorinstead ofRuntimeError.send()/send_with_response()with a generator, iterator or file body are no longer retried.Client(middleware=[...])raisesTypeErrorfor async middleware.AsyncClientdecodes an unread async-stream response returned by middleware.Judgement calls for review
timeout=Nonewithhttpx2_client=raises the conflictTypeError. The caller-owned client cannot honour it, so I treated it as a set option. The alternative is to ignore it silently, which would bring back the bug from item 1.send()rather than in_build_request. Applying the stream-based rule in_build_requestwould also markpost(content=[b"a", b"b"]): httpx2 wraps a list in the same iterator stream as a generator, andtests/test_retry_sync.py::test_client_post_with_list_content_does_not_mark_extensionsrequires that a list stays retryable. The verb methods keep the argument-based check, which can tell a list from a generator. Known limitation: a list body sent throughbuild_request()+send()is marked and not retried.docs/resilience.mdsays so._is_streaming_body_async/_is_streaming_body_syncin_internal/status.pyare still used by_prepare_request, so nothing became dead code.build_request()raisinghttpx2.InvalidURL.RuntimeErrorraised inside astream()block toTransportErroronly when the httpx2 client is closed at that point, which matches what_terminalalready did.AsyncClientnow fails with httpx2'sRuntimeErrorfromaread()instead ofDecodeError. That middleware was already broken; flagging it because the exception type changes.Tests
New
tests/test_client_error_paths.py(48 tests, both clients) and updatedtests/test_client_options.py. Each test was run red before its fix:test_explicit_timeout_none_is_forwarded_as_no_timeoutfailed withExpected: Client(timeout=None) / Actual: Client();test_unset_options_are_not_forwardedno longer passestimeout=None.test_sync_remote_protocol_error_maps_to_network_errorfailed withhttpware.errors.TransportError: peer closed connection ....test_sync_invalid_url_on_verb_maps_to_transport_error[get]failed withhttpx2.InvalidURL: Invalid non-printable ASCII character in URL.test_sync_stream_on_closed_client_raises_transport_errorfailed withRuntimeError: Cannot send a request, as the client has been closed.test_sync_send_of_raw_streaming_request_refuses_retryfailed withassert 3 == 1(three attempts).test_sync_client_rejects_async_middleware[AsyncRetry]failed withDID NOT RAISE TypeError.test_async_send_decodes_unread_async_stream_from_middlewarefailed withDecodeError: ... without having called read().just lint,just lint-ciandjust testpass: 1061 tests. Every commit passes the suite on its own.Performance
End-to-end timing of a sync
get()againstMockTransportis about 47 to 60 µs on this machine, and the noise between interleaved base/head runs was over 10 µs, so it cannot show a 0.1 µs change. No regression was visible. Per added operation:get()etc.):_sendreplaces thesendcall at the same depth; theInvalidURLtry/except costs nothing unless it raises; the mapper takes one extra argument (enter/exit cost the same within noise); theRemoteProtocolErrorcheck runs only on the error path.send()with a raw request:_mark_streaming_bodyis about 80 ns (108 ns including the timing lambda).AsyncClientdecode path: theresponse.is_closedguard is about 30 ns. An unguardedawait response.aread()on an already-read response cost about 144 ns, which is why the guard is there.Client.__init__: the coroutine check runs once per middleware at construction only.