Repository navigation
fix!: validate retry and budget config, case-insensitive retry_methods, Retry-After give-up event - #177
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Four fixes to
Retry,AsyncRetry, the shared_RetryPolicyandRetryBudget. Each one was written test first, and sync and async are covered alike.1. Config is validated at construction (breaking)
Retry,AsyncRetryandRetryBudgetnow raiseValueErrorwhen constructed with a value that would crash or quietly misbehave later. Before this,base_delay=-1constructed fine and the first sync retry crashed intime.sleepwith a negative delay (async slept 0). A negativemax_delaymade everyRetry-Afterlook too long and produced negative backoff delays. Bad budget values either turned retries off without saying so or crashed insidetry_withdraw.The bounds, and the reason for each:
max_attempts>= 1base_delay>= 0, includinginfmin()and the backoff silently usesmax_delaymax_delay>= 0Retry-After; NaN breaks the comparisons;inflets an unboundedRetry-After(see fix 4) reachtime.sleep(inf), which raisesOverflowError, or hangasyncio.sleepforeverRetryBudget.ttl> 00or negative purges every deposit and makes the floor<= 0, so retries are refused; NaN never purges (unbounded memory) and crashes inint();infnever purges and crashes inint()RetryBudget.min_retries_per_sec>= 0infcrash inint()RetryBudget.percent_can_retry>= 0, values above1.0includedinfcrash inmath.ceil()0stays valid wherever it already worked:base_delay=0andmax_delay=0mean no waiting, andmin_retries_per_sec=0orpercent_can_retry=0switch off that half of the budget formula.The validation follows
_validate_bulkhead_config: one module-level function per module and module-level message constants._validate_retry_configlives inretry.pyand_validate_budget_configinbudget.py.2.
retry_methodsignores case_RetryPolicyupper-casesretry_methodsonce at construction, soretry_methods={"post"}now retries POST.httpx2.Requestalready upper-casesrequest.method, so the per-request.upper()call indecideis gone.3.
retry.giving_upfires whenRetry-Afterexceedsmax_delayBefore this, the give-up in that case only added the PEP 678 note and emitted nothing. It now emits
retry.giving_upat WARNING with the same attributes as the attempts-exhausted emission (attempts,method,url,last_status,last_exception_type) plusreason="retry_after_exceeds_max_delay",retry_afterandmax_delay. The existing attempts-exhausted emission gainsreason="max_attempts"so a consumer can tell the two apart with one field. That is an added attribute; nothing was renamed or removed.4. An oversized integer
Retry-Afteris an infinite waitA
Retry-Afterof 309 or more digits overflowedfloat(), the parser returnedNone, and the call was retried after the normal jittered backoff. A value past Python's 4300-digitint()limit took the same path through aValueError. Both now parse asmath.inf, which exceedsmax_delayand goes down the give-up path from fix 3. A hugely negative integer now counts as 0, matching the documented rule for negative integers. Whitespace, past HTTP dates, small negative integers and non-integer values parse exactly as before.Release notes (0.22.0)
Breaking:
Retry,AsyncRetryandRetryBudgetreject invalid configuration at construction withValueError. Migration: if construction now fails, the old value was already broken at request time. Usebase_delay >= 0and a finitemax_delay >= 0. Replacemax_delay=float("inf")with a large finite number. ForRetryBudget, use a finitettl > 0and finite, non-negativemin_retries_per_secandpercent_can_retry. To turn the budget off in practice, pass a largemin_retries_per_secinstead ofinf.Fixed:
retry_methodsmatches regardless of case.Fixed: giving up because
Retry-Afterexceededmax_delaynow emitsretry.giving_upwithreason="retry_after_exceeds_max_delay". Everyretry.giving_upevent now carries areasonattribute.Fixed: a
Retry-Afterinteger too large for a float makes the retry give up instead of retrying immediately.Docs
docs/resilience.md,AsyncRetryandRetryBudgetsections: the parameter bounds, case-insensitiveretry_methods, the event on theRetry-Aftergive-up, and oversized integers.docs/observability.md: thereason,retry_afterandmax_delayattributes ofretry.giving_up.Performance
Validation and method upper-casing run once, at construction. The per-request path is unchanged apart from dropping one
str.upper()call indecide. Measured withtimeit(best of 7 x 50,000, CPython, three alternating runs againstorigin/main):Retry.__call__, success on first try_RetryPolicy.decide, retryable 503The differences are within run-to-run noise.
Tests
just lint,just lint-ciandjust test-cipass: 1229 tests, 100% coverage, no newpragma: no cover. The retry and budget test files also pass on Python 3.11, andjust docs-buildpasses.Two existing tests,
test_retry_after_huge_digit_string_does_not_crash(sync and async), asserted the old retry-with-jitter behaviour for a 400-digitRetry-After. They are nowtest_retry_after_huge_digit_string_gives_upand assert the give-up.