perf: cheaper hot paths in the circuit breakers, AsyncBulkhead and the event-loop guard - #183

Four independent changes, one commit each. None of them is meant to change behaviour. The one thing a user could notice is that a KeyedCircuitBreaker key function no longer runs under the breaker lock (see "Notes for 0.22.0").

What changed

1. Sync breakers admit into a closed circuit without the lock; keyed breakers look up existing circuits without it. _CircuitBreakerState has a new attribute, closed_ticket: the epoch while the circuit is CLOSED, None otherwise. _call_sync reads it once without the lock and takes the lock only when it is None. record() and every transition still run under the lock. _KeyedStates.get reads the circuit map without a lock. On a miss it takes a separate _create_lock, checks again and inserts. The async breakers use the same shape (one read, admit() only on None), which saves a method call and a try block per call.
2. The default circuit key is a (scheme, raw_host, port) tuple instead of an "https://host:port" string built for every request. The string is still what events carry in circuit_key. It is built once, when the circuit is created, with the same expression as before.
3. AsyncBulkhead skips asyncio.timeout when a slot is free.
4. The async middlewares check the bound loop inline (if self._loop is not asyncio.get_running_loop()) and call check_event_loop only on a mismatch. A call used to build two lambdas. The error message constants are unchanged.

Concurrency reasoning (please check this part)

Sync breaker, unlocked read of closed_ticket

The only unlocked read on the sync admission path is state.closed_ticket, one attribute load. Its only writer is _move_to, which always runs under the breaker lock and sets _state, _epoch and then closed_ticket. Under the lock, closed_ticket == (_epoch if _state is CLOSED else None) always holds. The sync Hypothesis test asserts this after every step and the stress test after every record.

On free-threaded 3.14t a load or store of an object reference in an instance attribute is atomic: a reader sees the old value or the new one, never a mix. The int is immutable, so it is complete when published. Under the GIL this is trivially true.

What the reader can see:

• An int E. E was the epoch of a CLOSED era, because closed_ticket only ever holds an int while the circuit is CLOSED. The call goes ahead with ticket E, which is exactly what admit() under the lock would have returned at the moment of the load. That load is the linearization point. Later, record(E, ...) runs under the lock:

  • If _epoch is still E, the circuit is still in that CLOSED era. Every transition adds 2 to the epoch and epochs are never reused, so the outcome applies as it always did.
  • If the circuit moved on, _epoch > E, so E is neither _epoch nor _epoch + 1 and record drops the outcome. A call admitted under the lock just before the transition and still in flight across it is treated the same way.

  A stale read is therefore indistinguishable from a locked admission that happened slightly earlier. The straggler fix (fix: circuit breaker ignores straggler outcomes, unwraps budget exhaustion, rate-limits circuit.rejected #167) already handles that case.

• None. The call falls back to admit() under the lock. admit() reads closed_ticket again, because the circuit may have closed in the meantime (test_locked_admission_into_a_circuit_closed_since_the_unlocked_read_gets_the_closed_ticket covers this), and otherwise runs the OPEN and HALF_OPEN logic exactly as before.

Why one packed attribute and not _state and _epoch read separately: that pair can tear. A reader could see _state is CLOSED at epoch E, then the circuit could go to OPEN and on to HALF_OPEN (E + 4) before the reader loads _epoch. The reader would then hold ticket E + 4, which equals the HALF_OPEN epoch, so record would take it as a CLOSED-era outcome while the circuit is half-open. It would count a failure toward failure_threshold, or reset the failure count, and go around the probe. One load closes that window.

The invariants the task listed:

• Never more than one probe in flight. _probe_in_flight is read and written only inside admit() and record(), both under the lock. The fast path never hands out a probe ticket: probe tickets are odd (epoch + 1) and come only from admit().
• success_threshold respected. _consecutive_successes changes only in record() for the current probe ticket, under the lock.
• No outcome from a stale epoch changes state. record() compares the ticket against _epoch under the lock, as before.
• Counters consistent. Every counter write is still under the lock. The fast path writes nothing.

Keyed lookup without the lock

_KeyedStates.get calls self._states.get(key) without a lock. On 3.14t, dict reads are safe against a concurrent insert: writes lock the dict, and reads that race a write retry. Under the GIL a dict.get is atomic. Inserts happen only in _create, under _create_lock, with a second lookup, so each key gets exactly one state. Entries are never replaced or removed, so a successful unlocked lookup returns the single state that key will ever have. A miss takes the locked path.

_create_lock is separate from the breaker lock so that creating a circuit does not wait behind record() calls on other keys. Keyed transitions still share one breaker-wide lock, as before.

Event-loop guard

self._loop goes from None to a loop once, under _loop_lock, and never changes after that. The inline check reads it without the lock. If it is the running loop we are done. Otherwise check_event_loop reads it again under the lock and either binds the running loop (None) or raises. Because every caller now checks first, the helper no longer does its own unlocked read, and its unit test changed to match.

AsyncBulkhead fast path

acquire() in CPython 3.11.16 and 3.14.7 (read from inspect.getsource) starts with if not self.locked(): self._value -= 1; return True. It does not suspend, so the timeout has nothing to bound. In both versions locked() is self._value == 0 or any(not w.cancelled() for w in self._waiters). It is true whenever a live waiter is queued, so the fast path is taken only when acquire() itself would take it, and a new caller cannot get ahead of a queued waiter. No await runs between our locked() call and the one inside acquire(), so the two see the same state. test_a_freed_slot_goes_to_the_queued_waiter_before_a_new_call pins the FIFO order. acquire_timeout=0 with a free slot already succeeded before this change, because the timeout was cancelled before it could fire, so that case is unchanged.

Numbers

Best-of-7 or best-of-15 timeit with a trivial next, several alternating runs, Apple Silicon (8 cores), CPython 3.14.7. The machine was shared with other jobs (load average 3 to 10), so these are ranges across runs.

Whole branch against origin/main, ns per call, three alternating runs:

Each change against the commit before it:

1. Lock-free admission: CircuitBreaker 742 to 851 down to 557 to 639 ns (measured while the machine was busier), about 25% less. KeyedCircuitBreaker 1818 to 2391 down to 1564 to 1964 ns. The async admission shape, measured in one process against the old shape, saves 50 to 75 ns (392/490 down to 340/416 ns). Eight threads on 3.14t: CircuitBreaker 0.78 to 0.93 up to 1.27 to 1.34 Mcalls/s, KeyedCircuitBreaker 0.30 to 0.33 up to 0.49 to 0.55 Mcalls/s.
2. Tuple key: key function 408 to 540 down to 165 to 215 ns. KeyedCircuitBreaker 717 to 950 down to 513 to 633 ns, AsyncKeyedCircuitBreaker 1078 to 1445 down to 852 to 1174 ns.
3. Bulkhead fast path: 1978 to 2088 down to 809 to 855 ns.
4. Inline loop check: the check itself 206 to 218 down to 47 ns. AsyncCircuitBreaker 626 to 786 down to 390 to 409 ns, AsyncKeyedCircuitBreaker 940 to 1021 down to 654 to 701 ns, AsyncBulkhead about 300 ns less in two of three runs (one noisy run showed no gain).

Thread throughput for the whole branch, 8 threads, Mcalls/s:

The plain breaker on 3.14t was too noisy to call. Every call still takes the lock once, in record(), and that lock is now the bottleneck. I did not reproduce the earlier estimates of about 60% less per sync call and 2.9x keyed throughput on 3.14t. What I measured is about 25 to 35% less per sync call and 1.5x to 2x keyed throughput.

Tests

• tests/test_circuit_breaker_sync_props.py (new): the sync twin of the async interleaving property. Each call runs on its own thread and blocks in next until the test finishes it, so Hypothesis picks the overlap order. After every step it checks that a stale outcome leaves the state alone, that probes never overlap, that HALF_OPEN to CLOSED happens after exactly success_threshold probe successes, and that closed_ticket, _probe_in_flight and the counters agree.
• test_lock_free_admission_keeps_the_circuit_invariants (stress): 16 threads, 2000 calls each, reset_timeout=0 so the circuit cycles constantly. It wraps the state's admit and record, which run under the breaker lock, and checks the same invariants there: at most one probe ticket outstanding, no stale ticket changing any state or emitting events, consistent counters, and each half-open era being success_threshold successes or a run of successes ending in one failure.
• test_racing_first_requests_share_one_circuit_per_origin (stress): threads race to create circuits for four origins. Every recorded failure must land on the one shared circuit per origin.
• Keyed circuit_key for a custom key is its str() (sync and async). There is also the FIFO test for the bulkhead.

Mutation checks on 3.14t: dropping the stale-epoch check in record fails both the property test and the stress test. Creating keyed circuits without the lock and the second lookup fails the keyed stress test in about one run in three. A torn read of _state and _epoch was not caught by the stress test, because the window between two loads is a few nanoseconds. That case is covered by the reasoning above, not by a test.

just lint, just lint-ci and just test-ci pass: 1275 tests, 100% coverage. The full suite also passes on 3.11 and on 3.14t, and the stress tests passed in repeated runs on 3.14t.

tests/test_bulkhead_props.py::test_fail_fast_rejects_when_at_capacity failed twice in four test-ci runs while the machine was heavily loaded, and never failed in later runs. It depends on timing (holders sleep 50 ms while the extras must be rejected) and goes through the locked path, which this branch does not change. I left it alone.

Notes for 0.22.0

• No intended behaviour change. circuit_key in events keeps its exact format.
• The key function of KeyedCircuitBreaker now runs outside the breaker lock, so several threads can call it at once. Pure functions of the request, the documented use, are unaffected.