You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test_fail_fast_rejects_when_at_capacity is timing-dependent and flaky under load #184
tests/test_bulkhead_props.py::test_fail_fast_rejects_when_at_capacity failed 2 of 4 local just test-ci runs while the machine's load average was about 10, then passed in 8 later runs. The bulkhead code it exercises did not change between those runs.
The test synchronizes by time:
It starts max_concurrent holder requests, then await asyncio.sleep(0.005) and assumes every holder has acquired its slot by then.
The holders keep their slots for a fixed delay=0.05 in the handler, and the extra requests must run inside that window.
Under load either assumption can fail. If a holder has not reached acquire() after 5 ms, an extra request takes the free slot and pytest.raises(BulkheadFullError) fails. If the extras start after 50 ms, the holders have already released.
Fix: synchronize on events instead of sleeps. The handler counts in-flight requests, sets an asyncio.Event once max_concurrent are inside, and then waits on a second event that the test sets after the extra requests have been rejected. Other tests in the file that rely on _InFlightHandler(delay=...) timing may need the same treatment.
How to confirm: run the test under artificial load, e.g. stress-ng --cpu 0 or several parallel just test-ci runs, with --count from pytest-repeat or a shell loop.
Activity