Skip to content

test_fail_fast_rejects_when_at_capacity is timing-dependent and flaky under load #184

Closed

Description

@lesnik512

tests/test_bulkhead_props.py::test_fail_fast_rejects_when_at_capacity failed 2 of 4 local just test-ci runs while the machine's load average was about 10, then passed in 8 later runs. The bulkhead code it exercises did not change between those runs.

The test synchronizes by time:

  • It starts max_concurrent holder requests, then await asyncio.sleep(0.005) and assumes every holder has acquired its slot by then.
  • The holders keep their slots for a fixed delay=0.05 in the handler, and the extra requests must run inside that window.

Under load either assumption can fail. If a holder has not reached acquire() after 5 ms, an extra request takes the free slot and pytest.raises(BulkheadFullError) fails. If the extras start after 50 ms, the holders have already released.

Fix: synchronize on events instead of sleeps. The handler counts in-flight requests, sets an asyncio.Event once max_concurrent are inside, and then waits on a second event that the test sets after the extra requests have been rejected. Other tests in the file that rely on _InFlightHandler(delay=...) timing may need the same treatment.

How to confirm: run the test under artificial load, e.g. stress-ng --cpu 0 or several parallel just test-ci runs, with --count from pytest-repeat or a shell loop.

Activity

  1. added
    bugSomething isn't working
    ready-for-agentFully specified, ready for an AFK agent
    on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions