Flaky test: test_segment_uses_downsampled_rate races the monitor's background thread #8031

Flaky test

tests/test_monitor.py::test_segment_uses_downsampled_rate

CI failure

• Gevent (3.10, ubuntu-22.04) on master at 7e8594c (Oct 13). That commit didn't touch the monitor.

tests/test_monitor.py:116: in test_segment_uses_downsampled_rate
    assert segment._sample_rate == 0.5
E   assert 0.25 == 0.5
E    +  where 0.25 = <NoOpStreamedSpan(sampled=False)>._sample_rate

In that job the test took about 0.34 s: the previous test passed at 11:59:39.629 and this one failed at 11:59:39.968.

Expected vs actual

• Expected: after the test's single manual monitor.run(), downsample_factor is 1, so the segment's sample rate is 1.0 / 2**1 = 0.5.
• Actual: downsample_factor is 2 by the time the segment is sampled, giving 0.25.

Cause (from reading the code)

The test sets monitor.interval = 0.1 and then drives the health check by hand with monitor.run(). But the first monitor.is_healthy() call (_ensure_running) also starts the monitor's background thread, which calls run() every interval seconds. If more than 0.1 s passes between that first call and the sampling decision, the background thread runs another unhealthy check, and downsample_factor goes from 1 to 2. That is easy to hit on a slow or busy runner, and under gevent.

The same race can hit test_transaction_uses_downsampled_rate (same setup) and test_monitor_unhealthy, which asserts exact factors over 15 manual runs with a 0.1 s background interval.

Possibly related

In the same run, tests/tracing/test_decorator.py::test_trace_decorator_no_trx failed with leftover monitor debug logs (downsampling with a factor of 10) (job). That points to monitor threads from earlier tests that keep running after the test ends. This is still to be checked.

via johdax.

--

View Junior Session [Sentry]