Replace the inactivity cop's per-second sweep with a timer wheel - #13821

Drives the real cop against mock NetEvents on the test_net Catch2 thread,
so timeout dispatch cost can be measured at high connection counts without
a load generator.

Freeze the mock's padded footprint instead of tying it to a live
sizeof(UnixNetVConnection), which the timer wheel refactor will change;
make the touch counters plain (single-writer) instead of atomic so they
stop inflating the current baseline with instrumentation the wheel
removes; make the lock-contention handshake unwind-safe via RAII plus an
unconditional release in the destructor; time the measured region with
steady_clock instead of ink_get_hrtime()'s CLOCK_REALTIME; tighten the
idle/mass_expiry/lock_contention assertions to exact equality instead of
loose bounds; rename the misleading p99 column to min/max; and print a
one-time provenance/size stamp from a Catch2 listener plus a derived
ns/conn column.

The five scenarios' construct/arm/warmup/sample/summarize/print sequence
had already started to drift (keepalive re-armed its deadlines twice per
warmup iteration; idle/mass_expiry had no per-tick step with nothing
marking that as deliberate). Route idle/keepalive/mass_expiry/churn
through one Scenario-driven helper so every scenario is warmed up and
sampled identically by construction. lock_contention still needs to
bracket each individual timed call rather than just mutate state before
it, so it shares only the setup/warmup half of the helper and keeps its
own sampling loop.

The cop took a MUTEX_TRY_LOCK on every scanned NetEvent before it knew
whether anything needed doing, costing an atomic RMW per connection per
second on a cache line shared with the owning HttpSM. Only the callback,
free, and default-timeout paths need the lock.

Also fixes benchmark_InactivityCop.cc's lock_contention scenario: it used
to hold 10% of mutexes on idle connections, but idle connections no longer
attempt the lock at all, so the held fraction was never exercised and the
exact-equality assertion started failing. Switch it to expired deadlines
(like mass_expiry) so the cop still locks every connection and the held
fraction still measures the try-lock-failure path.

Doing this lazily in the cop forced it to visit every open connection at
least once per second regardless of deadlines, and left the connection
with no usable deadline at registration time.

Also switches the benchmark fixture to register mocks through the real
startCop() instead of enqueuing onto open_list directly, so this path
is exercised there too; this is setup-only and does not affect the
timed region.

A ring of intrusive lists bucketed by second, so expiry costs what the
due buckets hold rather than what the whole set holds. Lives in tscore
with no event system dependency so it can be tested directly.

… lag

A fire callback that reschedules could re-enter the bucket being
drained and fire repeatedly; cancel() on an element owned by another
wheel spliced both rings; and expire() walked one iteration per
elapsed tick, so an uninitialised cursor spun about 1.8 billion times.

Added a [!benchmark]-tagged Catch2 case that drives 100,000
continuously re-arming elements through a simulated hour (3600
one-second ticks) for wheel sizes 256/512/1024/4096 and timeouts
30s, 120s, and a 4h tunnel active timeout, counting visits via
deadline_of() (called once per element popped, on both a genuine
fire and a clamp-driven lazy rearm).

Results (build-bench, N=100000, 3600 ticks, seed=0xBEEF):

buckets   timeout_s   total_visits   visits/elem/period   mem/wheel  mem*32threads
256              30       12000000               1.0000      2 KiB          64 KiB
256             120        3000000               1.0000      2 KiB          64 KiB
256           14400        1403140              56.1256      2 KiB          64 KiB
512              30       12000000               1.0000      4 KiB         128 KiB
512             120        3000000               1.0000      4 KiB         128 KiB
512           14400         701306              28.0522      4 KiB         128 KiB
1024             30       12000000               1.0000      8 KiB         256 KiB
1024            120        3000000               1.0000      8 KiB         256 KiB
1024          14400         314857              12.5943      8 KiB         256 KiB
4096             30       12000000               1.0000     32 KiB           1 MiB
4096            120        3000000               1.0000     32 KiB           1 MiB
4096          14400          25183               1.0073     32 KiB           1 MiB

At 30s/120s every candidate size already exceeds the range, so
visits/elem/period is flat at 1.0 across all sizes, as expected. The
4h row confirms the timeout/range shape and is where size matters;
the 1-hour observation window is shorter than the 4h timeout, so
this exact row underestimates true steady state for the larger
sizes (confirmed separately with longer windows: 1024 converges to
~14.9 revisits/element/period, 4096 to ~4.0), but the direction and
relative magnitude are unaffected.

1024 imposes real, avoidable churn on multi-hour timeouts (tunnels)
for no benefit to the dominant 30s/120s case, at a memory cost
(8 KiB vs 32 KiB per wheel, one wheel per ET_NET thread) that is
negligible either way. Bumping the default to 4096 is therefore
strictly better and is applied here.

Routing every deadline write through rearm_timer() makes the wheel's
cached deadline never-later-than the true one by construction.
netActivity is the deliberate exception: it only extends, so lazy rearm
covers it.

Nothing fires from the wheel yet; landing the bookkeeping on its own
means a schedule/cancel bug surfaces before the switchover. Closed
NetEvents are re-armed to the next tick so an fd cannot linger until a
deadline minutes away.

The cop keeps its cadence, debug tags, metrics, and queue trimming; only
the scan is replaced, so cost per tick is now proportional to the due
buckets rather than to every open connection on the thread.

Arming the default inactivity timeout moved to set_enabled/netActivity:
the wheel only visits scheduled elements, so a connection whose deadline
was applied lazily by the old sweep would otherwise never time out.

The first-sample check failed with 0 >= 1000: the harness advances real
time via sleeps, so whether a call crosses a tick boundary depends on OS
scheduling. Only the total across samples is stable enough to assert.

Two edits to the same comment landed back to back and concatenated.

Also drops the TS-4612 epoll hook that existed only to shrink the sweep,
taking a branch and a store off the per-event path, and reclaims two
pointers per NetEvent with cop_link.

open_list stays: it is no longer walked for timeouts, but it is the only
enumeration of all NetEvents on a thread and backs startCop's
double-registration assert. The wheel cannot substitute, since a NetEvent
with no deadline is not scheduled in it.

The cop's own cost had no telemetry, which is how an O(N)-per-second
sweep regressed unnoticed at high connection counts. Also re-arms the
timer when netActivity arms a default deadline on a vc that was never
scheduled: the old sweep visited every connection so it caught that
case, but the wheel only visits what is scheduled.

The autest suite proves the counters are harmless but never reads their
values. Assert them against the mock's own counts instead: zero when
nothing is due, and equal to the recorded callbacks when everything is.

Covers the timer wheel, the never-later-than rule that makes lazy rearm
safe, why netActivity skips the re-arm and the one transition where it
cannot, and the two traps the wheel introduces: a closed connection must
be reaped on the next tick, and a connection with no deadline is not in
the wheel to be noticed later.

The file sets .. highlight:: cpp like its siblings, so a bare :: block
lexes as C++ and the quoted Catch2 tag fails the lexer, which sphinx -W
turns into a docs build failure.

The wheel's resolution is one second, so a harness advancing real time
either sleeps between every call or has most calls do nothing. Splitting
run(now, e) out of the event handler lets the benchmark advance a
synthetic clock exactly one tick per call, which is what production does.

churn measures 1% per tick again and asserts it exactly, the 1.1s settle
sleeps are gone, and the suite runs about 20x faster with no real-time
dependence.

It said raising this reduces pressure at high connection counts, which
was true when a check swept every open connection. A check now costs
what the due deadlines cost, so raising it only coarsens timeouts.

Timeout dispatch no longer walks every open connection, so raising this
setting to relieve load now only delays timeouts. Operators who tuned it
on the old advice are paying accuracy for nothing.

An unconstrained typename F that calls deadline_of() on its argument puts
the requirement in a comment and the diagnostic deep inside expire(). The
concept states it, and a delegate missing deadline_of() now fails at the
call site naming the missing member.

Shape follows the sketch NetTimeout.h left as a TODO for C++20.

A pass that exhausted the budget waited out the rest of the tick, so a
burst larger than the budget was retired at TIMEOUT_BUDGET per
inactivity_check_frequency. Worse, expire() does not advance its cursor
until a bucket drains, so an over-full bucket also delayed examining
every tick behind it: 100k connections coming due together would have
taken about 25 seconds to retire at the default settings.

The budget is meant to bound one pass, not to cap what a tick retires.

The wheel hands over an element once its deadline is <= now, but both
expiry tests are strict <, so a deadline exactly equal to now was popped,
not fired, and never rescheduled - that connection could never time out.

Re-arming rather than loosening the comparison matches the pre-wheel
sweep, which left such an element for the next tick.

inactivity_check_frequency may be raised above 1, which leaves the
wheel's one-second buckets alone and just means a pass advances several
ticks. Pins that every deadline in the skipped-over ticks fires on that
pass, not only the newest bucket's.

schedule_imm() did not do what the previous commit claimed. An immediate
event is dispatched inline by EThread::process_queue()'s own dequeue
loop, so rescheduling immediately re-entered the pass before the poll
events ran and bounded nothing.

A timed event lands in EventQueue, where dequeue_ready() will not return
it during the current loop iteration, so the tail handler polls first.

TS-4131 worked around the cop failing to close connections whose active
timeout had expired by forcing a full scan of the active queue every
tick. The wheel makes that scan unnecessary, but "the cop silently
stopped visiting a connection" is the failure mode this refactor hit
three separate times, so the check survives as diagnostics rather than as
a hot path.

Two states are anomalies: a live deadline with no wheel entry, which can
never fire, and a cached deadline later than the true one, which fires
late. Lagging behind TIMEOUT_BUDGET is neither, and is reported without
being counted, so a burst does not make the audit cry wolf.

Gated on the inactivity_cop_audit debug tag and every 60th pass, so it
costs a branch on a cached bool when off. audit() returns the count so it
is assertable; the tag carries per-offender detail.

This lands before the removal of the active-queue scan so the check is in
place when that risk is introduced, and can be shown to report nothing
against the current code first.

TS-4131 had the cop force a full active-queue scan each pass because the
cop of that era only compared next_inactivity_timeout_at and so never
fired active timeouts. The wheel schedules on _earliest_deadline(), the
non-zero minimum of both deadlines, so that fallback has nothing left to
find and was an unconditional O(active_queue) walk per thread per second
-- the sweep the wheel exists to avoid, reintroduced behind a config
value.

The scan was also actively harmful: its _close_ne re-armed connections it
had just closed, so a due connection was fired twice in one pass and left
scheduled afterwards. The first new test covers that and was red before
the removal.

The benchmark cannot see this change, since its scenarios leave
max_requests_per_thread_in at 0 and the scan early-returned there;
measuring the cost needs the limits exercised.

TIMEOUT_BUDGET caps a pass at 4096 fires, a number picked without data.
Since a pass runs on an event thread, a full-budget pass delays that
thread's poll loop for its whole duration, so choosing the budget needs
the per-close cost -- which nothing reported.

Times expire() only, so pass_time_us over fired is a per-close figure and
not diluted by the keep-alive sweep. Two clock reads per pass per thread
is cheap enough to leave always on, which also makes the cop's cost
visible in production rather than only under a debug tag.

pass_max_us is a best-effort high-water mark: a concurrent update can lose
a sample, understating the worst pass but never reporting a value no pass
took. A CAS loop is not worth it for a diagnostic.

The budget is a latency policy, not a correctness setting: a pass runs on
an event thread, so firing N timeouts stalls that thread's poll loop for
as long as those N closes take. Reaping idle connections is low priority
work, so trading a longer total reap for smaller slices is usually the
right call -- but it was a compile-time constant, so no deployment could
make that trade.

Default stays 4096 so behaviour is unchanged. Measured in the lab,
dropping it to 256 cut the worst pass 41% for 33% more total cop CPU,
with everything still reaped.

expire() treats a budget of 0 as "do nothing and leave the wheel alone",
which would silently stop every timeout in the process with no error and
no metric moving. The records schema rejects 0 and budget() falls back to
the default for a value that arrives another way.

Ubuntu CI still builds with clang 12, whose standard library has no
<concepts>. The delegate concept only needed std::convertible_to, so
spell the check with std::is_convertible_v and keep the constraint.

The records regex accepted any size, and budget() narrowed the value to
int, so anything above INT_MAX became a negative budget: expire() did
no work while the cop rescheduled itself every millisecond, stopping
all timeouts. Restrict the record to 1-1048576 and clamp in budget().

The budget only counted fired timeouts, but a connection whose deadline
moved out is re-armed without firing. A bucket of connections opened in
the same second and still busy would then be walked in one unbounded
pass. Stop a pass after 16x its timeout budget of visits too, and treat
either limit as behind.

The metric took the wheel's return value, which counts every due element
handed to Fire, including lock failures that are re-armed and already
closed NetEvents that are only freed. That overstated timeouts and the
pass_time_us / fired per-close figure. Count the callbacks in Fire.

The cross-thread close handoff wrote to the NetEvent after publishing
closed, but the owner may free a closed NetEvent through its ready list
at any time, so that write could land in freed memory. Claim it on the
handoff list before marking it closed; free_netevent() leaves a claimed
NetEvent to process_handoffs(), which frees it once the push is done.

A per-remap default_inactivity_timeout arrives after the global default
has armed the connection, but set_default_inactivity_timeout() only
stored the duration. netActivity() never re-arms, so a shorter override
waited out the global deadline, a day by default. Move a deadline the
default produced to the new duration, keeping the idle time accrued.