From 82c95908fe81badaa7ee7bb081a798f0fb2246be Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 14:23:02 +0100 Subject: [PATCH 1/4] docs(run-engine): reconcile fairness spike with the caps plan of record The plan of record ships concurrency caps (per-task total, per-key limit) and leaves the fair-selection scoring untouched, the opposite lever from the earlier spike's SFQ recommendation. Adds the reconciliation design and the queueing-theory research grounding it (WFQ/SFQ wait bounds, concurrency-cap fairness limits, CoDel scope, real multi-tenant systems, Little's law). --- ...7-23-fairness-caps-vs-scheduling-design.md | 121 ++++++++++++++++ .../run-queue/fairness-spike-ck/RESEARCH.md | 133 ++++++++++++++++++ 2 files changed, 254 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-23-fairness-caps-vs-scheduling-design.md create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/RESEARCH.md diff --git a/docs/superpowers/specs/2026-07-23-fairness-caps-vs-scheduling-design.md b/docs/superpowers/specs/2026-07-23-fairness-caps-vs-scheduling-design.md new file mode 100644 index 0000000000..5472c9620a --- /dev/null +++ b/docs/superpowers/specs/2026-07-23-fairness-caps-vs-scheduling-design.md @@ -0,0 +1,121 @@ +# Caps vs scheduling: reconciling the fairness spike with the plan of record + +Throwaway spike design. Ships nothing; delete before any merge to main. + +## Why + +Two prior spikes (base-queue grain, then the real CK-dequeue grain) concluded: +score `ckIndex` by a fair discipline (SFQ/stride virtual time, or DRR) instead of +by head timestamp, to fix per-concurrency-key starvation (#2617). That is a +change to the fair-selection SCORING / serve order. + +The plan of record for "Queue multi-tenant fairness" does something different. It +ships bounded concurrency CAP primitives and deliberately leaves the +fair-selection scoring untouched: + +- Phase 1: a per-base-queue TOTAL concurrency cap (`:groupConcurrency` SET SCARD + gated against a `:totalConcurrency` limit). +- Phase 2: a per-KEY concurrency limit (`:ckLimits` HASH, HGET per CK variant). + +The only scheduler-adjacent change is negative: the fair-selection strategy drops +base queues already at their total cap so it stops picking them. Scoring within +the contended region is still head-timestamp order. + +So the spiked mechanism (scheduling) and the shipped mechanism (caps) are +different knobs. This spike proves or disproves whether the caps deliver the +fairness the scheduling disciplines deliver, and reconciles the two. + +## The thesis to test + +From the research (see `RESEARCH.md`): occupancy caps and fair scheduling are +orthogonal. A cap bounds a tenant's occupancy and (via Little's Law) its +throughput share; it does not bound wait unless the dequeue is +oldest-ELIGIBLE-first AND freed slots exist. A scheduler bounds a starved +tenant's first-serve wait but is work-conserving (bounds no occupancy). Neither +substitutes for the other; production systems layer them (Kubernetes APF: seats + +fair queueing). + +Falsifiable claims: + +1. On a simple skew (one heavy key, light keys), a per-key cap on the heavy key + cuts the light keys' wait about as well as SFQ, BECAUSE Trigger's CK dequeue is + eligibility-aware (a variant at its per-key limit is skipped). Predict: + per-key cap ~= SFQ on light-key wait here. +2. A per-key cap is NOT work-conserving: when the heavy key is alone (siblings + idle), the cap throttles it below the env limit and idles slots, inflating + makespan. SFQ/baseline uses the whole env. Predict: per-key cap makespan >> + SFQ makespan on a heavy-alone workload. +3. A per-key cap fails the sybil split: a heavy tenant spreading its backlog over + many CK variants, each under its per-key cap, still starves a late light key + under oldest-first, because sum of per-key caps is unbounded relative to the + queue. Predict: per-key cap light-key wait stays high on the sybil scenario; + SFQ still protects the light key. +4. A total cap (per-task) does NOT fix cross-key starvation within one task: it + lowers the whole task's ceiling uniformly, and age-order still serves the heavy + backlog first within it. Predict: total-cap-only light-key wait ~= baseline. +5. Layered per-key cap + SFQ ordering (the APF pattern) is at least as good as + either alone on every scenario. Predict: layered ~= SFQ on wait, and inherits + the cap's occupancy bound. + +## Harness + +Reuse the CK harness (`fairness-spike-ck`) that drives the real +`dequeueMessagesFromCkQueueTracked` Lua at `maxCount = 1`, rescoring `ckIndex` to +express a discipline's order. Model the caps as admission gates: + +- Per-key cap: a discipline marks any CK variant whose in-flight >= its per-key + cap as INELIGIBLE. The driver writes ineligible variants a beyond-window + `ckIndex` score so the real Lua's `ZRANGEBYSCORE -inf now` skips them, exactly + as the real per-key gate would. Among eligible variants, keep baseline age + order (or an inner scheduler for the layered discipline). +- Total cap: the driver refuses to dequeue when total in-flight (holding.length, + = group SCARD for one base queue) >= the total cap, modelling the group gate. + +This is faithful to the plan's eligibility-aware semantics; the fidelity gap is +the same `maxCount = 1` one the CK spike already documents (production dequeues in +batches; a real per-key/total gate lives inside the batched Lua). + +New driver state: `inFlightByCk` and `runToCk`, maintained on serve/ack. + +## Disciplines + +- `baseline` (existing): age order, no cap. +- `perKeyCap(capOf)`: eligibility by per-key in-flight cap; age order among + eligible. Heavy key capped low, others uncapped (env-bounded). +- `totalCap(n)`: no per-key differentiation, no rescore ordering change; driver + gate on total in-flight. Models Phase 1 within one task. +- `sfq`, `stride`, `drr` (existing): scheduling. +- `perKeyCap+sfq` (layered): eligibility by per-key cap, SFQ order among eligible. +- (CoDel already disproven in the CK spike; not re-run here.) + +## Scenarios + +Existing: `ckSkew`, `ckBalanced`, `ckTrickle`. New: + +- `ckSybil`: heavy tenant as MANY CK variants (~20), each a modest backlog, all + enqueued early (old heads); one light key arrives later via poisson. Tests the + sybil split. +- `ckHeavyIdle`: one heavy key with a large backlog, siblings absent or arriving + very late and few. Tests work-conservation (makespan / idle slots). + +## Metrics + +Reuse `computeMetrics` (per-key wait = headline, contention share = directional). +Add `makespanMs` (max dequeue logical time) as the work-conservation signal: +a non-work-conserving cap inflates makespan on `ckHeavyIdle`. + +## Success bar + +Relative ranking only (as with the prior spikes), multi-seed, driving the real +Lua. The deliverable is a reconciliation verdict: which knob each mechanism is, +what each fixes and fails, and whether the plan's caps + the spike's scheduling +should layer. + +## Not building + +- A production implementation (spike is throwaway). +- A separate multi-base-queue (cross-task) harness for the total cap's real job + (protecting one task's env budget from another task). The total cap's cross-task + isolation is argued analytically from the research; within-task it is shown not + to fix cross-key starvation. A cross-task harness is noted as future work. +- Re-running CoDel (already disproven at both grains). diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/RESEARCH.md b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/RESEARCH.md new file mode 100644 index 0000000000..4539c2a412 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/RESEARCH.md @@ -0,0 +1,133 @@ +# Queue-fairness research (grounding for the caps-vs-scheduling reconciliation) + +Throwaway spike notes. Five Fable research passes, distilled. Citations kept so +the findings write-up and report can point at real sources. This grounds the +central claim: occupancy caps and fair scheduling are orthogonal knobs, and the +plan-of-record ships the cap knob while the earlier spike measured the scheduler +knob. + +## The orthogonality result (theory) + +Caps bound occupancy, not wait. A tenant capped at C in-flight with mean service +time S has long-run throughput <= C/S (Little's Law, L = lambda*W). That is an +upper bound on the capped tenant's share; it reserves no lower bound for anyone +else and says nothing about any tenant's waiting time (Little relates averages in +a stable system, not tails, and if the capped tenant's arrival rate exceeds C/S +its queue never stabilises so the law does not even apply to it). + +Scheduling bounds wait, not occupancy. WFQ/PGPS tracks GPS within one max job +(Parekh-Gallager finish-time bound L_max/r); SFQ gives a starved flow's head item +a hard wait bound of "one max-size job from every other active tenant" (Goyal-Vin +SFQ Theorem 2), with no server-rate assumption. But all of family A is +work-conserving: a lone backlogged tenant takes 100% of the server. Nothing in a +scheduler limits how many slots a tenant holds. + +Parekh-Gallager is the canonical joint statement: a worst-case per-flow delay +bound is the product of arrival regulation (leaky/token bucket = the admission +knob) AND a scheduling discipline (GPS/WFQ = the order knob). Neither alone yields +the bound. Cruz network-calculus caveat: plain FIFO does get a delay bound IF every +input is burstiness-constrained (arrival regulator on ingress) and aggregate rho < +C, but a concurrency cap is not an ingress regulator (it bounds in-flight, not +queue admission, and queue depth stays unbounded), so under adversarial arrival +FIFO wait is unbounded and the orthogonality holds without qualification. + +Sources: Little 1961 (Oper. Res. 9:383-387); Parekh & Gallager 1993/1994 (GPS, +IEEE/ACM ToN); Goyal, Vin & Cheng, Start-time Fair Queueing (SIGCOMM'96 / ToN'97); +Shreedhar & Varghese, DRR (SIGCOMM'95); Waldspurger & Weihl, Stride Scheduling +(MIT TM-528, 1995); Cruz, A Calculus for Network Delay (IEEE T-IT 1991); Kingman's +formula; Harchol-Balter, Performance Modeling and Design of Computer Systems (CUP +2013). + +## When a cap alone DOES cut a starved tenant's wait (the load-bearing condition) + +A per-tenant concurrency cap on heavy tenant H cuts light tenant L's wait to +near-zero iff BOTH: + +1. Slot availability: sum of caps of all backlogged tenants other than L is < N + (the binding aggregate limit), so freed slots exist that capped tenants can + never occupy; and +2. Eligibility-aware serve order: the dequeue selects the oldest ELIGIBLE item, + skipping items whose tenant is at cap, so L's head is reachable without + draining H's older items first. + +If (2) fails (single global age-ordered list with a head-blocking consumer), the +cap does NOT help L and with strict head-blocking makes L's wait WORSE: H's +backlog drains at k slots instead of N while the freed N-k slots sit idle +(head-of-line blocking; Parekh-Gallager's FCFS-gives-no-isolation remark). + +Trigger's CK dequeue is oldest-ELIGIBLE-first: the CK Lua gates each variant at +its per-key concurrencyLimit and skips a variant that is at its limit, moving to +the next-oldest eligible. So condition (2) holds structurally. That is WHY per-key +caps can work for wait here, and would not work on a head-blocking FIFO. + +## Where caps fail even with eligibility-aware order (the sybil split) + +Concurrency keys are client-chosen. A heavy tenant spreads its backlog across many +CK variants, each under its own per-key cap, all "eligible". The binding +constraint becomes the base-queue total cap (or env limit); oldest-first then +serves the adversary's older backlog across its many keys before a newcomer's +head. Per-key caps bound nothing in aggregate, because sum of per-key caps is +unbounded relative to the queue cap when keys are dynamic. The light tenant's wait +scales with the adversary's total queued backlog, which no per-key cap regulates. +Within a base queue, the fix under adversarial arrival is an order change +(round-robin / fair queueing across keys), not another cap. + +## Known static-cap failure modes (practice) + +2DFQ (Mace et al., SIGCOMM 2016): "Rate limiters, typically implemented as token +buckets, are not designed to provide fairness at short time intervals ... they can +either underutilize the system or concurrent bursts can overload it without +providing any further fairness guarantees." Their desirable-properties section +requires the scheduler be work-conserving, which "precludes the use of ad-hoc +throttling mechanisms to control misbehaving tenants." Pisces (OSDI 2012) and DRF +(NSDI 2011) both exist because static slot partitioning under/over-utilises under +skewed demand. Netflix concurrency-limits: static limits (Limit = RPS * latency, +i.e. Little's Law) "quickly go out of date"; hence adaptive. + +The price of caps when they DO give order-independent wait bounds: they degenerate +into a static partition (sum of caps <= K), which is non-work-conserving +(utilisation ceiling of sum-of-caps even when one tenant could use all K) and +needs bounded, pre-known tenant cardinality. + +## Production precedent: caps and scheduling are LAYERED, not either/or + +- Kubernetes API Priority & Fairness (the closest analog): total server + concurrency split into per-priority-level "seats" (a cap), THEN shuffle-sharded + fair queueing decides dispatch order within a level. Kubernetes hit exactly the + cap-alone failure with max-inflight before APF, and the fix ADDED fair queueing + on top of the existing cap rather than replacing it. (K8s docs; KEP-1040.) +- SQL Server Resource Governor: pool MIN/MAX/CAP percent (caps + reservation) + + workload-group IMPORTANCE biasing the scheduler's order. Same shape. +- YARN Fair/Capacity scheduler: minShare floor + maxResources cap + weighted + fair-share ordering, with preemption to reclaim the floor. +- Mesos/DRF, Borg: quota/admission caps layered with fair-share or priority order. +- Amazon SQS fair queues: fairness metric is DWELL TIME, fixed by reprioritising + delivery ORDER when a tenant's in-flight share is disproportionate, and + explicitly does NOT rate-limit per tenant. Order is the dwell-time knob; + occupancy limits alone were not it. +- Envoy / gRPC / Postgres connection caps: caps ALONE, and they claim overload + PROTECTION, not fairness. AWS token-bucket quotas claim "fairness" only in the + weaker selective-throttling sense, and rely on an elastic, rarely-saturated + fleet. + +Condition for caps-alone to suffice in practice: sum of caps comfortably below (or +elastically kept below) real capacity, and shed/rejected work acceptable, i.e. the +system never holds a contended backlog it must drain in some order. The moment you +hold a queue of admitted-but-waiting work at saturation, the serve order IS the +fairness policy. + +## CoDel (confirms the prior spike's "CoDel disproven" verdict) + +CoDel is an AQM: it bounds standing-queue delay by DROPPING packets when the +minimum sojourn over a window stays above target (5ms/100ms defaults; RFC 8289). +It is not a scheduler and gives no inter-flow fairness (RFC 7567: queue management +and scheduling are complementary, not substitutes). FQ-CoDel gets all its fairness +from a DRR scheduler; CoDel is only the per-flow AQM inside each sub-queue (RFC +8290). In a durable run queue that can never drop work, CoDel's actuator is gone: +reordering conserves total queue and total work, so hoisting one item's sojourn +down pushes others' up. Hoisting the stalest item to the front of an already +age-ordered queue re-applies the age bias the base already has, so it is a no-op +at best and a dominant-tenant amplifier at worst (a tenant that dumps a big +backlog owns the entire stale set). Facebook's server-side CoDel adaptation ("Fail +at Scale", ACM Queue 2015) keeps the drop (stale requests expire) and reorders +adaptive-LIFO (newest first), the opposite of stalest-first hoisting. From 1afa1476dff009a8e0a4b4da6f71897400735713 Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 14:23:53 +0100 Subject: [PATCH 2/4] test(run-engine): caps-vs-scheduling bench over the real CK Lua gates Runs the plan's concurrency caps (per-key limit via the real per-queue gate, total cap via a driver group gate) head-to-head with SFQ/DRR through the real CK-dequeue Lua. Per-key caps fix a starved key's wait when one key floods (eligibility-aware dequeue) but fail when a tenant shards its backlog across many keys (the sybil split), and are not work-conserving; a total cap only lowers the ceiling and worsens cross-key wait; scheduling fixes every case including sybil and stays work-conserving. Adds a makespan work-conservation metric. --- .../fairness-spike-ck/CAPS_FINDINGS.md | 134 +++ .../capsFairness.bench.test.ts | 233 +++++ .../fairness-spike-ck/harness/ckDriver.ts | 32 +- .../results/caps-ckHeavyIdle.json | 233 +++++ .../results/caps-ckSkew.json | 501 ++++++++++ .../results/caps-ckSybil.json | 903 ++++++++++++++++++ .../results/caps-ckTrickle.json | 367 +++++++ .../fairness-spike/harness/metrics.ts | 7 + 8 files changed, 2408 insertions(+), 2 deletions(-) create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/capsFairness.bench.test.ts create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckHeavyIdle.json create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSkew.json create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSybil.json create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckTrickle.json diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md new file mode 100644 index 0000000000..4d2cf434c0 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md @@ -0,0 +1,134 @@ +# Caps vs scheduling: reconciliation findings + +Throwaway spike. Ships nothing; delete before any merge to main. + +Bottom line: the plan-of-record's concurrency CAPS and the earlier spike's fair +SCHEDULING are different knobs, and the data on the real CK-dequeue Lua matches +the queueing theory (see `RESEARCH.md`). A per-key cap fixes a starved key's wait +when ONE key floods, because Trigger's CK dequeue is oldest-eligible-first, but it +FAILS when a tenant shards its backlog across many concurrency keys (the sybil +split), and it is not work-conserving. Fair scheduling (SFQ/DRR) fixes the wait on +every scenario, including the sybil split, and stays work-conserving. A total +(per-task) cap does not address cross-key starvation at all: applied inside a task +it only lowers the ceiling and makes the starved key's wait worse. The two +mechanisms are complementary, and every production system that needs fairness +under saturation layers them (Kubernetes APF: seats + fair queueing). + +## How the caps were modelled (fidelity) + +- Per-key cap (Phase 2): the REAL Lua gate. `updateQueueConcurrencyLimits` sets + the base queue's concurrencyLimit, and the CK-dequeue Lua caps each ck variant's + in-flight at it and skips an at-limit variant (oldest-eligible-first, true age + order, no rescore involved). Uniform across variants: Phase 2's per-key HGET + override would cap only the heavy key, but a light key never approaches the cap + so the effect is equivalent here. (This also means "just lower the existing + per-queue concurrency limit" is itself a per-key cap; Phase 2 makes it + per-key-specific.) +- Total cap (Phase 1): driver-side. The real Lua has no group gate yet, so the + driver refuses to admit while total in-flight across all variants of the base + queue (= `:groupConcurrency` SCARD in one base queue) is at the cap. +- Ordering disciplines (baseline age order, SFQ, DRR) are unchanged from the CK + scheduling spike, driven through the same real Lua at `maxCount = 1`. +- Same `maxCount = 1` fidelity caveat as the CK spike: production dequeues in + batches, so a real per-key/total gate lives inside the batched Lua. + +## Results + +env=4, per-key cap=2, total cap=2, 3 seeds. `lightWait` = the starved key's mean +wait (logical ms), the headline. `makespan` = drain time (work-conservation +signal). `contWorstS/W` = worst contention share over weight (directional). + +| scenario | treatment | lightWait | worstWait | makespan | contWorstS/W | +| ----------- | -------------- | --------- | --------- | -------- | ------------ | +| ckSkew | baseline | 1098 | 1261 | 2083 | 0.187 | +| ckSkew | perKeyCap | 20 | 1555 | 3038 | 0.814 | +| ckSkew | totalCap | 2840 | 2974 | 3947 | 0.213 | +| ckSkew | sfq | 14 | 1069 | 2083 | 0.723 | +| ckSkew | drr | 17 | 1067 | 2083 | 0.608 | +| ckSkew | perKeyCap+sfq | 7 | 1628 | 3114 | 0.800 | +| ckTrickle | baseline | 1107 | 1134 | 1940 | 0.279 | +| ckTrickle | perKeyCap | 19 | 1555 | 3038 | 0.922 | +| ckTrickle | totalCap | 2852 | 2861 | 3905 | 0.279 | +| ckTrickle | sfq | 17 | 1070 | 1942 | 0.909 | +| ckTrickle | drr | 23 | 1069 | 1941 | 0.790 | +| ckTrickle | perKeyCap+sfq | 8 | 1606 | 3097 | 0.658 | +| ckSybil | baseline | 1767 | 1823 | 2067 | 0.000 | +| ckSybil | perKeyCap | 1718 | 1831 | 2148 | 0.367 | +| ckSybil | totalCap | 3796 | 3796 | 4147 | 0.000 | +| ckSybil | sfq | 462 | 1062 | 2068 | 0.690 | +| ckSybil | drr | 496 | 1081 | 2071 | 0.690 | +| ckSybil | perKeyCap+sfq | 462 | 1062 | 2068 | 0.690 | +| ckHeavyIdle | baseline | 633 | 633 | 1240 | 1.000 | +| ckHeavyIdle | perKeyCap | 1291 | 1291 | 2507 | 1.000 | +| ckHeavyIdle | totalCap | 1291 | 1291 | 2507 | 1.000 | +| ckHeavyIdle | sfq | 633 | 633 | 1240 | 1.000 | +| ckHeavyIdle | drr | 633 | 633 | 1240 | 1.000 | +| ckHeavyIdle | perKeyCap+sfq | 1291 | 1291 | 2507 | 1.000 | + +(ckHeavyIdle is a single key, so "lightWait" is the heavy key's own wait and the +contention metric is degenerate at 1.0; the signal there is makespan.) + +## Verdicts + +- Per-key cap (Phase 2): PROVEN for the single-heavy case, DISPROVEN for the + sybil case, and it is not work-conserving. + - Single heavy key (ckSkew/ckTrickle): cuts the light key's wait like a + scheduler (1098 to 20, 1107 to 19) because capping the one heavy key frees + slots and the CK Lua is oldest-eligible-first, so the light key's head is + reachable. This works ONLY because the dequeue skips at-cap variants; on a + head-blocking FIFO the same cap would idle the freed slots and make wait worse. + - Sybil split (ckSybil): barely moves the light key's wait (1767 to 1718) + because the attacker's 10 keys keep env saturated with older heads and no + per-key cap bounds their aggregate. Only the fair order rescues the light key + (sfq 462, ~3.7x better than perKeyCap). Concurrency keys are client-chosen, so + this is cheap to trigger. + - Not work-conserving: throttles the capped key even with the env idle + (ckHeavyIdle makespan 1240 to 2507, 2x; ckSkew 2083 to 3038, +46%). +- Total cap (Phase 1) for cross-KEY fairness: DISPROVEN. Applied inside one task + it only lowers the whole task's ceiling and, with age order unchanged, makes the + starved key's wait worse on every contended scenario (ckSkew 1098 to 2840, + ckSybil 1767 to 3796). The total cap's real job is cross-TASK isolation + (reservation between base queues when the sum of per-task caps is below the env + limit); that is a different problem from #2617's within-task cross-key + starvation and is not exercised by this single-base-queue harness (noted as + future work). +- Scheduling (SFQ/DRR): PROVEN on every scenario including the sybil split, and + work-conserving (makespan stays at the baseline optimum 2083/1240). SFQ and DRR + track each other within noise, as in the CK spike. +- Layered per-key cap + SFQ (the Kubernetes-APF pattern): best of both on the + single-heavy case (lowest light wait AND the cap's occupancy bound), but + inherits the static cap's makespan penalty (ckSkew 3114, ckHeavyIdle 2507). On + the sybil case the cap adds nothing and SFQ does all the work (462). APF avoids + the work-conservation penalty by making the cap ELASTIC (borrow/lend seats); a + static cap cannot. + +## Reconciliation with the earlier spike and the plan of record + +- The earlier spike's recommendation (score `ckIndex` by a fair discipline) is the + general fix: it is the only mechanism here that survives the sybil split and it + is work-conserving. +- The plan of record ships caps first, and that is a defensible sequencing, not a + contradiction. A per-key cap is bounded, predictable, operator-controlled, and + self-healing (a Redis SET), and it fully fixes the common single-heavy-key case + with far less engine risk than reworking the dequeue scoring. Its limits are + real (defeated by key sharding, not work-conserving), which is exactly why the + plan calls automatic elastic fairness a later, opt-in phase. +- The honest layered conclusion (matching Kubernetes APF, SQL Server Resource + Governor, YARN, and the Parekh-Gallager result that a delay bound needs BOTH an + admission regulator AND a scheduler): keep the caps for isolation and + entitlements, and add a fair dequeue order for the contended region when + saturation and key-sharding make caps alone insufficient. Not either/or. + +## Caveats + +- Relative ranking only; single shard, single base queue, single sequential + consumer; simulated holds on a logical clock; 3 seeds; equal weights. +- `maxCount = 1` (see fidelity note); the total cap is driver-modelled, not the + real (unbuilt) group gate. +- Per-key cap is modelled uniformly (real per-queue gate); a per-key-specific + Phase-2 override is equivalent here only because the light key never approaches + the cap. +- Cross-task isolation (the total cap's real purpose) is argued from the research, + not measured; a multi-base-queue harness is future work. +- Wait is the trustworthy signal; contention share is volume-confounded for + low-volume keys (same caveat as the CK spike). diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/capsFairness.bench.test.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/capsFairness.bench.test.ts new file mode 100644 index 0000000000..99e1cab180 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/capsFairness.bench.test.ts @@ -0,0 +1,233 @@ +import { redisTest } from "@internal/testcontainers"; +import type { RedisOptions } from "@internal/redis"; +import { describe } from "vitest"; +import { mkdirSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { runCkScenario } from "./harness/ckDriver.js"; +import { + buildWorkload, + weightsOf, + type WorkloadConfig, +} from "../fairness-spike/harness/workload.js"; +import type { RunMetrics, GroupMetrics } from "../fairness-spike/harness/metrics.js"; +import { BaselineCk, SfqCk, DrrCk, type CkDiscipline } from "./disciplines.js"; + +/** + * Caps vs scheduling. Runs the plan-of-record's concurrency CAPS (Phase-2 per-key + * limit via the real per-queue concurrency gate, Phase-1 total cap via a + * driver-side group gate) head-to-head against the scheduling disciplines + * (SFQ/DRR) on identical scenarios, through the real CK-dequeue Lua at + * maxCount = 1. + * + * A "treatment" is (order discipline, per-key cap?, total cap?). Caps are + * admission settings, not disciplines: the per-key cap is the real Lua's native + * per-variant gate (oldest-eligible-first, true age order), the total cap is the + * driver refusing to admit past the group ceiling. + * + * Headline = the light (starved) key's wait. makespan = work-conservation signal. + * contention share = directional. + */ + +const RESULTS_DIR = join(dirname(fileURLToPath(import.meta.url)), "results"); +const SEEDS = ["seed-a", "seed-b", "seed-c"]; +const ENV_LIMIT = 4; +const PER_KEY_CAP = 2; // heavy key(s) bound to half the env +const TOTAL_CAP = 2; // per-task total ceiling, below env + +type Treatment = { + label: string; + makeDiscipline: () => CkDiscipline; + perKeyCap?: number; + totalCap?: number; +}; + +const TREATMENTS: Treatment[] = [ + { label: "baseline", makeDiscipline: () => new BaselineCk() }, + { label: "perKeyCap", makeDiscipline: () => new BaselineCk(), perKeyCap: PER_KEY_CAP }, + { label: "totalCap", makeDiscipline: () => new BaselineCk(), totalCap: TOTAL_CAP }, + { label: "sfq", makeDiscipline: () => new SfqCk() }, + { label: "drr", makeDiscipline: () => new DrrCk() }, + { label: "perKeyCap+sfq", makeDiscipline: () => new SfqCk(), perKeyCap: PER_KEY_CAP }, +]; + +type CapScenario = { + config: Omit; + /** the key whose wait is the headline (a starved light key, or the heavy key for heavy-idle) */ + lightKey: string; +}; + +function sybilHeavy(count: number, runsEach: number) { + return Array.from({ length: count }, (_, i) => ({ + tenantId: `heavy-${i}`, + runCount: runsEach, + holdMsMean: 25, + })); +} + +const SCENARIOS: Record = { + // one heavy key floods (old head), four light keys trickle in later. One heavy + // key => a per-key cap frees slots the light keys can take. + ckSkew: { + lightKey: "light-1", + config: { + envConcurrencyLimit: ENV_LIMIT, + tenants: [ + { tenantId: "heavy", runCount: 240, holdMsMean: 25 }, + { tenantId: "light-1", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + { tenantId: "light-2", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + { tenantId: "light-3", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + { tenantId: "light-4", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + ], + }, + }, + + // a bulk backlog plus two keys trickling in slowly + ckTrickle: { + lightKey: "trickle-1", + config: { + envConcurrencyLimit: ENV_LIMIT, + tenants: [ + { tenantId: "bulk", runCount: 240, holdMsMean: 25 }, + { tenantId: "trickle-1", runCount: 30, arrival: "poisson", ratePerSec: 25, holdMsMean: 25 }, + { tenantId: "trickle-2", runCount: 30, arrival: "poisson", ratePerSec: 25, holdMsMean: 25 }, + ], + }, + }, + + // sybil split: one attacker spreads its backlog across 10 concurrency keys, each + // with a large backlog that stays non-empty through the light key's whole + // arrival window. Each attacker key is under the same per-key cap, but the cap + // frees no aggregate slot (2 attacker keys fill env, and as one empties the next + // attacker key's old head is served before the newer light key). Only a fair + // order rescues the light key. + ckSybil: { + lightKey: "light", + config: { + envConcurrencyLimit: ENV_LIMIT, + tenants: [ + ...sybilHeavy(10, 30), + { tenantId: "light", runCount: 20, arrival: "poisson", ratePerSec: 40, holdMsMean: 25 }, + ], + }, + }, + + // work-conservation: one heavy key alone with a big backlog. A per-key or total + // cap throttles it below the env limit and idles slots, inflating makespan; a + // scheduler uses the whole env. Headline here is makespan, not wait. + ckHeavyIdle: { + lightKey: "heavy", + config: { + envConcurrencyLimit: ENV_LIMIT, + tenants: [{ tenantId: "heavy", runCount: 200, holdMsMean: 25 }], + }, + }, +}; + +function stats(xs: number[]) { + return { + mean: xs.reduce((a, b) => a + b, 0) / xs.length, + min: Math.min(...xs), + max: Math.max(...xs), + }; +} + +function fmt(n: number, d = 0): string { + return Number.isFinite(n) ? n.toFixed(d) : String(n); +} + +function waitOf(metrics: RunMetrics, key: string): number { + return metrics.perGroup.find((g) => g.groupId === key)?.meanWait ?? 0; +} + +function worstWaitOf(metrics: RunMetrics): number { + return metrics.perGroup.length ? Math.max(...metrics.perGroup.map((g) => g.meanWait)) : 0; +} + +describe("caps vs scheduling bench", () => { + mkdirSync(RESULTS_DIR, { recursive: true }); + + for (const [scenarioName, scenario] of Object.entries(SCENARIOS)) { + redisTest( + `caps scenario: ${scenarioName}`, + async ({ redisContainer }) => { + const runs = new Map>(); + + for (const seed of SEEDS) { + const config: WorkloadConfig = { ...scenario.config, seed }; + const workload = buildWorkload(config); + const expectedTotal = workload.tenants.reduce((n, t) => n + t.runCount, 0); + + for (const treatment of TREATMENTS) { + const redis: RedisOptions = { + keyPrefix: `rq:caps:${scenarioName}:${treatment.label}:${seed}:`, + host: redisContainer.getHost(), + port: redisContainer.getPort(), + }; + const metrics = await runCkScenario({ + redis, + discipline: treatment.makeDiscipline(), + workload, + perKeyCap: treatment.perKeyCap, + totalCap: treatment.totalCap, + }); + if (metrics.totalDequeued !== expectedTotal) { + throw new Error( + `${scenarioName}/${treatment.label}/${seed}: dequeued ${metrics.totalDequeued} of ${expectedTotal}` + ); + } + const arr = runs.get(treatment.label) ?? []; + arr.push({ seed, metrics }); + runs.set(treatment.label, arr); + } + } + + const perTreatment = [...runs.entries()].map(([label, rs]) => ({ + treatment: label, + lightWait: stats(rs.map((r) => waitOf(r.metrics, scenario.lightKey))), + worstWait: stats(rs.map((r) => worstWaitOf(r.metrics))), + makespan: stats(rs.map((r) => r.metrics.makespanMs)), + contentionWorst: stats(rs.map((r) => r.metrics.contentionWorstShareOverWeight)), + detailSeed0: rs[0].metrics.perGroup as GroupMetrics[], + })); + + const firstWorkload = buildWorkload({ ...scenario.config, seed: SEEDS[0] }); + writeFileSync( + join(RESULTS_DIR, `caps-${scenarioName}.json`), + JSON.stringify( + { + scenario: scenarioName, + seeds: SEEDS, + envLimit: ENV_LIMIT, + perKeyCap: PER_KEY_CAP, + totalCap: TOTAL_CAP, + lightKey: scenario.lightKey, + weights: weightsOf(firstWorkload), + perTreatment, + }, + null, + 2 + ) + ); + + const lines = [ + ``, + `### caps: ${scenarioName} (${SEEDS.length} seeds, env=${ENV_LIMIT}, perKeyCap=${PER_KEY_CAP}, totalCap=${TOTAL_CAP}, light=${scenario.lightKey})`, + `treatment lightWait worstWait makespan contWorstS/W`, + ...perTreatment.map( + (r) => + `${r.treatment.padEnd(16)} ${fmt(r.lightWait.mean).padStart(8)} ${fmt( + r.worstWait.mean + ).padStart(8)} ${fmt(r.makespan.mean).padStart(7)} ${fmt( + r.contentionWorst.mean, + 3 + ).padStart(7)}` + ), + ``, + ]; + process.stdout.write(lines.join("\n") + "\n"); + }, + 300_000 + ); + } +}); diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts index 12a926fe14..18277ccb33 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts @@ -32,6 +32,21 @@ export type CkDriverConfig = { discipline: CkDiscipline; workload: WorkloadSpec; maxLogicalMs?: number; + /** + * Phase-2 per-key limit, modelled with the REAL Lua gate: sets the base queue's + * concurrencyLimit so the CK-dequeue Lua caps EACH ck variant's in-flight at + * this and skips an at-limit variant (oldest-eligible-first). Uniform across + * variants (Phase 2's per-key HGET override would cap only the heavy key, but + * since a light key never approaches the cap the effect is equivalent here). + * Undefined = no per-key cap (env limit is the only per-variant ceiling). + */ + perKeyCap?: number; + /** + * Phase-1 total cap, modelled at the driver: do not admit while total in-flight + * across all CK variants of the base queue (= :groupConcurrency SCARD) is at or + * above this. The real Lua has no group gate yet, so this one is driver-side. + */ + totalCap?: number; }; function authenticatedEnv(limit: number) { @@ -73,6 +88,13 @@ export async function runCkScenario(config: CkDriverConfig): Promise queueSelectionStrategy: new FairQueueSelectionStrategy({ redis: config.redis, keys }), }); + // Phase-2 per-key cap: set the real per-queue concurrency limit the CK Lua + // gates each variant against. Faithful (native gate, true age order, no rescore + // pollution). + if (config.perKeyCap !== undefined) { + await queue.updateQueueConcurrencyLimits(env, BASE_QUEUE, config.perKeyCap); + } + const reader = new CkReader(admin, keys, config.redis.keyPrefix ?? ""); config.discipline.reset(); @@ -118,13 +140,19 @@ export async function runCkScenario(config: CkDriverConfig): Promise await queue.enqueueMessage({ env, message, workerQueue: ENV, skipDequeueProcessing: true }); } + const baseQueue = keys.queueKey(env, BASE_QUEUE, "any"); + const totalCap = config.totalCap; let progressed = true; while (progressed) { + // Phase-1 total cap: refuse to admit while group in-flight is at the cap + // (models the :groupConcurrency SCARD gate). Wait for a completion. + if (totalCap !== undefined && holding.length >= totalCap) break; + if (config.discipline.rescore) { - const active = await reader.readActiveCks(keys.queueKey(env, BASE_QUEUE, "any")); + const active = await reader.readActiveCks(baseQueue); if (active.length > 0) { const order = config.discipline.order(active, scoreBase + t); - await rescoreCkIndex(admin, keys, keys.queueKey(env, BASE_QUEUE, "any"), order, Date.now()); + await rescoreCkIndex(admin, keys, baseQueue, order, Date.now()); } } diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckHeavyIdle.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckHeavyIdle.json new file mode 100644 index 0000000000..14c6d06dcc --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckHeavyIdle.json @@ -0,0 +1,233 @@ +{ + "scenario": "ckHeavyIdle", + "seeds": [ + "seed-a", + "seed-b", + "seed-c" + ], + "envLimit": 4, + "perKeyCap": 2, + "totalCap": 2, + "lightKey": "heavy", + "weights": { + "heavy": 1 + }, + "perTreatment": [ + { + "treatment": "baseline", + "lightWait": { + "mean": 632.8383333333334, + "min": 574.22, + "max": 702.875 + }, + "worstWait": { + "mean": 632.8383333333334, + "min": 574.22, + "max": 702.875 + }, + "makespan": { + "mean": 1240.3333333333333, + "min": 1159, + "max": 1311 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 200, + "weight": 1, + "share": 1, + "contentionShareOverWeight": 0, + "meanWait": 702.875, + "waitP50": 723, + "waitP99": 1309, + "waitMax": 1311 + } + ] + }, + { + "treatment": "perKeyCap", + "lightWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "worstWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "makespan": { + "mean": 2507, + "min": 2344, + "max": 2638 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 200, + "weight": 1, + "share": 1, + "contentionShareOverWeight": 0, + "meanWait": 1431.305, + "waitP50": 1501, + "waitP99": 2623, + "waitMax": 2638 + } + ] + }, + { + "treatment": "totalCap", + "lightWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "worstWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "makespan": { + "mean": 2507, + "min": 2344, + "max": 2638 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 200, + "weight": 1, + "share": 1, + "contentionShareOverWeight": 0, + "meanWait": 1431.305, + "waitP50": 1501, + "waitP99": 2623, + "waitMax": 2638 + } + ] + }, + { + "treatment": "sfq", + "lightWait": { + "mean": 632.8383333333334, + "min": 574.22, + "max": 702.875 + }, + "worstWait": { + "mean": 632.8383333333334, + "min": 574.22, + "max": 702.875 + }, + "makespan": { + "mean": 1240.3333333333333, + "min": 1159, + "max": 1311 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 200, + "weight": 1, + "share": 1, + "contentionShareOverWeight": 0, + "meanWait": 702.875, + "waitP50": 723, + "waitP99": 1309, + "waitMax": 1311 + } + ] + }, + { + "treatment": "drr", + "lightWait": { + "mean": 632.8383333333334, + "min": 574.22, + "max": 702.875 + }, + "worstWait": { + "mean": 632.8383333333334, + "min": 574.22, + "max": 702.875 + }, + "makespan": { + "mean": 1240.3333333333333, + "min": 1159, + "max": 1311 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 200, + "weight": 1, + "share": 1, + "contentionShareOverWeight": 0, + "meanWait": 702.875, + "waitP50": 723, + "waitP99": 1309, + "waitMax": 1311 + } + ] + }, + { + "treatment": "perKeyCap+sfq", + "lightWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "worstWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "makespan": { + "mean": 2507, + "min": 2344, + "max": 2638 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 200, + "weight": 1, + "share": 1, + "contentionShareOverWeight": 0, + "meanWait": 1431.305, + "waitP50": 1501, + "waitP99": 2623, + "waitMax": 2638 + } + ] + } + ] +} \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSkew.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSkew.json new file mode 100644 index 0000000000..ee566536b7 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSkew.json @@ -0,0 +1,501 @@ +{ + "scenario": "ckSkew", + "seeds": [ + "seed-a", + "seed-b", + "seed-c" + ], + "envLimit": 4, + "perKeyCap": 2, + "totalCap": 2, + "lightKey": "light-1", + "weights": { + "heavy": 1, + "light-1": 1, + "light-2": 1, + "light-3": 1, + "light-4": 1 + }, + "perTreatment": [ + { + "treatment": "baseline", + "lightWait": { + "mean": 1098.2666666666667, + "min": 857.4, + "max": 1321 + }, + "worstWait": { + "mean": 1261.3777777777777, + "min": 1175.1333333333334, + "max": 1321 + }, + "makespan": { + "mean": 2082.6666666666665, + "min": 1765, + "max": 2285 + }, + "contentionWorst": { + "mean": 0.1866549331190391, + "min": 0.1858108108108108, + "max": 0.1877133105802048 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 4.067796610169491, + "meanWait": 871.9875, + "waitP50": 875, + "waitP99": 1644, + "waitMax": 1646 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2542372881355932, + "meanWait": 1321, + "waitP50": 1239, + "waitP99": 1653, + "waitMax": 1653 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.23728813559322035, + "meanWait": 981.6666666666666, + "waitP50": 1043, + "waitP99": 1659, + "waitMax": 1659 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.1864406779661017, + "meanWait": 679.1333333333333, + "waitP50": 753, + "waitP99": 1661, + "waitMax": 1661 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2542372881355932, + "meanWait": 915, + "waitP50": 839, + "waitP99": 1677, + "waitMax": 1677 + } + ] + }, + { + "treatment": "perKeyCap", + "lightWait": { + "mean": 19.622222222222224, + "min": 8.266666666666667, + "max": 38.13333333333333 + }, + "worstWait": { + "mean": 1555.2430555555557, + "min": 1426.3708333333334, + "max": 1770.1916666666666 + }, + "makespan": { + "mean": 3038.3333333333335, + "min": 2863, + "max": 3309 + }, + "contentionWorst": { + "mean": 0.8140259655565085, + "min": 0.7009345794392522, + "max": 0.9259259259259258 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.296296296296296, + "meanWait": 1770.1916666666666, + "waitP50": 1765, + "waitP99": 3293, + "waitMax": 3309 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.9259259259259258, + "meanWait": 8.266666666666667, + "waitP50": 0, + "waitP99": 42, + "waitMax": 42 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.9259259259259258, + "meanWait": 3.933333333333333, + "waitP50": 0, + "waitP99": 55, + "waitMax": 55 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.9259259259259258, + "meanWait": 2.466666666666667, + "waitP50": 0, + "waitP99": 24, + "waitMax": 24 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.9259259259259258, + "meanWait": 8.333333333333334, + "waitP50": 0, + "waitP99": 37, + "waitMax": 37 + } + ] + }, + { + "treatment": "totalCap", + "lightWait": { + "mean": 2840.2000000000003, + "min": 2670.3333333333335, + "max": 3138.4666666666667 + }, + "worstWait": { + "mean": 2973.6222222222223, + "min": 2766.9333333333334, + "max": 3138.4666666666667 + }, + "makespan": { + "mean": 3947.3333333333335, + "min": 3532, + "max": 4290 + }, + "contentionWorst": { + "mean": 0.21268177618484008, + "min": 0.1858108108108108, + "max": 0.23411371237458192 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 4.013377926421405, + "meanWait": 1770.1916666666666, + "waitP50": 1765, + "waitP99": 3293, + "waitMax": 3309 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2508361204013378, + "meanWait": 3138.4666666666667, + "waitP50": 3102, + "waitP99": 3367, + "waitMax": 3367 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.23411371237458192, + "meanWait": 2885.3333333333335, + "waitP50": 2913, + "waitP99": 3380, + "waitMax": 3380 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2508361204013378, + "meanWait": 2674, + "waitP50": 2769, + "waitP99": 3374, + "waitMax": 3374 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2508361204013378, + "meanWait": 2852.0666666666666, + "waitP50": 2841, + "waitP99": 3400, + "waitMax": 3400 + } + ] + }, + { + "treatment": "sfq", + "lightWait": { + "mean": 14.311111111111112, + "min": 11.333333333333334, + "max": 16.2 + }, + "worstWait": { + "mean": 1068.6944444444443, + "min": 937.3291666666667, + "max": 1149.6333333333334 + }, + "makespan": { + "mean": 2082.6666666666665, + "min": 1765, + "max": 2285 + }, + "contentionWorst": { + "mean": 0.7228911750188346, + "min": 0.6547619047619048, + "max": 0.7692307692307693 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.8617021276595742, + "meanWait": 1149.6333333333334, + "waitP50": 1193, + "waitP99": 2099, + "waitMax": 2110 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 16.2, + "waitP50": 16, + "waitP99": 40, + "waitMax": 40 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.7446808510638298, + "meanWait": 16.066666666666666, + "waitP50": 14, + "waitP99": 48, + "waitMax": 48 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 13, + "waitP50": 10, + "waitP99": 41, + "waitMax": 41 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 9.533333333333333, + "waitP50": 9, + "waitP99": 24, + "waitMax": 24 + } + ] + }, + { + "treatment": "drr", + "lightWait": { + "mean": 16.8, + "min": 14.6, + "max": 20.866666666666667 + }, + "worstWait": { + "mean": 1066.875, + "min": 935.9833333333333, + "max": 1149.1125 + }, + "makespan": { + "mean": 2082.6666666666665, + "min": 1765, + "max": 2285 + }, + "contentionWorst": { + "mean": 0.6077242434174587, + "min": 0.5555555555555555, + "max": 0.648148148148148 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 2.3893805309734515, + "meanWait": 1149.1125, + "waitP50": 1190, + "waitP99": 2099, + "waitMax": 2110 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.6637168141592921, + "meanWait": 20.866666666666667, + "waitP50": 20, + "waitP99": 45, + "waitMax": 45 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.6194690265486725, + "meanWait": 16.866666666666667, + "waitP50": 13, + "waitP99": 35, + "waitMax": 35 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.6637168141592921, + "meanWait": 12.266666666666667, + "waitP50": 11, + "waitP99": 42, + "waitMax": 42 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.6637168141592921, + "meanWait": 16, + "waitP50": 12, + "waitP99": 57, + "waitMax": 57 + } + ] + }, + { + "treatment": "perKeyCap+sfq", + "lightWait": { + "mean": 6.844444444444444, + "min": 6, + "max": 8.133333333333333 + }, + "worstWait": { + "mean": 1627.55, + "min": 1473.6708333333333, + "max": 1797.925 + }, + "makespan": { + "mean": 3114, + "min": 2915, + "max": 3338 + }, + "contentionWorst": { + "mean": 0.8002825914644652, + "min": 0.6521739130434782, + "max": 0.974025974025974 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 0.6521739130434782, + "meanWait": 1797.925, + "waitP50": 1794, + "waitP99": 3322, + "waitMax": 3338 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1.0869565217391304, + "meanWait": 6, + "waitP50": 0, + "waitP99": 31, + "waitMax": 31 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1.0869565217391304, + "meanWait": 2.1333333333333333, + "waitP50": 0, + "waitP99": 22, + "waitMax": 22 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1.0869565217391304, + "meanWait": 0.6, + "waitP50": 0, + "waitP99": 9, + "waitMax": 9 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1.0869565217391304, + "meanWait": 7.533333333333333, + "waitP50": 0, + "waitP99": 44, + "waitMax": 44 + } + ] + } + ] +} \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSybil.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSybil.json new file mode 100644 index 0000000000..458a0f33b2 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSybil.json @@ -0,0 +1,903 @@ +{ + "scenario": "ckSybil", + "seeds": [ + "seed-a", + "seed-b", + "seed-c" + ], + "envLimit": 4, + "perKeyCap": 2, + "totalCap": 2, + "lightKey": "light", + "weights": { + "heavy-0": 1, + "heavy-1": 1, + "heavy-2": 1, + "heavy-3": 1, + "heavy-4": 1, + "heavy-5": 1, + "heavy-6": 1, + "heavy-7": 1, + "heavy-8": 1, + "heavy-9": 1, + "light": 1 + }, + "perTreatment": [ + { + "treatment": "baseline", + "lightWait": { + "mean": 1767.1833333333334, + "min": 1681.25, + "max": 1926.3 + }, + "worstWait": { + "mean": 1823.0333333333335, + "min": 1717.5, + "max": 1956.5666666666666 + }, + "makespan": { + "mean": 2067, + "min": 1891, + "max": 2296 + }, + "contentionWorst": { + "mean": 0, + "min": 0, + "max": 0 + }, + "detailSeed0": [ + { + "groupId": "heavy-0", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 112.16666666666667, + "waitP50": 103, + "waitP99": 252, + "waitMax": 252 + }, + { + "groupId": "heavy-1", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 319.4, + "waitP50": 321, + "waitP99": 395, + "waitMax": 395 + }, + { + "groupId": "heavy-2", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 491.7, + "waitP50": 497, + "waitP99": 565, + "waitMax": 565 + }, + { + "groupId": "heavy-3", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 673.7, + "waitP50": 670, + "waitP99": 760, + "waitMax": 760 + }, + { + "groupId": "heavy-4", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 885.6333333333333, + "waitP50": 893, + "waitP99": 1005, + "waitMax": 1005 + }, + { + "groupId": "heavy-5", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 1104.0333333333333, + "waitP50": 1105, + "waitP99": 1183, + "waitMax": 1183 + }, + { + "groupId": "heavy-6", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 1288.5, + "waitP50": 1284, + "waitP99": 1378, + "waitMax": 1378 + }, + { + "groupId": "heavy-7", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 1532.5333333333333, + "waitP50": 1538, + "waitP99": 1653, + "waitMax": 1653 + }, + { + "groupId": "heavy-8", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 1734.2666666666667, + "waitP50": 1731, + "waitP99": 1811, + "waitMax": 1811 + }, + { + "groupId": "heavy-9", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 1956.5666666666666, + "waitP50": 1954, + "waitP99": 2108, + "waitMax": 2108 + }, + { + "groupId": "light", + "dequeued": 20, + "weight": 1, + "share": 0.0625, + "contentionShareOverWeight": 0, + "meanWait": 1926.3, + "waitP50": 1872, + "waitP99": 2127, + "waitMax": 2127 + } + ] + }, + { + "treatment": "perKeyCap", + "lightWait": { + "mean": 1718.3833333333332, + "min": 1671.6, + "max": 1749.3 + }, + "worstWait": { + "mean": 1830.788888888889, + "min": 1674.1333333333334, + "max": 2068.9333333333334 + }, + "makespan": { + "mean": 2148.3333333333335, + "min": 1911, + "max": 2406 + }, + "contentionWorst": { + "mean": 0.36687717667535047, + "min": 0.0728476821192053, + "max": 0.7073954983922829 + }, + "detailSeed0": [ + { + "groupId": "heavy-0", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 264.96666666666664, + "waitP50": 256, + "waitP99": 531, + "waitMax": 531 + }, + { + "groupId": "heavy-1", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 119.06666666666666, + "waitP50": 114, + "waitP99": 297, + "waitMax": 297 + }, + { + "groupId": "heavy-2", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 465.8666666666667, + "waitP50": 458, + "waitP99": 610, + "waitMax": 610 + }, + { + "groupId": "heavy-3", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 721, + "waitP50": 719, + "waitP99": 885, + "waitMax": 885 + }, + { + "groupId": "heavy-4", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 863.9333333333333, + "waitP50": 853, + "waitP99": 1089, + "waitMax": 1089 + }, + { + "groupId": "heavy-5", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 1120.2, + "waitP50": 1115, + "waitP99": 1285, + "waitMax": 1285 + }, + { + "groupId": "heavy-6", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 1319.8666666666666, + "waitP50": 1319, + "waitP99": 1537, + "waitMax": 1537 + }, + { + "groupId": "heavy-7", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 1536.7333333333333, + "waitP50": 1577, + "waitP99": 1788, + "waitMax": 1788 + }, + { + "groupId": "heavy-8", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0610932475884243, + "meanWait": 1688.3, + "waitP50": 1697, + "waitP99": 1858, + "waitMax": 1858 + }, + { + "groupId": "heavy-9", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 0.7427652733118971, + "meanWait": 2068.9333333333334, + "waitP50": 2038, + "waitP99": 2406, + "waitMax": 2406 + }, + { + "groupId": "light", + "dequeued": 20, + "weight": 1, + "share": 0.0625, + "contentionShareOverWeight": 0.7073954983922829, + "meanWait": 1734.25, + "waitP50": 1693, + "waitP99": 1867, + "waitMax": 1867 + } + ] + }, + { + "treatment": "totalCap", + "lightWait": { + "mean": 3795.7999999999997, + "min": 3553.5, + "max": 4165.45 + }, + "worstWait": { + "mean": 3795.7999999999997, + "min": 3553.5, + "max": 4165.45 + }, + "makespan": { + "mean": 4147.333333333333, + "min": 3793, + "max": 4605 + }, + "contentionWorst": { + "mean": 0, + "min": 0, + "max": 0 + }, + "detailSeed0": [ + { + "groupId": "heavy-0", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 264.96666666666664, + "waitP50": 256, + "waitP99": 531, + "waitMax": 531 + }, + { + "groupId": "heavy-1", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 658.4666666666667, + "waitP50": 650, + "waitP99": 833, + "waitMax": 833 + }, + { + "groupId": "heavy-2", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 1004.7, + "waitP50": 994, + "waitP99": 1146, + "waitMax": 1146 + }, + { + "groupId": "heavy-3", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 1365.9666666666667, + "waitP50": 1366, + "waitP99": 1531, + "waitMax": 1531 + }, + { + "groupId": "heavy-4", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 1805.8666666666666, + "waitP50": 1801, + "waitP99": 2028, + "waitMax": 2028 + }, + { + "groupId": "heavy-5", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 2228.5333333333333, + "waitP50": 2215, + "waitP99": 2391, + "waitMax": 2391 + }, + { + "groupId": "heavy-6", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 2611.9, + "waitP50": 2612, + "waitP99": 2830, + "waitMax": 2830 + }, + { + "groupId": "heavy-7", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 3094.9333333333334, + "waitP50": 3140, + "waitP99": 3347, + "waitMax": 3347 + }, + { + "groupId": "heavy-8", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 3496.6666666666665, + "waitP50": 3507, + "waitP99": 3666, + "waitMax": 3666 + }, + { + "groupId": "heavy-9", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.1, + "meanWait": 3950.733333333333, + "waitP50": 3922, + "waitP99": 4288, + "waitMax": 4288 + }, + { + "groupId": "light", + "dequeued": 20, + "weight": 1, + "share": 0.0625, + "contentionShareOverWeight": 0, + "meanWait": 4165.45, + "waitP50": 4122, + "waitP99": 4307, + "waitMax": 4307 + } + ] + }, + { + "treatment": "sfq", + "lightWait": { + "mean": 462.3833333333334, + "min": 438.85, + "max": 489.6 + }, + "worstWait": { + "mean": 1061.6888888888889, + "min": 952.7, + "max": 1178.3 + }, + "makespan": { + "mean": 2067.6666666666665, + "min": 1877, + "max": 2295 + }, + "contentionWorst": { + "mean": 0.689655172413793, + "min": 0.689655172413793, + "max": 0.689655172413793 + }, + "detailSeed0": [ + { + "groupId": "heavy-0", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1173.3, + "waitP50": 1209, + "waitP99": 2263, + "waitMax": 2263 + }, + { + "groupId": "heavy-1", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1175.1, + "waitP50": 1241, + "waitP99": 2247, + "waitMax": 2247 + }, + { + "groupId": "heavy-2", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1172.5666666666666, + "waitP50": 1224, + "waitP99": 2283, + "waitMax": 2283 + }, + { + "groupId": "heavy-3", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1178.3, + "waitP50": 1240, + "waitP99": 2271, + "waitMax": 2271 + }, + { + "groupId": "heavy-4", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1171.1333333333334, + "waitP50": 1236, + "waitP99": 2264, + "waitMax": 2264 + }, + { + "groupId": "heavy-5", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1173.9666666666667, + "waitP50": 1255, + "waitP99": 2245, + "waitMax": 2245 + }, + { + "groupId": "heavy-6", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1176.1666666666667, + "waitP50": 1167, + "waitP99": 2291, + "waitMax": 2291 + }, + { + "groupId": "heavy-7", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1, + "meanWait": 1169.2, + "waitP50": 1173, + "waitP99": 2295, + "waitMax": 2295 + }, + { + "groupId": "heavy-8", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1171.2, + "waitP50": 1165, + "waitP99": 2286, + "waitMax": 2286 + }, + { + "groupId": "heavy-9", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1171.2666666666667, + "waitP50": 1218, + "waitP99": 2292, + "waitMax": 2292 + }, + { + "groupId": "light", + "dequeued": 20, + "weight": 1, + "share": 0.0625, + "contentionShareOverWeight": 0.689655172413793, + "meanWait": 489.6, + "waitP50": 386, + "waitP99": 1026, + "waitMax": 1026 + } + ] + }, + { + "treatment": "drr", + "lightWait": { + "mean": 495.8833333333334, + "min": 471.3, + "max": 528.95 + }, + "worstWait": { + "mean": 1081.4777777777779, + "min": 972.5666666666667, + "max": 1198 + }, + "makespan": { + "mean": 2071.3333333333335, + "min": 1883, + "max": 2300 + }, + "contentionWorst": { + "mean": 0.689655172413793, + "min": 0.689655172413793, + "max": 0.689655172413793 + }, + "detailSeed0": [ + { + "groupId": "heavy-0", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1132.0333333333333, + "waitP50": 1159, + "waitP99": 2239, + "waitMax": 2239 + }, + { + "groupId": "heavy-1", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1141.4333333333334, + "waitP50": 1163, + "waitP99": 2281, + "waitMax": 2281 + }, + { + "groupId": "heavy-2", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1146.4, + "waitP50": 1165, + "waitP99": 2240, + "waitMax": 2240 + }, + { + "groupId": "heavy-3", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1157.6333333333334, + "waitP50": 1191, + "waitP99": 2285, + "waitMax": 2285 + }, + { + "groupId": "heavy-4", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1163.4, + "waitP50": 1195, + "waitP99": 2247, + "waitMax": 2247 + }, + { + "groupId": "heavy-5", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1171.7666666666667, + "waitP50": 1196, + "waitP99": 2283, + "waitMax": 2283 + }, + { + "groupId": "heavy-6", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1176.1666666666667, + "waitP50": 1199, + "waitP99": 2249, + "waitMax": 2249 + }, + { + "groupId": "heavy-7", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1, + "meanWait": 1185.9666666666667, + "waitP50": 1224, + "waitP99": 2300, + "waitMax": 2300 + }, + { + "groupId": "heavy-8", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1191.6, + "waitP50": 1241, + "waitP99": 2274, + "waitMax": 2274 + }, + { + "groupId": "heavy-9", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1198, + "waitP50": 1243, + "waitP99": 2285, + "waitMax": 2285 + }, + { + "groupId": "light", + "dequeued": 20, + "weight": 1, + "share": 0.0625, + "contentionShareOverWeight": 0.689655172413793, + "meanWait": 528.95, + "waitP50": 459, + "waitP99": 1077, + "waitMax": 1077 + } + ] + }, + { + "treatment": "perKeyCap+sfq", + "lightWait": { + "mean": 462.3833333333334, + "min": 438.85, + "max": 489.6 + }, + "worstWait": { + "mean": 1061.6888888888889, + "min": 952.7, + "max": 1178.3 + }, + "makespan": { + "mean": 2067.6666666666665, + "min": 1877, + "max": 2295 + }, + "contentionWorst": { + "mean": 0.689655172413793, + "min": 0.689655172413793, + "max": 0.689655172413793 + }, + "detailSeed0": [ + { + "groupId": "heavy-0", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1173.3, + "waitP50": 1209, + "waitP99": 2263, + "waitMax": 2263 + }, + { + "groupId": "heavy-1", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1175.1, + "waitP50": 1241, + "waitP99": 2247, + "waitMax": 2247 + }, + { + "groupId": "heavy-2", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1172.5666666666666, + "waitP50": 1224, + "waitP99": 2283, + "waitMax": 2283 + }, + { + "groupId": "heavy-3", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1178.3, + "waitP50": 1240, + "waitP99": 2271, + "waitMax": 2271 + }, + { + "groupId": "heavy-4", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1171.1333333333334, + "waitP50": 1236, + "waitP99": 2264, + "waitMax": 2264 + }, + { + "groupId": "heavy-5", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1173.9666666666667, + "waitP50": 1255, + "waitP99": 2245, + "waitMax": 2245 + }, + { + "groupId": "heavy-6", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1176.1666666666667, + "waitP50": 1167, + "waitP99": 2291, + "waitMax": 2291 + }, + { + "groupId": "heavy-7", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1, + "meanWait": 1169.2, + "waitP50": 1173, + "waitP99": 2295, + "waitMax": 2295 + }, + { + "groupId": "heavy-8", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1171.2, + "waitP50": 1165, + "waitP99": 2286, + "waitMax": 2286 + }, + { + "groupId": "heavy-9", + "dequeued": 30, + "weight": 1, + "share": 0.09375, + "contentionShareOverWeight": 1.0344827586206897, + "meanWait": 1171.2666666666667, + "waitP50": 1218, + "waitP99": 2292, + "waitMax": 2292 + }, + { + "groupId": "light", + "dequeued": 20, + "weight": 1, + "share": 0.0625, + "contentionShareOverWeight": 0.689655172413793, + "meanWait": 489.6, + "waitP50": 386, + "waitP99": 1026, + "waitMax": 1026 + } + ] + } + ] +} \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckTrickle.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckTrickle.json new file mode 100644 index 0000000000..bc1a3d8ed7 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckTrickle.json @@ -0,0 +1,367 @@ +{ + "scenario": "ckTrickle", + "seeds": [ + "seed-a", + "seed-b", + "seed-c" + ], + "envLimit": 4, + "perKeyCap": 2, + "totalCap": 2, + "lightKey": "trickle-1", + "weights": { + "bulk": 1, + "trickle-1": 1, + "trickle-2": 1 + }, + "perTreatment": [ + { + "treatment": "baseline", + "lightWait": { + "mean": 1107.3222222222223, + "min": 836.3333333333334, + "max": 1338.6666666666667 + }, + "worstWait": { + "mean": 1133.6222222222223, + "min": 872.8, + "max": 1338.6666666666667 + }, + "makespan": { + "mean": 1940.3333333333333, + "min": 1821, + "max": 2142 + }, + "contentionWorst": { + "mean": 0.27872569582223233, + "min": 0.2542372881355932, + "max": 0.29096989966555187 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 2.440677966101695, + "meanWait": 871.9875, + "waitP50": 875, + "waitP99": 1644, + "waitMax": 1646 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.3050847457627119, + "meanWait": 1338.6666666666667, + "waitP50": 1399, + "waitP99": 1705, + "waitMax": 1705 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.2542372881355932, + "meanWait": 1165.8666666666666, + "waitP50": 1162, + "waitP99": 1718, + "waitMax": 1718 + } + ] + }, + { + "treatment": "perKeyCap", + "lightWait": { + "mean": 18.933333333333334, + "min": 8.8, + "max": 26.066666666666666 + }, + "worstWait": { + "mean": 1555.2430555555557, + "min": 1426.3708333333334, + "max": 1770.1916666666666 + }, + "makespan": { + "mean": 3038.3333333333335, + "min": 2863, + "max": 3309 + }, + "contentionWorst": { + "mean": 0.921778350515464, + "min": 0.9, + "max": 0.9375 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.2000000000000002, + "meanWait": 1770.1916666666666, + "waitP50": 1765, + "waitP99": 3293, + "waitMax": 3309 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9, + "meanWait": 26.066666666666666, + "waitP50": 0, + "waitP99": 146, + "waitMax": 146 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9, + "meanWait": 15.766666666666667, + "waitP50": 0, + "waitP99": 159, + "waitMax": 159 + } + ] + }, + { + "treatment": "totalCap", + "lightWait": { + "mean": 2851.788888888889, + "min": 2528.4666666666667, + "max": 3236.766666666667 + }, + "worstWait": { + "mean": 2860.7444444444445, + "min": 2535.3, + "max": 3236.766666666667 + }, + "makespan": { + "mean": 3904.6666666666665, + "min": 3653, + "max": 4342 + }, + "contentionWorst": { + "mean": 0.27872569582223233, + "min": 0.2542372881355932, + "max": 0.29096989966555187 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 2.440677966101695, + "meanWait": 1770.1916666666666, + "waitP50": 1765, + "waitP99": 3293, + "waitMax": 3309 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.3050847457627119, + "meanWait": 3236.766666666667, + "waitP50": 3266, + "waitP99": 3523, + "waitMax": 3523 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.2542372881355932, + "meanWait": 3129.5, + "waitP50": 3132, + "waitP99": 3536, + "waitMax": 3536 + } + ] + }, + { + "treatment": "sfq", + "lightWait": { + "mean": 17.444444444444443, + "min": 11.633333333333333, + "max": 23.933333333333334 + }, + "worstWait": { + "mean": 1070.3680555555557, + "min": 957.8, + "max": 1237.4333333333334 + }, + "makespan": { + "mean": 1942, + "min": 1818, + "max": 2140 + }, + "contentionWorst": { + "mean": 0.9092746009294808, + "min": 0.8910891089108911, + "max": 0.9183673469387755 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.163265306122449, + "meanWait": 1237.4333333333334, + "waitP50": 1317, + "waitP99": 2139, + "waitMax": 2140 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9183673469387755, + "meanWait": 23.933333333333334, + "waitP50": 13, + "waitP99": 96, + "waitMax": 96 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9183673469387755, + "meanWait": 20.3, + "waitP50": 14, + "waitP99": 62, + "waitMax": 62 + } + ] + }, + { + "treatment": "drr", + "lightWait": { + "mean": 22.755555555555556, + "min": 16.1, + "max": 30.433333333333334 + }, + "worstWait": { + "mean": 1068.9722222222222, + "min": 957.0166666666667, + "max": 1235.5583333333334 + }, + "makespan": { + "mean": 1941.3333333333333, + "min": 1818, + "max": 2140 + }, + "contentionWorst": { + "mean": 0.7900070943549204, + "min": 0.7692307692307692, + "max": 0.8181818181818181 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.4347826086956523, + "meanWait": 1235.5583333333334, + "waitP50": 1315, + "waitP99": 2139, + "waitMax": 2140 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.782608695652174, + "meanWait": 30.433333333333334, + "waitP50": 18, + "waitP99": 128, + "waitMax": 128 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.782608695652174, + "meanWait": 26.333333333333332, + "waitP50": 25, + "waitP99": 61, + "waitMax": 61 + } + ] + }, + { + "treatment": "perKeyCap+sfq", + "lightWait": { + "mean": 8.011111111111111, + "min": 3.6, + "max": 14.333333333333334 + }, + "worstWait": { + "mean": 1606.2347222222222, + "min": 1438.425, + "max": 1844.2416666666666 + }, + "makespan": { + "mean": 3097.3333333333335, + "min": 2876, + "max": 3387 + }, + "contentionWorst": { + "mean": 0.6577540106951871, + "min": 0.42857142857142855, + "max": 0.8823529411764707 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 0.6623376623376623, + "meanWait": 1844.2416666666666, + "waitP50": 1843, + "waitP99": 3371, + "waitMax": 3387 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 1.168831168831169, + "meanWait": 14.333333333333334, + "waitP50": 0, + "waitP99": 96, + "waitMax": 96 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 1.168831168831169, + "meanWait": 5.1, + "waitP50": 0, + "waitP99": 31, + "waitMax": 31 + } + ] + } + ] +} \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike/harness/metrics.ts b/internal-packages/run-engine/src/run-queue/fairness-spike/harness/metrics.ts index 1543a93c2b..3dc91e4b6c 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike/harness/metrics.ts +++ b/internal-packages/run-engine/src/run-queue/fairness-spike/harness/metrics.ts @@ -33,6 +33,12 @@ export type RunMetrics = { */ contentionWorstShareOverWeight: number; contentionJain: number; + /** + * Logical time of the last dequeue (drain time). Work-conservation signal: on a + * fixed workload a non-work-conserving discipline (e.g. a static cap that idles + * slots when the capped tenant is alone) drains slower, so makespan is larger. + */ + makespanMs: number; /** * The largest per-group p99/max wait. NOTE: this is NOT an anti-staleness win * signal. It is dominated by the highest-volume tenant, which a fair selector @@ -172,6 +178,7 @@ export function computeMetrics(input: { wallClockMs: input.wallClockMs, contentionWorstShareOverWeight: cowVec.length ? Math.min(...cowVec) : 1, contentionJain: jain(cowVec), + makespanMs: events.length ? Math.max(...events.map((e) => e.dequeueAtMs)) : 0, worstWaitP99: perGroup.length ? Math.max(...perGroup.map((p) => p.waitP99)) : 0, worstWaitMax: perGroup.length ? Math.max(...perGroup.map((p) => p.waitMax)) : 0, }; From c59a159ffe43dd2af32afd9d23bd89a3056447ae Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 15:02:39 +0100 Subject: [PATCH 3/4] test(run-engine): address caps-vs-scheduling review Blind two-model review (a third stalled). Folds in: consistent relative-ranking register instead of proven/disproven; reframes the total cap as a capacity- confounded cross-task knob, not a cross-key fairness failure; leads the sybil result with the mechanism (unbounded per-key sum) plus the real Lua's 3-wide scan window (ZRANGEBYSCORE ... LIMIT 0, maxCount*3), which governs why per-key caps work with one heavy key and give no wait improvement once a tenant shards (sybil bumped to 20 attacker keys, now flat vs baseline). Adds the shipped combined total+per-key config and a fully-layered total+per-key+sfq treatment. Marks the caps-first sequencing as interpretation, not measured. Corrects the makespan doc (last dequeue, arrival-confounded off ckHeavyIdle), defines worstWait, notes the contention-share seed spread, and comments the speculative order() coupling. --- .../fairness-spike-ck/CAPS_FINDINGS.md | 256 +- .../capsFairness.bench.test.ts | 20 +- .../fairness-spike-ck/harness/ckDriver.ts | 6 + .../results/caps-ckHeavyIdle.json | 72 + .../results/caps-ckSkew.json | 160 ++ .../results/caps-ckSybil.json | 2234 +++++++++++++---- .../results/caps-ckTrickle.json | 116 + .../fairness-spike/harness/metrics.ts | 9 +- 8 files changed, 2236 insertions(+), 637 deletions(-) diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md index 4d2cf434c0..5dd82de233 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md @@ -1,134 +1,188 @@ # Caps vs scheduling: reconciliation findings -Throwaway spike. Ships nothing; delete before any merge to main. +Throwaway spike. Ships nothing; delete before any merge to main. Relative ranking +on a small simulation, not a statistical or load study: read the verdicts as "what +this harness supports", not proofs. Went through a blind two-model adversarial +review (a third stalled); the review's fixes are folded in below. Bottom line: the plan-of-record's concurrency CAPS and the earlier spike's fair -SCHEDULING are different knobs, and the data on the real CK-dequeue Lua matches -the queueing theory (see `RESEARCH.md`). A per-key cap fixes a starved key's wait -when ONE key floods, because Trigger's CK dequeue is oldest-eligible-first, but it -FAILS when a tenant shards its backlog across many concurrency keys (the sybil -split), and it is not work-conserving. Fair scheduling (SFQ/DRR) fixes the wait on -every scenario, including the sybil split, and stays work-conserving. A total -(per-task) cap does not address cross-key starvation at all: applied inside a task -it only lowers the ceiling and makes the starved key's wait worse. The two -mechanisms are complementary, and every production system that needs fairness -under saturation layers them (Kubernetes APF: seats + fair queueing). - -## How the caps were modelled (fidelity) +SCHEDULING are different knobs, and the data on the real CK-dequeue Lua lines up +with the queueing theory (see `RESEARCH.md`). A per-key cap cuts a starved key's +wait when ONE key floods, because Trigger's CK dequeue is oldest-eligible-first; +it gives no wait improvement once a tenant shards its backlog across many +concurrency keys, and it is not work-conserving. Fair scheduling (SFQ/DRR) is the +only knob here that improves the starved key on every scenario including the +sharded one, and it stays work-conserving. A total (per-task) cap is a +cross-task knob; inside one task it only lowers the ceiling and is not a fairness +lever at all. The mechanisms are complementary, and production systems that need +fairness under saturation layer them (Kubernetes APF: seats + fair queueing). + +## How the mechanisms were modelled (fidelity) - Per-key cap (Phase 2): the REAL Lua gate. `updateQueueConcurrencyLimits` sets - the base queue's concurrencyLimit, and the CK-dequeue Lua caps each ck variant's + the base queue's concurrencyLimit; the CK-dequeue Lua caps each ck variant's in-flight at it and skips an at-limit variant (oldest-eligible-first, true age - order, no rescore involved). Uniform across variants: Phase 2's per-key HGET - override would cap only the heavy key, but a light key never approaches the cap - so the effect is equivalent here. (This also means "just lower the existing - per-queue concurrency limit" is itself a per-key cap; Phase 2 makes it - per-key-specific.) + order, no rescore). Uniform across variants: Phase 2's per-key HGET override + would cap only the heavy key, but a light key never approaches the cap so the + effect is equivalent here. (So "just lower the existing per-queue concurrency + limit" already IS a per-key cap; Phase 2 makes it per-key-specific.) - Total cap (Phase 1): driver-side. The real Lua has no group gate yet, so the driver refuses to admit while total in-flight across all variants of the base queue (= `:groupConcurrency` SCARD in one base queue) is at the cap. - Ordering disciplines (baseline age order, SFQ, DRR) are unchanged from the CK scheduling spike, driven through the same real Lua at `maxCount = 1`. -- Same `maxCount = 1` fidelity caveat as the CK spike: production dequeues in - batches, so a real per-key/total gate lives inside the batched Lua. +- Two fidelity limits matter for reading the numbers, both driver-independent: + - `maxCount = 1` (same as the CK spike): production dequeues in batches, so a + real per-key/total gate lives inside the batched Lua. + - The `*3` scan window: the real CK Lua reads `ZRANGEBYSCORE ckIndexKey -inf now + LIMIT 0, actualMaxCount*3` (`index.ts:4041/4193`). At `maxCount = 1` that is + the 3 oldest-scored variants per call. So a per-key cap frees the light key + only when the light head lands inside that 3-wide window after the at-cap + variants ahead of it. With one heavy key it does; with many old-headed + attacker variants it never does. This window governs the skew-works / + sharded-fails split, and it scales with `maxCount` in production (batches), + not fixed at 3, so the sharded result's exact severity would differ on the + real batched path (direction not established). ## Results -env=4, per-key cap=2, total cap=2, 3 seeds. `lightWait` = the starved key's mean -wait (logical ms), the headline. `makespan` = drain time (work-conservation -signal). `contWorstS/W` = worst contention share over weight (directional). - -| scenario | treatment | lightWait | worstWait | makespan | contWorstS/W | -| ----------- | -------------- | --------- | --------- | -------- | ------------ | -| ckSkew | baseline | 1098 | 1261 | 2083 | 0.187 | -| ckSkew | perKeyCap | 20 | 1555 | 3038 | 0.814 | -| ckSkew | totalCap | 2840 | 2974 | 3947 | 0.213 | -| ckSkew | sfq | 14 | 1069 | 2083 | 0.723 | -| ckSkew | drr | 17 | 1067 | 2083 | 0.608 | -| ckSkew | perKeyCap+sfq | 7 | 1628 | 3114 | 0.800 | -| ckTrickle | baseline | 1107 | 1134 | 1940 | 0.279 | -| ckTrickle | perKeyCap | 19 | 1555 | 3038 | 0.922 | -| ckTrickle | totalCap | 2852 | 2861 | 3905 | 0.279 | -| ckTrickle | sfq | 17 | 1070 | 1942 | 0.909 | -| ckTrickle | drr | 23 | 1069 | 1941 | 0.790 | -| ckTrickle | perKeyCap+sfq | 8 | 1606 | 3097 | 0.658 | -| ckSybil | baseline | 1767 | 1823 | 2067 | 0.000 | -| ckSybil | perKeyCap | 1718 | 1831 | 2148 | 0.367 | -| ckSybil | totalCap | 3796 | 3796 | 4147 | 0.000 | -| ckSybil | sfq | 462 | 1062 | 2068 | 0.690 | -| ckSybil | drr | 496 | 1081 | 2071 | 0.690 | -| ckSybil | perKeyCap+sfq | 462 | 1062 | 2068 | 0.690 | -| ckHeavyIdle | baseline | 633 | 633 | 1240 | 1.000 | -| ckHeavyIdle | perKeyCap | 1291 | 1291 | 2507 | 1.000 | -| ckHeavyIdle | totalCap | 1291 | 1291 | 2507 | 1.000 | -| ckHeavyIdle | sfq | 633 | 633 | 1240 | 1.000 | -| ckHeavyIdle | drr | 633 | 633 | 1240 | 1.000 | -| ckHeavyIdle | perKeyCap+sfq | 1291 | 1291 | 2507 | 1.000 | +env=4, per-key cap=2, total cap=2, 3 seeds. Columns: `lightWait` = the starved +key's mean wait (logical ms, the headline where it is not confounded); +`worstWait` = the largest per-group mean wait (i.e. the busiest key, which a fair +discipline deliberately makes wait its turn, so higher here is often correct); +`makespan` = logical time of the last dequeue (a work-conservation signal ONLY on +ckHeavyIdle, arrival-confounded elsewhere); `contWorstS/W` = worst contention +share over weight (directional; volume-confounded and, for per-key cap on sybil, +seed-noisy). + +| scenario | treatment | lightWait | worstWait | makespan | contWorstS/W | +| ----------- | ------------------ | --------- | --------- | -------- | ------------ | +| ckSkew | baseline | 1098 | 1261 | 2083 | 0.187 | +| ckSkew | perKeyCap | 20 | 1555 | 3038 | 0.814 | +| ckSkew | totalCap | 2840 | 2974 | 3947 | 0.213 | +| ckSkew | total+perKey | 2840 | 2974 | 3947 | 0.213 | +| ckSkew | sfq | 14 | 1069 | 2083 | 0.723 | +| ckSkew | drr | 17 | 1067 | 2083 | 0.608 | +| ckSkew | perKeyCap+sfq | 7 | 1628 | 3114 | 0.800 | +| ckSkew | total+perKey+sfq | 52 | 2363 | 3939 | 0.803 | +| ckTrickle | baseline | 1107 | 1134 | 1940 | 0.279 | +| ckTrickle | perKeyCap | 19 | 1555 | 3038 | 0.922 | +| ckTrickle | totalCap | 2852 | 2861 | 3905 | 0.279 | +| ckTrickle | total+perKey | 2852 | 2861 | 3905 | 0.279 | +| ckTrickle | sfq | 17 | 1070 | 1942 | 0.909 | +| ckTrickle | drr | 23 | 1069 | 1941 | 0.790 | +| ckTrickle | perKeyCap+sfq | 8 | 1606 | 3097 | 0.658 | +| ckTrickle | total+perKey+sfq | 106 | 2337 | 3911 | 0.883 | +| ckSybil | baseline | 1765 | 1876 | 2070 | 0.000 | +| ckSybil | perKeyCap | 1776 | 1869 | 2128 | 0.403 | +| ckSybil | totalCap | 3793 | 3801 | 4147 | 0.000 | +| ckSybil | total+perKey | 3793 | 3801 | 4147 | 0.000 | +| ckSybil | sfq | 1009 | 1019 | 2065 | 1.000 | +| ckSybil | drr | 1061 | 1068 | 2055 | 0.994 | +| ckSybil | perKeyCap+sfq | 1010 | 1019 | 2072 | 1.000 | +| ckSybil | total+perKey+sfq | 2292 | 2292 | 4146 | 1.000 | +| ckHeavyIdle | baseline | 633 | 633 | 1240 | 1.000 | +| ckHeavyIdle | perKeyCap | 1291 | 1291 | 2507 | 1.000 | +| ckHeavyIdle | totalCap | 1291 | 1291 | 2507 | 1.000 | +| ckHeavyIdle | total+perKey | 1291 | 1291 | 2507 | 1.000 | +| ckHeavyIdle | sfq | 633 | 633 | 1240 | 1.000 | +| ckHeavyIdle | drr | 633 | 633 | 1240 | 1.000 | +| ckHeavyIdle | perKeyCap+sfq | 1291 | 1291 | 2507 | 1.000 | +| ckHeavyIdle | total+perKey+sfq | 1291 | 1291 | 2507 | 1.000 | (ckHeavyIdle is a single key, so "lightWait" is the heavy key's own wait and the -contention metric is degenerate at 1.0; the signal there is makespan.) +contention metric is degenerate at 1.0; makespan is the signal there. ckSybil is +20 attacker keys plus one light key.) -## Verdicts +## What each mechanism does, at the cross-key grain -- Per-key cap (Phase 2): PROVEN for the single-heavy case, DISPROVEN for the - sybil case, and it is not work-conserving. +- Per-key cap (Phase 2). SUPPORTED for the single-heavy case; NOT a wait fix once + the tenant shards; not work-conserving. - Single heavy key (ckSkew/ckTrickle): cuts the light key's wait like a scheduler (1098 to 20, 1107 to 19) because capping the one heavy key frees - slots and the CK Lua is oldest-eligible-first, so the light key's head is - reachable. This works ONLY because the dequeue skips at-cap variants; on a - head-blocking FIFO the same cap would idle the freed slots and make wait worse. - - Sybil split (ckSybil): barely moves the light key's wait (1767 to 1718) - because the attacker's 10 keys keep env saturated with older heads and no - per-key cap bounds their aggregate. Only the fair order rescues the light key - (sfq 462, ~3.7x better than perKeyCap). Concurrency keys are client-chosen, so - this is cheap to trigger. + slots and the CK Lua serves the light head as the next eligible one. This + depends on the light head being reachable inside the Lua's 3-wide scan window; + with one at-cap variant ahead of it, it is. + - Sharded / sybil (ckSybil, 20 attacker keys): NO wait improvement (baseline + 1765, perKeyCap 1776; the difference is within the per-seed spread, baseline + 1681..1926, perKeyCap 1672..1861). Contention share nudges up (0.000 to a + mean 0.403) but that mean hides a ~10x seed swing (0.07..0.71), so it is not a + dependable improvement. The reason is structural: the sum of per-key caps is + unbounded relative to the queue when keys are client-chosen, and the 3-wide + scan window is always full of older attacker heads, so the light head is never + reached. Concurrency keys are client-chosen, so this is cheap to trigger. - Not work-conserving: throttles the capped key even with the env idle - (ckHeavyIdle makespan 1240 to 2507, 2x; ckSkew 2083 to 3038, +46%). -- Total cap (Phase 1) for cross-KEY fairness: DISPROVEN. Applied inside one task - it only lowers the whole task's ceiling and, with age order unchanged, makes the - starved key's wait worse on every contended scenario (ckSkew 1098 to 2840, - ckSybil 1767 to 3796). The total cap's real job is cross-TASK isolation - (reservation between base queues when the sum of per-task caps is below the env - limit); that is a different problem from #2617's within-task cross-key - starvation and is not exercised by this single-base-queue harness (noted as - future work). -- Scheduling (SFQ/DRR): PROVEN on every scenario including the sybil split, and - work-conserving (makespan stays at the baseline optimum 2083/1240). SFQ and DRR - track each other within noise, as in the CK spike. -- Layered per-key cap + SFQ (the Kubernetes-APF pattern): best of both on the - single-heavy case (lowest light wait AND the cap's occupancy bound), but - inherits the static cap's makespan penalty (ckSkew 3114, ckHeavyIdle 2507). On - the sybil case the cap adds nothing and SFQ does all the work (462). APF avoids - the work-conservation penalty by making the cap ELASTIC (borrow/lend seats); a - static cap cannot. + (ckHeavyIdle makespan 1240 to 2507, 2x, the cleanest single result in the + spike; ckSkew 2083 to 3038, though that scenario's makespan is partly arrival- + confounded). +- Total cap (Phase 1) at the cross-key grain: NOT a fairness lever, and the + in-task comparison is capacity-confounded. `totalCap=2` caps the whole task's + aggregate at half of env=4, so it simply halves throughput: light's wait rises + (ckSkew 1098 to 2840) for the same reason heavy's does (both now share half the + server), which is Little's-Law throughput loss, not a fairness effect. The total + cap's real purpose is cross-TASK isolation (reservation between base queues when + the sum of per-task caps is below the env limit), a different problem from + #2617's within-task cross-key starvation and one this single-base-queue harness + does not exercise (noted as future work). Do not read the "worse" numbers as + "total caps harm fairness"; read them as "wrong knob, and measured on a lower + ceiling." +- Combined total + per-key (the shipped Phase-1+2 config): in this toy the total + cap (2) is below a single per-key cap's reach, so it dominates and the per-key + cap is non-binding (`total+perKey` equals `totalCap` to the digit). This toy + therefore does not exercise the combined config's real regime (total >> per-key, + cross-task). What it does show: adding a fair order on top (`total+perKey+sfq`) + restores fair share within the throttled aggregate (contWorstS/W 0.80..1.0) but + still pays the total cap's throughput loss (makespan ~3900+). +- Scheduling (SFQ/DRR): the only knob that improves the starved key on every + scenario, and work-conserving (makespan stays at the baseline optimum + 2083/1240). On the sharded case SFQ takes the light key from fully starved to + its full fair share (contWorstS/W 0.000 to 1.000, seed-stable) and roughly + halves its wait (1765 to 1009); the residual wait is real saturation shared + fairly across 21 keys, not starvation. SFQ and DRR track each other within + noise, as in the CK spike. +- Layered per-key cap + SFQ: best light-key wait on the single-heavy case (7, 8) + and it carries the cap's occupancy bound, at the cap's makespan cost (matches or + slightly exceeds perKeyCap makespan: ckSkew 3038 to 3114). On the sharded case + the cap adds nothing and SFQ does all the work (1010, same as SFQ alone). This + is the Kubernetes-APF shape; APF avoids the work-conservation cost by making the + cap ELASTIC (borrow/lend seats), which a static cap cannot. ## Reconciliation with the earlier spike and the plan of record -- The earlier spike's recommendation (score `ckIndex` by a fair discipline) is the - general fix: it is the only mechanism here that survives the sybil split and it - is work-conserving. -- The plan of record ships caps first, and that is a defensible sequencing, not a - contradiction. A per-key cap is bounded, predictable, operator-controlled, and - self-healing (a Redis SET), and it fully fixes the common single-heavy-key case - with far less engine risk than reworking the dequeue scoring. Its limits are - real (defeated by key sharding, not work-conserving), which is exactly why the - plan calls automatic elastic fairness a later, opt-in phase. -- The honest layered conclusion (matching Kubernetes APF, SQL Server Resource - Governor, YARN, and the Parekh-Gallager result that a delay bound needs BOTH an - admission regulator AND a scheduler): keep the caps for isolation and - entitlements, and add a fair dequeue order for the contended region when - saturation and key-sharding make caps alone insufficient. Not either/or. +Measured here: caps and scheduling fix different things and can be layered +(the per-key-cap+SFQ and total+perKey+sfq rows). Fair scheduling is the only +mechanism in this harness that improves the starved key on the sharded case and +stays work-conserving, which is what the earlier spike recommended (score +`ckIndex` by virtual time). + +Interpretation, NOT measured by this benchmark (it measures wait/makespan/share on +a simulation, not engineering cost or rollout risk): shipping the caps first still +reads as defensible. A per-key cap is bounded, operator-controlled, self-healing +(a Redis SET), and it fully fixes the common single-heavy-key case, which is a +smaller engine change than reworking the dequeue scoring. Its limits are real +(no help once a tenant shards its keys, not work-conserving), which is the case +for treating automatic fair scheduling as a later phase rather than never. + +The layered end state matches Kubernetes APF, SQL Server Resource Governor, YARN, +and the Parekh-Gallager result that a worst-case delay bound needs BOTH an +admission regulator AND a scheduler: keep the caps for isolation and entitlements, +add a fair dequeue order for the contended region when saturation and key-sharding +make caps alone insufficient. Not either/or. ## Caveats -- Relative ranking only; single shard, single base queue, single sequential - consumer; simulated holds on a logical clock; 3 seeds; equal weights. -- `maxCount = 1` (see fidelity note); the total cap is driver-modelled, not the - real (unbuilt) group gate. +- Relative ranking on a simulation; single shard, single base queue, single + sequential consumer; simulated holds on a logical clock; 3 seeds; equal weights. + The verdict words ("supported", "not a wait fix") are relative to this harness. +- `maxCount = 1` and the `*3` scan window (see fidelity section); the total cap is + driver-modelled, not the real (unbuilt) group gate. - Per-key cap is modelled uniformly (real per-queue gate); a per-key-specific Phase-2 override is equivalent here only because the light key never approaches the cap. +- makespan is the last dequeue, not completion, and is arrival-confounded on the + poisson scenarios; trust it only on ckHeavyIdle. +- Contention share is volume-confounded for low-volume keys, and for the per-key + cap on the sharded case it is seed-noisy (0.07..0.71); wait is the trustworthy + signal, share is directional. - Cross-task isolation (the total cap's real purpose) is argued from the research, not measured; a multi-base-queue harness is future work. -- Wait is the trustworthy signal; contention share is volume-confounded for - low-volume keys (same caveat as the CK spike). diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/capsFairness.bench.test.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/capsFairness.bench.test.ts index 99e1cab180..0521dc7ded 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/capsFairness.bench.test.ts +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/capsFairness.bench.test.ts @@ -46,9 +46,13 @@ const TREATMENTS: Treatment[] = [ { label: "baseline", makeDiscipline: () => new BaselineCk() }, { label: "perKeyCap", makeDiscipline: () => new BaselineCk(), perKeyCap: PER_KEY_CAP }, { label: "totalCap", makeDiscipline: () => new BaselineCk(), totalCap: TOTAL_CAP }, + // the plan-of-record's shipped combined config: total cap AND per-key cap + { label: "total+perKey", makeDiscipline: () => new BaselineCk(), perKeyCap: PER_KEY_CAP, totalCap: TOTAL_CAP }, { label: "sfq", makeDiscipline: () => new SfqCk() }, { label: "drr", makeDiscipline: () => new DrrCk() }, { label: "perKeyCap+sfq", makeDiscipline: () => new SfqCk(), perKeyCap: PER_KEY_CAP }, + // both caps plus a fair order (the fully-layered end state) + { label: "total+perKey+sfq", makeDiscipline: () => new SfqCk(), perKeyCap: PER_KEY_CAP, totalCap: TOTAL_CAP }, ]; type CapScenario = { @@ -95,18 +99,20 @@ const SCENARIOS: Record = { }, }, - // sybil split: one attacker spreads its backlog across 10 concurrency keys, each - // with a large backlog that stays non-empty through the light key's whole - // arrival window. Each attacker key is under the same per-key cap, but the cap - // frees no aggregate slot (2 attacker keys fill env, and as one empties the next - // attacker key's old head is served before the newer light key). Only a fair - // order rescues the light key. + // sybil split: one attacker spreads its backlog across 20 concurrency keys, each + // with a backlog that stays non-empty through the light key's whole arrival + // window. Each attacker key is under the same per-key cap, but the cap frees no + // aggregate slot (attacker keys fill env, and as one empties the next attacker + // key's old head is served before the newer light key). The real CK Lua also + // only scans the 3 oldest-scored variants per call (ZRANGEBYSCORE ... LIMIT 0, + // maxCount*3), so with 20 attacker heads ahead of it the light head is never in + // the window. Only a fair order rescues the light key. ckSybil: { lightKey: "light", config: { envConcurrencyLimit: ENV_LIMIT, tenants: [ - ...sybilHeavy(10, 30), + ...sybilHeavy(20, 15), { tenantId: "light", runCount: 20, arrival: "poisson", ratePerSec: 40, holdMsMean: 25 }, ], }, diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts index 18277ccb33..cc89f62b8b 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts @@ -151,6 +151,12 @@ export async function runCkScenario(config: CkDriverConfig): Promise if (config.discipline.rescore) { const active = await reader.readActiveCks(baseQueue); if (active.length > 0) { + // NOTE: order() runs before every dequeue attempt, including the + // terminal iteration whose dequeue returns nothing, so it can advance + // a discipline's state with no matching onServiced. For the shipped + // SFQ/DRR this is idempotent (SFQ floor already at the min clock; DRR's + // winner already has deficit >= 1). A non-idempotent discipline dropped + // in here would need its accounting made robust to that speculative call. const order = config.discipline.order(active, scoreBase + t); await rescoreCkIndex(admin, keys, baseQueue, order, Date.now()); } diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckHeavyIdle.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckHeavyIdle.json index 14c6d06dcc..a03f6eea06 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckHeavyIdle.json +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckHeavyIdle.json @@ -121,6 +121,42 @@ } ] }, + { + "treatment": "total+perKey", + "lightWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "worstWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "makespan": { + "mean": 2507, + "min": 2344, + "max": 2638 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 200, + "weight": 1, + "share": 1, + "contentionShareOverWeight": 0, + "meanWait": 1431.305, + "waitP50": 1501, + "waitP99": 2623, + "waitMax": 2638 + } + ] + }, { "treatment": "sfq", "lightWait": { @@ -228,6 +264,42 @@ "waitMax": 2638 } ] + }, + { + "treatment": "total+perKey+sfq", + "lightWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "worstWait": { + "mean": 1290.6516666666669, + "min": 1173.15, + "max": 1431.305 + }, + "makespan": { + "mean": 2507, + "min": 2344, + "max": 2638 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 200, + "weight": 1, + "share": 1, + "contentionShareOverWeight": 0, + "meanWait": 1431.305, + "waitP50": 1501, + "waitP99": 2623, + "waitMax": 2638 + } + ] } ] } \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSkew.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSkew.json index ee566536b7..310e248fd5 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSkew.json +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSkew.json @@ -257,6 +257,86 @@ } ] }, + { + "treatment": "total+perKey", + "lightWait": { + "mean": 2840.2000000000003, + "min": 2670.3333333333335, + "max": 3138.4666666666667 + }, + "worstWait": { + "mean": 2973.6222222222223, + "min": 2766.9333333333334, + "max": 3138.4666666666667 + }, + "makespan": { + "mean": 3947.3333333333335, + "min": 3532, + "max": 4290 + }, + "contentionWorst": { + "mean": 0.21268177618484008, + "min": 0.1858108108108108, + "max": 0.23411371237458192 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 4.013377926421405, + "meanWait": 1770.1916666666666, + "waitP50": 1765, + "waitP99": 3293, + "waitMax": 3309 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2508361204013378, + "meanWait": 3138.4666666666667, + "waitP50": 3102, + "waitP99": 3367, + "waitMax": 3367 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.23411371237458192, + "meanWait": 2885.3333333333335, + "waitP50": 2913, + "waitP99": 3380, + "waitMax": 3380 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2508361204013378, + "meanWait": 2674, + "waitP50": 2769, + "waitP99": 3374, + "waitMax": 3374 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2508361204013378, + "meanWait": 2852.0666666666666, + "waitP50": 2841, + "waitP99": 3400, + "waitMax": 3400 + } + ] + }, { "treatment": "sfq", "lightWait": { @@ -496,6 +576,86 @@ "waitMax": 44 } ] + }, + { + "treatment": "total+perKey+sfq", + "lightWait": { + "mean": 51.888888888888886, + "min": 45.06666666666667, + "max": 55.4 + }, + "worstWait": { + "mean": 2363.016666666667, + "min": 2045.6083333333333, + "max": 2573.3708333333334 + }, + "makespan": { + "mean": 3939.3333333333335, + "min": 3564, + "max": 4238 + }, + "contentionWorst": { + "mean": 0.8028035775713794, + "min": 0.588235294117647, + "max": 0.9868421052631579 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 0.588235294117647, + "meanWait": 2573.3708333333334, + "waitP50": 2694, + "waitP99": 4222, + "waitMax": 4238 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1.102941176470588, + "meanWait": 55.2, + "waitP50": 32, + "waitP99": 149, + "waitMax": 149 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1.102941176470588, + "meanWait": 23.333333333333332, + "waitP50": 18, + "waitP99": 55, + "waitMax": 55 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1.102941176470588, + "meanWait": 14.4, + "waitP50": 11, + "waitP99": 44, + "waitMax": 44 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1.102941176470588, + "meanWait": 21.6, + "waitP50": 17, + "waitP99": 65, + "waitMax": 65 + } + ] } ] } \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSybil.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSybil.json index 458a0f33b2..2fb2f04448 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSybil.json +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckSybil.json @@ -20,25 +20,35 @@ "heavy-7": 1, "heavy-8": 1, "heavy-9": 1, + "heavy-10": 1, + "heavy-11": 1, + "heavy-12": 1, + "heavy-13": 1, + "heavy-14": 1, + "heavy-15": 1, + "heavy-16": 1, + "heavy-17": 1, + "heavy-18": 1, + "heavy-19": 1, "light": 1 }, "perTreatment": [ { "treatment": "baseline", "lightWait": { - "mean": 1767.1833333333334, - "min": 1681.25, - "max": 1926.3 + "mean": 1764.7666666666664, + "min": 1679.1, + "max": 1925.8 }, "worstWait": { - "mean": 1823.0333333333335, - "min": 1717.5, - "max": 1956.5666666666666 + "mean": 1876.1333333333332, + "min": 1778.4, + "max": 2046.3333333333333 }, "makespan": { - "mean": 2067, - "min": 1891, - "max": 2296 + "mean": 2070, + "min": 1893, + "max": 2302 }, "contentionWorst": { "mean": 0, @@ -48,113 +58,223 @@ "detailSeed0": [ { "groupId": "heavy-0", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 112.16666666666667, - "waitP50": 103, - "waitP99": 252, - "waitMax": 252 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 53.93333333333333, + "waitP50": 44, + "waitP99": 128, + "waitMax": 128 }, { "groupId": "heavy-1", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 319.4, - "waitP50": 321, - "waitP99": 395, - "waitMax": 395 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 167.93333333333334, + "waitP50": 163, + "waitP99": 205, + "waitMax": 205 }, { "groupId": "heavy-2", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 491.7, - "waitP50": 497, - "waitP99": 565, - "waitMax": 565 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1408.7333333333333, + "waitP50": 1401, + "waitP99": 1446, + "waitMax": 1446 }, { "groupId": "heavy-3", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 673.7, - "waitP50": 670, - "waitP99": 760, - "waitMax": 760 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1506.2, + "waitP50": 1509, + "waitP99": 1541, + "waitMax": 1541 }, { "groupId": "heavy-4", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 885.6333333333333, - "waitP50": 893, - "waitP99": 1005, - "waitMax": 1005 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1582.4, + "waitP50": 1582, + "waitP99": 1626, + "waitMax": 1626 }, { "groupId": "heavy-5", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 1104.0333333333333, - "waitP50": 1105, - "waitP99": 1183, - "waitMax": 1183 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1668.6666666666667, + "waitP50": 1668, + "waitP99": 1708, + "waitMax": 1708 }, { "groupId": "heavy-6", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 1288.5, - "waitP50": 1284, - "waitP99": 1378, - "waitMax": 1378 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1749.9333333333334, + "waitP50": 1755, + "waitP99": 1786, + "waitMax": 1786 }, { "groupId": "heavy-7", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 1532.5333333333333, - "waitP50": 1538, - "waitP99": 1653, - "waitMax": 1653 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1847.4666666666667, + "waitP50": 1852, + "waitP99": 1899, + "waitMax": 1899 }, { "groupId": "heavy-8", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 1734.2666666666667, - "waitP50": 1731, - "waitP99": 1811, - "waitMax": 1811 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1953.3333333333333, + "waitP50": 1947, + "waitP99": 1991, + "waitMax": 1991 }, { "groupId": "heavy-9", - "dequeued": 30, + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2046.3333333333333, + "waitP50": 2039, + "waitP99": 2104, + "waitMax": 2104 + }, + { + "groupId": "heavy-10", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 286.4, + "waitP50": 270, + "waitP99": 353, + "waitMax": 353 + }, + { + "groupId": "heavy-11", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 394.53333333333336, + "waitP50": 398, + "waitP99": 437, + "waitMax": 437 + }, + { + "groupId": "heavy-12", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 475.8666666666667, + "waitP50": 478, + "waitP99": 519, + "waitMax": 519 + }, + { + "groupId": "heavy-13", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 597.8666666666667, + "waitP50": 599, + "waitP99": 653, + "waitMax": 653 + }, + { + "groupId": "heavy-14", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 697.4666666666667, + "waitP50": 697, + "waitP99": 749, + "waitMax": 749 + }, + { + "groupId": "heavy-15", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 818.3333333333334, + "waitP50": 799, + "waitP99": 915, + "waitMax": 915 + }, + { + "groupId": "heavy-16", + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 1956.5666666666666, - "waitP50": 1954, - "waitP99": 2108, - "waitMax": 2108 + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 952.8666666666667, + "waitP50": 950, + "waitP99": 989, + "waitMax": 989 + }, + { + "groupId": "heavy-17", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1054.4, + "waitP50": 1058, + "waitP99": 1084, + "waitMax": 1084 + }, + { + "groupId": "heavy-18", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1123.2, + "waitP50": 1111, + "waitP99": 1180, + "waitMax": 1180 + }, + { + "groupId": "heavy-19", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1274.3333333333333, + "waitP50": 1248, + "waitP99": 1379, + "waitMax": 1379 }, { "groupId": "light", @@ -162,175 +282,285 @@ "weight": 1, "share": 0.0625, "contentionShareOverWeight": 0, - "meanWait": 1926.3, - "waitP50": 1872, - "waitP99": 2127, - "waitMax": 2127 + "meanWait": 1925.8, + "waitP50": 1869, + "waitP99": 2133, + "waitMax": 2133 } ] }, { "treatment": "perKeyCap", "lightWait": { - "mean": 1718.3833333333332, - "min": 1671.6, - "max": 1749.3 + "mean": 1776.05, + "min": 1691.5, + "max": 1891.35 }, "worstWait": { - "mean": 1830.788888888889, - "min": 1674.1333333333334, - "max": 2068.9333333333334 + "mean": 1869.3555555555556, + "min": 1753.2, + "max": 2095.5333333333333 }, "makespan": { - "mean": 2148.3333333333335, - "min": 1911, - "max": 2406 + "mean": 2128.3333333333335, + "min": 1931, + "max": 2330 }, "contentionWorst": { - "mean": 0.36687717667535047, - "min": 0.0728476821192053, - "max": 0.7073954983922829 + "mean": 0.40264026402640263, + "min": 0, + "max": 1 }, "detailSeed0": [ { "groupId": "heavy-0", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 264.96666666666664, - "waitP50": 256, - "waitP99": 531, - "waitMax": 531 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 135.8, + "waitP50": 124, + "waitP99": 287, + "waitMax": 287 }, { "groupId": "heavy-1", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 119.06666666666666, - "waitP50": 114, - "waitP99": 297, - "waitMax": 297 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 70.6, + "waitP50": 57, + "waitP99": 209, + "waitMax": 209 }, { "groupId": "heavy-2", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 465.8666666666667, - "waitP50": 458, - "waitP99": 610, - "waitMax": 610 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1381.5333333333333, + "waitP50": 1371, + "waitP99": 1470, + "waitMax": 1470 }, { "groupId": "heavy-3", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 721, - "waitP50": 719, - "waitP99": 885, - "waitMax": 885 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1533.1333333333334, + "waitP50": 1529, + "waitP99": 1590, + "waitMax": 1590 }, { "groupId": "heavy-4", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 863.9333333333333, - "waitP50": 853, - "waitP99": 1089, - "waitMax": 1089 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1560.8, + "waitP50": 1551, + "waitP99": 1643, + "waitMax": 1643 }, { "groupId": "heavy-5", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 1120.2, - "waitP50": 1115, - "waitP99": 1285, - "waitMax": 1285 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1680.6666666666667, + "waitP50": 1675, + "waitP99": 1752, + "waitMax": 1752 }, { "groupId": "heavy-6", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 1319.8666666666666, - "waitP50": 1319, - "waitP99": 1537, - "waitMax": 1537 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1745.1333333333334, + "waitP50": 1767, + "waitP99": 1813, + "waitMax": 1813 }, { "groupId": "heavy-7", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 1536.7333333333333, - "waitP50": 1577, - "waitP99": 1788, - "waitMax": 1788 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1858.8, + "waitP50": 1863, + "waitP99": 1942, + "waitMax": 1942 }, { "groupId": "heavy-8", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0610932475884243, - "meanWait": 1688.3, - "waitP50": 1697, - "waitP99": 1858, - "waitMax": 1858 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1936.7333333333333, + "waitP50": 1929, + "waitP99": 2018, + "waitMax": 2018 }, { "groupId": "heavy-9", - "dequeued": 30, + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2095.5333333333333, + "waitP50": 2085, + "waitP99": 2257, + "waitMax": 2257 + }, + { + "groupId": "heavy-10", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 328.4, + "waitP50": 316, + "waitP99": 427, + "waitMax": 427 + }, + { + "groupId": "heavy-11", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 371.4, + "waitP50": 371, + "waitP99": 441, + "waitMax": 441 + }, + { + "groupId": "heavy-12", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 530, + "waitP50": 506, + "waitP99": 633, + "waitMax": 633 + }, + { + "groupId": "heavy-13", + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 0.7427652733118971, - "meanWait": 2068.9333333333334, - "waitP50": 2038, - "waitP99": 2406, - "waitMax": 2406 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 570.8, + "waitP50": 564, + "waitP99": 678, + "waitMax": 678 + }, + { + "groupId": "heavy-14", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 725.6, + "waitP50": 711, + "waitP99": 837, + "waitMax": 837 + }, + { + "groupId": "heavy-15", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 811.6, + "waitP50": 768, + "waitP99": 987, + "waitMax": 987 + }, + { + "groupId": "heavy-16", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 937.2666666666667, + "waitP50": 932, + "waitP99": 1063, + "waitMax": 1063 + }, + { + "groupId": "heavy-17", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1052.8666666666666, + "waitP50": 1065, + "waitP99": 1098, + "waitMax": 1098 + }, + { + "groupId": "heavy-18", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1176.5333333333333, + "waitP50": 1172, + "waitP99": 1315, + "waitMax": 1315 + }, + { + "groupId": "heavy-19", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1239.5333333333333, + "waitP50": 1234, + "waitP99": 1423, + "waitMax": 1423 }, { "groupId": "light", "dequeued": 20, "weight": 1, "share": 0.0625, - "contentionShareOverWeight": 0.7073954983922829, - "meanWait": 1734.25, - "waitP50": 1693, - "waitP99": 1867, - "waitMax": 1867 + "contentionShareOverWeight": 1, + "meanWait": 1891.35, + "waitP50": 1850, + "waitP99": 2031, + "waitMax": 2031 } ] }, { "treatment": "totalCap", "lightWait": { - "mean": 3795.7999999999997, - "min": 3553.5, - "max": 4165.45 + "mean": 3793.066666666667, + "min": 3551.4, + "max": 4164.25 }, "worstWait": { - "mean": 3795.7999999999997, - "min": 3553.5, - "max": 4165.45 + "mean": 3800.5333333333333, + "min": 3573.8, + "max": 4164.25 }, "makespan": { - "mean": 4147.333333333333, - "min": 3793, - "max": 4605 + "mean": 4146.666666666667, + "min": 3790, + "max": 4604 }, "contentionWorst": { "mean": 0, @@ -340,113 +570,479 @@ "detailSeed0": [ { "groupId": "heavy-0", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 264.96666666666664, - "waitP50": 256, - "waitP99": 531, - "waitMax": 531 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 135.8, + "waitP50": 124, + "waitP99": 287, + "waitMax": 287 }, { "groupId": "heavy-1", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 658.4666666666667, - "waitP50": 650, - "waitP99": 833, - "waitMax": 833 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 366.6, + "waitP50": 350, + "waitP99": 507, + "waitMax": 507 }, { "groupId": "heavy-2", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 1004.7, - "waitP50": 994, - "waitP99": 1146, - "waitMax": 1146 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2846.6, + "waitP50": 2832, + "waitP99": 2931, + "waitMax": 2931 }, { "groupId": "heavy-3", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 1365.9666666666667, - "waitP50": 1366, - "waitP99": 1531, - "waitMax": 1531 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3029, + "waitP50": 3026, + "waitP99": 3087, + "waitMax": 3087 }, { "groupId": "heavy-4", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 1805.8666666666666, - "waitP50": 1801, - "waitP99": 2028, - "waitMax": 2028 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3188.133333333333, + "waitP50": 3177, + "waitP99": 3270, + "waitMax": 3270 }, { "groupId": "heavy-5", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 2228.5333333333333, - "waitP50": 2215, - "waitP99": 2391, - "waitMax": 2391 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3355.866666666667, + "waitP50": 3354, + "waitP99": 3427, + "waitMax": 3427 }, { "groupId": "heavy-6", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 2611.9, - "waitP50": 2612, - "waitP99": 2830, - "waitMax": 2830 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3520.0666666666666, + "waitP50": 3531, + "waitP99": 3587, + "waitMax": 3587 }, { "groupId": "heavy-7", - "dequeued": 30, - "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 3094.9333333333334, - "waitP50": 3140, - "waitP99": 3347, - "waitMax": 3347 + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3725.133333333333, + "waitP50": 3734, + "waitP99": 3813, + "waitMax": 3813 }, { "groupId": "heavy-8", - "dequeued": 30, + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3925.6, + "waitP50": 3912, + "waitP99": 4000, + "waitMax": 4000 + }, + { + "groupId": "heavy-9", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 4134.2, + "waitP50": 4129, + "waitP99": 4297, + "waitMax": 4297 + }, + { + "groupId": "heavy-10", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 624.0666666666667, + "waitP50": 614, + "waitP99": 721, + "waitMax": 721 + }, + { + "groupId": "heavy-11", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 803.6, + "waitP50": 803, + "waitP99": 873, + "waitMax": 873 + }, + { + "groupId": "heavy-12", + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 3496.6666666666665, - "waitP50": 3507, - "waitP99": 3666, - "waitMax": 3666 + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 983.0666666666667, + "waitP50": 956, + "waitP99": 1085, + "waitMax": 1085 + }, + { + "groupId": "heavy-13", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1222.3333333333333, + "waitP50": 1215, + "waitP99": 1329, + "waitMax": 1329 + }, + { + "groupId": "heavy-14", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1413.8666666666666, + "waitP50": 1394, + "waitP99": 1528, + "waitMax": 1528 + }, + { + "groupId": "heavy-15", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1673, + "waitP50": 1632, + "waitP99": 1848, + "waitMax": 1848 + }, + { + "groupId": "heavy-16", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1935.8, + "waitP50": 1932, + "waitP99": 2060, + "waitMax": 2060 + }, + { + "groupId": "heavy-17", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2133.266666666667, + "waitP50": 2143, + "waitP99": 2180, + "waitMax": 2180 + }, + { + "groupId": "heavy-18", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2276.6, + "waitP50": 2273, + "waitP99": 2414, + "waitMax": 2414 + }, + { + "groupId": "heavy-19", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2580.866666666667, + "waitP50": 2568, + "waitP99": 2759, + "waitMax": 2759 + }, + { + "groupId": "light", + "dequeued": 20, + "weight": 1, + "share": 0.0625, + "contentionShareOverWeight": 0, + "meanWait": 4164.25, + "waitP50": 4123, + "waitP99": 4297, + "waitMax": 4297 + } + ] + }, + { + "treatment": "total+perKey", + "lightWait": { + "mean": 3793.066666666667, + "min": 3551.4, + "max": 4164.25 + }, + "worstWait": { + "mean": 3800.5333333333333, + "min": 3573.8, + "max": 4164.25 + }, + "makespan": { + "mean": 4146.666666666667, + "min": 3790, + "max": 4604 + }, + "contentionWorst": { + "mean": 0, + "min": 0, + "max": 0 + }, + "detailSeed0": [ + { + "groupId": "heavy-0", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 135.8, + "waitP50": 124, + "waitP99": 287, + "waitMax": 287 + }, + { + "groupId": "heavy-1", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 366.6, + "waitP50": 350, + "waitP99": 507, + "waitMax": 507 + }, + { + "groupId": "heavy-2", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2846.6, + "waitP50": 2832, + "waitP99": 2931, + "waitMax": 2931 + }, + { + "groupId": "heavy-3", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3029, + "waitP50": 3026, + "waitP99": 3087, + "waitMax": 3087 + }, + { + "groupId": "heavy-4", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3188.133333333333, + "waitP50": 3177, + "waitP99": 3270, + "waitMax": 3270 + }, + { + "groupId": "heavy-5", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3355.866666666667, + "waitP50": 3354, + "waitP99": 3427, + "waitMax": 3427 + }, + { + "groupId": "heavy-6", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3520.0666666666666, + "waitP50": 3531, + "waitP99": 3587, + "waitMax": 3587 + }, + { + "groupId": "heavy-7", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3725.133333333333, + "waitP50": 3734, + "waitP99": 3813, + "waitMax": 3813 + }, + { + "groupId": "heavy-8", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 3925.6, + "waitP50": 3912, + "waitP99": 4000, + "waitMax": 4000 }, { "groupId": "heavy-9", - "dequeued": 30, + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 4134.2, + "waitP50": 4129, + "waitP99": 4297, + "waitMax": 4297 + }, + { + "groupId": "heavy-10", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 624.0666666666667, + "waitP50": 614, + "waitP99": 721, + "waitMax": 721 + }, + { + "groupId": "heavy-11", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 803.6, + "waitP50": 803, + "waitP99": 873, + "waitMax": 873 + }, + { + "groupId": "heavy-12", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 983.0666666666667, + "waitP50": 956, + "waitP99": 1085, + "waitMax": 1085 + }, + { + "groupId": "heavy-13", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1222.3333333333333, + "waitP50": 1215, + "waitP99": 1329, + "waitMax": 1329 + }, + { + "groupId": "heavy-14", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1413.8666666666666, + "waitP50": 1394, + "waitP99": 1528, + "waitMax": 1528 + }, + { + "groupId": "heavy-15", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1673, + "waitP50": 1632, + "waitP99": 1848, + "waitMax": 1848 + }, + { + "groupId": "heavy-16", + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.1, - "meanWait": 3950.733333333333, - "waitP50": 3922, - "waitP99": 4288, - "waitMax": 4288 + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 1935.8, + "waitP50": 1932, + "waitP99": 2060, + "waitMax": 2060 + }, + { + "groupId": "heavy-17", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2133.266666666667, + "waitP50": 2143, + "waitP99": 2180, + "waitMax": 2180 + }, + { + "groupId": "heavy-18", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2276.6, + "waitP50": 2273, + "waitP99": 2414, + "waitMax": 2414 + }, + { + "groupId": "heavy-19", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1.05, + "meanWait": 2580.866666666667, + "waitP50": 2568, + "waitP99": 2759, + "waitMax": 2759 }, { "groupId": "light", @@ -454,448 +1050,1034 @@ "weight": 1, "share": 0.0625, "contentionShareOverWeight": 0, - "meanWait": 4165.45, - "waitP50": 4122, - "waitP99": 4307, - "waitMax": 4307 + "meanWait": 4164.25, + "waitP50": 4123, + "waitP99": 4297, + "waitMax": 4297 } ] }, { "treatment": "sfq", "lightWait": { - "mean": 462.3833333333334, - "min": 438.85, - "max": 489.6 + "mean": 1009.25, + "min": 957.95, + "max": 1103.7 }, "worstWait": { - "mean": 1061.6888888888889, - "min": 952.7, - "max": 1178.3 + "mean": 1018.9888888888889, + "min": 966.1, + "max": 1117.4 }, "makespan": { - "mean": 2067.6666666666665, - "min": 1877, - "max": 2295 + "mean": 2064.6666666666665, + "min": 1889, + "max": 2297 }, "contentionWorst": { - "mean": 0.689655172413793, - "min": 0.689655172413793, - "max": 0.689655172413793 + "mean": 1, + "min": 1, + "max": 1 }, "detailSeed0": [ { "groupId": "heavy-0", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1173.3, - "waitP50": 1209, - "waitP99": 2263, - "waitMax": 2263 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1095.7333333333333, + "waitP50": 1009, + "waitP99": 2202, + "waitMax": 2202 }, { "groupId": "heavy-1", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1175.1, - "waitP50": 1241, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1105.4, + "waitP50": 1110, "waitP99": 2247, "waitMax": 2247 }, { "groupId": "heavy-2", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1172.5666666666666, - "waitP50": 1224, - "waitP99": 2283, - "waitMax": 2283 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1113.1333333333334, + "waitP50": 1007, + "waitP99": 2145, + "waitMax": 2145 }, { "groupId": "heavy-3", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1178.3, - "waitP50": 1240, - "waitP99": 2271, - "waitMax": 2271 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1114.3333333333333, + "waitP50": 1065, + "waitP99": 2163, + "waitMax": 2163 }, { "groupId": "heavy-4", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1171.1333333333334, - "waitP50": 1236, - "waitP99": 2264, - "waitMax": 2264 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1108.4, + "waitP50": 1033, + "waitP99": 2133, + "waitMax": 2133 }, { "groupId": "heavy-5", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1173.9666666666667, - "waitP50": 1255, - "waitP99": 2245, - "waitMax": 2245 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1112.6666666666667, + "waitP50": 1106, + "waitP99": 2153, + "waitMax": 2153 }, { "groupId": "heavy-6", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1176.1666666666667, - "waitP50": 1167, - "waitP99": 2291, - "waitMax": 2291 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1117.4, + "waitP50": 1030, + "waitP99": 2186, + "waitMax": 2186 }, { "groupId": "heavy-7", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, + "share": 0.046875, "contentionShareOverWeight": 1, - "meanWait": 1169.2, - "waitP50": 1173, - "waitP99": 2295, - "waitMax": 2295 + "meanWait": 1114.2, + "waitP50": 1009, + "waitP99": 2257, + "waitMax": 2257 }, { "groupId": "heavy-8", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1171.2, - "waitP50": 1165, - "waitP99": 2286, - "waitMax": 2286 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1114.3333333333333, + "waitP50": 1032, + "waitP99": 2220, + "waitMax": 2220 }, { "groupId": "heavy-9", - "dequeued": 30, + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1112.6666666666667, + "waitP50": 1046, + "waitP99": 2239, + "waitMax": 2239 + }, + { + "groupId": "heavy-10", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1101.4, + "waitP50": 1097, + "waitP99": 2142, + "waitMax": 2142 + }, + { + "groupId": "heavy-11", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1101.4, + "waitP50": 1126, + "waitP99": 2131, + "waitMax": 2131 + }, + { + "groupId": "heavy-12", + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1171.2666666666667, - "waitP50": 1218, - "waitP99": 2292, - "waitMax": 2292 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1110.4, + "waitP50": 1140, + "waitP99": 2203, + "waitMax": 2203 + }, + { + "groupId": "heavy-13", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1107.4, + "waitP50": 1004, + "waitP99": 2239, + "waitMax": 2239 + }, + { + "groupId": "heavy-14", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1106.6, + "waitP50": 1124, + "waitP99": 2153, + "waitMax": 2153 + }, + { + "groupId": "heavy-15", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1105.6666666666667, + "waitP50": 1096, + "waitP99": 2203, + "waitMax": 2203 + }, + { + "groupId": "heavy-16", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1109.8666666666666, + "waitP50": 1109, + "waitP99": 2144, + "waitMax": 2144 + }, + { + "groupId": "heavy-17", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1116.3333333333333, + "waitP50": 1084, + "waitP99": 2201, + "waitMax": 2201 + }, + { + "groupId": "heavy-18", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1105.4, + "waitP50": 1112, + "waitP99": 2220, + "waitMax": 2220 + }, + { + "groupId": "heavy-19", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1101.5333333333333, + "waitP50": 1035, + "waitP99": 2219, + "waitMax": 2219 }, { "groupId": "light", "dequeued": 20, "weight": 1, "share": 0.0625, - "contentionShareOverWeight": 0.689655172413793, - "meanWait": 489.6, - "waitP50": 386, - "waitP99": 1026, - "waitMax": 1026 + "contentionShareOverWeight": 1, + "meanWait": 1103.7, + "waitP50": 1204, + "waitP99": 1832, + "waitMax": 1832 } ] }, { "treatment": "drr", "lightWait": { - "mean": 495.8833333333334, - "min": 471.3, - "max": 528.95 + "mean": 1061.3999999999999, + "min": 1008.8, + "max": 1163.15 }, "worstWait": { - "mean": 1081.4777777777779, - "min": 972.5666666666667, - "max": 1198 + "mean": 1068.1499999999999, + "min": 1012.25, + "max": 1173.4666666666667 }, "makespan": { - "mean": 2071.3333333333335, - "min": 1883, + "mean": 2055, + "min": 1864, "max": 2300 }, "contentionWorst": { - "mean": 0.689655172413793, - "min": 0.689655172413793, - "max": 0.689655172413793 + "mean": 0.9936908517350158, + "min": 0.9936908517350158, + "max": 0.9936908517350158 }, "detailSeed0": [ { "groupId": "heavy-0", - "dequeued": 30, + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1037.4, + "waitP50": 990, + "waitP99": 2147, + "waitMax": 2147 + }, + { + "groupId": "heavy-1", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1046.0666666666666, + "waitP50": 991, + "waitP99": 2189, + "waitMax": 2189 + }, + { + "groupId": "heavy-2", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1123, + "waitP50": 1095, + "waitP99": 2162, + "waitMax": 2162 + }, + { + "groupId": "heavy-3", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1133.2666666666667, + "waitP50": 1101, + "waitP99": 2211, + "waitMax": 2211 + }, + { + "groupId": "heavy-4", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1138.1333333333334, + "waitP50": 1106, + "waitP99": 2180, + "waitMax": 2180 + }, + { + "groupId": "heavy-5", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1151.8, + "waitP50": 1121, + "waitP99": 2270, + "waitMax": 2270 + }, + { + "groupId": "heavy-6", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1149.8666666666666, + "waitP50": 1124, + "waitP99": 2184, + "waitMax": 2184 + }, + { + "groupId": "heavy-7", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1158.3333333333333, + "waitP50": 1129, + "waitP99": 2213, + "waitMax": 2213 + }, + { + "groupId": "heavy-8", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1163.2666666666667, + "waitP50": 1133, + "waitP99": 2186, + "waitMax": 2186 + }, + { + "groupId": "heavy-9", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1173.4666666666667, + "waitP50": 1141, + "waitP99": 2242, + "waitMax": 2242 + }, + { + "groupId": "heavy-10", + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1132.0333333333333, - "waitP50": 1159, - "waitP99": 2239, - "waitMax": 2239 + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1048, + "waitP50": 993, + "waitP99": 2148, + "waitMax": 2148 + }, + { + "groupId": "heavy-11", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1063.4666666666667, + "waitP50": 1012, + "waitP99": 2220, + "waitMax": 2220 + }, + { + "groupId": "heavy-12", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1061.8, + "waitP50": 1027, + "waitP99": 2150, + "waitMax": 2150 + }, + { + "groupId": "heavy-13", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1073.7333333333333, + "waitP50": 1042, + "waitP99": 2199, + "waitMax": 2199 + }, + { + "groupId": "heavy-14", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1079.1333333333334, + "waitP50": 1047, + "waitP99": 2151, + "waitMax": 2151 + }, + { + "groupId": "heavy-15", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1092.6, + "waitP50": 1059, + "waitP99": 2253, + "waitMax": 2253 + }, + { + "groupId": "heavy-16", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1091, + "waitP50": 1063, + "waitP99": 2153, + "waitMax": 2153 + }, + { + "groupId": "heavy-17", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1105.3333333333333, + "waitP50": 1069, + "waitP99": 2210, + "waitMax": 2210 + }, + { + "groupId": "heavy-18", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1107.6, + "waitP50": 1087, + "waitP99": 2161, + "waitMax": 2161 + }, + { + "groupId": "heavy-19", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 0.9936908517350158, + "meanWait": 1119.1333333333334, + "waitP50": 1092, + "waitP99": 2237, + "waitMax": 2237 + }, + { + "groupId": "light", + "dequeued": 20, + "weight": 1, + "share": 0.0625, + "contentionShareOverWeight": 1.1261829652996846, + "meanWait": 1163.15, + "waitP50": 1280, + "waitP99": 1826, + "waitMax": 1826 + } + ] + }, + { + "treatment": "perKeyCap+sfq", + "lightWait": { + "mean": 1010.0166666666665, + "min": 957.95, + "max": 1106 + }, + "worstWait": { + "mean": 1018.9888888888889, + "min": 966.1, + "max": 1117.4 + }, + "makespan": { + "mean": 2072, + "min": 1889, + "max": 2319 + }, + "contentionWorst": { + "mean": 1, + "min": 1, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy-0", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1095.7333333333333, + "waitP50": 1009, + "waitP99": 2202, + "waitMax": 2202 }, { "groupId": "heavy-1", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1141.4333333333334, - "waitP50": 1163, - "waitP99": 2281, - "waitMax": 2281 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1105.4, + "waitP50": 1110, + "waitP99": 2247, + "waitMax": 2247 }, { "groupId": "heavy-2", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1146.4, - "waitP50": 1165, - "waitP99": 2240, - "waitMax": 2240 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1113.1333333333334, + "waitP50": 1007, + "waitP99": 2145, + "waitMax": 2145 }, { "groupId": "heavy-3", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1157.6333333333334, - "waitP50": 1191, - "waitP99": 2285, - "waitMax": 2285 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1114.3333333333333, + "waitP50": 1065, + "waitP99": 2163, + "waitMax": 2163 }, { "groupId": "heavy-4", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1163.4, - "waitP50": 1195, - "waitP99": 2247, - "waitMax": 2247 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1108.4, + "waitP50": 1033, + "waitP99": 2133, + "waitMax": 2133 }, { "groupId": "heavy-5", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1171.7666666666667, - "waitP50": 1196, - "waitP99": 2283, - "waitMax": 2283 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1112.6666666666667, + "waitP50": 1106, + "waitP99": 2153, + "waitMax": 2153 }, { "groupId": "heavy-6", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1176.1666666666667, - "waitP50": 1199, - "waitP99": 2249, - "waitMax": 2249 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1117.4, + "waitP50": 1030, + "waitP99": 2186, + "waitMax": 2186 }, { "groupId": "heavy-7", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, + "share": 0.046875, "contentionShareOverWeight": 1, - "meanWait": 1185.9666666666667, - "waitP50": 1224, - "waitP99": 2300, - "waitMax": 2300 + "meanWait": 1114.2, + "waitP50": 1009, + "waitP99": 2257, + "waitMax": 2257 }, { "groupId": "heavy-8", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1191.6, - "waitP50": 1241, - "waitP99": 2274, - "waitMax": 2274 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1114.3333333333333, + "waitP50": 1032, + "waitP99": 2220, + "waitMax": 2220 }, { "groupId": "heavy-9", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1198, - "waitP50": 1243, - "waitP99": 2285, - "waitMax": 2285 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1112.6666666666667, + "waitP50": 1046, + "waitP99": 2239, + "waitMax": 2239 + }, + { + "groupId": "heavy-10", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1101.4, + "waitP50": 1097, + "waitP99": 2142, + "waitMax": 2142 + }, + { + "groupId": "heavy-11", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1101.4, + "waitP50": 1126, + "waitP99": 2131, + "waitMax": 2131 + }, + { + "groupId": "heavy-12", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1110.4, + "waitP50": 1140, + "waitP99": 2203, + "waitMax": 2203 + }, + { + "groupId": "heavy-13", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1107.4, + "waitP50": 1004, + "waitP99": 2239, + "waitMax": 2239 + }, + { + "groupId": "heavy-14", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1106.6, + "waitP50": 1124, + "waitP99": 2153, + "waitMax": 2153 + }, + { + "groupId": "heavy-15", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1105.6666666666667, + "waitP50": 1096, + "waitP99": 2203, + "waitMax": 2203 + }, + { + "groupId": "heavy-16", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1109.8666666666666, + "waitP50": 1109, + "waitP99": 2144, + "waitMax": 2144 + }, + { + "groupId": "heavy-17", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1116.3333333333333, + "waitP50": 1084, + "waitP99": 2201, + "waitMax": 2201 + }, + { + "groupId": "heavy-18", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1105.4, + "waitP50": 1112, + "waitP99": 2220, + "waitMax": 2220 + }, + { + "groupId": "heavy-19", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 1101.5333333333333, + "waitP50": 1035, + "waitP99": 2219, + "waitMax": 2219 }, { "groupId": "light", "dequeued": 20, "weight": 1, "share": 0.0625, - "contentionShareOverWeight": 0.689655172413793, - "meanWait": 528.95, - "waitP50": 459, - "waitP99": 1077, - "waitMax": 1077 + "contentionShareOverWeight": 1, + "meanWait": 1106, + "waitP50": 1204, + "waitP99": 1832, + "waitMax": 1832 } ] }, { - "treatment": "perKeyCap+sfq", + "treatment": "total+perKey+sfq", "lightWait": { - "mean": 462.3833333333334, - "min": 438.85, - "max": 489.6 + "mean": 2291.7499999999995, + "min": 2143.7, + "max": 2524.85 }, "worstWait": { - "mean": 1061.6888888888889, - "min": 952.7, - "max": 1178.3 + "mean": 2291.7499999999995, + "min": 2143.7, + "max": 2524.85 }, "makespan": { - "mean": 2067.6666666666665, - "min": 1877, - "max": 2295 + "mean": 4146.333333333333, + "min": 3792, + "max": 4602 }, "contentionWorst": { - "mean": 0.689655172413793, - "min": 0.689655172413793, - "max": 0.689655172413793 + "mean": 1, + "min": 1, + "max": 1 }, "detailSeed0": [ { "groupId": "heavy-0", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1173.3, - "waitP50": 1209, - "waitP99": 2263, - "waitMax": 2263 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2234.6666666666665, + "waitP50": 2154, + "waitP99": 4379, + "waitMax": 4379 }, { "groupId": "heavy-1", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1175.1, - "waitP50": 1241, - "waitP99": 2247, - "waitMax": 2247 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2235.9333333333334, + "waitP50": 2313, + "waitP99": 4380, + "waitMax": 4380 }, { "groupId": "heavy-2", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1172.5666666666666, - "waitP50": 1224, - "waitP99": 2283, - "waitMax": 2283 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2247.866666666667, + "waitP50": 2083, + "waitP99": 4403, + "waitMax": 4403 }, { "groupId": "heavy-3", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1178.3, - "waitP50": 1240, - "waitP99": 2271, - "waitMax": 2271 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2254.0666666666666, + "waitP50": 2135, + "waitP99": 4427, + "waitMax": 4427 }, { "groupId": "heavy-4", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1171.1333333333334, - "waitP50": 1236, - "waitP99": 2264, - "waitMax": 2264 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2250.6666666666665, + "waitP50": 2081, + "waitP99": 4502, + "waitMax": 4502 }, { "groupId": "heavy-5", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1173.9666666666667, - "waitP50": 1255, - "waitP99": 2245, - "waitMax": 2245 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2253.0666666666666, + "waitP50": 2080, + "waitP99": 4511, + "waitMax": 4511 }, { "groupId": "heavy-6", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1176.1666666666667, - "waitP50": 1167, - "waitP99": 2291, - "waitMax": 2291 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2255.9333333333334, + "waitP50": 2076, + "waitP99": 4512, + "waitMax": 4512 }, { "groupId": "heavy-7", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, + "share": 0.046875, "contentionShareOverWeight": 1, - "meanWait": 1169.2, - "waitP50": 1173, - "waitP99": 2295, - "waitMax": 2295 + "meanWait": 2248.266666666667, + "waitP50": 2059, + "waitP99": 4527, + "waitMax": 4527 }, { "groupId": "heavy-8", - "dequeued": 30, + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1171.2, - "waitP50": 1165, - "waitP99": 2286, - "waitMax": 2286 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2251.266666666667, + "waitP50": 2012, + "waitP99": 4521, + "waitMax": 4521 }, { "groupId": "heavy-9", - "dequeued": 30, + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2248.6666666666665, + "waitP50": 2040, + "waitP99": 4483, + "waitMax": 4483 + }, + { + "groupId": "heavy-10", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2237.9333333333334, + "waitP50": 2237, + "waitP99": 4287, + "waitMax": 4287 + }, + { + "groupId": "heavy-11", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2236.4666666666667, + "waitP50": 2331, + "waitP99": 4289, + "waitMax": 4289 + }, + { + "groupId": "heavy-12", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2242.133333333333, + "waitP50": 2315, + "waitP99": 4367, + "waitMax": 4367 + }, + { + "groupId": "heavy-13", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2241.3333333333335, + "waitP50": 2167, + "waitP99": 4484, + "waitMax": 4484 + }, + { + "groupId": "heavy-14", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2234.133333333333, + "waitP50": 2221, + "waitP99": 4334, + "waitMax": 4334 + }, + { + "groupId": "heavy-15", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2233, + "waitP50": 2205, + "waitP99": 4362, + "waitMax": 4362 + }, + { + "groupId": "heavy-16", + "dequeued": 15, "weight": 1, - "share": 0.09375, - "contentionShareOverWeight": 1.0344827586206897, - "meanWait": 1171.2666666666667, - "waitP50": 1218, - "waitP99": 2292, - "waitMax": 2292 + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2236.6, + "waitP50": 2233, + "waitP99": 4325, + "waitMax": 4325 + }, + { + "groupId": "heavy-17", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2248.8, + "waitP50": 2109, + "waitP99": 4483, + "waitMax": 4483 + }, + { + "groupId": "heavy-18", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2246.6666666666665, + "waitP50": 2191, + "waitP99": 4464, + "waitMax": 4464 + }, + { + "groupId": "heavy-19", + "dequeued": 15, + "weight": 1, + "share": 0.046875, + "contentionShareOverWeight": 1, + "meanWait": 2242.9333333333334, + "waitP50": 2172, + "waitP99": 4401, + "waitMax": 4401 }, { "groupId": "light", "dequeued": 20, "weight": 1, "share": 0.0625, - "contentionShareOverWeight": 0.689655172413793, - "meanWait": 489.6, - "waitP50": 386, - "waitP99": 1026, - "waitMax": 1026 + "contentionShareOverWeight": 1, + "meanWait": 2524.85, + "waitP50": 2807, + "waitP99": 4112, + "waitMax": 4112 } ] } diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckTrickle.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckTrickle.json index bc1a3d8ed7..90b0efdbc4 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckTrickle.json +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/caps-ckTrickle.json @@ -189,6 +189,64 @@ } ] }, + { + "treatment": "total+perKey", + "lightWait": { + "mean": 2851.788888888889, + "min": 2528.4666666666667, + "max": 3236.766666666667 + }, + "worstWait": { + "mean": 2860.7444444444445, + "min": 2535.3, + "max": 3236.766666666667 + }, + "makespan": { + "mean": 3904.6666666666665, + "min": 3653, + "max": 4342 + }, + "contentionWorst": { + "mean": 0.27872569582223233, + "min": 0.2542372881355932, + "max": 0.29096989966555187 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 2.440677966101695, + "meanWait": 1770.1916666666666, + "waitP50": 1765, + "waitP99": 3293, + "waitMax": 3309 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.3050847457627119, + "meanWait": 3236.766666666667, + "waitP50": 3266, + "waitP99": 3523, + "waitMax": 3523 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.2542372881355932, + "meanWait": 3129.5, + "waitP50": 3132, + "waitP99": 3536, + "waitMax": 3536 + } + ] + }, { "treatment": "sfq", "lightWait": { @@ -362,6 +420,64 @@ "waitMax": 31 } ] + }, + { + "treatment": "total+perKey+sfq", + "lightWait": { + "mean": 105.68888888888888, + "min": 77.96666666666667, + "max": 143.8 + }, + "worstWait": { + "mean": 2337.3555555555554, + "min": 2159.9333333333334, + "max": 2685.2916666666665 + }, + "makespan": { + "mean": 3911.3333333333335, + "min": 3662, + "max": 4298 + }, + "contentionWorst": { + "mean": 0.8826988839201834, + "min": 0.6923076923076924, + "max": 0.9782608695652174 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.0434782608695652, + "meanWait": 2685.2916666666665, + "waitP50": 2754, + "waitP99": 4282, + "waitMax": 4298 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9782608695652174, + "meanWait": 143.8, + "waitP50": 156, + "waitP99": 304, + "waitMax": 304 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9782608695652174, + "meanWait": 63.3, + "waitP50": 58, + "waitP99": 161, + "waitMax": 161 + } + ] } ] } \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike/harness/metrics.ts b/internal-packages/run-engine/src/run-queue/fairness-spike/harness/metrics.ts index 3dc91e4b6c..ed109a5ec5 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike/harness/metrics.ts +++ b/internal-packages/run-engine/src/run-queue/fairness-spike/harness/metrics.ts @@ -34,9 +34,12 @@ export type RunMetrics = { contentionWorstShareOverWeight: number; contentionJain: number; /** - * Logical time of the last dequeue (drain time). Work-conservation signal: on a - * fixed workload a non-work-conserving discipline (e.g. a static cap that idles - * slots when the capped tenant is alone) drains slower, so makespan is larger. + * Logical time of the LAST dequeue (not completion; ~one holdMs before the final + * ack). Work-conservation signal, but read it ONLY on a service-bound workload + * with no late arrivals (e.g. ckHeavyIdle): there a non-work-conserving + * discipline that idles slots drains slower, so makespan is larger. On scenarios + * with poisson arrivals the last dequeue is partly set by the arrival tail, not + * the discipline, so makespan is arrival-confounded and not a clean signal there. */ makespanMs: number; /** From 9b8440cfee6edce6e17fd2de6cd4a8548201d527 Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 17:15:49 +0100 Subject: [PATCH 4/4] test(run-engine): cross-task total-cap bench (the total cap's real job) Measures the claim the caps bench left argued-not-measured: the per-task total cap's real purpose is cross-TASK isolation, not cross-key. Two keyless tasks share one env; capping the heavy task (its real per-queue concurrency gate, which for a keyless task equals its total cap) cuts the light task's wait 475 to 2 under the production FairQueueSelectionStrategy, versus failing at the cross-key grain. Still not work-conserving (makespan 2039 to 3039); SFQ protects the light task too and stays work-conserving. Adds a perQueueCap hook to the base-queue driver. --- .../fairness-spike-ck/CAPS_FINDINGS.md | 28 ++- .../crossTaskCaps.bench.test.ts | 193 ++++++++++++++++++ .../results/xtask-crossTaskSkew.json | 192 +++++++++++++++++ .../fairness-spike/harness/driver.ts | 13 ++ 4 files changed, 417 insertions(+), 9 deletions(-) create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/crossTaskCaps.bench.test.ts create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/xtask-crossTaskSkew.json diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md index 5dd82de233..b1daf94ee8 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/CAPS_FINDINGS.md @@ -119,13 +119,21 @@ contention metric is degenerate at 1.0; makespan is the signal there. ckSybil is in-task comparison is capacity-confounded. `totalCap=2` caps the whole task's aggregate at half of env=4, so it simply halves throughput: light's wait rises (ckSkew 1098 to 2840) for the same reason heavy's does (both now share half the - server), which is Little's-Law throughput loss, not a fairness effect. The total - cap's real purpose is cross-TASK isolation (reservation between base queues when - the sum of per-task caps is below the env limit), a different problem from - #2617's within-task cross-key starvation and one this single-base-queue harness - does not exercise (noted as future work). Do not read the "worse" numbers as - "total caps harm fairness"; read them as "wrong knob, and measured on a lower - ceiling." + server), which is Little's-Law throughput loss, not a fairness effect. It is the + wrong knob for cross-key starvation, measured on a lower ceiling; do not read + the "worse" numbers as "total caps harm fairness." +- Total cap (Phase 1) at the cross-TASK grain: this IS its job, and it works. + Measured in a separate multi-base-queue bench (`crossTaskCaps.bench.test.ts`): + two keyless tasks share one env, a heavy task floods it, and capping the heavy + task (its per-queue concurrency limit, the real native gate, which for a keyless + task equals its total cap) cuts the light TASK's wait from 475 to 2 under the + production `FairQueueSelectionStrategy`. So the total cap protects a light task + from a heavy task, the reservation-isolation role the research describes. It is + still not work-conserving (makespan 2039 to 3039), and SFQ at the task grain + protects the light task too (wait 14) while staying work-conserving (2039). The + fidelity note: this models a KEYLESS task, so the per-queue limit is the total; + a task WITH concurrency keys needs the group SET to sum across variants (the + unbuilt Phase-1 gate). - Combined total + per-key (the shipped Phase-1+2 config): in this toy the total cap (2) is below a single per-key cap's reach, so it dominates and the per-key cap is non-binding (`total+perKey` equals `totalCap` to the digit). This toy @@ -184,5 +192,7 @@ make caps alone insufficient. Not either/or. - Contention share is volume-confounded for low-volume keys, and for the per-key cap on the sharded case it is seed-noisy (0.07..0.71); wait is the trustworthy signal, share is directional. -- Cross-task isolation (the total cap's real purpose) is argued from the research, - not measured; a multi-base-queue harness is future work. +- Cross-task isolation (the total cap's real purpose) is now measured in + `crossTaskCaps.bench.test.ts` for KEYLESS tasks (per-queue limit = total cap). + A task with concurrency keys needs the unbuilt group-SET gate to sum across + variants; that batched, keyed path is still not exercised. diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/crossTaskCaps.bench.test.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/crossTaskCaps.bench.test.ts new file mode 100644 index 0000000000..fd78732a9d --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/crossTaskCaps.bench.test.ts @@ -0,0 +1,193 @@ +import { redisTest } from "@internal/testcontainers"; +import { describe } from "vitest"; +import { mkdirSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { createRedisClient } from "@internal/redis"; +import { RunQueueFullKeyProducer } from "../keyProducer.js"; +import { FairQueueSelectionStrategy } from "../fairQueueSelectionStrategy.js"; +import { runScenario } from "../fairness-spike/harness/driver.js"; +import { SfqStrategy } from "../fairness-spike/strategies/sfqStrategy.js"; +import { GROUP_SEPARATOR } from "../fairness-spike/types.js"; +import { + buildWorkload, + weightsOf, + type WorkloadConfig, +} from "../fairness-spike/harness/workload.js"; +import type { RunMetrics, GroupMetrics } from "../fairness-spike/harness/metrics.js"; + +/** + * The total cap's REAL job: cross-TASK isolation. Two keyless tasks (base queues) + * share one env; a heavy task floods it and starves a light task. This is the + * problem #2617's total cap is for, and it is a DIFFERENT problem from the + * cross-KEY starvation the caps bench showed the total cap does not fix. + * + * The total cap on the heavy task is the real per-queue concurrency gate + * (updateQueueConcurrencyLimits); for a keyless task the per-queue limit is the + * per-task total (one base queue, no ck variants to sum), so this is faithful. + * Compared against the production FairQueueSelectionStrategy (baseline) and the + * spike SFQ selector, both driving the real RunQueue + testDequeueFromMasterQueue. + */ + +const RESULTS_DIR = join(dirname(fileURLToPath(import.meta.url)), "results"); +const SEEDS = ["seed-a", "seed-b", "seed-c"]; +const ENV_LIMIT = 4; +const HEAVY_TOTAL_CAP = 2; + +const keys = new RunQueueFullKeyProducer(); +const q = (tenant: string) => `${tenant}${GROUP_SEPARATOR}0`; + +type CrossScenario = { + config: Omit; + heavy: string; + lightKey: string; +}; + +const SCENARIOS: Record = { + // heavy task floods, two light tasks trickle in. All keyless (queueCount 1). + crossTaskSkew: { + heavy: "heavy", + lightKey: "light-1", + config: { + envConcurrencyLimit: ENV_LIMIT, + tenants: [ + { tenantId: "heavy", runCount: 240, holdMsMean: 25 }, + { tenantId: "light-1", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + { tenantId: "light-2", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + ], + }, + }, +}; + +type RedisOpts = { keyPrefix: string; host: string; port: number }; +type Treatment = { + label: string; + // returns the strategy and an optional client to quit afterwards. The real + // FairQueueSelectionStrategy takes RedisOptions and owns its own client; the + // spike SfqStrategy takes a live client. + makeStrategy: (redis: RedisOpts) => { strategy: any; client?: ReturnType }; + capHeavy?: boolean; +}; + +const TREATMENTS: Treatment[] = [ + { + label: "baseline(fairqueue)", + makeStrategy: (redis) => ({ strategy: new FairQueueSelectionStrategy({ redis, keys }) }), + }, + { + label: "heavyTotalCap", + makeStrategy: (redis) => ({ strategy: new FairQueueSelectionStrategy({ redis, keys }) }), + capHeavy: true, + }, + { + label: "sfq", + makeStrategy: (redis) => { + const client = createRedisClient(redis); + return { strategy: new SfqStrategy({ redis: client, keys }), client }; + }, + }, +]; + +function stats(xs: number[]) { + return { + mean: xs.reduce((a, b) => a + b, 0) / xs.length, + min: Math.min(...xs), + max: Math.max(...xs), + }; +} + +function fmt(n: number, d = 0): string { + return Number.isFinite(n) ? n.toFixed(d) : String(n); +} + +function waitOf(metrics: RunMetrics, key: string): number { + return metrics.perGroup.find((g) => g.groupId === key)?.meanWait ?? 0; +} + +describe("cross-task total cap bench", () => { + mkdirSync(RESULTS_DIR, { recursive: true }); + + for (const [scenarioName, scenario] of Object.entries(SCENARIOS)) { + redisTest( + `cross-task scenario: ${scenarioName}`, + async ({ redisContainer }) => { + const runs = new Map>(); + + for (const seed of SEEDS) { + const config: WorkloadConfig = { ...scenario.config, seed }; + const workload = buildWorkload(config); + const expectedTotal = workload.tenants.reduce((n, t) => n + t.runCount, 0); + + for (const treatment of TREATMENTS) { + const redis = { + keyPrefix: `rq:xtask:${scenarioName}:${treatment.label}:${seed}:`, + host: redisContainer.getHost(), + port: redisContainer.getPort(), + }; + const { strategy, client } = treatment.makeStrategy(redis); + const metrics = await runScenario({ + redis, + strategy, + workload, + perQueueCap: treatment.capHeavy ? { [q(scenario.heavy)]: HEAVY_TOTAL_CAP } : undefined, + }); + await client?.quit(); + if (metrics.totalDequeued !== expectedTotal) { + throw new Error( + `${scenarioName}/${treatment.label}/${seed}: dequeued ${metrics.totalDequeued} of ${expectedTotal}` + ); + } + const arr = runs.get(treatment.label) ?? []; + arr.push({ seed, metrics }); + runs.set(treatment.label, arr); + } + } + + const perTreatment = [...runs.entries()].map(([label, rs]) => ({ + treatment: label, + lightWait: stats(rs.map((r) => waitOf(r.metrics, scenario.lightKey))), + heavyWait: stats(rs.map((r) => waitOf(r.metrics, scenario.heavy))), + makespan: stats(rs.map((r) => r.metrics.makespanMs)), + contentionWorst: stats(rs.map((r) => r.metrics.contentionWorstShareOverWeight)), + detailSeed0: rs[0].metrics.perGroup as GroupMetrics[], + })); + + const firstWorkload = buildWorkload({ ...scenario.config, seed: SEEDS[0] }); + writeFileSync( + join(RESULTS_DIR, `xtask-${scenarioName}.json`), + JSON.stringify( + { + scenario: scenarioName, + seeds: SEEDS, + envLimit: ENV_LIMIT, + heavyTotalCap: HEAVY_TOTAL_CAP, + lightKey: scenario.lightKey, + weights: weightsOf(firstWorkload), + perTreatment, + }, + null, + 2 + ) + ); + + const lines = [ + ``, + `### cross-task: ${scenarioName} (${SEEDS.length} seeds, env=${ENV_LIMIT}, heavyTotalCap=${HEAVY_TOTAL_CAP}, light=${scenario.lightKey})`, + `treatment lightWait heavyWait makespan contWorstS/W`, + ...perTreatment.map( + (r) => + `${r.treatment.padEnd(19)} ${fmt(r.lightWait.mean).padStart(8)} ${fmt( + r.heavyWait.mean + ).padStart(8)} ${fmt(r.makespan.mean).padStart(7)} ${fmt( + r.contentionWorst.mean, + 3 + ).padStart(7)}` + ), + ``, + ]; + process.stdout.write(lines.join("\n") + "\n"); + }, + 300_000 + ); + } +}); diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/xtask-crossTaskSkew.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/xtask-crossTaskSkew.json new file mode 100644 index 0000000000..5ef0270251 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/xtask-crossTaskSkew.json @@ -0,0 +1,192 @@ +{ + "scenario": "crossTaskSkew", + "seeds": [ + "seed-a", + "seed-b", + "seed-c" + ], + "envLimit": 4, + "heavyTotalCap": 2, + "lightKey": "light-1", + "weights": { + "heavy": 1, + "light-1": 1, + "light-2": 1 + }, + "perTreatment": [ + { + "treatment": "baseline(fairqueue)", + "lightWait": { + "mean": 474.9555555555556, + "min": 352, + "max": 597.7333333333333 + }, + "heavyWait": { + "mean": 833.5527777777778, + "min": 761.4041666666667, + "max": 956.3708333333333 + }, + "makespan": { + "mean": 2039, + "min": 1765, + "max": 2285 + }, + "contentionWorst": { + "mean": 0.09807312252964427, + "min": 0.0703125, + "max": 0.1171875 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8888888888888888, + "contentionShareOverWeight": 2.70703125, + "meanWait": 956.3708333333333, + "waitP50": 961, + "waitP99": 1791, + "waitMax": 1796 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05555555555555555, + "contentionShareOverWeight": 0.17578125, + "meanWait": 597.7333333333333, + "waitP50": 709, + "waitP99": 935, + "waitMax": 935 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05555555555555555, + "contentionShareOverWeight": 0.1171875, + "meanWait": 529.6, + "waitP50": 695, + "waitP99": 919, + "waitMax": 919 + } + ] + }, + { + "treatment": "heavyTotalCap", + "lightWait": { + "mean": 2, + "min": 0, + "max": 4.533333333333333 + }, + "heavyWait": { + "mean": 1556.2055555555555, + "min": 1426.3708333333334, + "max": 1773.0791666666667 + }, + "makespan": { + "mean": 3039.3333333333335, + "min": 2863, + "max": 3312 + }, + "contentionWorst": { + "mean": 0.43014705882352944, + "min": 0.1875, + "max": 0.75 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8888888888888888, + "contentionShareOverWeight": 0.75, + "meanWait": 1773.0791666666667, + "waitP50": 1768, + "waitP99": 3296, + "waitMax": 3312 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05555555555555555, + "contentionShareOverWeight": 1.125, + "meanWait": 4.533333333333333, + "waitP50": 0, + "waitP99": 32, + "waitMax": 32 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05555555555555555, + "contentionShareOverWeight": 1.125, + "meanWait": 3.533333333333333, + "waitP50": 0, + "waitP99": 49, + "waitMax": 49 + } + ] + }, + { + "treatment": "sfq", + "lightWait": { + "mean": 13.555555555555555, + "min": 7.933333333333334, + "max": 21.466666666666665 + }, + "heavyWait": { + "mean": 896.6930555555555, + "min": 813.3458333333333, + "max": 1030.9541666666667 + }, + "makespan": { + "mean": 2039, + "min": 1765, + "max": 2285 + }, + "contentionWorst": { + "mean": 0.695078031212485, + "min": 0.6470588235294118, + "max": 0.7647058823529411 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8888888888888888, + "contentionShareOverWeight": 1.3529411764705883, + "meanWait": 1030.9541666666667, + "waitP50": 1071, + "waitP99": 1864, + "waitMax": 1866 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05555555555555555, + "contentionShareOverWeight": 0.8823529411764707, + "meanWait": 21.466666666666665, + "waitP50": 15, + "waitP99": 79, + "waitMax": 79 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05555555555555555, + "contentionShareOverWeight": 0.7647058823529411, + "meanWait": 11.266666666666667, + "waitP50": 13, + "waitP99": 26, + "waitMax": 26 + } + ] + } + ] +} \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike/harness/driver.ts b/internal-packages/run-engine/src/run-queue/fairness-spike/harness/driver.ts index bbe26e21a2..84e216c107 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike/harness/driver.ts +++ b/internal-packages/run-engine/src/run-queue/fairness-spike/harness/driver.ts @@ -22,6 +22,13 @@ export type DriverConfig = { workload: WorkloadSpec; /** ceiling on logical time (ms); the loop is event-driven so this only guards runaway starvation */ maxLogicalMs?: number; + /** + * Per-base-queue concurrency limit, keyed by the workload queue name + * (e.g. `heavy~0`). Sets the real per-queue concurrency gate the dequeue Lua + * enforces. For a keyless task (one base queue) this IS the per-task total cap, + * so it models the plan's Phase-1 total cap for cross-task isolation. + */ + perQueueCap?: Record; }; function authenticatedEnv(limit: number) { @@ -72,6 +79,12 @@ export async function runScenario(config: DriverConfig): Promise { await config.strategy.reset?.(); + if (config.perQueueCap) { + for (const [queueName, cap] of Object.entries(config.perQueueCap)) { + await queue.updateQueueConcurrencyLimits(env, queueName, cap); + } + } + const sorted = expandEvents(config.workload); const total = sorted.length; const holdByRun = new Map();