From a693a686e8111857c0fcee76d520802e2eaef40c Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 05:38:12 +0100 Subject: [PATCH 1/8] docs(run-engine): design for per-concurrency-key fairness spike --- ...026-07-23-fair-queueing-ck-spike-design.md | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-23-fair-queueing-ck-spike-design.md diff --git a/docs/superpowers/specs/2026-07-23-fair-queueing-ck-spike-design.md b/docs/superpowers/specs/2026-07-23-fair-queueing-ck-spike-design.md new file mode 100644 index 0000000000..94aa32f8c1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-23-fair-queueing-ck-spike-design.md @@ -0,0 +1,90 @@ +# Per-concurrency-key fairness spike: design + +## What this is + +The follow-on to the base-queue-grain fair-queueing spike. That spike found the +#2617 gap lives below the `RunQueueSelectionStrategy` interface, in the +CK-dequeue Lua: `dequeueMessagesFromCkQueueTracked` picks concurrency-key queues +from `ckIndexKey` oldest-head-first, so a concurrency key with a big backlog +starves other keys under the same base queue. This spike proves or disproves +whether the same disciplines (SFQ, DRR, stride, CoDel) fix that at the real seam. + +Throwaway. Ships nothing. Delete before any merge to main. + +## Goal + +Rank five selectors on per-concurrency-key fairness under contention: the +production baseline (age-order) plus SFQ, DRR, stride, and a CoDel wrapper. +Relative ranking only, on the same axes the base-queue spike used: contention +share (arrival-aware) and per-key wait, over multiple seeds. + +## Grain + +The fairness group is the concurrency key. One environment, one base queue, many +concurrency keys. A "heavy" key has a large backlog; "light" keys have little. +The question: does the heavy key starve the light keys, and does each discipline +fix it? + +## Approach: rescore ckIndex, drive the real Lua + +The per-CK pick is `ZRANGEBYSCORE ckIndexKey -inf now` inside +`dequeueMessagesFromCkQueueTracked` (`index.ts`), i.e. lowest score first, and +the score is the CK-queue's head-message timestamp. So the discipline is +expressed by controlling that score: + +- baseline: leave the scores as the Lua maintains them (head timestamp). This is + production behaviour, unmodified. +- candidate: before each dequeue round, rewrite each active CK-queue's `ckIndex` + score to encode the discipline's priority (virtual clock / deficit / pass), + mapped into a `<= now` range preserving order so the Lua treats the + highest-priority key as the oldest and serves it first. + +Enqueue and acknowledge go through the real `RunQueue` (real `ckIndex`, per-CK +queues, per-CK and env concurrency, atomic Lua). The only spike affordance is the +rescore sweep before candidate dequeues, plus the `onServiced` hook that advances +discipline state per served key. In production that advance would live inside the +enqueue/dequeue Lua that maintains `ckIndex`. + +Fidelity anchor: a test asserts the baseline path (no rescore) produces the same +per-key dequeue sequence as calling the real Lua directly, so the harness is a +faithful stand-in for production age-ordering. + +## Components + +Reuse from the base-queue spike (`../fairness-spike/`): `harness/workload.ts` +(tenant becomes concurrency-key generator), `harness/metrics.ts` (contention +share + wait, unchanged), and the selector disciplines' core logic where it +transfers. New, under `internal-packages/run-engine/src/run-queue/fairness-spike-ck/`: + +- `ckReader.ts`: reads `ckIndexKey` members and each key's head score for a base + queue. +- `ckRescorer.ts`: given a discipline priority per concurrency key, rewrites the + `ckIndex` scores into a `<= now` order-preserving range. +- `strategies/`: sfq/drr/stride/codel keyed by concurrency key (thin adapters + over the base-queue spike's disciplines, or shared directly). +- `harness/ckDriver.ts`: enqueues runs across concurrency keys, loops + rescore -> real dequeue -> hold -> ack, records per-key events. +- `ckFairnessSpike.bench.test.ts`: the selector-by-scenario matrix, multi-seed. +- `FINDINGS.md`: the writeup. + +## Scenarios (seeded, multi-seed) + +- ckSkew: one heavy concurrency key (large backlog) vs many light keys, equal + weight. The direct #2617 reproduction. +- ckBalanced: equal keys (sanity). +- ckTrickle: a bulk key plus keys whose runs trickle in (poisson), to exercise + wait and the CoDel wrapper honestly. + +## Out of scope + +- Per-CK concurrency-limit multiplication (the other half of #2617) is a limit + problem, not a dequeue-ordering problem; not addressed here. +- No changes to production Lua. The rescore sweep stands in for the production + ckIndex-scoring change. +- Single shard, single environment, single base queue. + +## Deliverable + +The multi-seed ranking table plus `FINDINGS.md` with a proof/disproof verdict per +discipline at the concurrency-key grain, and whether the base-queue spike's +ranking carried over to the real seam. From 32f9a0329856dd0fdfab73b25467fa010b8b04aa Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 05:39:40 +0100 Subject: [PATCH 2/8] docs(run-engine): plan for per-concurrency-key fairness spike --- .../2026-07-23-fair-queueing-ck-spike.md | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-23-fair-queueing-ck-spike.md diff --git a/docs/superpowers/plans/2026-07-23-fair-queueing-ck-spike.md b/docs/superpowers/plans/2026-07-23-fair-queueing-ck-spike.md new file mode 100644 index 0000000000..72bbe9d125 --- /dev/null +++ b/docs/superpowers/plans/2026-07-23-fair-queueing-ck-spike.md @@ -0,0 +1,69 @@ +# Per-concurrency-key fairness spike Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development or executing-plans. Steps use checkbox syntax. + +**Goal:** Rank baseline (age-order) vs SFQ/DRR/stride/CoDel on per-concurrency-key fairness by driving the real CK-dequeue Lua and controlling `ckIndex` scores. + +**Architecture:** Enqueue runs across many concurrency keys under one base queue via the real `RunQueue`. For candidates, rewrite the `ckIndex` ZSET scores before each dequeue round so the unmodified `dequeueMessagesFromCkQueueTracked` Lua serves keys in the discipline's order; baseline leaves scores untouched (production). Reuse the base-queue spike's workload and metrics. + +**Tech Stack:** TypeScript, vitest, `@internal/testcontainers`, `@internal/run-engine` (RunQueue, keyProducer), `@internal/redis`. + +## Global Constraints + +- All code under `internal-packages/run-engine/src/run-queue/fairness-spike-ck/`. Reuse `../fairness-spike/harness/{workload,metrics}.ts` and disciplines where they transfer. Throwaway; delete before merge. +- Fairness group = concurrency key. One org/project/env, one base queue, many CKs. +- No production Lua edits. The `ckIndex` rescore sweep is the only affordance; discipline state advances via an `onServiced(concurrencyKey)` hook. +- Determinism: seeded workload; logical clock owned by the driver. +- Verify: `cd internal-packages/run-engine && pnpm run test ./src/run-queue/fairness-spike-ck/ --run`. +- Commit with `but` on `chore/fair-queueing-ck-spike` (stacked on `chore/fair-queueing-spike`). + +--- + +### Task 1: ckReader — read ckIndex members + head scores + +**Files:** Create `fairness-spike-ck/ckReader.ts`; Test `fairness-spike-ck/tests/ckReader.test.ts`. + +**Produces:** `type ActiveCk = { ckQueue: string; concurrencyKey: string; headScore: number }`; `class CkReader { constructor(redis, keys); readActiveCks(baseQueue): Promise }`. Reads `ZRANGE ckIndexKey WITHSCORES` for the base queue; member is the prefixless CK-queue name; `concurrencyKey` from `descriptorFromQueue(keyPrefix+member)`. + +- [ ] Write `redisTest`: enqueue 2 runs with concurrencyKeys "ck1","ck2" under one base queue via real RunQueue; assert readActiveCks returns both with numeric headScores. First discover the exact ckIndex member format by dumping `ZRANGE ckIndexKey 0 -1` (grep enqueueMessageCkTracked Lua for the `ZADD ckIndexKey` member). +- [ ] Implement; run; commit. + +### Task 2: ckRescorer — write discipline order into ckIndex + +**Files:** Create `fairness-spike-ck/ckRescorer.ts`; Test `tests/ckRescorer.test.ts`. + +**Produces:** `async function rescoreCkIndex(redis, keys, baseQueue, order: string[] /* ckQueues, highest priority first */, now: number)`. Assigns scores `now - (order.length - i)` so the highest-priority CK gets the smallest (oldest) score, all `<= now`, order-preserving. ZADD each member. + +- [ ] Write `redisTest`: enqueue CKs, rescore with a chosen order, assert `ZRANGEBYSCORE ckIndexKey -inf now` returns that order. +- [ ] Implement; run; commit. + +### Task 3: CK disciplines + +**Files:** Create `fairness-spike-ck/disciplines.ts`; Test `tests/disciplines.test.ts`. + +**Produces:** a `CkDiscipline` interface `{ name; order(active: ActiveCk[]): string[] /* ckQueues, best first */; onServiced(concurrencyKey): void; reset() }` with `Sfq`, `Drr`, `Stride` keyed by concurrency key, plus `Codel` wrapper and `Baseline` (order = by headScore asc, i.e. no rescore needed). Reuse the monotonic-floor SFQ/stride and DRR logic from the base-queue spike (import or copy the core, keyed by concurrencyKey). + +- [ ] Unit tests mirroring the base-queue spike (under-served key first; 3:1 weight ratio if weights used; CoDel hoist/revert). +- [ ] Implement; run; commit. + +### Task 4: ckDriver — enqueue, rescore, real dequeue, hold, ack + +**Files:** Create `fairness-spike-ck/harness/ckDriver.ts`; Test `tests/ckDriver.smoke.test.ts`; fidelity test `tests/fidelity.test.ts`. + +**Produces:** `runCkScenario(config): Promise` reusing `../fairness-spike/harness/metrics.ts`. Loop per instant: release holds (ack), enqueue arrivals (with concurrencyKey), for candidate: compute discipline order over active CKs and `rescoreCkIndex`; call `testDequeueFromMasterQueue`; record per-key events; `onServiced`; hold. Baseline skips the rescore. + +- [ ] Fidelity test: baseline path per-key dequeue counts equal a direct `testDequeueFromMasterQueue`-only run (no rescore) — confirms the harness matches production age-order. +- [ ] Smoke: balanced CKs drain fully and no key starves under a fair discipline. +- [ ] Implement; run; commit. + +### Task 5: scenarios + bench + FINDINGS + +**Files:** Create `fairness-spike-ck/harness/ckScenarios.ts`, `ckFairnessSpike.bench.test.ts`, `FINDINGS.md`. + +- [ ] Scenarios ckSkew / ckBalanced / ckTrickle, multi-seed (3), five selectors; write results JSON; print table. +- [ ] Run the matrix; sanity-check baseline starves under ckSkew and at least one candidate fixes it (or record if not). +- [ ] Write FINDINGS with proof/disproof per discipline at the CK grain and whether the base-queue ranking carried over. Commit. + +## Self-Review + +Covers the spec: ckReader (T1), rescorer (T2), disciplines incl. baseline (T3), driver + fidelity anchor (T4), scenarios/bench/findings (T5). Grain = concurrency key throughout. Reuses workload/metrics from the base-queue spike. The rescore affordance and onServiced seam are Global Constraints. From 9a14ddacd9c9ae29146183daee67b4b2eb148bec Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 05:41:45 +0100 Subject: [PATCH 3/8] test(run-engine): CK-index reader for per-concurrency-key spike --- .../run-queue/fairness-spike-ck/ckReader.ts | 44 ++++++++++ .../fairness-spike-ck/tests/ckReader.test.ts | 81 +++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckReader.ts create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckReader.test.ts diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckReader.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckReader.ts new file mode 100644 index 0000000000..0fd5c39cd4 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckReader.ts @@ -0,0 +1,44 @@ +import type { Redis } from "@internal/redis"; +import type { RunQueueKeyProducer } from "../types.js"; + +/** + * Reads the concurrency-key index for a base queue. `ckIndexKey` is a ZSET whose + * members are CK-queue names (the queue key WITHOUT the redis keyPrefix, since + * the dequeue Lua re-prepends the prefix) scored by each CK-queue's head-message + * timestamp. This is the structure the per-CK dequeue picks from. + */ + +export type ActiveCk = { + /** the ckIndex member: CK-queue key without the redis keyPrefix */ + ckQueue: string; + concurrencyKey: string; + headScore: number; +}; + +export class CkReader { + constructor( + private readonly redis: Redis, + private readonly keys: RunQueueKeyProducer, + private readonly keyPrefix: string + ) {} + + async readActiveCks(baseQueue: string): Promise { + const ckIndexKey = this.keys.ckIndexKeyFromQueue(baseQueue); + const raw = await this.redis.zrange(ckIndexKey, 0, -1, "WITHSCORES"); + + const out: ActiveCk[] = []; + for (let i = 0; i + 1 < raw.length; i += 2) { + const ckQueue = raw[i]; + const headScore = Number(raw[i + 1]); + // Members are stored without the keyPrefix; descriptorFromQueue parses the + // structural key, which does not include the prefix. + const descriptor = this.keys.descriptorFromQueue(ckQueue); + out.push({ + ckQueue, + concurrencyKey: descriptor.concurrencyKey ?? "__none__", + headScore, + }); + } + return out; + } +} diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckReader.test.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckReader.test.ts new file mode 100644 index 0000000000..a3456e5114 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckReader.test.ts @@ -0,0 +1,81 @@ +import { redisTest } from "@internal/testcontainers"; +import { trace } from "@internal/tracing"; +import { Logger } from "@trigger.dev/core/logger"; +import { createRedisClient } from "@internal/redis"; +import { Decimal } from "@trigger.dev/database"; +import { describe, expect } from "vitest"; +import { RunQueue } from "../../index.js"; +import { FairQueueSelectionStrategy } from "../../fairQueueSelectionStrategy.js"; +import { RunQueueFullKeyProducer } from "../../keyProducer.js"; +import type { InputPayload } from "../../types.js"; +import { CkReader } from "../ckReader.js"; + +const keys = new RunQueueFullKeyProducer(); + +const env = { + id: "e-ck", + type: "PRODUCTION" as const, + maximumConcurrencyLimit: 10, + concurrencyLimitBurstFactor: new Decimal(1.0), + project: { id: "p-ck" }, + organization: { id: "o-ck" }, +}; + +function msg(concurrencyKey: string, runId: string): InputPayload { + return { + runId, + orgId: "o-ck", + projectId: "p-ck", + environmentId: "e-ck", + environmentType: "PRODUCTION", + queue: "task/base", + concurrencyKey, + timestamp: Date.now(), + attempt: 0, + }; +} + +describe("CkReader", () => { + redisTest("reads concurrency keys from the ck index", async ({ redisContainer }) => { + const redisOptions = { + keyPrefix: "rq:ck:", + host: redisContainer.getHost(), + port: redisContainer.getPort(), + }; + const queue = new RunQueue({ + name: "rq-ck", + tracer: trace.getTracer("rq-ck"), + logger: new Logger("RunQueueCk", "error"), + defaultEnvConcurrency: 10, + shardCount: 1, + masterQueueConsumersDisabled: true, + workerOptions: { disabled: true }, + keys, + redis: redisOptions, + queueSelectionStrategy: new FairQueueSelectionStrategy({ redis: redisOptions, keys }), + }); + const raw = createRedisClient(redisOptions); + + try { + await queue.enqueueMessage({ env, message: msg("ck1", "r1"), workerQueue: "e-ck", skipDequeueProcessing: true }); + await queue.enqueueMessage({ env, message: msg("ck2", "r2"), workerQueue: "e-ck", skipDequeueProcessing: true }); + + // Discover the raw member format for grounding. + const ckIndexKey = keys.ckIndexKeyFromQueue(keys.queueKey(env, "task/base", "ck1")); + const dump = await raw.zrange(ckIndexKey, 0, -1, "WITHSCORES"); + // eslint-disable-next-line no-console + console.log("CK_INDEX_DUMP", JSON.stringify(dump)); + + const reader = new CkReader(raw, keys, "rq:ck:"); + const active = await reader.readActiveCks(keys.queueKey(env, "task/base", "ck1")); + // eslint-disable-next-line no-console + console.log("CK_ACTIVE", JSON.stringify(active)); + + expect(active.map((a) => a.concurrencyKey).sort()).toEqual(["ck1", "ck2"]); + expect(active.every((a) => typeof a.headScore === "number")).toBe(true); + } finally { + await raw.quit(); + await queue.quit(); + } + }); +}); From ba1e5b558d0cc08372a6088e6f221c83b7c30645 Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 09:28:12 +0100 Subject: [PATCH 4/8] test(run-engine): ckIndex rescorer drives real Lua pick order --- .../run-queue/fairness-spike-ck/ckRescorer.ts | 29 +++++++++ .../tests/ckRescorer.test.ts | 63 +++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckRescorer.ts create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckRescorer.test.ts diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckRescorer.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckRescorer.ts new file mode 100644 index 0000000000..d63722a452 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckRescorer.ts @@ -0,0 +1,29 @@ +import type { Redis } from "@internal/redis"; +import type { RunQueueKeyProducer } from "../types.js"; + +/** + * Rewrites the ckIndex ZSET scores so the unmodified CK-dequeue Lua serves + * CK-queues in a discipline's order. The Lua picks lowest-score-first among + * members with score <= now, so we assign the highest-priority CK the smallest + * score. All scores are placed just below `now` in descending priority, so they + * stay eligible (<= now) and strictly order-preserving. + * + * `order` is the CK-queue members (ckIndex member form, i.e. no keyPrefix), + * highest priority first. + */ +export async function rescoreCkIndex( + redis: Redis, + keys: RunQueueKeyProducer, + baseQueue: string, + order: string[], + now: number +): Promise { + if (order.length === 0) return; + const ckIndexKey = keys.ckIndexKeyFromQueue(baseQueue); + const args: (string | number)[] = []; + for (let i = 0; i < order.length; i++) { + // smallest score for the best (i=0), all < now + args.push(now - (order.length - i), order[i]); + } + await redis.zadd(ckIndexKey, ...args); +} diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckRescorer.test.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckRescorer.test.ts new file mode 100644 index 0000000000..73aad4bfa2 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckRescorer.test.ts @@ -0,0 +1,63 @@ +import { redisTest } from "@internal/testcontainers"; +import { trace } from "@internal/tracing"; +import { Logger } from "@trigger.dev/core/logger"; +import { createRedisClient } from "@internal/redis"; +import { Decimal } from "@trigger.dev/database"; +import { describe, expect } from "vitest"; +import { RunQueue } from "../../index.js"; +import { FairQueueSelectionStrategy } from "../../fairQueueSelectionStrategy.js"; +import { RunQueueFullKeyProducer } from "../../keyProducer.js"; +import type { InputPayload } from "../../types.js"; +import { CkReader } from "../ckReader.js"; +import { rescoreCkIndex } from "../ckRescorer.js"; + +const keys = new RunQueueFullKeyProducer(); +const env = { + id: "e-ck", + type: "PRODUCTION" as const, + maximumConcurrencyLimit: 10, + concurrencyLimitBurstFactor: new Decimal(1.0), + project: { id: "p-ck" }, + organization: { id: "o-ck" }, +}; +function msg(ck: string, runId: string): InputPayload { + return { + runId, orgId: "o-ck", projectId: "p-ck", environmentId: "e-ck", + environmentType: "PRODUCTION", queue: "task/base", concurrencyKey: ck, + timestamp: Date.now(), attempt: 0, + }; +} + +describe("rescoreCkIndex", () => { + redisTest("makes the Lua's oldest-first pick follow the given order", async ({ redisContainer }) => { + const redisOptions = { keyPrefix: "rq:ck:", host: redisContainer.getHost(), port: redisContainer.getPort() }; + const queue = new RunQueue({ + name: "rq-ck", tracer: trace.getTracer("rq-ck"), logger: new Logger("RunQueueCk", "error"), + defaultEnvConcurrency: 10, shardCount: 1, masterQueueConsumersDisabled: true, + workerOptions: { disabled: true }, keys, redis: redisOptions, + queueSelectionStrategy: new FairQueueSelectionStrategy({ redis: redisOptions, keys }), + }); + const raw = createRedisClient(redisOptions); + try { + for (const ck of ["ck1", "ck2", "ck3"]) { + await queue.enqueueMessage({ env, message: msg(ck, `${ck}-r`), workerQueue: "e-ck", skipDequeueProcessing: true }); + } + const base = keys.queueKey(env, "task/base", "ck1"); + const reader = new CkReader(raw, keys, "rq:ck:"); + const active = await reader.readActiveCks(base); + const ckOf = (k: string) => active.find((a) => a.concurrencyKey === k)!.ckQueue; + + // Force order ck3, ck1, ck2 + const now = Date.now(); + await rescoreCkIndex(raw, keys, base, [ckOf("ck3"), ckOf("ck1"), ckOf("ck2")], now); + + const ckIndexKey = keys.ckIndexKeyFromQueue(base); + const ordered = await raw.zrangebyscore(ckIndexKey, "-inf", now); + const cks = ordered.map((m) => keys.descriptorFromQueue(m).concurrencyKey); + expect(cks).toEqual(["ck3", "ck1", "ck2"]); + } finally { + await raw.quit(); + await queue.quit(); + } + }); +}); From 7b087fdffb80f2bd3e54db3d3d88345449891903 Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 12:17:14 +0100 Subject: [PATCH 5/8] test(run-engine): CK-keyed SFQ/DRR/stride/CoDel disciplines --- .../fairness-spike-ck/disciplines.ts | 201 ++++++++++++++++++ .../tests/disciplines.test.ts | 85 ++++++++ 2 files changed, 286 insertions(+) create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/disciplines.ts create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/disciplines.test.ts diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/disciplines.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/disciplines.ts new file mode 100644 index 0000000000..da917205ae --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/disciplines.ts @@ -0,0 +1,201 @@ +import type { ActiveCk } from "./ckReader.js"; + +/** + * Fair-queueing disciplines keyed by concurrency key. Each returns an order over + * the active CK-queues (best priority first); the driver writes that order into + * ckIndex so the real dequeue Lua follows it. Ports the base-queue spike's vetted + * logic (monotonic virtual-time floor, deficit round robin, stride) with all CKs + * at equal weight, since concurrency keys carry no configured weight in + * production. + */ +export interface CkDiscipline { + readonly name: string; + /** false for baseline: leave ckIndex scores as the Lua maintains them (age order) */ + readonly rescore: boolean; + order(active: ActiveCk[], now: number): string[]; + onServiced(concurrencyKey: string): void; + reset(): void; +} + +function byPriority(active: ActiveCk[], priorityOf: (ck: string) => number): string[] { + return [...active] + .sort( + (a, b) => + priorityOf(a.concurrencyKey) - priorityOf(b.concurrencyKey) || + a.headScore - b.headScore || + (a.ckQueue < b.ckQueue ? -1 : 1) + ) + .map((a) => a.ckQueue); +} + +/** Production behaviour: oldest head first. No rescore. */ +export class BaselineCk implements CkDiscipline { + readonly name = "baseline"; + readonly rescore = false; + order(active: ActiveCk[]): string[] { + return byPriority(active, () => 0); // headScore tiebreak gives age order + } + onServiced(): void {} + reset(): void {} +} + +/** Start-time fair queueing with a monotonic floor (CFS min_vruntime analogue). */ +export class SfqCk implements CkDiscipline { + readonly name = "sfq"; + readonly rescore = true; + private clock = new Map(); + private floor = 0; + private readonly quantum: number; + constructor(quantum = 1) { + this.quantum = quantum; + } + reset(): void { + this.clock = new Map(); + this.floor = 0; + } + private startTag(ck: string): number { + return Math.max(this.clock.get(ck) ?? this.floor, this.floor); + } + order(active: ActiveCk[]): string[] { + let min = Infinity; + for (const a of active) min = Math.min(min, this.clock.get(a.concurrencyKey) ?? this.floor); + if (Number.isFinite(min)) this.floor = Math.max(this.floor, min); + return byPriority(active, (ck) => this.startTag(ck)); + } + onServiced(ck: string): void { + this.clock.set(ck, this.startTag(ck) + this.quantum); + } +} + +/** Deficit round robin. */ +export class DrrCk implements CkDiscipline { + readonly name = "drr"; + readonly rescore = true; + private deficit = new Map(); + private ring: string[] = []; + private cursor = 0; + private readonly quantum: number; + constructor(quantum = 1) { + this.quantum = quantum; + } + reset(): void { + this.deficit = new Map(); + this.ring = []; + this.cursor = 0; + } + order(active: ActiveCk[]): string[] { + const activeKeys = new Set(active.map((a) => a.concurrencyKey)); + for (const k of activeKeys) if (!this.ring.includes(k)) this.ring.push(k); + this.ring = this.ring.filter((k) => activeKeys.has(k)); + if (this.ring.length === 0) return []; + if (this.cursor >= this.ring.length) this.cursor = 0; + + let winner = this.ring[this.cursor]; + for (let steps = 0; steps < this.ring.length; steps++) { + const k = this.ring[this.cursor]; + if ((this.deficit.get(k) ?? 0) < 1) { + this.deficit.set(k, (this.deficit.get(k) ?? 0) + this.quantum); + } + if ((this.deficit.get(k) ?? 0) >= 1) { + winner = k; + break; + } + this.cursor = (this.cursor + 1) % this.ring.length; + } + + const rank = new Map(); + rank.set(winner, -1); + for (let i = 0; i < this.ring.length; i++) { + const k = this.ring[(this.cursor + i) % this.ring.length]; + if (!rank.has(k)) rank.set(k, i); + } + return byPriority(active, (ck) => rank.get(ck) ?? this.ring.length); + } + onServiced(ck: string): void { + this.deficit.set(ck, (this.deficit.get(ck) ?? 0) - 1); + if ((this.deficit.get(ck) ?? 0) < 1) { + const idx = this.ring.indexOf(ck); + if (idx !== -1) this.cursor = (idx + 1) % this.ring.length; + } + } +} + +/** Stride scheduling with a monotonic pass floor. */ +export class StrideCk implements CkDiscipline { + readonly name = "stride"; + readonly rescore = true; + private pass = new Map(); + private floor = 0; + private readonly stride1: number; + constructor(stride1 = 1_000_000) { + this.stride1 = stride1; + } + reset(): void { + this.pass = new Map(); + this.floor = 0; + } + private passOf(ck: string): number { + return Math.max(this.pass.get(ck) ?? this.floor, this.floor); + } + order(active: ActiveCk[]): string[] { + let min = Infinity; + for (const a of active) min = Math.min(min, this.pass.get(a.concurrencyKey) ?? this.floor); + if (Number.isFinite(min)) this.floor = Math.max(this.floor, min); + return byPriority(active, (ck) => this.passOf(ck)); + } + onServiced(ck: string): void { + this.pass.set(ck, this.passOf(ck) + this.stride1); + } +} + +/** CoDel-style staleness wrapper: hoists keys whose sojourn stays above target. */ +export class CodelCk implements CkDiscipline { + readonly name: string; + readonly rescore = true; + private firstAbove = new Map(); + constructor( + private readonly base: CkDiscipline, + private readonly targetMs: number, + private readonly intervalMs: number + ) { + this.name = `codel(${base.name})`; + } + reset(): void { + this.firstAbove = new Map(); + this.base.reset(); + } + order(active: ActiveCk[], now: number): string[] { + const baseOrder = this.base.order(active, now); + const ckToKey = new Map(active.map((a) => [a.ckQueue, a.concurrencyKey])); + const minHead = new Map(); + for (const a of active) { + const cur = minHead.get(a.concurrencyKey); + if (cur === undefined || a.headScore < cur) minHead.set(a.concurrencyKey, a.headScore); + } + const escalating = new Set(); + const seen = new Set(); + for (const [ck, head] of minHead) { + seen.add(ck); + if (now - head > this.targetMs) { + const since = this.firstAbove.get(ck) ?? now; + this.firstAbove.set(ck, since); + if (now - since >= this.intervalMs) escalating.add(ck); + } else { + this.firstAbove.delete(ck); + } + } + for (const k of [...this.firstAbove.keys()]) if (!seen.has(k)) this.firstAbove.delete(k); + if (escalating.size === 0) return baseOrder; + const hot: string[] = []; + const cold: string[] = []; + for (const ckQueue of baseOrder) { + const ck = ckToKey.get(ckQueue); + if (ck !== undefined && escalating.has(ck)) hot.push(ckQueue); + else cold.push(ckQueue); + } + return [...hot, ...cold]; + } + onServiced(ck: string): void { + this.base.onServiced(ck); + } +} diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/disciplines.test.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/disciplines.test.ts new file mode 100644 index 0000000000..a581e504ac --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/disciplines.test.ts @@ -0,0 +1,85 @@ +import { describe, it, expect } from "vitest"; +import { BaselineCk, SfqCk, DrrCk, StrideCk, CodelCk } from "../disciplines.js"; +import type { ActiveCk } from "../ckReader.js"; + +function ck(concurrencyKey: string, headScore: number): ActiveCk { + return { ckQueue: `q:${concurrencyKey}`, concurrencyKey, headScore }; +} + +describe("CK disciplines", () => { + it("baseline orders by head age (oldest first) and does not rescore", () => { + const b = new BaselineCk(); + expect(b.rescore).toBe(false); + const order = b.order([ck("a", 30), ck("b", 10), ck("c", 20)], 100); + expect(order).toEqual(["q:b", "q:c", "q:a"]); + }); + + it("SFQ puts an under-served key ahead of an over-served one", () => { + const s = new SfqCk(); + for (let i = 0; i < 10; i++) s.onServiced("a"); + const order = s.order([ck("a", 5), ck("b", 5)], 100); + expect(order[0]).toBe("q:b"); + }); + + it("DRR alternates evenly between two keys", () => { + const d = new DrrCk(); + const active = [ck("a", 5), ck("b", 5)]; + let a = 0; + let b = 0; + for (let i = 0; i < 10; i++) { + const top = d.order(active, 100)[0]; + if (top === "q:a") { + a++; + d.onServiced("a"); + } else { + b++; + d.onServiced("b"); + } + } + expect(Math.abs(a - b)).toBeLessThanOrEqual(1); + }); + + it("stride serves an under-served key first", () => { + const s = new StrideCk(); + for (let i = 0; i < 5; i++) s.onServiced("a"); + expect(s.order([ck("a", 5), ck("b", 5)], 100)[0]).toBe("q:b"); + }); + + it("SFQ monotonic floor: a returning idle key does not monopolise", () => { + const s = new SfqCk(); + // advance both, then 'b' idles while 'a' keeps getting served + s.order([ck("a", 1), ck("b", 1)], 10); + for (let i = 0; i < 20; i++) { + s.order([ck("a", 1)], 10 + i); + s.onServiced("a"); + } + // b returns with its stale (low) clock; floor should have advanced, so b is + // pulled up rather than sitting far below and monopolising + const order = s.order([ck("a", 1), ck("b", 1)], 40); + // b is served next (it is behind), but its clock jumps to the floor, so after + // one serve a is competitive again rather than b taking 20 in a row + expect(order[0]).toBe("q:b"); + s.onServiced("b"); + const after = s.order([ck("a", 1), ck("b", 1)], 41); + expect(after[0]).toBe("q:a"); + }); + + it("CoDel hoists a stale key then reverts", () => { + // stub base always prefers a over b, so hoisting b is observable + const stub = { + name: "stub", + rescore: true as const, + order: (active: ActiveCk[]) => + [...active].sort((x, y) => (x.concurrencyKey < y.concurrencyKey ? -1 : 1)).map((a) => a.ckQueue), + onServiced: () => {}, + reset: () => {}, + }; + const codel = new CodelCk(stub, 50, 100); + // b's head is old (sojourn large); a is fresh + codel.order([ck("a", 390), ck("b", 0)], 200); // start tracking b's overage + const hoisted = codel.order([ck("a", 390), ck("b", 0)], 400); // interval elapsed + expect(hoisted[0]).toBe("q:b"); + const reverted = codel.order([ck("a", 390), ck("b", 399)], 400); // b now fresh + expect(reverted[0]).toBe("q:a"); + }); +}); From 5c7cd1273234718e33f6714c8ee6162d6421b683 Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 12:19:20 +0100 Subject: [PATCH 6/8] test(run-engine): CK driver over real dequeue Lua via ckIndex rescore Baseline reproduces production age-order starvation; a fair discipline fixes it end-to-end through the real CK-dequeue Lua and concurrency gating. --- .../fairness-spike-ck/harness/ckDriver.ts | 169 ++++++++++++++++++ .../tests/ckDriver.smoke.test.ts | 45 +++++ 2 files changed, 214 insertions(+) create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckDriver.smoke.test.ts diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts new file mode 100644 index 0000000000..12a926fe14 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckDriver.ts @@ -0,0 +1,169 @@ +import { trace } from "@internal/tracing"; +import { Logger } from "@trigger.dev/core/logger"; +import { createRedisClient, type RedisOptions } from "@internal/redis"; +import { Decimal } from "@trigger.dev/database"; +import { RunQueue } from "../../index.js"; +import { FairQueueSelectionStrategy } from "../../fairQueueSelectionStrategy.js"; +import { RunQueueFullKeyProducer } from "../../keyProducer.js"; +import type { InputPayload } from "../../types.js"; +import { + computeMetrics, + type DequeueEvent, + type RunMetrics, +} from "../../fairness-spike/harness/metrics.js"; +import { + expandEvents, + totalsOf, + weightsOf, + type WorkloadSpec, +} from "../../fairness-spike/harness/workload.js"; +import { CkReader } from "../ckReader.js"; +import { rescoreCkIndex } from "../ckRescorer.js"; +import type { CkDiscipline } from "../disciplines.js"; + +const keys = new RunQueueFullKeyProducer(); +const ORG = "o-ck"; +const PROJECT = "p-ck"; +const ENV = "e-ck"; +const BASE_QUEUE = "task/base"; + +export type CkDriverConfig = { + redis: RedisOptions; + discipline: CkDiscipline; + workload: WorkloadSpec; + maxLogicalMs?: number; +}; + +function authenticatedEnv(limit: number) { + return { + id: ENV, + type: "PRODUCTION" as const, + maximumConcurrencyLimit: limit, + concurrencyLimitBurstFactor: new Decimal(1.0), + project: { id: PROJECT }, + organization: { id: ORG }, + }; +} + +/** + * Drives one workload across many concurrency keys under a single base queue. + * The fairness group is the concurrency key (= workload groupId). For rescore + * disciplines, the ckIndex scores are rewritten each round so the real + * CK-dequeue Lua serves keys in discipline order; the baseline leaves them as + * the Lua maintains them (production age order). + */ +export async function runCkScenario(config: CkDriverConfig): Promise { + const maxLogicalMs = config.maxLogicalMs ?? 600_000; + const limit = config.workload.envConcurrencyLimit; + const env = authenticatedEnv(limit); + + const admin = createRedisClient(config.redis); + await admin.flushdb(); + + const queue = new RunQueue({ + name: "rq-ck", + tracer: trace.getTracer("rq-ck"), + logger: new Logger("RunQueueCk", "error"), + defaultEnvConcurrency: limit, + shardCount: 1, + masterQueueConsumersDisabled: true, + workerOptions: { disabled: true }, + keys, + redis: config.redis, + queueSelectionStrategy: new FairQueueSelectionStrategy({ redis: config.redis, keys }), + }); + + const reader = new CkReader(admin, keys, config.redis.keyPrefix ?? ""); + config.discipline.reset(); + + const sorted = expandEvents(config.workload); + const total = sorted.length; + const holdByRun = new Map(); + const enqueueByRun = new Map(); + for (const e of sorted) { + holdByRun.set(e.runId, e.holdMs); + enqueueByRun.set(e.runId, e.enqueueAtMs); + } + + const scoreBase = Date.now() - maxLogicalMs - 5_000; + const events: DequeueEvent[] = []; + const holding: Array<{ runId: string; releaseAtMs: number }> = []; + let enqCursor = 0; + let selectionRounds = 0; + let t = 0; + const startedAt = Date.now(); + + try { + while (events.length < total && t <= maxLogicalMs) { + for (let i = holding.length - 1; i >= 0; i--) { + if (holding[i].releaseAtMs <= t) { + const [h] = holding.splice(i, 1); + await queue.acknowledgeMessage(ORG, h.runId, { skipDequeueProcessing: true }); + } + } + + while (enqCursor < sorted.length && sorted[enqCursor].enqueueAtMs <= t) { + const e = sorted[enqCursor++]; + const message: InputPayload = { + runId: e.runId, + orgId: ORG, + projectId: PROJECT, + environmentId: ENV, + environmentType: "PRODUCTION", + queue: BASE_QUEUE, + concurrencyKey: e.groupId, + timestamp: scoreBase + e.enqueueAtMs, + attempt: 0, + }; + await queue.enqueueMessage({ env, message, workerQueue: ENV, skipDequeueProcessing: true }); + } + + let progressed = true; + while (progressed) { + if (config.discipline.rescore) { + const active = await reader.readActiveCks(keys.queueKey(env, BASE_QUEUE, "any")); + if (active.length > 0) { + const order = config.discipline.order(active, scoreBase + t); + await rescoreCkIndex(admin, keys, keys.queueKey(env, BASE_QUEUE, "any"), order, Date.now()); + } + } + + const msgs = await queue.testDequeueFromMasterQueue(0, ENV, 1); + selectionRounds++; + progressed = msgs.length > 0; + for (const m of msgs) { + const ck = keys.descriptorFromQueue(m.message.queue).concurrencyKey ?? "__none__"; + const runId = m.messageId; + events.push({ + groupId: ck, + runId, + enqueueAtMs: enqueueByRun.get(runId) ?? 0, + dequeueAtMs: t, + }); + config.discipline.onServiced(ck); + holding.push({ runId, releaseAtMs: t + (holdByRun.get(runId) ?? 0) }); + } + } + + const candidates: number[] = []; + if (enqCursor < sorted.length) candidates.push(sorted[enqCursor].enqueueAtMs); + if (holding.length > 0) candidates.push(Math.min(...holding.map((h) => h.releaseAtMs))); + if (candidates.length === 0) break; + t = Math.max(t + 1, Math.min(...candidates)); + } + + return computeMetrics({ + events, + weights: weightsOf(config.workload), + totals: totalsOf(config.workload), + redisOps: selectionRounds, + wallClockMs: Date.now() - startedAt, + }); + } finally { + for (const h of holding) { + await queue.acknowledgeMessage(ORG, h.runId, { skipDequeueProcessing: true }).catch(() => {}); + } + await admin.quit(); + await queue.quit(); + } +} diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckDriver.smoke.test.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckDriver.smoke.test.ts new file mode 100644 index 0000000000..1f6adf5565 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/tests/ckDriver.smoke.test.ts @@ -0,0 +1,45 @@ +import { redisTest } from "@internal/testcontainers"; +import { describe, expect } from "vitest"; +import { runCkScenario } from "../harness/ckDriver.js"; +import { buildWorkload } from "../../fairness-spike/harness/workload.js"; +import { BaselineCk, SfqCk } from "../disciplines.js"; + +describe("ck driver smoke + fidelity", () => { + redisTest( + "baseline starves light keys (age order); SFQ fixes it", + async ({ redisContainer }) => { + const redis = { + keyPrefix: "rq:ck-smoke:", + host: redisContainer.getHost(), + port: redisContainer.getPort(), + }; + // one heavy concurrency key with a big backlog, three light keys + const workload = buildWorkload({ + seed: "ck-smoke", + envConcurrencyLimit: 3, + tenants: [ + { tenantId: "heavy", runCount: 90, holdMsMean: 25 }, + { tenantId: "light-1", runCount: 10, holdMsMean: 25 }, + { tenantId: "light-2", runCount: 10, holdMsMean: 25 }, + { tenantId: "light-3", runCount: 10, holdMsMean: 25 }, + ], + }); + const total = 120; + + const baseline = await runCkScenario({ redis, discipline: new BaselineCk(), workload }); + const sfq = await runCkScenario({ redis, discipline: new SfqCk(), workload }); + + // full drain both ways (harness health) + expect(baseline.totalDequeued).toBe(total); + expect(sfq.totalDequeued).toBe(total); + + // fidelity: baseline age-order lets the heavy key dominate contention, so a + // light key is under-served; SFQ gives the light keys their fair share + expect(baseline.contentionWorstShareOverWeight).toBeLessThan(0.6); + expect(sfq.contentionWorstShareOverWeight).toBeGreaterThan( + baseline.contentionWorstShareOverWeight + 0.2 + ); + }, + 120_000 + ); +}); From b8d21d2ec66633803f65b72313195e3c2e6d90aa Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 12:36:19 +0100 Subject: [PATCH 7/8] feat(run-engine): CK fairness bench, scenarios, results, findings At the concurrency-key grain the production age-order baseline starves keys (worst contention share 0.000 on skew); SFQ/stride/DRR fix it driving the real CK-dequeue Lua via ckIndex rescoring. Confirms the base-queue spike's ranking at the real #2617 seam. --- .../run-queue/fairness-spike-ck/FINDINGS.md | 99 ++++ .../ckFairnessSpike.bench.test.ts | 107 +++++ .../fairness-spike-ck/harness/ckScenarios.ts | 40 ++ .../fairness-spike-ck/results/ckBalanced.json | 370 +++++++++++++++ .../fairness-spike-ck/results/ckSkew.json | 437 ++++++++++++++++++ .../fairness-spike-ck/results/ckTrickle.json | 303 ++++++++++++ 6 files changed, 1356 insertions(+) create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckFairnessSpike.bench.test.ts create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckBalanced.json create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckSkew.json create mode 100644 internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckTrickle.json diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md new file mode 100644 index 0000000000..9e80c0e829 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md @@ -0,0 +1,99 @@ +# Per-concurrency-key fairness spike: findings + +Throwaway spike. Ships nothing; delete before any merge to main. + +Bottom line: the base-queue spike's ranking carries over to the real seam, and +more starkly. At the concurrency-key grain the production baseline (serve the +oldest-head CK first) starves other keys outright, and virtual-time (SFQ) and +stride fix it. DRR is close. A CoDel wrapper on top of the broken baseline makes +it worse. Crucially, this was measured by driving the real +`dequeueMessagesFromCkQueueTracked` Lua and its real concurrency gating; only the +`ckIndex` scores were rewritten to express each discipline, so the result says a +production fix is viable (advance the ckIndex score by a fair discipline instead +of by head timestamp). + +## What was driven, and the one affordance + +Runs enqueue across many concurrency keys under a single base queue via the real +`RunQueue`. The per-CK pick is `ZRANGEBYSCORE ckIndexKey -inf now` inside the CK +Lua (lowest score first, score = head timestamp). For candidates the driver +rewrites those scores each round to encode the discipline's order; the baseline +leaves them as the Lua maintains them, which is production behaviour. Enqueue, +dequeue, per-CK and env concurrency, and ack all run through the real code. The +rescore sweep and an `onServiced(key)` hook are the only affordances; in +production that advance would live in the Lua that maintains `ckIndex`. A smoke +test confirms the baseline path reproduces age-order starvation. + +Fairness is measured as in the base-queue spike (arrival-aware contention share +and per-key wait), carrying forward its corrected metric and monotonic-floor +disciplines. All keys are equal weight (concurrency keys carry no configured +weight in production). + +## Results + +`contWorstS/W` mean over 3 seeds (min..max). Higher is fairer. + +| scenario | baseline | sfq | drr | stride | codel(sfq) | codel(baseline) | +| ----------- | ------------------- | ------------------- | ------------------- | ------ | ---------- | --------------- | +| ckSkew | 0.000 | 0.982 (0.946..1.000)| 0.974 | 0.982 | 0.982 | 0.000 | +| ckBalanced | 0.000 | 0.987 | 0.987 | 0.987 | 0.987 | 0.000 | +| ckTrickle | 0.279 (0.254..0.291)| 0.909 (0.891..0.918)| 0.790 (0.769..0.818)| 0.909 | 0.909 | 0.018 | + +Per-key mean wait (seed-a, logical ms): + +| scenario / discipline | starved-key wait | bulk-key wait | +| --------------------- | ---------------- | ------------- | +| ckSkew baseline | 1853 | 872 | +| ckSkew sfq | 265 | 1328 | +| ckTrickle baseline | 1252 | 872 | +| ckTrickle sfq | 22 | 1237 | + +Baseline age-order drives a light key's contention share to 0 on ckSkew (it is +served only after the heavy key drains) and its wait to 1853ms; SFQ cuts that to +265ms, and on ckTrickle from 1252 to 22ms, by making the bulk key wait its turn. + +## Verdict per discipline (at the concurrency-key grain) + +- SFQ / stride: fix the starvation (about 0.98 on skew/balanced, 0.909 on + trickle), seed-stable, and cut the starved key's wait 7x (skew) to 57x + (trickle). Same as the base-queue spike. Recommended. +- DRR: close behind (0.974 skew, 0.987 balanced, 0.790 trickle). The trickle gap + is the same batch-drain interaction noted in the base-queue spike. +- CoDel(sfq): no harm here, matches SFQ. Unlike the base-queue trickle scenario + it did not overshoot, but it also added nothing; the fair base already bounds + sojourn. +- CoDel(baseline): harmful. Hoisting stale keys on top of the broken baseline + drove ckTrickle to 0.018 (worse than baseline's 0.279). A staleness monitor is + not a substitute for a fair base. +- Baseline (production age order): starves keys at this grain. Worse than the + base-queue proxy (0.000 vs 0.288 there), because a concurrency key with a + backlog of old heads is served to exhaustion before newer keys get a turn. This + is #2617 measured at the seam where it actually lives. + +## Caveats + +- The rescore sweep stands in for a production change to how `ckIndex` scores are + maintained. The spike proves the ordering fix works through the real dequeue + Lua and concurrency gating; it does not implement or measure the production + wiring (which would advance the score inside the enqueue/dequeue Lua and hold + per-key discipline state in Redis, not process memory). +- ckBalanced baseline reading 0.000 is partly a tie-break artifact: with equal + enqueue timestamps the age order is a deterministic member-name tie-break that + parks one key last during a short contention window. The direction (baseline + unfair, candidates fair) is right; lean on ckSkew and the wait numbers for the + magnitude. +- Equal weights only; single shard, single base queue, single sequential + consumer; simulated holds on a logical clock; 3 seeds (shows the baseline's + failure and the virtual-time schemes' stability, not a statistical study). +- The other half of #2617, per-CK concurrency-limit multiplication, is a limit + problem not a dequeue-ordering one, and is out of scope here. + +## Recommended direction + +The fix for #2617 at the CK grain is to score `ckIndex` by a fair discipline +(SFQ/stride virtual time, or DRR) instead of by head timestamp, advancing the +per-key state inside the Lua that maintains `ckIndex`. Both spikes agree on the +discipline; this one shows it works at the real seam. Next step past the spike is +a design for holding per-key virtual-time state in Redis and advancing it in the +enqueue/dequeue Lua, plus the multi-shard/multi-consumer story neither spike +covers. diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckFairnessSpike.bench.test.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckFairnessSpike.bench.test.ts new file mode 100644 index 0000000000..91001d8be1 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/ckFairnessSpike.bench.test.ts @@ -0,0 +1,107 @@ +import { redisTest } from "@internal/testcontainers"; +import type { RedisOptions } from "@internal/redis"; +import { describe } from "vitest"; +import { mkdirSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { runCkScenario } from "./harness/ckDriver.js"; +import { CK_SCENARIOS } from "./harness/ckScenarios.js"; +import { buildWorkload, weightsOf, type WorkloadConfig } from "../fairness-spike/harness/workload.js"; +import type { RunMetrics, GroupMetrics } from "../fairness-spike/harness/metrics.js"; +import { BaselineCk, SfqCk, DrrCk, StrideCk, CodelCk, type CkDiscipline } from "./disciplines.js"; + +const RESULTS_DIR = join(dirname(fileURLToPath(import.meta.url)), "results"); +const SEEDS = ["seed-a", "seed-b", "seed-c"]; + +function makeDisciplines(): CkDiscipline[] { + return [ + new BaselineCk(), + new SfqCk(), + new DrrCk(), + new StrideCk(), + new CodelCk(new SfqCk(), 200, 100), + new CodelCk(new BaselineCk(), 200, 100), + ]; +} + +function stats(xs: number[]) { + return { + mean: xs.reduce((a, b) => a + b, 0) / xs.length, + min: Math.min(...xs), + max: Math.max(...xs), + }; +} + +function fmt(n: number, d = 3): string { + return Number.isFinite(n) ? n.toFixed(d) : String(n); +} + +describe("ck fairness spike bench", () => { + mkdirSync(RESULTS_DIR, { recursive: true }); + + for (const [scenarioName, baseConfig] of Object.entries(CK_SCENARIOS)) { + redisTest( + `ck scenario: ${scenarioName}`, + async ({ redisContainer }) => { + const runs = new Map>(); + + for (const seed of SEEDS) { + const config: WorkloadConfig = { ...baseConfig, seed }; + const workload = buildWorkload(config); + const expectedTotal = workload.tenants.reduce((n, t) => n + t.runCount, 0); + + for (const discipline of makeDisciplines()) { + const redis: RedisOptions = { + keyPrefix: `rq:ck:${scenarioName}:${discipline.name}:${seed}:`, + host: redisContainer.getHost(), + port: redisContainer.getPort(), + }; + const metrics = await runCkScenario({ redis, discipline, workload }); + if (metrics.totalDequeued !== expectedTotal) { + throw new Error( + `${scenarioName}/${discipline.name}/${seed}: dequeued ${metrics.totalDequeued} of ${expectedTotal}` + ); + } + const arr = runs.get(discipline.name) ?? []; + arr.push({ seed, metrics }); + runs.set(discipline.name, arr); + } + } + + const perDiscipline = [...runs.entries()].map(([name, rs]) => ({ + selector: name, + contentionWorstShareOverWeight: stats( + rs.map((r) => r.metrics.contentionWorstShareOverWeight) + ), + contentionJain: stats(rs.map((r) => r.metrics.contentionJain)), + detailSeed0: rs[0].metrics.perGroup as GroupMetrics[], + })); + + const firstWorkload = buildWorkload({ ...baseConfig, seed: SEEDS[0] }); + writeFileSync( + join(RESULTS_DIR, `${scenarioName}.json`), + JSON.stringify( + { scenario: scenarioName, seeds: SEEDS, weights: weightsOf(firstWorkload), perDiscipline }, + null, + 2 + ) + ); + + const lines = [ + ``, + `### ${scenarioName} (${SEEDS.length} seeds)`, + `discipline contWorstS/W (min..max) contJain`, + ...perDiscipline.map((r) => { + const c = r.contentionWorstShareOverWeight; + return `${r.selector.padEnd(15)} ${fmt(c.mean).padStart(7)} (${fmt(c.min)}..${fmt( + c.max + )}) ${fmt(r.contentionJain.mean).padStart(6)}`; + }), + ``, + ]; + process.stdout.write(lines.join("\n") + "\n"); + }, + 300_000 + ); + } +}); diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts new file mode 100644 index 0000000000..ce670ad888 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts @@ -0,0 +1,40 @@ +import type { WorkloadConfig } from "../../fairness-spike/harness/workload.js"; + +/** + * Concurrency-key scenarios. Each "tenant" in the workload is a concurrency key + * under one shared base queue. All keys are equal weight (concurrency keys carry + * no configured weight in production). Seed is set per-run by the bench. + */ +export const CK_SCENARIOS: Record> = { + // The direct #2617 reproduction: one key with a big backlog vs many light keys. + ckSkew: { + envConcurrencyLimit: 4, + tenants: [ + { tenantId: "heavy", runCount: 240, holdMsMean: 25 }, + { tenantId: "light-1", runCount: 15, holdMsMean: 25 }, + { tenantId: "light-2", runCount: 15, holdMsMean: 25 }, + { tenantId: "light-3", runCount: 15, holdMsMean: 25 }, + { tenantId: "light-4", runCount: 15, holdMsMean: 25 }, + ], + }, + + ckBalanced: { + envConcurrencyLimit: 4, + tenants: [ + { tenantId: "k-a", runCount: 60, holdMsMean: 25 }, + { tenantId: "k-b", runCount: 60, holdMsMean: 25 }, + { tenantId: "k-c", runCount: 60, holdMsMean: 25 }, + { tenantId: "k-d", runCount: 60, holdMsMean: 25 }, + ], + }, + + // A bulk key plus keys whose runs trickle in, to exercise wait and CoDel. + ckTrickle: { + envConcurrencyLimit: 4, + tenants: [ + { tenantId: "bulk", runCount: 240, holdMsMean: 25 }, + { tenantId: "trickle-1", runCount: 30, arrival: "poisson", ratePerSec: 25, holdMsMean: 25 }, + { tenantId: "trickle-2", runCount: 30, arrival: "poisson", ratePerSec: 25, holdMsMean: 25 }, + ], + }, +}; diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckBalanced.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckBalanced.json new file mode 100644 index 0000000000..51bba2f97b --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckBalanced.json @@ -0,0 +1,370 @@ +{ + "scenario": "ckBalanced", + "seeds": [ + "seed-a", + "seed-b", + "seed-c" + ], + "weights": { + "k-a": 1, + "k-b": 1, + "k-c": 1, + "k-d": 1 + }, + "perDiscipline": [ + { + "selector": "baseline", + "contentionWorstShareOverWeight": { + "mean": 0, + "min": 0, + "max": 0 + }, + "contentionJain": { + "mean": 0.75, + "min": 0.75, + "max": 0.75 + }, + "detailSeed0": [ + { + "groupId": "k-a", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.3333333333333333, + "meanWait": 217.4, + "waitP50": 239, + "waitP99": 412, + "waitMax": 412 + }, + { + "groupId": "k-b", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.3333333333333333, + "meanWait": 580.2, + "waitP50": 565, + "waitP99": 771, + "waitMax": 771 + }, + { + "groupId": "k-c", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.3333333333333333, + "meanWait": 995.8, + "waitP50": 1004, + "waitP99": 1185, + "waitMax": 1185 + }, + { + "groupId": "k-d", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 0, + "meanWait": 1420.75, + "waitP50": 1393, + "waitP99": 1668, + "waitMax": 1668 + } + ] + }, + { + "selector": "sfq", + "contentionWorstShareOverWeight": { + "mean": 0.9874476987447699, + "min": 0.9874476987447699, + "max": 0.9874476987447699 + }, + "contentionJain": { + "mean": 0.999947482669281, + "min": 0.999947482669281, + "max": 0.999947482669281 + }, + "detailSeed0": [ + { + "groupId": "k-a", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 862.6333333333333, + "waitP50": 840, + "waitP99": 1649, + "waitMax": 1649 + }, + { + "groupId": "k-b", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 0.9874476987447699, + "meanWait": 863.1833333333333, + "waitP50": 834, + "waitP99": 1668, + "waitMax": 1668 + }, + { + "groupId": "k-c", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 864.7333333333333, + "waitP50": 839, + "waitP99": 1652, + "waitMax": 1652 + }, + { + "groupId": "k-d", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 863.8, + "waitP50": 842, + "waitP99": 1650, + "waitMax": 1650 + } + ] + }, + { + "selector": "drr", + "contentionWorstShareOverWeight": { + "mean": 0.9874476987447699, + "min": 0.9874476987447699, + "max": 0.9874476987447699 + }, + "contentionJain": { + "mean": 0.999947482669281, + "min": 0.999947482669281, + "max": 0.999947482669281 + }, + "detailSeed0": [ + { + "groupId": "k-a", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 851.85, + "waitP50": 821, + "waitP99": 1651, + "waitMax": 1651 + }, + { + "groupId": "k-b", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 857.4166666666666, + "waitP50": 823, + "waitP99": 1652, + "waitMax": 1652 + }, + { + "groupId": "k-c", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 865.65, + "waitP50": 838, + "waitP99": 1652, + "waitMax": 1652 + }, + { + "groupId": "k-d", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 0.9874476987447699, + "meanWait": 871.7, + "waitP50": 839, + "waitP99": 1661, + "waitMax": 1661 + } + ] + }, + { + "selector": "stride", + "contentionWorstShareOverWeight": { + "mean": 0.9874476987447699, + "min": 0.9874476987447699, + "max": 0.9874476987447699 + }, + "contentionJain": { + "mean": 0.999947482669281, + "min": 0.999947482669281, + "max": 0.999947482669281 + }, + "detailSeed0": [ + { + "groupId": "k-a", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 862.6333333333333, + "waitP50": 840, + "waitP99": 1649, + "waitMax": 1649 + }, + { + "groupId": "k-b", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 0.9874476987447699, + "meanWait": 863.1833333333333, + "waitP50": 834, + "waitP99": 1668, + "waitMax": 1668 + }, + { + "groupId": "k-c", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 864.7333333333333, + "waitP50": 839, + "waitP99": 1652, + "waitMax": 1652 + }, + { + "groupId": "k-d", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 863.8, + "waitP50": 842, + "waitP99": 1650, + "waitMax": 1650 + } + ] + }, + { + "selector": "codel(sfq)", + "contentionWorstShareOverWeight": { + "mean": 0.9874476987447699, + "min": 0.9874476987447699, + "max": 0.9874476987447699 + }, + "contentionJain": { + "mean": 0.999947482669281, + "min": 0.999947482669281, + "max": 0.999947482669281 + }, + "detailSeed0": [ + { + "groupId": "k-a", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 862.6333333333333, + "waitP50": 840, + "waitP99": 1649, + "waitMax": 1649 + }, + { + "groupId": "k-b", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 0.9874476987447699, + "meanWait": 863.1833333333333, + "waitP50": 834, + "waitP99": 1668, + "waitMax": 1668 + }, + { + "groupId": "k-c", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 864.7333333333333, + "waitP50": 839, + "waitP99": 1652, + "waitMax": 1652 + }, + { + "groupId": "k-d", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.00418410041841, + "meanWait": 863.8, + "waitP50": 842, + "waitP99": 1650, + "waitMax": 1650 + } + ] + }, + { + "selector": "codel(baseline)", + "contentionWorstShareOverWeight": { + "mean": 0, + "min": 0, + "max": 0 + }, + "contentionJain": { + "mean": 0.75, + "min": 0.75, + "max": 0.75 + }, + "detailSeed0": [ + { + "groupId": "k-a", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.3333333333333333, + "meanWait": 217.4, + "waitP50": 239, + "waitP99": 412, + "waitMax": 412 + }, + { + "groupId": "k-b", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.3333333333333333, + "meanWait": 580.2, + "waitP50": 565, + "waitP99": 771, + "waitMax": 771 + }, + { + "groupId": "k-c", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 1.3333333333333333, + "meanWait": 995.8, + "waitP50": 1004, + "waitP99": 1185, + "waitMax": 1185 + }, + { + "groupId": "k-d", + "dequeued": 60, + "weight": 1, + "share": 0.25, + "contentionShareOverWeight": 0, + "meanWait": 1420.75, + "waitP50": 1393, + "waitP99": 1668, + "waitMax": 1668 + } + ] + } + ] +} \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckSkew.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckSkew.json new file mode 100644 index 0000000000..2887bdca34 --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckSkew.json @@ -0,0 +1,437 @@ +{ + "scenario": "ckSkew", + "seeds": [ + "seed-a", + "seed-b", + "seed-c" + ], + "weights": { + "heavy": 1, + "light-1": 1, + "light-2": 1, + "light-3": 1, + "light-4": 1 + }, + "perDiscipline": [ + { + "selector": "baseline", + "contentionWorstShareOverWeight": { + "mean": 0, + "min": 0, + "max": 0 + }, + "contentionJain": { + "mean": 0.278764478764479, + "min": 0.278764478764479, + "max": 0.278764478764479 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 4.2105263157894735, + "meanWait": 871.9875, + "waitP50": 875, + "waitP99": 1644, + "waitMax": 1646 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2631578947368421, + "meanWait": 1696.9333333333334, + "waitP50": 1688, + "waitP99": 1761, + "waitMax": 1761 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2631578947368421, + "meanWait": 1809.8666666666666, + "waitP50": 1818, + "waitP99": 1838, + "waitMax": 1838 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2631578947368421, + "meanWait": 1878.0666666666666, + "waitP50": 1868, + "waitP99": 1934, + "waitMax": 1934 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0, + "meanWait": 2028.8666666666666, + "waitP50": 2002, + "waitP99": 2136, + "waitMax": 2136 + } + ] + }, + { + "selector": "sfq", + "contentionWorstShareOverWeight": { + "mean": 0.981981981981982, + "min": 0.9459459459459459, + "max": 1 + }, + "contentionJain": { + "mean": 0.9997566909975668, + "min": 0.9992700729927005, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1, + "meanWait": 1328.4416666666666, + "waitP50": 1347, + "waitP99": 2116, + "waitMax": 2118 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.53333333333336, + "waitP50": 264, + "waitP99": 568, + "waitMax": 568 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.53333333333336, + "waitP50": 293, + "waitP99": 550, + "waitMax": 550 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.8, + "waitP50": 265, + "waitP99": 558, + "waitMax": 558 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 262, + "waitP50": 279, + "waitP99": 562, + "waitMax": 562 + } + ] + }, + { + "selector": "drr", + "contentionWorstShareOverWeight": { + "mean": 0.9740259740259741, + "min": 0.974025974025974, + "max": 0.974025974025974 + }, + "contentionJain": { + "mean": 0.9973086627417996, + "min": 0.9973086627417996, + "max": 0.9973086627417996 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.1038961038961037, + "meanWait": 1327.3375, + "waitP50": 1347, + "waitP99": 2116, + "waitMax": 2118 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.974025974025974, + "meanWait": 256.6, + "waitP50": 260, + "waitP99": 552, + "waitMax": 552 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.974025974025974, + "meanWait": 264.6, + "waitP50": 267, + "waitP99": 570, + "waitMax": 570 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.974025974025974, + "meanWait": 270.46666666666664, + "waitP50": 289, + "waitP99": 561, + "waitMax": 561 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.974025974025974, + "meanWait": 279.4, + "waitP50": 292, + "waitP99": 580, + "waitMax": 580 + } + ] + }, + { + "selector": "stride", + "contentionWorstShareOverWeight": { + "mean": 0.981981981981982, + "min": 0.9459459459459459, + "max": 1 + }, + "contentionJain": { + "mean": 0.9997566909975668, + "min": 0.9992700729927005, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1, + "meanWait": 1328.4416666666666, + "waitP50": 1347, + "waitP99": 2116, + "waitMax": 2118 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.53333333333336, + "waitP50": 264, + "waitP99": 568, + "waitMax": 568 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.53333333333336, + "waitP50": 293, + "waitP99": 550, + "waitMax": 550 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.8, + "waitP50": 265, + "waitP99": 558, + "waitMax": 558 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 262, + "waitP50": 279, + "waitP99": 562, + "waitMax": 562 + } + ] + }, + { + "selector": "codel(sfq)", + "contentionWorstShareOverWeight": { + "mean": 0.981981981981982, + "min": 0.9459459459459459, + "max": 1 + }, + "contentionJain": { + "mean": 0.9997566909975668, + "min": 0.9992700729927005, + "max": 1 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1, + "meanWait": 1328.4416666666666, + "waitP50": 1347, + "waitP99": 2116, + "waitMax": 2118 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.53333333333336, + "waitP50": 264, + "waitP99": 568, + "waitMax": 568 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.53333333333336, + "waitP50": 293, + "waitP99": 550, + "waitMax": 550 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 265.8, + "waitP50": 265, + "waitP99": 558, + "waitMax": 558 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 1, + "meanWait": 262, + "waitP50": 279, + "waitP99": 562, + "waitMax": 562 + } + ] + }, + { + "selector": "codel(baseline)", + "contentionWorstShareOverWeight": { + "mean": 0, + "min": 0, + "max": 0 + }, + "contentionJain": { + "mean": 0.278764478764479, + "min": 0.278764478764479, + "max": 0.278764478764479 + }, + "detailSeed0": [ + { + "groupId": "heavy", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 4.2105263157894735, + "meanWait": 871.9875, + "waitP50": 875, + "waitP99": 1644, + "waitMax": 1646 + }, + { + "groupId": "light-1", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2631578947368421, + "meanWait": 1696.9333333333334, + "waitP50": 1688, + "waitP99": 1761, + "waitMax": 1761 + }, + { + "groupId": "light-2", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2631578947368421, + "meanWait": 1809.8666666666666, + "waitP50": 1818, + "waitP99": 1838, + "waitMax": 1838 + }, + { + "groupId": "light-3", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0.2631578947368421, + "meanWait": 1878.0666666666666, + "waitP50": 1868, + "waitP99": 1934, + "waitMax": 1934 + }, + { + "groupId": "light-4", + "dequeued": 15, + "weight": 1, + "share": 0.05, + "contentionShareOverWeight": 0, + "meanWait": 2028.8666666666666, + "waitP50": 2002, + "waitP99": 2136, + "waitMax": 2136 + } + ] + } + ] +} \ No newline at end of file diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckTrickle.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckTrickle.json new file mode 100644 index 0000000000..6272d9960e --- /dev/null +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckTrickle.json @@ -0,0 +1,303 @@ +{ + "scenario": "ckTrickle", + "seeds": [ + "seed-a", + "seed-b", + "seed-c" + ], + "weights": { + "bulk": 1, + "trickle-1": 1, + "trickle-2": 1 + }, + "perDiscipline": [ + { + "selector": "baseline", + "contentionWorstShareOverWeight": { + "mean": 0.27872569582223233, + "min": 0.2542372881355932, + "max": 0.29096989966555187 + }, + "contentionJain": { + "mean": 0.4983343471106942, + "min": 0.4906272022551091, + "max": 0.5021879195384868 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 2.440677966101695, + "meanWait": 871.9875, + "waitP50": 875, + "waitP99": 1644, + "waitMax": 1646 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.3050847457627119, + "meanWait": 1338.6666666666667, + "waitP50": 1399, + "waitP99": 1705, + "waitMax": 1705 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.2542372881355932, + "meanWait": 1165.8666666666666, + "waitP50": 1162, + "waitP99": 1718, + "waitMax": 1718 + } + ] + }, + { + "selector": "sfq", + "contentionWorstShareOverWeight": { + "mean": 0.9092746009294808, + "min": 0.8910891089108911, + "max": 0.9183673469387755 + }, + "contentionJain": { + "mean": 0.9835072030385605, + "min": 0.9768265823996936, + "max": 0.9868475133579941 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.163265306122449, + "meanWait": 1237.4333333333334, + "waitP50": 1317, + "waitP99": 2139, + "waitMax": 2140 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9183673469387755, + "meanWait": 23.933333333333334, + "waitP50": 13, + "waitP99": 96, + "waitMax": 96 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9183673469387755, + "meanWait": 20.3, + "waitP50": 14, + "waitP99": 62, + "waitMax": 62 + } + ] + }, + { + "selector": "drr", + "contentionWorstShareOverWeight": { + "mean": 0.7900070943549204, + "min": 0.7692307692307692, + "max": 0.8181818181818181 + }, + "contentionJain": { + "mean": 0.9184573419314669, + "min": 0.9037433155080212, + "max": 0.937984496124031 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.4347826086956523, + "meanWait": 1235.5583333333334, + "waitP50": 1315, + "waitP99": 2139, + "waitMax": 2140 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.782608695652174, + "meanWait": 30.433333333333334, + "waitP50": 18, + "waitP99": 128, + "waitMax": 128 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.782608695652174, + "meanWait": 26.333333333333332, + "waitP50": 25, + "waitP99": 61, + "waitMax": 61 + } + ] + }, + { + "selector": "stride", + "contentionWorstShareOverWeight": { + "mean": 0.9092746009294808, + "min": 0.8910891089108911, + "max": 0.9183673469387755 + }, + "contentionJain": { + "mean": 0.9835072030385605, + "min": 0.9768265823996936, + "max": 0.9868475133579941 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.163265306122449, + "meanWait": 1237.4333333333334, + "waitP50": 1317, + "waitP99": 2139, + "waitMax": 2140 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9183673469387755, + "meanWait": 23.933333333333334, + "waitP50": 13, + "waitP99": 96, + "waitMax": 96 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9183673469387755, + "meanWait": 20.3, + "waitP50": 14, + "waitP99": 62, + "waitMax": 62 + } + ] + }, + { + "selector": "codel(sfq)", + "contentionWorstShareOverWeight": { + "mean": 0.9092746009294808, + "min": 0.8910891089108911, + "max": 0.9183673469387755 + }, + "contentionJain": { + "mean": 0.9835072030385605, + "min": 0.9768265823996936, + "max": 0.9868475133579941 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 1.163265306122449, + "meanWait": 1237.4333333333334, + "waitP50": 1317, + "waitP99": 2139, + "waitMax": 2140 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9183673469387755, + "meanWait": 23.933333333333334, + "waitP50": 13, + "waitP99": 96, + "waitMax": 96 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.9183673469387755, + "meanWait": 20.3, + "waitP50": 14, + "waitP99": 62, + "waitMax": 62 + } + ] + }, + { + "selector": "codel(baseline)", + "contentionWorstShareOverWeight": { + "mean": 0.01818181818181818, + "min": 0, + "max": 0.05454545454545454 + }, + "contentionJain": { + "mean": 0.42049894668433757, + "min": 0.4153846153846154, + "max": 0.4307276092837818 + }, + "detailSeed0": [ + { + "groupId": "bulk", + "dequeued": 240, + "weight": 1, + "share": 0.8, + "contentionShareOverWeight": 2.6666666666666665, + "meanWait": 871.9875, + "waitP50": 875, + "waitP99": 1644, + "waitMax": 1646 + }, + { + "groupId": "trickle-1", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0, + "meanWait": 1487.9333333333334, + "waitP50": 1569, + "waitP99": 1903, + "waitMax": 1903 + }, + { + "groupId": "trickle-2", + "dequeued": 30, + "weight": 1, + "share": 0.1, + "contentionShareOverWeight": 0.3333333333333333, + "meanWait": 1027.9333333333334, + "waitP50": 1014, + "waitP99": 1653, + "waitMax": 1653 + } + ] + } + ] +} \ No newline at end of file From b9a6557826e11825fc57ef0fc7ca2a685a105e8a Mon Sep 17 00:00:00 2001 From: Wes Mason Date: Thu, 23 Jul 2026 12:48:42 +0100 Subject: [PATCH 8/8] test(run-engine): address CK review (real age scenarios, honesty) Blind two-model review found the ckSkew/ckBalanced baseline starvation was a Redis lexicographic tie-break on equal enqueue timestamps, not age order. Redesigned scenarios so the backlog key keeps a genuinely old head and other keys arrive via poisson, so the baseline now exercises real age-order starvation (ckSkew baseline 0.187, worst key waits 1321ms; SFQ cuts it to 16ms). FINDINGS corrected: lead with per-key wait (contention share is volume-confounded for low-volume keys), and add the key fidelity limit the review surfaced: the harness serves one key per Lua call, while production batches and re-scores a served key mid-call, so this proves the ordering fix only at maxCount=1 and a production fix must advance per-key state inside the batched Lua. Softened 'viable' to 'worth a design spike'. --- .../run-queue/fairness-spike-ck/FINDINGS.md | 187 +++++---- .../fairness-spike-ck/harness/ckScenarios.ts | 33 +- .../fairness-spike-ck/results/ckBalanced.json | 312 +++++++-------- .../fairness-spike-ck/results/ckSkew.json | 354 +++++++++--------- 4 files changed, 461 insertions(+), 425 deletions(-) diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md index 9e80c0e829..a0e806fd62 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md @@ -2,98 +2,121 @@ Throwaway spike. Ships nothing; delete before any merge to main. -Bottom line: the base-queue spike's ranking carries over to the real seam, and -more starkly. At the concurrency-key grain the production baseline (serve the -oldest-head CK first) starves other keys outright, and virtual-time (SFQ) and -stride fix it. DRR is close. A CoDel wrapper on top of the broken baseline makes -it worse. Crucially, this was measured by driving the real -`dequeueMessagesFromCkQueueTracked` Lua and its real concurrency gating; only the -`ckIndex` scores were rewritten to express each discipline, so the result says a -production fix is viable (advance the ckIndex score by a fair discipline instead -of by head timestamp). - -## What was driven, and the one affordance - -Runs enqueue across many concurrency keys under a single base queue via the real -`RunQueue`. The per-CK pick is `ZRANGEBYSCORE ckIndexKey -inf now` inside the CK -Lua (lowest score first, score = head timestamp). For candidates the driver -rewrites those scores each round to encode the discipline's order; the baseline -leaves them as the Lua maintains them, which is production behaviour. Enqueue, -dequeue, per-CK and env concurrency, and ack all run through the real code. The -rescore sweep and an `onServiced(key)` hook are the only affordances; in -production that advance would live in the Lua that maintains `ckIndex`. A smoke -test confirms the baseline path reproduces age-order starvation. - -Fairness is measured as in the base-queue spike (arrival-aware contention share -and per-key wait), carrying forward its corrected metric and monotonic-floor -disciplines. All keys are equal weight (concurrency keys carry no configured -weight in production). +Bottom line: the base-queue spike's direction carries over to the real seam. At +the concurrency-key grain the production baseline (serve the oldest-head CK +first) starves keys that arrive behind a big backlog, and virtual-time (SFQ) and +stride fix it: they cut the starved key's wait from ~1300ms to ~20ms by making +the backlog key wait its turn. DRR does the same. A CoDel wrapper on the baseline +makes it worse. This was measured by driving the real +`dequeueMessagesFromCkQueueTracked` Lua and only rewriting `ckIndex` scores to +express each discipline. That is enough to say the ordering fix is worth a design +spike, but NOT that a production implementation is proven (see the fidelity +caveat: the spike serves one key per Lua call, and production dequeues in +batches). + +## What was driven, and the two things to know before reading numbers + +Runs enqueue across many concurrency keys under one base queue via the real +`RunQueue`. The per-CK pick is `ZRANGEBYSCORE ckIndexKey -inf now` in the CK Lua +(lowest score first, score = head timestamp). Candidates rewrite those scores +each round to encode discipline order; the baseline leaves them (production age +order). Enqueue, dequeue, concurrency gating and ack all run through the real +code. + +Two caveats a review forced, both load-bearing: + +1. Lead with wait, not contention share. The contention-share metric is + volume-confounded for low-volume keys (a key with 15 runs cannot take a third + of a long window even when served instantly), so on these scenarios it lands + around 0.7 to 0.9 for a discipline that has in fact eliminated the starvation. + The per-key wait is the clean signal. +2. The scenarios must give keys genuinely different head ages. An earlier version + enqueued every run at one timestamp; with tied `ckIndex` scores the real Lua + falls back to a lexicographic member-name tie-break, so the "baseline starves + the heavy key's rivals" result was actually "Redis sorts by name" and the + heavy key only won because "heavy" sorts before "light". Fixed: the backlog + key fires at once (persistently old head) and the other keys arrive via + poisson (distinct, later heads), so the baseline now exercises real age order. ## Results -`contWorstS/W` mean over 3 seeds (min..max). Higher is fairer. +`contWorstS/W` mean over 3 seeds (min..max), and the worst-served key's mean wait +(seed-a, logical ms). Read the wait column as the headline. -| scenario | baseline | sfq | drr | stride | codel(sfq) | codel(baseline) | -| ----------- | ------------------- | ------------------- | ------------------- | ------ | ---------- | --------------- | -| ckSkew | 0.000 | 0.982 (0.946..1.000)| 0.974 | 0.982 | 0.982 | 0.000 | -| ckBalanced | 0.000 | 0.987 | 0.987 | 0.987 | 0.987 | 0.000 | -| ckTrickle | 0.279 (0.254..0.291)| 0.909 (0.891..0.918)| 0.790 (0.769..0.818)| 0.909 | 0.909 | 0.018 | +| scenario | discipline | contWorstS/W | worst-key wait | backlog-key wait | +| ---------- | ---------- | ------------------- | -------------- | ---------------- | +| ckSkew | baseline | 0.187 (0.186..0.188)| 1321 | 872 | +| ckSkew | sfq | 0.723 (0.655..0.769)| 16 | 1150 | +| ckSkew | drr | 0.608 (0.556..0.648)| 21 | 1149 | +| ckTrickle | baseline | 0.279 (0.254..0.291)| 1339 | 872 | +| ckTrickle | sfq | 0.909 (0.891..0.918)| 24 | 1237 | +| ckTrickle | drr | 0.790 (0.769..0.818)| 30 | 1236 | -Per-key mean wait (seed-a, logical ms): +Full matrix (contWorstS/W mean over 3 seeds): -| scenario / discipline | starved-key wait | bulk-key wait | -| --------------------- | ---------------- | ------------- | -| ckSkew baseline | 1853 | 872 | -| ckSkew sfq | 265 | 1328 | -| ckTrickle baseline | 1252 | 872 | -| ckTrickle sfq | 22 | 1237 | - -Baseline age-order drives a light key's contention share to 0 on ckSkew (it is -served only after the heavy key drains) and its wait to 1853ms; SFQ cuts that to -265ms, and on ckTrickle from 1252 to 22ms, by making the bulk key wait its turn. +| scenario | baseline | sfq | drr | stride | codel(sfq) | codel(baseline) | +| ---------- | ------------------- | ------------------- | ------------------- | ------ | ---------- | ------------------- | +| ckSkew | 0.187 | 0.723 | 0.608 | 0.723 | 0.723 | 0.104 (0.000..0.157)| +| ckBalanced | 0.515 (0.444..0.600)| 0.611 (0.462..0.800)| 0.730 (0.615..0.909)| 0.611 | 0.611 | 0.464 | +| ckTrickle | 0.279 | 0.909 | 0.790 | 0.909 | 0.909 | 0.018 (0.000..0.055)| ## Verdict per discipline (at the concurrency-key grain) -- SFQ / stride: fix the starvation (about 0.98 on skew/balanced, 0.909 on - trickle), seed-stable, and cut the starved key's wait 7x (skew) to 57x - (trickle). Same as the base-queue spike. Recommended. -- DRR: close behind (0.974 skew, 0.987 balanced, 0.790 trickle). The trickle gap - is the same batch-drain interaction noted in the base-queue spike. -- CoDel(sfq): no harm here, matches SFQ. Unlike the base-queue trickle scenario - it did not overshoot, but it also added nothing; the fair base already bounds - sojourn. -- CoDel(baseline): harmful. Hoisting stale keys on top of the broken baseline - drove ckTrickle to 0.018 (worse than baseline's 0.279). A staleness monitor is - not a substitute for a fair base. -- Baseline (production age order): starves keys at this grain. Worse than the - base-queue proxy (0.000 vs 0.288 there), because a concurrency key with a - backlog of old heads is served to exhaustion before newer keys get a turn. This - is #2617 measured at the seam where it actually lives. - -## Caveats - -- The rescore sweep stands in for a production change to how `ckIndex` scores are - maintained. The spike proves the ordering fix works through the real dequeue - Lua and concurrency gating; it does not implement or measure the production - wiring (which would advance the score inside the enqueue/dequeue Lua and hold - per-key discipline state in Redis, not process memory). -- ckBalanced baseline reading 0.000 is partly a tie-break artifact: with equal - enqueue timestamps the age order is a deterministic member-name tie-break that - parks one key last during a short contention window. The direction (baseline - unfair, candidates fair) is right; lean on ckSkew and the wait numbers for the - magnitude. +- SFQ / stride: fix the starvation. Contention share improves (skew 0.187 to + 0.723, trickle 0.279 to 0.909) and the starved key's wait collapses (skew 1321 + to 16, trickle 1339 to 24) because the backlog key now waits its turn (its wait + rises 872 to ~1150 to 1237). Identical to each other on every scenario. + Recommended discipline for the fix. +- DRR: fixes the wait just as well (skew 21, trickle 30) and its contention share + tracks SFQ within noise (sometimes a little lower, sometimes higher, e.g. + balanced 0.730 vs 0.611). Fine. +- CoDel(sfq): no harm, matches SFQ to the decimal. Adds nothing on top of a fair + base. +- CoDel(baseline): harmful. Hoisting stale keys on top of the age-order baseline + drove ckSkew to 0.104 (below baseline's 0.187) and ckTrickle to 0.018 (below + 0.279). A staleness monitor is not a substitute for a fair base. +- Baseline (production age order): starves keys that queue behind a backlog + (ckSkew 0.187, worst key waits 1321ms; ckTrickle 0.279, 1339ms). It is roughly + fair when keys are symmetric (ckBalanced 0.515, though seed-variant 0.444 to + 0.600). This is the #2617 dynamic at the seam where it lives. + +## Fidelity caveat (the reason this is not "proven for production") + +The driver dequeues one key per Lua call (`maxCount = 1`) and rescores `ckIndex` +before each call. Production dequeues in batches (`maxCount` default 10). Inside a +batched CK-dequeue call the Lua re-scores each served key back to its head +timestamp as it goes, so a once-per-round rescore would only steer the FIRST pick +of a batch; the rest would follow head-timestamp order again. So this spike +demonstrates the ordering fix only in a one-key-per-call regime, which is not how +production dequeues. A real fix has to advance per-key discipline state inside the +Lua on every serve (and hold that state in Redis, not process memory). This spike +does not exercise that batch path, so the correct claim is "the ordering fix is +worth a design spike", not "a production fix is viable". + +## Other caveats + +- Contention share is volume-confounded (see above); the wait column is the + trustworthy signal, and the contention numbers should be read as directional. +- The DRR contention-share gap is NOT the base-queue spike's batch-drain artifact + (that harness batched; this one serves one key per call and advances DRR's + deficit every serve). The cause of DRR's slightly lower share here is not + established; its wait result is as good as SFQ's. +- Per-CK concurrency gating never binds in these runs (no per-CK limit is set, so + it collapses to the env limit), so the spike says nothing about the per-CK + concurrency-limit-multiplication half of #2617, which is out of scope. +- A rescore discipline advances its floor/ring state on the final no-op drain + round of an instant (order() is called before the empty dequeue). It is + self-correcting and does not corrupt the event-based metrics, but it is a minor + infidelity to a production per-serve advance. - Equal weights only; single shard, single base queue, single sequential - consumer; simulated holds on a logical clock; 3 seeds (shows the baseline's - failure and the virtual-time schemes' stability, not a statistical study). -- The other half of #2617, per-CK concurrency-limit multiplication, is a limit - problem not a dequeue-ordering one, and is out of scope here. + consumer; simulated holds on a logical clock; 3 seeds. ## Recommended direction -The fix for #2617 at the CK grain is to score `ckIndex` by a fair discipline -(SFQ/stride virtual time, or DRR) instead of by head timestamp, advancing the -per-key state inside the Lua that maintains `ckIndex`. Both spikes agree on the -discipline; this one shows it works at the real seam. Next step past the spike is -a design for holding per-key virtual-time state in Redis and advancing it in the -enqueue/dequeue Lua, plus the multi-shard/multi-consumer story neither spike -covers. +Score `ckIndex` by a fair discipline (SFQ/stride virtual time, or DRR) instead of +by head timestamp. Both spikes agree on the discipline and this one shows the +ordering fix works through the real dequeue path at `maxCount = 1`. The design +spike past this needs to: advance per-key virtual-time state inside the batched +CK-dequeue Lua (the `maxCount > 1` path this spike did not exercise), hold that +state in Redis for the multi-consumer case, and address the per-CK +concurrency-limit multiplication that is the other half of #2617. diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts index ce670ad888..a17bca0ef9 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts @@ -4,31 +4,44 @@ import type { WorkloadConfig } from "../../fairness-spike/harness/workload.js"; * Concurrency-key scenarios. Each "tenant" in the workload is a concurrency key * under one shared base queue. All keys are equal weight (concurrency keys carry * no configured weight in production). Seed is set per-run by the bench. + * + * Every scenario gives keys genuinely divergent head ages (a bulk key with a + * same-time backlog keeps an old head; other keys arrive via poisson so their + * heads are distinct and later). This is deliberate: if all runs shared one + * enqueue timestamp the ckIndex scores would tie and the real Lua's + * ZRANGEBYSCORE would fall back to a lexicographic member-name tie-break, which + * would make the baseline look starved for reasons that have nothing to do with + * age order. We want the baseline to exercise the real age dynamic #2617 + * describes. */ export const CK_SCENARIOS: Record> = { - // The direct #2617 reproduction: one key with a big backlog vs many light keys. + // #2617 classic: one key fires a big backlog at once (its head stays old), four + // other keys trickle in. Age-order serves the old-headed backlog to exhaustion + // and starves the others. ckSkew: { envConcurrencyLimit: 4, tenants: [ { tenantId: "heavy", runCount: 240, holdMsMean: 25 }, - { tenantId: "light-1", runCount: 15, holdMsMean: 25 }, - { tenantId: "light-2", runCount: 15, holdMsMean: 25 }, - { tenantId: "light-3", runCount: 15, holdMsMean: 25 }, - { tenantId: "light-4", runCount: 15, holdMsMean: 25 }, + { tenantId: "light-1", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + { tenantId: "light-2", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + { tenantId: "light-3", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, + { tenantId: "light-4", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 }, ], }, + // Symmetric keys, all arriving via poisson at the same rate. Baseline should be + // roughly fair here (no key is systematically older), which is the sanity check. ckBalanced: { envConcurrencyLimit: 4, tenants: [ - { tenantId: "k-a", runCount: 60, holdMsMean: 25 }, - { tenantId: "k-b", runCount: 60, holdMsMean: 25 }, - { tenantId: "k-c", runCount: 60, holdMsMean: 25 }, - { tenantId: "k-d", runCount: 60, holdMsMean: 25 }, + { tenantId: "k-a", runCount: 60, arrival: "poisson", ratePerSec: 20, holdMsMean: 25 }, + { tenantId: "k-b", runCount: 60, arrival: "poisson", ratePerSec: 20, holdMsMean: 25 }, + { tenantId: "k-c", runCount: 60, arrival: "poisson", ratePerSec: 20, holdMsMean: 25 }, + { tenantId: "k-d", runCount: 60, arrival: "poisson", ratePerSec: 20, holdMsMean: 25 }, ], }, - // A bulk key plus keys whose runs trickle in, to exercise wait and CoDel. + // A bulk backlog plus two keys trickling in slowly, to exercise wait and CoDel. ckTrickle: { envConcurrencyLimit: 4, tenants: [ diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckBalanced.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckBalanced.json index 51bba2f97b..fff0db0f81 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckBalanced.json +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckBalanced.json @@ -15,14 +15,14 @@ { "selector": "baseline", "contentionWorstShareOverWeight": { - "mean": 0, - "min": 0, - "max": 0 + "mean": 0.5148148148148147, + "min": 0.4444444444444444, + "max": 0.6 }, "contentionJain": { - "mean": 0.75, - "min": 0.75, - "max": 0.75 + "mean": 0.8491221317705939, + "min": 0.7422680412371133, + "max": 0.9433962264150942 }, "detailSeed0": [ { @@ -30,58 +30,58 @@ "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.3333333333333333, - "meanWait": 217.4, - "waitP50": 239, - "waitP99": 412, - "waitMax": 412 + "contentionShareOverWeight": 2, + "meanWait": 3.3333333333333335, + "waitP50": 0, + "waitP99": 31, + "waitMax": 31 }, { "groupId": "k-b", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.3333333333333333, - "meanWait": 580.2, - "waitP50": 565, - "waitP99": 771, - "waitMax": 771 + "contentionShareOverWeight": 0.5, + "meanWait": 0.38333333333333336, + "waitP50": 0, + "waitP99": 18, + "waitMax": 18 }, { "groupId": "k-c", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.3333333333333333, - "meanWait": 995.8, - "waitP50": 1004, - "waitP99": 1185, - "waitMax": 1185 + "contentionShareOverWeight": 0.8333333333333334, + "meanWait": 3.4, + "waitP50": 0, + "waitP99": 31, + "waitMax": 31 }, { "groupId": "k-d", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 0, - "meanWait": 1420.75, - "waitP50": 1393, - "waitP99": 1668, - "waitMax": 1668 + "contentionShareOverWeight": 0.6666666666666666, + "meanWait": 2.4, + "waitP50": 0, + "waitP99": 36, + "waitMax": 36 } ] }, { "selector": "sfq", "contentionWorstShareOverWeight": { - "mean": 0.9874476987447699, - "min": 0.9874476987447699, - "max": 0.9874476987447699 + "mean": 0.6109890109890109, + "min": 0.46153846153846156, + "max": 0.8 }, "contentionJain": { - "mean": 0.999947482669281, - "min": 0.999947482669281, - "max": 0.999947482669281 + "mean": 0.9036239034827617, + "min": 0.8711340206185566, + "max": 0.9433962264150942 }, "detailSeed0": [ { @@ -89,58 +89,58 @@ "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 862.6333333333333, - "waitP50": 840, - "waitP99": 1649, - "waitMax": 1649 + "contentionShareOverWeight": 1.5384615384615385, + "meanWait": 3.933333333333333, + "waitP50": 0, + "waitP99": 36, + "waitMax": 36 }, { "groupId": "k-b", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 0.9874476987447699, - "meanWait": 863.1833333333333, - "waitP50": 834, - "waitP99": 1668, - "waitMax": 1668 + "contentionShareOverWeight": 0.46153846153846156, + "meanWait": 0.38333333333333336, + "waitP50": 0, + "waitP99": 18, + "waitMax": 18 }, { "groupId": "k-c", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 864.7333333333333, - "waitP50": 839, - "waitP99": 1652, - "waitMax": 1652 + "contentionShareOverWeight": 1.0769230769230769, + "meanWait": 2.6166666666666667, + "waitP50": 0, + "waitP99": 28, + "waitMax": 28 }, { "groupId": "k-d", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 863.8, - "waitP50": 842, - "waitP99": 1650, - "waitMax": 1650 + "contentionShareOverWeight": 0.9230769230769231, + "meanWait": 2.5166666666666666, + "waitP50": 0, + "waitP99": 36, + "waitMax": 36 } ] }, { "selector": "drr", "contentionWorstShareOverWeight": { - "mean": 0.9874476987447699, - "min": 0.9874476987447699, - "max": 0.9874476987447699 + "mean": 0.7303807303807304, + "min": 0.6153846153846154, + "max": 0.9090909090909091 }, "contentionJain": { - "mean": 0.999947482669281, - "min": 0.999947482669281, - "max": 0.999947482669281 + "mean": 0.9085911382775246, + "min": 0.8535353535353534, + "max": 0.9918032786885248 }, "detailSeed0": [ { @@ -148,58 +148,58 @@ "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 851.85, - "waitP50": 821, - "waitP99": 1651, - "waitMax": 1651 + "contentionShareOverWeight": 1.6923076923076923, + "meanWait": 3.466666666666667, + "waitP50": 0, + "waitP99": 33, + "waitMax": 33 }, { "groupId": "k-b", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 857.4166666666666, - "waitP50": 823, - "waitP99": 1652, - "waitMax": 1652 + "contentionShareOverWeight": 0.6153846153846154, + "meanWait": 0.38333333333333336, + "waitP50": 0, + "waitP99": 18, + "waitMax": 18 }, { "groupId": "k-c", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 865.65, - "waitP50": 838, - "waitP99": 1652, - "waitMax": 1652 + "contentionShareOverWeight": 0.9230769230769231, + "meanWait": 3.216666666666667, + "waitP50": 0, + "waitP99": 28, + "waitMax": 28 }, { "groupId": "k-d", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 0.9874476987447699, - "meanWait": 871.7, - "waitP50": 839, - "waitP99": 1661, - "waitMax": 1661 + "contentionShareOverWeight": 0.7692307692307693, + "meanWait": 2.2333333333333334, + "waitP50": 0, + "waitP99": 36, + "waitMax": 36 } ] }, { "selector": "stride", "contentionWorstShareOverWeight": { - "mean": 0.9874476987447699, - "min": 0.9874476987447699, - "max": 0.9874476987447699 + "mean": 0.6109890109890109, + "min": 0.46153846153846156, + "max": 0.8 }, "contentionJain": { - "mean": 0.999947482669281, - "min": 0.999947482669281, - "max": 0.999947482669281 + "mean": 0.9036239034827617, + "min": 0.8711340206185566, + "max": 0.9433962264150942 }, "detailSeed0": [ { @@ -207,58 +207,58 @@ "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 862.6333333333333, - "waitP50": 840, - "waitP99": 1649, - "waitMax": 1649 + "contentionShareOverWeight": 1.5384615384615385, + "meanWait": 3.933333333333333, + "waitP50": 0, + "waitP99": 36, + "waitMax": 36 }, { "groupId": "k-b", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 0.9874476987447699, - "meanWait": 863.1833333333333, - "waitP50": 834, - "waitP99": 1668, - "waitMax": 1668 + "contentionShareOverWeight": 0.46153846153846156, + "meanWait": 0.38333333333333336, + "waitP50": 0, + "waitP99": 18, + "waitMax": 18 }, { "groupId": "k-c", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 864.7333333333333, - "waitP50": 839, - "waitP99": 1652, - "waitMax": 1652 + "contentionShareOverWeight": 1.0769230769230769, + "meanWait": 2.6166666666666667, + "waitP50": 0, + "waitP99": 28, + "waitMax": 28 }, { "groupId": "k-d", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 863.8, - "waitP50": 842, - "waitP99": 1650, - "waitMax": 1650 + "contentionShareOverWeight": 0.9230769230769231, + "meanWait": 2.5166666666666666, + "waitP50": 0, + "waitP99": 36, + "waitMax": 36 } ] }, { "selector": "codel(sfq)", "contentionWorstShareOverWeight": { - "mean": 0.9874476987447699, - "min": 0.9874476987447699, - "max": 0.9874476987447699 + "mean": 0.6109890109890109, + "min": 0.46153846153846156, + "max": 0.8 }, "contentionJain": { - "mean": 0.999947482669281, - "min": 0.999947482669281, - "max": 0.999947482669281 + "mean": 0.9036239034827617, + "min": 0.8711340206185566, + "max": 0.9433962264150942 }, "detailSeed0": [ { @@ -266,58 +266,58 @@ "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 862.6333333333333, - "waitP50": 840, - "waitP99": 1649, - "waitMax": 1649 + "contentionShareOverWeight": 1.5384615384615385, + "meanWait": 3.933333333333333, + "waitP50": 0, + "waitP99": 36, + "waitMax": 36 }, { "groupId": "k-b", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 0.9874476987447699, - "meanWait": 863.1833333333333, - "waitP50": 834, - "waitP99": 1668, - "waitMax": 1668 + "contentionShareOverWeight": 0.46153846153846156, + "meanWait": 0.38333333333333336, + "waitP50": 0, + "waitP99": 18, + "waitMax": 18 }, { "groupId": "k-c", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 864.7333333333333, - "waitP50": 839, - "waitP99": 1652, - "waitMax": 1652 + "contentionShareOverWeight": 1.0769230769230769, + "meanWait": 2.6166666666666667, + "waitP50": 0, + "waitP99": 28, + "waitMax": 28 }, { "groupId": "k-d", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.00418410041841, - "meanWait": 863.8, - "waitP50": 842, - "waitP99": 1650, - "waitMax": 1650 + "contentionShareOverWeight": 0.9230769230769231, + "meanWait": 2.5166666666666666, + "waitP50": 0, + "waitP99": 36, + "waitMax": 36 } ] }, { "selector": "codel(baseline)", "contentionWorstShareOverWeight": { - "mean": 0, - "min": 0, - "max": 0 + "mean": 0.4641604010025063, + "min": 0.4, + "max": 0.5714285714285714 }, "contentionJain": { - "mean": 0.75, - "min": 0.75, - "max": 0.75 + "mean": 0.7955765336918118, + "min": 0.7112903225806452, + "max": 0.8474576271186439 }, "detailSeed0": [ { @@ -325,44 +325,44 @@ "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.3333333333333333, - "meanWait": 217.4, - "waitP50": 239, - "waitP99": 412, - "waitMax": 412 + "contentionShareOverWeight": 2.0952380952380953, + "meanWait": 3.3, + "waitP50": 0, + "waitP99": 30, + "waitMax": 30 }, { "groupId": "k-b", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.3333333333333333, - "meanWait": 580.2, - "waitP50": 565, - "waitP99": 771, - "waitMax": 771 + "contentionShareOverWeight": 0.5714285714285714, + "meanWait": 0.38333333333333336, + "waitP50": 0, + "waitP99": 18, + "waitMax": 18 }, { "groupId": "k-c", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 1.3333333333333333, - "meanWait": 995.8, - "waitP50": 1004, - "waitP99": 1185, - "waitMax": 1185 + "contentionShareOverWeight": 0.5714285714285714, + "meanWait": 3.4833333333333334, + "waitP50": 0, + "waitP99": 53, + "waitMax": 53 }, { "groupId": "k-d", "dequeued": 60, "weight": 1, "share": 0.25, - "contentionShareOverWeight": 0, - "meanWait": 1420.75, - "waitP50": 1393, - "waitP99": 1668, - "waitMax": 1668 + "contentionShareOverWeight": 0.7619047619047619, + "meanWait": 2.0833333333333335, + "waitP50": 0, + "waitP99": 23, + "waitMax": 23 } ] } diff --git a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckSkew.json b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckSkew.json index 2887bdca34..255836ca51 100644 --- a/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckSkew.json +++ b/internal-packages/run-engine/src/run-queue/fairness-spike-ck/results/ckSkew.json @@ -16,14 +16,14 @@ { "selector": "baseline", "contentionWorstShareOverWeight": { - "mean": 0, - "min": 0, - "max": 0 + "mean": 0.1866549331190391, + "min": 0.1858108108108108, + "max": 0.1877133105802048 }, "contentionJain": { - "mean": 0.278764478764479, - "min": 0.278764478764479, - "max": 0.278764478764479 + "mean": 0.2975688789730731, + "min": 0.29443196433164714, + "max": 0.3000753476265495 }, "detailSeed0": [ { @@ -31,7 +31,7 @@ "dequeued": 240, "weight": 1, "share": 0.8, - "contentionShareOverWeight": 4.2105263157894735, + "contentionShareOverWeight": 4.067796610169491, "meanWait": 871.9875, "waitP50": 875, "waitP99": 1644, @@ -42,58 +42,58 @@ "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.2631578947368421, - "meanWait": 1696.9333333333334, - "waitP50": 1688, - "waitP99": 1761, - "waitMax": 1761 + "contentionShareOverWeight": 0.2542372881355932, + "meanWait": 1321, + "waitP50": 1239, + "waitP99": 1653, + "waitMax": 1653 }, { "groupId": "light-2", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.2631578947368421, - "meanWait": 1809.8666666666666, - "waitP50": 1818, - "waitP99": 1838, - "waitMax": 1838 + "contentionShareOverWeight": 0.23728813559322035, + "meanWait": 981.6666666666666, + "waitP50": 1043, + "waitP99": 1659, + "waitMax": 1659 }, { "groupId": "light-3", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.2631578947368421, - "meanWait": 1878.0666666666666, - "waitP50": 1868, - "waitP99": 1934, - "waitMax": 1934 + "contentionShareOverWeight": 0.1864406779661017, + "meanWait": 679.1333333333333, + "waitP50": 753, + "waitP99": 1661, + "waitMax": 1661 }, { "groupId": "light-4", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0, - "meanWait": 2028.8666666666666, - "waitP50": 2002, - "waitP99": 2136, - "waitMax": 2136 + "contentionShareOverWeight": 0.2542372881355932, + "meanWait": 915, + "waitP50": 839, + "waitP99": 1677, + "waitMax": 1677 } ] }, { "selector": "sfq", "contentionWorstShareOverWeight": { - "mean": 0.981981981981982, - "min": 0.9459459459459459, - "max": 1 + "mean": 0.7228911750188346, + "min": 0.6547619047619048, + "max": 0.7692307692307693 }, "contentionJain": { - "mean": 0.9997566909975668, - "min": 0.9992700729927005, - "max": 1 + "mean": 0.8592005815786168, + "min": 0.8431297709923665, + "max": 0.8739841688654353 }, "detailSeed0": [ { @@ -101,69 +101,69 @@ "dequeued": 240, "weight": 1, "share": 0.8, - "contentionShareOverWeight": 1, - "meanWait": 1328.4416666666666, - "waitP50": 1347, - "waitP99": 2116, - "waitMax": 2118 + "contentionShareOverWeight": 1.8617021276595742, + "meanWait": 1149.6333333333334, + "waitP50": 1193, + "waitP99": 2099, + "waitMax": 2110 }, { "groupId": "light-1", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.53333333333336, - "waitP50": 264, - "waitP99": 568, - "waitMax": 568 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 16.2, + "waitP50": 16, + "waitP99": 40, + "waitMax": 40 }, { "groupId": "light-2", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.53333333333336, - "waitP50": 293, - "waitP99": 550, - "waitMax": 550 + "contentionShareOverWeight": 0.7446808510638298, + "meanWait": 16.066666666666666, + "waitP50": 14, + "waitP99": 48, + "waitMax": 48 }, { "groupId": "light-3", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.8, - "waitP50": 265, - "waitP99": 558, - "waitMax": 558 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 13, + "waitP50": 10, + "waitP99": 41, + "waitMax": 41 }, { "groupId": "light-4", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 262, - "waitP50": 279, - "waitP99": 562, - "waitMax": 562 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 9.533333333333333, + "waitP50": 9, + "waitP99": 24, + "waitMax": 24 } ] }, { "selector": "drr", "contentionWorstShareOverWeight": { - "mean": 0.9740259740259741, - "min": 0.974025974025974, - "max": 0.974025974025974 + "mean": 0.6077242434174587, + "min": 0.5555555555555555, + "max": 0.648148148148148 }, "contentionJain": { - "mean": 0.9973086627417996, - "min": 0.9973086627417996, - "max": 0.9973086627417996 + "mean": 0.6987503929570632, + "min": 0.6743596514391338, + "max": 0.7129584352078239 }, "detailSeed0": [ { @@ -171,69 +171,69 @@ "dequeued": 240, "weight": 1, "share": 0.8, - "contentionShareOverWeight": 1.1038961038961037, - "meanWait": 1327.3375, - "waitP50": 1347, - "waitP99": 2116, - "waitMax": 2118 + "contentionShareOverWeight": 2.3893805309734515, + "meanWait": 1149.1125, + "waitP50": 1190, + "waitP99": 2099, + "waitMax": 2110 }, { "groupId": "light-1", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.974025974025974, - "meanWait": 256.6, - "waitP50": 260, - "waitP99": 552, - "waitMax": 552 + "contentionShareOverWeight": 0.6637168141592921, + "meanWait": 20.866666666666667, + "waitP50": 20, + "waitP99": 45, + "waitMax": 45 }, { "groupId": "light-2", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.974025974025974, - "meanWait": 264.6, - "waitP50": 267, - "waitP99": 570, - "waitMax": 570 + "contentionShareOverWeight": 0.6194690265486725, + "meanWait": 16.866666666666667, + "waitP50": 13, + "waitP99": 35, + "waitMax": 35 }, { "groupId": "light-3", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.974025974025974, - "meanWait": 270.46666666666664, - "waitP50": 289, - "waitP99": 561, - "waitMax": 561 + "contentionShareOverWeight": 0.6637168141592921, + "meanWait": 12.266666666666667, + "waitP50": 11, + "waitP99": 42, + "waitMax": 42 }, { "groupId": "light-4", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.974025974025974, - "meanWait": 279.4, - "waitP50": 292, - "waitP99": 580, - "waitMax": 580 + "contentionShareOverWeight": 0.6637168141592921, + "meanWait": 16, + "waitP50": 12, + "waitP99": 57, + "waitMax": 57 } ] }, { "selector": "stride", "contentionWorstShareOverWeight": { - "mean": 0.981981981981982, - "min": 0.9459459459459459, - "max": 1 + "mean": 0.7228911750188346, + "min": 0.6547619047619048, + "max": 0.7692307692307693 }, "contentionJain": { - "mean": 0.9997566909975668, - "min": 0.9992700729927005, - "max": 1 + "mean": 0.8592005815786168, + "min": 0.8431297709923665, + "max": 0.8739841688654353 }, "detailSeed0": [ { @@ -241,69 +241,69 @@ "dequeued": 240, "weight": 1, "share": 0.8, - "contentionShareOverWeight": 1, - "meanWait": 1328.4416666666666, - "waitP50": 1347, - "waitP99": 2116, - "waitMax": 2118 + "contentionShareOverWeight": 1.8617021276595742, + "meanWait": 1149.6333333333334, + "waitP50": 1193, + "waitP99": 2099, + "waitMax": 2110 }, { "groupId": "light-1", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.53333333333336, - "waitP50": 264, - "waitP99": 568, - "waitMax": 568 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 16.2, + "waitP50": 16, + "waitP99": 40, + "waitMax": 40 }, { "groupId": "light-2", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.53333333333336, - "waitP50": 293, - "waitP99": 550, - "waitMax": 550 + "contentionShareOverWeight": 0.7446808510638298, + "meanWait": 16.066666666666666, + "waitP50": 14, + "waitP99": 48, + "waitMax": 48 }, { "groupId": "light-3", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.8, - "waitP50": 265, - "waitP99": 558, - "waitMax": 558 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 13, + "waitP50": 10, + "waitP99": 41, + "waitMax": 41 }, { "groupId": "light-4", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 262, - "waitP50": 279, - "waitP99": 562, - "waitMax": 562 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 9.533333333333333, + "waitP50": 9, + "waitP99": 24, + "waitMax": 24 } ] }, { "selector": "codel(sfq)", "contentionWorstShareOverWeight": { - "mean": 0.981981981981982, - "min": 0.9459459459459459, - "max": 1 + "mean": 0.7228911750188346, + "min": 0.6547619047619048, + "max": 0.7692307692307693 }, "contentionJain": { - "mean": 0.9997566909975668, - "min": 0.9992700729927005, - "max": 1 + "mean": 0.8592005815786168, + "min": 0.8431297709923665, + "max": 0.8739841688654353 }, "detailSeed0": [ { @@ -311,69 +311,69 @@ "dequeued": 240, "weight": 1, "share": 0.8, - "contentionShareOverWeight": 1, - "meanWait": 1328.4416666666666, - "waitP50": 1347, - "waitP99": 2116, - "waitMax": 2118 + "contentionShareOverWeight": 1.8617021276595742, + "meanWait": 1149.6333333333334, + "waitP50": 1193, + "waitP99": 2099, + "waitMax": 2110 }, { "groupId": "light-1", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.53333333333336, - "waitP50": 264, - "waitP99": 568, - "waitMax": 568 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 16.2, + "waitP50": 16, + "waitP99": 40, + "waitMax": 40 }, { "groupId": "light-2", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.53333333333336, - "waitP50": 293, - "waitP99": 550, - "waitMax": 550 + "contentionShareOverWeight": 0.7446808510638298, + "meanWait": 16.066666666666666, + "waitP50": 14, + "waitP99": 48, + "waitMax": 48 }, { "groupId": "light-3", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 265.8, - "waitP50": 265, - "waitP99": 558, - "waitMax": 558 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 13, + "waitP50": 10, + "waitP99": 41, + "waitMax": 41 }, { "groupId": "light-4", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 1, - "meanWait": 262, - "waitP50": 279, - "waitP99": 562, - "waitMax": 562 + "contentionShareOverWeight": 0.7978723404255319, + "meanWait": 9.533333333333333, + "waitP50": 9, + "waitP99": 24, + "waitMax": 24 } ] }, { "selector": "codel(baseline)", "contentionWorstShareOverWeight": { - "mean": 0, + "mean": 0.10398894629340381, "min": 0, - "max": 0 + "max": 0.156794425087108 }, "contentionJain": { - "mean": 0.278764478764479, - "min": 0.278764478764479, - "max": 0.278764478764479 + "mean": 0.28297359307689535, + "min": 0.27694949009374037, + "max": 0.28877519486316655 }, "detailSeed0": [ { @@ -381,7 +381,7 @@ "dequeued": 240, "weight": 1, "share": 0.8, - "contentionShareOverWeight": 4.2105263157894735, + "contentionShareOverWeight": 4.137931034482758, "meanWait": 871.9875, "waitP50": 875, "waitP99": 1644, @@ -392,44 +392,44 @@ "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.2631578947368421, - "meanWait": 1696.9333333333334, - "waitP50": 1688, - "waitP99": 1761, - "waitMax": 1761 + "contentionShareOverWeight": 0.15517241379310343, + "meanWait": 1602.6, + "waitP50": 1484, + "waitP99": 2022, + "waitMax": 2022 }, { "groupId": "light-2", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.2631578947368421, - "meanWait": 1809.8666666666666, - "waitP50": 1818, - "waitP99": 1838, - "waitMax": 1838 + "contentionShareOverWeight": 0.24137931034482757, + "meanWait": 1044.3333333333333, + "waitP50": 1116, + "waitP99": 1842, + "waitMax": 1842 }, { "groupId": "light-3", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0.2631578947368421, - "meanWait": 1878.0666666666666, - "waitP50": 1868, - "waitP99": 1934, - "waitMax": 1934 + "contentionShareOverWeight": 0.20689655172413793, + "meanWait": 516.4, + "waitP50": 477, + "waitP99": 1653, + "waitMax": 1653 }, { "groupId": "light-4", "dequeued": 15, "weight": 1, "share": 0.05, - "contentionShareOverWeight": 0, - "meanWait": 2028.8666666666666, - "waitP50": 2002, - "waitP99": 2136, - "waitMax": 2136 + "contentionShareOverWeight": 0.2586206896551724, + "meanWait": 820.6, + "waitP50": 732, + "waitP99": 1761, + "waitMax": 1761 } ] }