Make the benchmarks able to resolve real effects - #15

An A/A run of the new workflow, with baseline-ref and candidate-ref both set to
main, reported a 0.859s improvement on identical code. The A/A control was flat
at -0.060s, so slot ordering was not the cause.

The cause was that each arm seeded its own cache entry. A cache entry's download
throughput depends on where the service placed the stored blob, and that
placement is fixed for the life of the entry. In one job on one runner the
baseline blob was served at 59.4 and 60.3 MB/s while the candidate blob was
served at 131.8 and 105.1 MB/s, and the same ordering held on every runner.
Because that bias is constant across runners rather than random, pairing cannot
remove it, and additional samples only tighten the interval around the wrong
answer. The arm was confounded with the blob.

A single entry is now seeded, from the candidate so the wrapper cache is
populated, and both arms restore it. The blob is therefore held constant across
the comparison by construction rather than assumed not to matter.

The README documents the confound and states that an A/A run is the check to
perform after changing the harness.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

With one shared cache entry, running the whole matrix at once puts every runner
on the same blob at the same moment. At 20 concurrent runners the mean restore
rose from about 3s to about 8s and the noise floor from 0.34s to 1.89s, which is
enough to hide the effect under test. That is contention no real workflow would
experience, and it costs sensitivity for nothing.

The matrix now runs in waves of four, so each measurement reflects an ordinary
restore.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

These two workflows still measured the way the focused benchmark did before it
was rebuilt: durations read from the Actions API's one-second clock, arms spread
across independent jobs, no intervals, and one seeded cache entry per arm. Their
sub-second results were therefore not evidence of anything, and the JDK cache
workflow carried the same arm-versus-blob confound that an A/A run exposed in
the focused benchmark.

scripts/paired.mjs now holds the pairing logic that all three share: reading
per-runner timing CSVs, grouping slots by arm, discarding runners that did not
complete every slot, deriving the noise floor from within-arm repeats, and
producing an interval, a permutation p-value and a verdict. The focused report is
rebuilt on it unchanged in behaviour.

JDK cache becomes a two-arm ABBA comparison. One seed job populates a single
Maven entry and a single JDK entry, and both arms restore the same Maven entry,
so only the JDK entry differs between them. cache-jdk false and true are then
measured in one job per runner, with the tool cache and the local repository
cleared before every slot. The action ref is now an input so a branch can be
measured.

The version sweep generalises ABBA to seven arms: every runner sets up v1 through
main and then main back through v1, which places each version's two slots
symmetrically about the middle of the job so drift cancels. Every version that
supports cache-dependency-path restores one shared entry. v3 predates that input
and keys on pom.xml, so it necessarily has its own entry and the report says to
treat its difference with more caution. Versions are compared against main with
a paired interval per runner, and main against itself supplies the A/A control.

cache-read-only is not used in the sweep. It exists only on main, so applying it
to some arms and not others would make post-job behaviour asymmetric; every slot
restores on an exact primary-key hit, which actions/cache already skips saving.

Both matrices run in waves of four, because every runner pulls the same seeded
blob and running them all at once measures contention on the cache service that
no real workflow would see.

Tests cover the two properties the design depends on: that differencing within a
runner removes between-runner speed differences, and that the mirrored order
cancels drift that is linear across the job.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

The version sweep and the JDK cache benchmark both checked PetClinic out over
the workspace root. The JDK cache workflow then had no copy of the benchmark
scripts to invoke, and the version sweep, which checked them out to .benchmark,
put them inside PetClinic's basedir, where its nohttp check failed the seed
build on http:// URLs in a toolchains template.

Check the benchmark repository out at the root and PetClinic below it instead.
The scripts are then addressable as scripts/, PetClinic's build only sees its
own tree, and the cache-identity markers move with it.

Also stop jdk-cache.sh from truncating the wrapper properties; the seed job
needs the real contents to build, and an idempotent append keeps the hashed
file identical between seeding and measuring.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

A measurement job is a couple of minutes of work, but a slot can stall
indefinitely on a cache download or a JDK fetch. One such runner in the version
sweep sat on a step that takes two seconds elsewhere, and with the default
six-hour job timeout it would have blocked the report for the rest of the day.

Bound the measurement jobs at 25 minutes. A stalled runner now fails quickly and
drops out of the report, which already discards runners that did not complete
every slot, and the remaining runners still produce a verdict.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

The JDK cache benchmark's A/A control failed: comparing the no-cache arm's own
first and last slot reported an improvement of 0.4s, 95% CI -0.7 to -0.1, on
slots that ran identical configuration. The headline verdict could not be
trusted while that was true.

The cause is that the first setup in a job pays costs the later ones do not:
DNS resolution, TLS handshakes to the cache service and to the JDK host, and a
cold page cache. That is a one-off spike at slot 1 rather than drift across the
job, so the mirrored slot order cannot cancel it, and it biases the arm that
happens to hold the first slot.

Run one unmeasured warm-up slot in every measurement job so the measured slots
all start from the same warmed state. Applied to all three workflows rather than
only the one where it was caught, since they share the structure that exposes it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

The sweep reported v2.5.1 as a 1.9s improvement over main. That number is real
but it means v2 does no caching and so never restores the Maven repository the
other versions spend most of their time on. Reported as an improvement it invites
exactly the wrong conclusion.

Report v1 and v2 as not comparable and say why. They are still measured, because
seeing what the caching versions spend their time on is the point of including
them.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

A slot in the version sweep sat on a cache restore for 214 seconds while its
neighbours in the same job took three. It is the second time a run has been held
open this way, and the version is incidental: the sweep restores the same 154 MiB
blob fifteen times per job across ten jobs, and the cache service occasionally
stalls one of them.

Cap each measured setup at three minutes. A stalled slot now fails its runner,
which the report discards along with the other incomplete ones, rather than
running out the job timeout. Narrow the sweep's waves to two, since it restores
three times as often per job as the two-arm workflows.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

A slot in the version sweep sat for 214 seconds on a setup that took 0 seconds
in the same job's mirrored slot, on the same runner and the same action version.
The action had already finished: it logged "maven cache is not found" and then
produced no further output until the job was cancelled.

The cause is in @actions/cache. Its promiseWithTimeout clears the timeout inside
a .then, which only runs when the raced promise fulfils. When the cache service
drops the blob mid-download, downloadToBuffer rejects, clearTimeout is skipped,
and the armed timer keeps the event loop alive for the remainder of the segment
timeout after all work is done. That default is 10 minutes.

Cap it at 2 minutes here. These restores move 154 MiB in about 3 seconds, so the
cap is far beyond any legitimate download and cannot abort real work; it only
bounds the idle. The stall filter already keeps such a runner out of the
estimate, so this is about not paying for the wait.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

The stall filter was only applied to the two-arm workflows, so the sweep still
formed its estimates from runners that had stalled on a restore. It showed:
its A/A control, comparing main with itself, reported 0.6s against a noise floor
of 0.157s.

Generalise the filter across arms. A runner whose two slots for any one version
disagree by more than the robust threshold is dropped, on the same reasoning as
before: which version a stall lands on is arbitrary, and the decision uses only
within-version spread, so it cannot bias the differences between versions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

The Maven configuration warm path was the last workflow still measuring the old
way. It ran the arms ABAB, which does not cancel drift across the job: with a
steady slowdown of d per slot the candidate is biased by +d, because its slots
sit later on average than the baseline's. It had no warm-up slot, so the first
setup's DNS, TLS and cold page cache costs landed entirely on the baseline. And
it summarised each configuration on its own, where one runner yields one paired
difference and no interval is possible.

Run the arms ABBA after a discarded warm-up slot, matching the other workflows,
and add a report job that pools across the matrix. Each of the 36 configurations
is a block: the arms are compared within it, on one runner, and the differences
are combined. That answers what this workflow actually asks — whether the
candidate differs from the baseline across configurations — and costs no extra
jobs. Breakdowns by operating system and cache profile get their own intervals,
and per-configuration numbers are published as single observations with no
verdict attached, because that is all they are.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

Two ways the reports could publish something misleading rather than fail.

formatInterval hard-coded "95% CI" while the interval it was given carries its
own confidence level, so a caller that widened or narrowed an interval would
have had it labelled 95% regardless. Read the level off the interval.

The reports also checked only some of the environment variables they go on to
render. A missing SETUP_JAVA_REPOSITORY or JAVA_VERSION reached the summary as
"undefined" instead of stopping the run. Check every variable a report renders,
in one shared helper, and name the ones that are missing.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

The version sweep used to place all seven versions in one table and compare each
against main. That is not a fair ranking: v1 and v2 are uncached, v3 uses an
older cache-key contract, and v5.6/main restore the Maven Wrapper in addition
to dependencies.

Keep the same-runner mirrored measurements, but report three cohorts: uncached
(v1/v2), dependency cache (v3/v4/v5.2), and dependency plus wrapper cache
(v5.6/main). Each cohort has its own reference and A/A control. Apply Holm's
step-down correction within a cohort so multiple comparisons do not turn an
uncertain p-value into a headline verdict. Retain the all-arm diagnostics for
runner quality and workload shape, explicitly not as a ranking.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

main is the newest code, so calling v5.6.0 a `regression` for being slower than
it states the finding backwards. Nothing regressed in v5.6.0; it shipped, and
then main got faster. The table was reading the difference in the direction that
made the older code look at fault for the improvement that came after it.

The two-arm workflows already have this right: they difference candidate minus
baseline, so the verdict describes the code under test. The sweep is the same
question with more baselines, so difference main minus each version and let the
verdict describe main. v5.6.0 now reads as a 1.303s `improvement` in main rather
than a 1.303s `regression` in v5.6.0, which is the same measurement said the
right way round.

Pin the direction with a test, because the sign is easy to flip back and the
resulting table is wrong in a way that still looks plausible.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1

The Maven configuration workflow was called a warm path and never had a seed
job, so its synthetic pom hashed to a key nothing had stored and every "warm"
restore was a miss. Its three cache profiles were three variations on a failed
lookup. The focused workflow, meanwhile, compared the same two refs on the same
warm restore as the version sweep.

Between them the benchmarks measured one quantity, repeatedly, and it is mostly
not a quantity setup-java controls: the transfer is handed to @actions/cache, so
a large fixture times the network. Identical code varied ninefold across runners
while the effect under test was a third of that spread.

Reorganise them by question. Cache value builds PetClinic for real, cached
against uncached, to establish the scale everything else is a fraction of. Cache
key stability asserts the key rather than timing it, because a key that moves
when it should not costs a whole miss. Action overhead keeps the configuration
matrix but seeds a megabyte-sized entry so restores genuinely hit and the
action's own work is what remains; its four cache profiles are now nested levels
that difference into a decomposition. Transfer overlap is the same measurement
at 160 MiB, where concurrency has something to hide. Cache save measures what
the first run pays, which nothing measured before.

Every action-overhead slot records cache-hit, and the report refuses to be read
normally when a profile did not do the work its name claims.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 71c45320-1417-4029-8402-69075d61dac1