Bench trace oracle: a marginal spot can redden CI on any line-shifting codegen change #346

Problem

./bench/ci pins deterministic LuaJIT counters against committed oracles. Two of the three are pure functions of the linked artifact and are byte-stable by construction. The third, the per-spec trace report, is not: it records which source locations reached a compiled trace (J* opcodes) or were blacklisted (I*), and trace formation is nondeterministic per process. trace_report.lua absorbs that with a majority vote over nine fresh LuaJIT processes, and the vote works — for spots whose per-process formation probability p sits near 0 or 1.

It does not fix a spot whose p sits in between, and such a spot can appear from nothing more than a line shift. LuaJIT's hot counters live in a small hashed table keyed by bytecode address, so removing a single local from an artifact reshuffles which counters collide. The tool's own header already records how sensitive this is: the same artifact measures p ~ 0.05 merely when loaded through a shorter relative path.

Bench.RefLoop.lua:32 JFUNCF — the entry of the ST thunk wrapping the hot loop — became such a spot when the single-use foreign-import fold (#251) removed one hoisted table from the artifact. Measured over raw single-process trials it forms with p roughly 0.6 to 0.85, drifting between measurement batches rather than sitting at a fixed value.

Why the existing knobs do not reach it

Raising the workload reps through 4, 8, 16 and 32 leaves p unchanged, which rules out the documented "not enough slack above the hot-call threshold" cause and confirms counter aliasing.

Raising the trial count does not help either, and the reason is worth stating: more trials sharpen the estimate of p, which only helps when p is a fixed value away from one half. Here p itself drifts, so more trials converge on a moving target, at linear cost in fresh processes.

What it costs

With nine trials, the probability that the vote lands on the wrong side is P(Binom(9, p) ≤ 4):

p = 0.8  ->  ~2% of runs flip
p = 0.7  ->  ~10%
p = 0.6  ->  ~27%

So this one golden line carries somewhere between a 2% and a 27% chance of reddening CI per run. Observed so far: of two CI runs that reached the bench step, one went red on it and one green. Ten consecutive local ./bench/ci runs matched, which is why "present" is the side currently pinned — it is what one machine produces every time and what CI reaches too, whereas pinning "absent" would fail locally on every run.

The general risk is the point: any codegen change that shifts lines in a bench artifact can push some borderline entry across, so the trace goldens carry a standing intermittent-red risk that is not attributable to the change under review.

Approach

Retry the bench step once on mismatch. Cheapest, and narrower than it looks: the only failures a retry absorbs are the coin-flip ones, because a genuine codegen change moves the counters deterministically and fails both attempts. Costs a doubled ~10-minute step on failure, and in principle masks a regression that itself lands near one half.

Quarantine named spots. Let a golden mark one line unpinnable and have the diff skip it. Keeps every other signal and makes the instability explicit and reviewed rather than silently absorbed. Costs a maintained exception list, and risks a real regression later hiding behind a quarantined line.

Classify marginal spots as a third state. Print a distinct token when the vote share falls in a middle band, so both machines emit the same thing for a borderline spot. This looks like a fix but is not: every band has edges, and a spot can sit on one. A 1/3–2/3 band leaves this spot at p ~ 0.6-0.85 straddling the upper edge; widening it to 0.1–0.9 catches this spot but puts the Bench.BindChain run wrapper the tool header records at p ~ 0.9 on the new edge.

Drop the signal class. Exclude artifact function-entry J*FUNCF states, keeping loop states and I*FUNCF blacklists. This does remove the failure mode — the marginal spots are entry traces racing their own inner loop's counter, which is exactly why the tool already excludes the spec harness's own wrappers. It also gives up real information: a compiled function entry is genuine codegen signal elsewhere in the corpus.

Suggested: the retry as the immediate mitigation, quarantine as the principled follow-up if it recurs on other specs. Avoid the band approach specifically because it resembles a fix without being one.

Prerequisites / Relations

Independent. Surfaced by #251 (PR #344), whose line shift pushed Bench.RefLoop.lua:32 onto the boundary, but the fragility is the oracle's own — any codegen change that shifts artifact lines can do the same to a different spot. Confined to bench/; no compiler code involved.

Acceptance criteria

• A codegen change that only shifts lines in a bench artifact does not redden CI through a marginal trace spot.
• A real change to which bytecodes trace or blacklist still fails the oracle.
• Whatever is chosen is recorded in bench/README.md next to the existing majority-vote explanation, since that text currently implies the vote is sufficient.