Problem
./bench/ci compares freshly generated LuaJIT counter reports against committed oracles in bench/goldens/. One of those reports, the per-spec trace report, records which artifact source locations ended up carrying a compiled trace or a blacklist, and whether a given location does is a per-process dice roll for spots sitting on LuaJIT's hot-count boundary. bench/tools/diff_counters absorbs that by re-measuring a report that differs from its golden only in the nondeterministic sets, passing if a fresh measurement matches.
When it does absorb a flip it prints the entries that flickered, deliberately — a golden entry that is not decisive is the one property a reviewer cannot see by reading the committed goldens, so swallowing it silently would hide the fact that the oracle is pinning something marginal. But the notice goes to stderr and the script exits 0, so it lands in a green step's log:
trace_ref_loop.txt: differs in the measured sets; re-measuring (1 of 2)
trace_ref_loop.txt: matched on re-measurement 1 -- the entries below are trace-formation noise, not a codegen change
@@ -5,7 +5,6 @@
Bench.RefLoop.lua:40 -- NYI: bytecode FNEW
bytecode end state (J*=compiled, I*=blacklisted):
Bench.RefLoop.lua:10 JFUNCF
- Bench.RefLoop.lua:32 JFUNCF
Bench.RefLoop.lua:34 JFORI
Bench.RefLoop.lua:34 JFORL
-counts: aborts=1 compiled=4 blacklisted=0
+counts: aborts=1 compiled=3 blacklisted=0
Nobody expands a green step. Measured at roughly 3% of runs on the current goldens, that means the signal is emitted a few times a year into a place no one reads.
Approach
Emit a GitHub Actions annotation alongside the stderr notice, so an absorbed flip appears on the PR itself rather than only in the step log:
if [ -n "${GITHUB_ACTIONS:-}" ]; then
printf '::warning::%s\n' "$name: absorbed a marginal trace spot on re-measurement $attempt"
fiGuarded on GITHUB_ACTIONS so a local ./bench/ci is unchanged.
The cost is that a vendor-neutral bench tool grows a branch that knows about one CI provider, which is why it was left out of the change that introduced the re-measurement rather than folded in. That is the decision to make here: accept the coupling for the visibility, or find a vendor-neutral equivalent (for instance, having bench/ci exit with a distinct non-failing status or write a marker file that a workflow step turns into an annotation, keeping the provider knowledge in the workflow).
Prerequisites / Relations
Independent; follows the re-measurement introduced in #346. Confined to bench/ plus possibly .github/workflows/ci.yaml; no compiler code involved.
Acceptance criteria
- An absorbed marginal trace spot is visible without expanding a green CI step's log.
- A local
./bench/ci run's output is unchanged.
- Whichever way the vendor coupling is resolved is recorded next to the existing explanation in
bench/README.md.
Problem
./bench/cicompares freshly generated LuaJIT counter reports against committed oracles inbench/goldens/. One of those reports, the per-spec trace report, records which artifact source locations ended up carrying a compiled trace or a blacklist, and whether a given location does is a per-process dice roll for spots sitting on LuaJIT's hot-count boundary.bench/tools/diff_countersabsorbs that by re-measuring a report that differs from its golden only in the nondeterministic sets, passing if a fresh measurement matches.When it does absorb a flip it prints the entries that flickered, deliberately — a golden entry that is not decisive is the one property a reviewer cannot see by reading the committed goldens, so swallowing it silently would hide the fact that the oracle is pinning something marginal. But the notice goes to stderr and the script exits 0, so it lands in a green step's log:
Nobody expands a green step. Measured at roughly 3% of runs on the current goldens, that means the signal is emitted a few times a year into a place no one reads.
Approach
Emit a GitHub Actions annotation alongside the stderr notice, so an absorbed flip appears on the PR itself rather than only in the step log:
Guarded on
GITHUB_ACTIONSso a local./bench/ciis unchanged.The cost is that a vendor-neutral bench tool grows a branch that knows about one CI provider, which is why it was left out of the change that introduced the re-measurement rather than folded in. That is the decision to make here: accept the coupling for the visibility, or find a vendor-neutral equivalent (for instance, having
bench/ciexit with a distinct non-failing status or write a marker file that a workflow step turns into an annotation, keeping the provider knowledge in the workflow).Prerequisites / Relations
Independent; follows the re-measurement introduced in #346. Confined to
bench/plus possibly.github/workflows/ci.yaml; no compiler code involved.Acceptance criteria
./bench/cirun's output is unchanged.bench/README.md.