bench_tools,ci: gate benchmark regressions on the confidence interval

[asaf-sw]

Problem

The benchmarking job fails on noise:

transfers_sequential_benchmark_vm: +31.59% (EXCEEDS 8.0% limit)

Same commit, two attempts:

Attempt	transfers_benchmark_cairo_native	transfers_sequential_benchmark_vm
1	-2.93% (196,992,050 ns)	+8.23%
4	-5.04% (159,334,692 ns)	+31.59%

The true change is 0%, so the observed noise floor reaches at least 31.59% while the limit is 8.0%. The cairo_native absolute time swung 197ms to 159ms between runs of the same binary, and the two benchmarks moved in opposite directions.

Two defects underneath:

1. The gate uses a point estimate and throws away the confidence interval. Criterion emits one and types/estimates.rs already deserializes it, but check_regressions compared mean.point_estimate against the limit, so a single noisy measurement fails the job.
2. Passing runs print no numbers. runner.rs printed per-benchmark results only after something had already failed; on success it printed just "All benchmarks passed regression check!". So the run-to-run noise was invisible, which is why the limit was never calibrated against it.

Change

• Gate on the lower bound of the change confidence interval instead of the point estimate. A benchmark counts as a regression only when even the most favourable reading of the measurement is a regression larger than the limit.
• Print one line per benchmark on both paths, with the point estimate, the interval, and the absolute time:

   ✓ transfers_benchmark_cairo_native: -2.93% [-6.40%, +0.55%] | 196992050.00ns
  ❌ transfers_benchmark_vm: +52.00% [+48.10%, +55.90%] | 500000000.00ns
        regression: confidence interval lower bound +48.10% EXCEEDS 40.0% limit
  

• Raise --regression-limit from 8.0 to 40.0, clear of observed noise with margin.
• Fix the absolute-time limits, which named benchmarks that do not exist and so had never applied (see below).
• Split the disk loading out of check_regressions into a pure compare_estimates, so the decision logic is unit testable at all. That is what the new tests exercise.

Verification

$ SEED=0 cargo nextest run -p bench_tools
Summary [0.021s] 17 tests run: 17 passed, 2 skipped

New tests in comparison_test.rs:

Test	Encodes
noisy_change_with_interval_below_the_limit_is_not_a_regression	A measurement too noisy to conclude from: point estimate above the limit, interval straddling it
change_with_interval_above_the_limit_is_a_regression	A real regression whose whole interval is above the limit MUST fail
change_below_the_limit_passes	The attempt-1 cairo_native numbers pass
absolute_limit_trips_independently_of_the_relative_check	The absolute backstop still works on its own
no_benchmarks_produces_no_comparisons	Empty input

cargo clippy -p bench_tools --all-targets -- -D warnings clean, YAML parses.

Which change actually rescues the observed failure

Worth being precise, because the two changes cover different failure modes. The observed run's intervals were tight: base vm [819.16, 872.45] ms, new vm [1091.6, 1127.5] ms. That puts the change interval at roughly [+25%, +38%], so the interval lower bound is about +25%.

Against the old 8.0 limit that still fails. It is the raised limit, not the confidence interval, that prevents this particular false regression. The interval gate covers the complementary case, where a single run is too noisy to support any conclusion, and it is a strict improvement in its own right, but it is not what rescues this run. Anyone reviewing the 35.0 number should weigh it on its own merits rather than assuming the interval gate is carrying it.

Note on the number

The limit is compared against the lower bound of the change confidence interval, not the point estimate. For the worst same-commit run observed, that lower bound is about +25%, so 40.0 clears it with roughly 15 points of margin, and a run noisier than anything seen so far still would not fail.

It is derived from four same-commit data points, which is thin. Now that passing runs print their numbers, the limit can be re-derived from the actual distribution after a few weeks. I left that instruction as a comment on the step.

The absolute-time backstop was dead (found in review)

An earlier revision of this PR claimed the absolute-time limits remained the backstop. They did not. They named transfers_benchmark_cairo_native and transfers_benchmark_vm, but the benchmarks are actually named transfers_sequential_benchmark_cairo_native and transfers_sequential_benchmark_vm (types/benchmark_config.rs:59,73). absolute_time_ns_limits.get(bench_name) never matched, so the limits guarded nothing.

Fixing the names alone would have made things worse. The real measurements from run 31621989610 are:

Benchmark	Base	PR (same commit, no causal change)	Old limit
transfers_sequential_benchmark_cairo_native	167.79 ms	159.33 ms	200 ms
transfers_sequential_benchmark_vm	843.24 ms	1109.60 ms	1000 ms

So the vm benchmark already exceeds its nominal 1000 ms limit on noise alone, and cairo_native has been observed at 197 ms against a 200 ms limit. Turning the limits on at those values would have converted a dead config into a live false-failure source.

This PR therefore fixes the names and sets values with headroom over the highest observed numbers: 300 ms and 1600 ms.

To stop this recurring, an absolute limit naming a benchmark that is not being compared is now a hard error rather than a silent no-op, covered by absolute_limit_naming_an_unknown_benchmark_panics.

🤖 Generated with Claude Code

[reviewable-StarkWare]

This change is 

[github-actions]

Artifacts upload workflows:

• Started at 2026-08-13T05:36:02Z
• Started at 2026-08-13T05:57:36Z
• Started at 2026-08-13T05:58:02Z
• Started at 2026-08-13T07:20:08Z
• Started at 2026-08-13T07:36:02Z
• Started at 2026-08-13T08:22:47Z

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e0c4df0. Configure here.}