ci: add a perf-movements reader for the benchmark pipelines

[antiguru]

Answering "what moved in performance" across the nightly, release-qualification and spec-sheet pipelines currently means opening a dozen job logs by hand and reading fixed-width tables. This adds mz-perf-movements, a skill plus a reader under misc/python/materialize/buildkite_insights/perf_movements/, that collects those numbers and ranks regressions and improvements together.

The pipelines publish two different kinds of evidence, and the reader keeps them apart. The feature benchmark, parallel benchmark and scalability benchmark each run both sides themselves against the merge base or a named ancestor and print a comparison table, so the movement is already computed in CI against a same-hardware baseline and only has to be recovered. The cluster spec sheet measures one build in isolation and uploads raw per-repetition CSV rows, so its baseline is reconstructed here as the median of preceding builds, which carries cross-run noise the self-comparing steps do not have. The skill says as much, so a spec-sheet movement is not read as equal evidence to a feature-benchmark movement of the same size.

Two decisions worth flagging for review. Direction is resolved per metric rather than taken from the sign of the change, because a parallel-benchmark qps rising is good while an avg rising is bad and both print as a positive number. And the caller's --threshold is the only floor applied: a step's own threshold is a build gate set well above the noise floor, so deferring to it would hide every movement CI tolerates, including the confirmation that an optimization landed.

Fetching reuses the existing buildkite_insights API, cache and builds layers, so a repeated run over the same window costs one fetch for the newest build. Where no Buildkite token is configured, --from-log and --from-csv parse files fetched by other means through the same code, and job logs are accepted both as plain text and in the JSON form the Buildkite MCP server saves.

Steps that publish neither a per-metric comparison nor per-repetition rows are excluded rather than approximated: limits, limits-instance-size, bounded-memory, bounded-memory-search, orchestratord-rolling-upgrade-downtime, cargo-bench, and the spec sheet's .cluster_object_limits.csv, whose headline number is a maximum healthy object count rather than a median. Richer history lives in the test_analytics database, which needs a separate credential; the skill points at it without depending on it.

Tests added in perf_movements_test.py cover the three log parsers, both spec-sheet CSV schemas, the CRLF line endings Buildkite actually serves, and the ranking and threshold behaviour. The fixtures are copied verbatim from nightly build 18282 and spec-sheet build 46, because the parsers depend on print statements elsewhere in the repo and go quiet rather than loud when those change.

Draft, because one path is still unexercised: the reader's own Buildkite fetching in collect.py has been read but not run, for want of a token in the development environment. Everything downstream of the fetch is verified against real build data through the offline flags. I will confirm the fetch path and undraft.

No release note, this is CI tooling with no user-visible surface.

🤖 Generated with Claude Code