[GLinnik21]

Closes #923

What

A test is flaky when one job of one run gave it both a pass and a fail (the same commit and build; in practice a re-run attempt). This tracks that in our own CI, no vendor.

• tools/ci-test-results.py (new): turns the libtest test <name> ... ok|FAILED lines of a host unit job's log into a small JSON (stable libtest has no JSON format). Both host unit jobs in ci.yml upload it as test-results-<slug>-<run_attempt> (14 days; the attempt in the name avoids re-run collisions).
• tools/ci-history.py: for new or re-run ci runs that failed or have attempt > 1 and whose head repo is this one, downloads those artifacts in memory (one validated JSON member, size and name limits, nothing extracted) and stores one event per flaky (job, test) under flaky. Only tests seen both ways are stored, never the ~6000 passes; 90-day retention (dropped in fold), at most 100 events per run (more is an outage, not 100 flakes). Fork PRs are skipped: a fork runs its own copy of ci.yml, so its artifact is attacker-authored. Jobs are never compared with each other (default vs hostsim are different builds).
• tools/ci-summary.py: derives flaky per (job, test): first/last seen, run count, status Active / Quarantined (in ci/flaky.txt) / Fixed (no flake for 30 days), and new (first flake within 24 h). No blocking gate.
• ci/flaky.txt (new, empty, # count: 0, shrink-only like ci/allow/layers.txt): quarantine list, test<TAB>reason. Read from main by both writers of the summary (ci-metrics.yml and build-bench.yml). Nothing skips quarantined tests yet.
• site/ci/index.html: a "Flaky tests" panel in the page's existing style (new-flake note, active list, fixed under a disclosure); every name is esc()'d.
• Docs: docs/agent-reference.md.

Verification

• Host tests: tools/test_ci_test_results.py (new, added to ci/check-python-steps.txt), tools/test_ci_history.py (+9 flaky cases incl. hostile zips, fork skip, rerun replacement, idempotent bytes, retention, per-run limit, failed download), tools/test_ci_summary.py (+9), ci/test_ci_workflows.py (pins both jobs' upload names against the collector's regex, and the quarantine wiring).
• make check on the committed tree: exit 0, tree clean afterwards. Last test result: line: test result: ok. 1857 passed; 0 failed; 64 ignored; 0 measured; 0 filtered out; finished in 55.72s.
• Page panel rendered under node with a stub DOM (hostile test name comes out escaped).
• Not proven: the live path (artifact upload on a real runner, gh api .../zip download) only runs in GitHub; the first re-run after merge will show it.

build-bench

This touches .github/workflows/ but changes no compiled input (no Cargo, Makefile or build.rs change); the workflow edits add two steps to existing test jobs. I did not run make build-bench: it takes every check slot exclusively and would stall the other lanes, and it would measure nothing this diff changes. No before/after table is given.

🤖 Generated with Claude Code