fix(ci): two REQUIRED contexts sat behind a single self-hosted runner, and it went offline - #1135
fix(ci): two REQUIRED contexts sat behind a single self-hosted runner, and it went offline#1135avrabe wants to merge 1 commit into
Conversation
…, and it went offline
MERGES WERE BLOCKED REPO-WIDE. `Format` and `Version Pin Sweep` — two of the
nine required contexts — both targeted [self-hosted, linux, x64, light], a pool
with EXACTLY ONE runner (pulseengine-ci-01-8). That runner went offline. The
Format job was assigned to it, sat 600 seconds, and failed having executed ZERO
steps (empty step list, missing log blob — the signature of a job that never
started, not a lint failure).
Branch protection requires the context to PASS. A dead pool cannot produce a
pass, so nothing could merge — the deadlock my own notes warn about, arriving
from the runner side rather than from a renamed check.
Diagnosed against the local oracle first: `cargo fmt --check` is CLEAN on the
branch that "failed" it, and that branch touches ZERO .rs files. So the failure
was never about formatting.
fleet at the time of the move
light 0 online / 1 total <- both required contexts here
lean-mem 4 online / 4 total idle
rust-cpu 7 online / 7 total idle
Both jobs moved to `lean-mem`. THE RULE THIS ENCODES: a required context must
not depend on a single-runner pool. Capacity was never the problem here —
redundancy was, and the CI-capacity work (#1062) had been optimising the wrong
axis for these two jobs. Recorded at both sites so a future retarget does not
put a required context back behind one machine.
NOT changed: the four required contexts on ubuntu-latest, and Clippy/Rivet
Validation on rust-cpu (7 runners, genuine redundancy).
Refs #1062
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Closing — the premise was fixed at the fleet level, and better than this PR fixed itWhen I opened this, Re-measured today, the fleet has changed: The dead runner is back and two Why close rather than merge anywayThe PR is green (59/59) and
That is now false. Landing prose that asserts something untrue about the The rule survives the PRThe finding was never really "move these two jobs". It was:
And the sharper half, which is why my CI-capacity measurements missed it: every That invariant is worth enforcing rather than remembering, so it goes to No code change needed today. Thanks to whoever relabelled the runners. |
…y invariant a live incident proved (#1137) * chore(rivet): RQ-62-CLAIMCHECK absorbs the required-context redundancy invariant a live incident proved PR #1135 is CLOSED, not merged — its premise was fixed better at the fleet level than in this repo. The finding it produced is worth keeping, so it moves here rather than dying with the branch. WHAT HAPPENED (2026-09-02): the `light` pool had EXACTLY ONE runner and it went offline. `Format` and `Version Pin Sweep` — two REQUIRED contexts — were routed there, were assigned to a dead machine, sat 600s, and failed having executed ZERO steps. Branch protection needs a required context to PASS, and a dead pool cannot produce one, so nothing in the repo could merge. Resolved at the fleet level: the runner returned and two `rust-cpu` machines gained the `light` label, so `light` now has 3 online. My PR would have moved the two jobs to `lean-mem` — marginally more redundant, but its comment asserts "a pool with EXACTLY ONE RUNNER", which is now FALSE, and landing prose that is untrue about the live fleet is the class v0.61 was named for. WHY EVERY #1062 MEASUREMENT MISSED IT, which is the part worth keeping: they all asked "is this pool IDLE?". `light` was idle, which reads as healthy. None asked "HOW MANY MACHINES CAN SATISFY THIS LABEL?" — and idle and unavailable are indistinguishable in a utilisation reading. That is the THIRD variant of the same error this issue has produced; the first two were reading a rate off a single instant, in opposite directions. THE INVARIANT, now in the done-when so it is checked rather than remembered: a required context must not target a label satisfiable by fewer than two ONLINE runners. Any retarget must demonstrate the destination's online-runner count, not merely that the pool was idle. Refs #1062 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L * plan(v0.62): scope RQ-62-ARCHMODEL (#1136) — feature-loop steps 1-2 are NOT permanently N/A (8 -> 9) Maintainer decision on #1136: "We should at least define the basic how the compiler runs on a system and things we can address in aadl if it is timing or other things but the general we should address." THE CASE IS STRONGER THAN THE QUESTION ASSUMED. synth already produces AADL-shaped facts that nothing owns: --emit-wcet sound per-function cycle bounds -> thread Expected_WCET linear-memory base / globals / stack -> memory components reserved R9 / R10 / R11 / R12 -> processor binding per-target MPU region model -> memory protection the --safety-bounds envelope -> a system property, today PROSE in CLAUDE.md And the timing ALREADY FLOWS — reconstructed. CLAUDE.md records that synth's WCET sidecar is "gale spar's T3/T4 C_i input", and `spar insight` compares traces against Expected_BCET/WCET/Mean. So the number already reaches a spar analysis by a CONSUMER hand-building a scheduling model around a sidecar synth emits. That is a hand-written mirror of a shipped artifact — the exact failure the North Star names. synth should DECLARE its timing and layout; consumers should not reconstruct them. Precedent is modest: scry/spar/scry.aadl is 374 lines and declares itself the source-of-truth architecture with WIT hand-derived for v0.1 and spar-codegen as the migration path. Same sequencing here — model as source of truth, generation as migration, not prerequisite. INCREMENT 1 IS DEFINE-AND-MAP: define how a compiled artifact executes on a target, map what AADL expresses natively (TIMING FIRST — the bound exists and has a consumer), and state what AADL CANNOT carry so the boundary is written down rather than rediscovered. NOBODY IS BLOCKED ON THE MODEL. #1131 does not wait for this: RQ-62-EMBEDDER ships the layout contract as a document derived from the emitter, on its own timeline, because an integrator is blocked TODAY and an architecture model is not the unblocking artifact. The model becomes that document's source of truth afterwards. Coupled consequence: steps 1-2 stop being an N/A, so RQ-62-LOOPCONFORM must count them as a real step rather than a blank. ARTIFACT_FLOOR 517 -> 518, re-derived with `rivet list`, not computed. Refs #1136, #1131 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Merges are blocked repo-wide, and it is not a code failure
FormatandVersion Pin Sweep— two of the nine required contexts — bothtargeted
[self-hosted, linux, x64, light], a pool with exactly one runner(
pulseengine-ci-01-8). That runner went offline.The Format job was assigned to it, sat 600 seconds, and failed having
executed zero steps — empty step list, missing log blob. That is the
signature of a job that never started, not a lint failure.
Branch protection requires the context to pass. A dead pool cannot produce a
pass, so nothing can merge.
Diagnosed against the local oracle first
The branch that "failed Format" touches no Rust at all. So the failure was never
about formatting.
Fleet at the time of the move
lightlean-memrust-cpuBoth jobs moved to
lean-mem.The rule this encodes
A required context must not depend on a single-runner pool.
Capacity was never the problem for these two jobs — redundancy was. The
CI-capacity work (#1062) had been optimising the wrong axis for them: it asked
"is this pool idle?" and never asked "how many machines can satisfy this
label?". Recorded at both job sites so a future retarget doesn't put a required
context back behind one machine.
Not changed
The four required contexts on
ubuntu-latest, andClippy/Rivet Validationonrust-cpu(7 runners — genuine redundancy).Note on the runner itself
pulseengine-ci-01-8is still offline. This PR removes the dependency on it;it does not fix the host. If that machine is meant to be in service, it needs
attention independently — and it is currently the only carrier of the
lightlabel, so anything else routed there later inherits the same single point of
failure.
Refs #1062