Skip to content

fix(ci): two REQUIRED contexts sat behind a single self-hosted runner, and it went offline - #1135

Closed
avrabe wants to merge 1 commit into
mainfrom
fix/light-runner-spof
Closed

fix(ci): two REQUIRED contexts sat behind a single self-hosted runner, and it went offline#1135
avrabe wants to merge 1 commit into
mainfrom
fix/light-runner-spof

Conversation

@avrabe

@avrabe avrabe commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Merges are blocked repo-wide, and it is not a code failure

Format and Version Pin Sweeptwo of the nine required contexts — both
targeted [self-hosted, linux, x64, light], a pool with exactly one runner
(pulseengine-ci-01-8). That runner went offline.

The Format job was assigned to it, sat 600 seconds, and failed having
executed zero steps — empty step list, missing log blob. That is the
signature of a job that never started, not a lint failure.

Branch protection requires the context to pass. A dead pool cannot produce a
pass, so nothing can merge.

Diagnosed against the local oracle first

cargo fmt --check      -> CLEAN
.rs files on branch    -> 0

The branch that "failed Format" touches no Rust at all. So the failure was never
about formatting.

Fleet at the time of the move

pool online / total
light 0 / 1 ← both required contexts here
lean-mem 4 / 4 idle
rust-cpu 7 / 7 idle

Both jobs moved to lean-mem.

The rule this encodes

A required context must not depend on a single-runner pool.

Capacity was never the problem for these two jobs — redundancy was. The
CI-capacity work (#1062) had been optimising the wrong axis for them: it asked
"is this pool idle?" and never asked "how many machines can satisfy this
label?". Recorded at both job sites so a future retarget doesn't put a required
context back behind one machine.

Not changed

The four required contexts on ubuntu-latest, and Clippy / Rivet Validation on rust-cpu (7 runners — genuine redundancy).

Note on the runner itself

pulseengine-ci-01-8 is still offline. This PR removes the dependency on it;
it does not fix the host. If that machine is meant to be in service, it needs
attention independently — and it is currently the only carrier of the light
label, so anything else routed there later inherits the same single point of
failure.

Refs #1062

…, and it went offline

MERGES WERE BLOCKED REPO-WIDE. `Format` and `Version Pin Sweep` — two of the
nine required contexts — both targeted [self-hosted, linux, x64, light], a pool
with EXACTLY ONE runner (pulseengine-ci-01-8). That runner went offline. The
Format job was assigned to it, sat 600 seconds, and failed having executed ZERO
steps (empty step list, missing log blob — the signature of a job that never
started, not a lint failure).

Branch protection requires the context to PASS. A dead pool cannot produce a
pass, so nothing could merge — the deadlock my own notes warn about, arriving
from the runner side rather than from a renamed check.

Diagnosed against the local oracle first: `cargo fmt --check` is CLEAN on the
branch that "failed" it, and that branch touches ZERO .rs files. So the failure
was never about formatting.

  fleet at the time of the move
    light      0 online / 1 total   <- both required contexts here
    lean-mem   4 online / 4 total   idle
    rust-cpu   7 online / 7 total   idle

Both jobs moved to `lean-mem`. THE RULE THIS ENCODES: a required context must
not depend on a single-runner pool. Capacity was never the problem here —
redundancy was, and the CI-capacity work (#1062) had been optimising the wrong
axis for these two jobs. Recorded at both sites so a future retarget does not
put a required context back behind one machine.

NOT changed: the four required contexts on ubuntu-latest, and Clippy/Rivet
Validation on rust-cpu (7 runners, genuine redundancy).

Refs #1062

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
@codecov

codecov Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@avrabe

avrabe commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Closing — the premise was fixed at the fleet level, and better than this PR fixed it

When I opened this, light had exactly one runner (pulseengine-ci-01-8) and
it was offline, so two required contexts could not run and nothing could merge.
This PR routed them to lean-mem.

Re-measured today, the fleet has changed:

pulseengine-ci-01-5    online   rust-cpu+light
pulseengine-ci-01-8    online   light
pulseengine-ci-01-10   online   rust-cpu+light

light: 3 online runners can satisfy it   (was 0 / 1)

The dead runner is back and two rust-cpu machines gained the light
label. That is the better fix: it removes the single point of failure for
everything routed to light, present and future, rather than moving two jobs
away from it.

Why close rather than merge anyway

The PR is green (59/59) and lean-mem has 4 runners vs light's 3, so merging
would be marginally defensible on redundancy. But the change carries a comment
that says:

[self-hosted, linux, x64, light], a pool with EXACTLY ONE RUNNER

That is now false. Landing prose that asserts something untrue about the
current fleet is precisely the defect class v0.61 was named for — stale claims
sitting behind a green gate — and it would be landed by the person who spent
that release removing them. Not worth two jobs' worth of marginal redundancy.

The rule survives the PR

The finding was never really "move these two jobs". It was:

A required context must not depend on a single-runner pool. Capacity was
never the problem — redundancy was.

And the sharper half, which is why my CI-capacity measurements missed it: every
sample I took asked "is this pool idle?". light was idle, which reads as
healthy. Nobody asked how many machines could satisfy the label. Idle and
unavailable look identical in a utilisation reading.

That invariant is worth enforcing rather than remembering, so it goes to
RQ-62-CLAIMCHECK (#1062), which already owns required-context routing —
any future retarget should have to show the destination label has more than one
online runner.

No code change needed today. Thanks to whoever relabelled the runners.

@avrabe avrabe closed this Sep 3, 2026
@avrabe
avrabe deleted the fix/light-runner-spof branch September 3, 2026 04:00
avrabe added a commit that referenced this pull request Sep 3, 2026
…y invariant a live incident proved (#1137)

* chore(rivet): RQ-62-CLAIMCHECK absorbs the required-context redundancy invariant a live incident proved

PR #1135 is CLOSED, not merged — its premise was fixed better at the fleet
level than in this repo. The finding it produced is worth keeping, so it moves
here rather than dying with the branch.

WHAT HAPPENED (2026-09-02): the `light` pool had EXACTLY ONE runner and it went
offline. `Format` and `Version Pin Sweep` — two REQUIRED contexts — were routed
there, were assigned to a dead machine, sat 600s, and failed having executed
ZERO steps. Branch protection needs a required context to PASS, and a dead pool
cannot produce one, so nothing in the repo could merge. Resolved at the fleet
level: the runner returned and two `rust-cpu` machines gained the `light`
label, so `light` now has 3 online. My PR would have moved the two jobs to
`lean-mem` — marginally more redundant, but its comment asserts "a pool with
EXACTLY ONE RUNNER", which is now FALSE, and landing prose that is untrue about
the live fleet is the class v0.61 was named for.

WHY EVERY #1062 MEASUREMENT MISSED IT, which is the part worth keeping: they
all asked "is this pool IDLE?". `light` was idle, which reads as healthy. None
asked "HOW MANY MACHINES CAN SATISFY THIS LABEL?" — and idle and unavailable
are indistinguishable in a utilisation reading. That is the THIRD variant of
the same error this issue has produced; the first two were reading a rate off a
single instant, in opposite directions.

THE INVARIANT, now in the done-when so it is checked rather than remembered: a
required context must not target a label satisfiable by fewer than two ONLINE
runners. Any retarget must demonstrate the destination's online-runner count,
not merely that the pool was idle.

Refs #1062

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L

* plan(v0.62): scope RQ-62-ARCHMODEL (#1136) — feature-loop steps 1-2 are NOT permanently N/A (8 -> 9)

Maintainer decision on #1136: "We should at least define the basic how the
compiler runs on a system and things we can address in aadl if it is timing or
other things but the general we should address."

THE CASE IS STRONGER THAN THE QUESTION ASSUMED. synth already produces
AADL-shaped facts that nothing owns:

  --emit-wcet sound per-function cycle bounds  -> thread Expected_WCET
  linear-memory base / globals / stack         -> memory components
  reserved R9 / R10 / R11 / R12                -> processor binding
  per-target MPU region model                  -> memory protection
  the --safety-bounds envelope                 -> a system property, today
                                                  PROSE in CLAUDE.md

And the timing ALREADY FLOWS — reconstructed. CLAUDE.md records that synth's
WCET sidecar is "gale spar's T3/T4 C_i input", and `spar insight` compares
traces against Expected_BCET/WCET/Mean. So the number already reaches a spar
analysis by a CONSUMER hand-building a scheduling model around a sidecar synth
emits. That is a hand-written mirror of a shipped artifact — the exact failure
the North Star names. synth should DECLARE its timing and layout; consumers
should not reconstruct them.

Precedent is modest: scry/spar/scry.aadl is 374 lines and declares itself the
source-of-truth architecture with WIT hand-derived for v0.1 and spar-codegen as
the migration path. Same sequencing here — model as source of truth, generation
as migration, not prerequisite.

INCREMENT 1 IS DEFINE-AND-MAP: define how a compiled artifact executes on a
target, map what AADL expresses natively (TIMING FIRST — the bound exists and
has a consumer), and state what AADL CANNOT carry so the boundary is written
down rather than rediscovered.

NOBODY IS BLOCKED ON THE MODEL. #1131 does not wait for this: RQ-62-EMBEDDER
ships the layout contract as a document derived from the emitter, on its own
timeline, because an integrator is blocked TODAY and an architecture model is
not the unblocking artifact. The model becomes that document's source of truth
afterwards.

Coupled consequence: steps 1-2 stop being an N/A, so RQ-62-LOOPCONFORM must
count them as a real step rather than a blank.

ARTIFACT_FLOOR 517 -> 518, re-derived with `rivet list`, not computed.

Refs #1136, #1131

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant