From c8b9554ce34b010b9b3e707fbfc42e54d0b9c16a Mon Sep 17 00:00:00 2001 From: Ralf Anton Beier Date: Wed, 2 Sep 2026 14:58:35 +0200 Subject: [PATCH] fix(ci): two REQUIRED contexts sat behind a single self-hosted runner, and it went offline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MERGES WERE BLOCKED REPO-WIDE. `Format` and `Version Pin Sweep` — two of the nine required contexts — both targeted [self-hosted, linux, x64, light], a pool with EXACTLY ONE runner (pulseengine-ci-01-8). That runner went offline. The Format job was assigned to it, sat 600 seconds, and failed having executed ZERO steps (empty step list, missing log blob — the signature of a job that never started, not a lint failure). Branch protection requires the context to PASS. A dead pool cannot produce a pass, so nothing could merge — the deadlock my own notes warn about, arriving from the runner side rather than from a renamed check. Diagnosed against the local oracle first: `cargo fmt --check` is CLEAN on the branch that "failed" it, and that branch touches ZERO .rs files. So the failure was never about formatting. fleet at the time of the move light 0 online / 1 total <- both required contexts here lean-mem 4 online / 4 total idle rust-cpu 7 online / 7 total idle Both jobs moved to `lean-mem`. THE RULE THIS ENCODES: a required context must not depend on a single-runner pool. Capacity was never the problem here — redundancy was, and the CI-capacity work (#1062) had been optimising the wrong axis for these two jobs. Recorded at both sites so a future retarget does not put a required context back behind one machine. NOT changed: the four required contexts on ubuntu-latest, and Clippy/Rivet Validation on rust-cpu (7 runners, genuine redundancy). Refs #1062 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L --- .github/workflows/ci.yml | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f6b3f0d5..0030ce5c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -140,7 +140,18 @@ jobs: fmt: name: Format - runs-on: [self-hosted, linux, x64, light] + # SINGLE POINT OF FAILURE REMOVED (2026-09-02). This was + # [self-hosted, linux, x64, light], a pool with EXACTLY ONE runner + # (pulseengine-ci-01-8). That runner went offline and this job — a REQUIRED + # context — sat 600s and failed with ZERO steps executed, which blocks every + # merge in the repo: branch protection needs the context to PASS, and a + # dead pool cannot produce a pass. `Version Pin Sweep` was behind the same + # single runner, so TWO of the nine required contexts shared one machine. + # + # Moved to `lean-mem` (4 runners, all online and idle at the time of the + # move). The rule this encodes: a REQUIRED context must not depend on a + # single-runner pool. Capacity was never the problem — redundancy was. + runs-on: [self-hosted, linux, x64, lean-mem] steps: - uses: actions/checkout@v7 - uses: dtolnay/rust-toolchain@stable @@ -161,7 +172,18 @@ jobs: pin-sweep: name: Version Pin Sweep - runs-on: [self-hosted, linux, x64, light] + # SINGLE POINT OF FAILURE REMOVED (2026-09-02). This was + # [self-hosted, linux, x64, light], a pool with EXACTLY ONE runner + # (pulseengine-ci-01-8). That runner went offline and this job — a REQUIRED + # context — sat 600s and failed with ZERO steps executed, which blocks every + # merge in the repo: branch protection needs the context to PASS, and a + # dead pool cannot produce a pass. `Version Pin Sweep` was behind the same + # single runner, so TWO of the nine required contexts shared one machine. + # + # Moved to `lean-mem` (4 runners, all online and idle at the time of the + # move). The rule this encodes: a REQUIRED context must not depend on a + # single-runner pool. Capacity was never the problem — redundancy was. + runs-on: [self-hosted, linux, x64, lean-mem] steps: - uses: actions/checkout@v7 # Issue #145: fail at PR time if [workspace.package].version drifts from