Skip to content

feat(certification): add durable Firstmate final-certification coordinator#983

Open
ChrisGutierrezNet wants to merge 6 commits into
kunchenguid:mainfrom
ChrisGutierrezNet:fm/certification-coordinator
Open

feat(certification): add durable Firstmate final-certification coordinator#983
ChrisGutierrezNet wants to merge 6 commits into
kunchenguid:mainfrom
ChrisGutierrezNet:fm/certification-coordinator

Conversation

@ChrisGutierrezNet

Copy link
Copy Markdown

Intent

Implement and publish a safe, durable Firstmate final-certification coordinator that serializes heavy No Mistakes validations, performs strict identity and delivery-mode preflight checks, survives restarts, automatically advances queued work, preserves worker ownership of No Mistakes gates, handles obsolete requests, crashed launches, malformed or version-skewed records, and bounded retention safely, remains backend- and adapter-neutral, and includes tracked design, documentation, skills, AGENTS updates, lint, and focused regression coverage. Push the feature branch to the authorized ChrisGutierrezNet fork and open an upstream PR against kunchenguid/firstmate main; do not merge.

What Changed

  • Adds bin/fm-certification.sh, a restart-durable coordinator that serializes heavy No Mistakes validations behind a machine-global lock, enforces strict identity and delivery-mode preflight checks (detached HEAD, wrong branch, changed HEAD, dirty work, non-no-mistakes delivery refusal), and auto-advances queued work in FIFO order while preserving worker ownership of the No Mistakes gates.
  • Handles the failure surface: obsolete/withdrawn requests, crashed launches via a launch grace window, malformed and version-skewed records through quarantine, stall escalation, and bounded retention for terminal/quarantined/withdrawn records.
  • Wires the feature into the surrounding harness — new certification-coordinator skill and harness-adapters note, AGENTS.md/CONTRIBUTING.md/config/scripts docs, a design doc (docs/certification-coordinator.md), touch-ups to fm-brief.sh, fm-session-start.sh, fm-test-run.sh, and fm-watch.sh, plus 18 focused regression tests in tests/fm-certification.test.sh (all passing).

Risk Assessment

✅ Low: The change is a well-bounded, single-owner shell coordinator that matches its documented invariants and ships comprehensive regression coverage for every state path; the only findings are minor, non-blocking hardening/simplification notes.

Testing

Baseline: the focused suite tests/fm-certification.test.sh passes all 18 cases covering the coordinator's safety invariants. To show the intent working the way an operator experiences it, I built and ran a CLI transcript that drives the real script against real git worktrees through the complete lifecycle — capacity-1 serialized admission, a single token-bound worker steer, worker-owned start, authoritative terminal reconciliation, and automatic FIFO advancement — and demonstrates the safety boundaries (direct-PR delivery refused, dirty unpublished work preserved with HEAD unchanged, and a call log proving the coordinator issued only read-only no-mistakes inspections and never a respond/abort/rerun/sync/recover/daemon/run command). Adjacent suites touched by the diff (fm-watch-triage, fm-watch-checkpoint, fm-brief, fm-test-run) pass. One fm-session-start assertion fails, but it reproduces identically on the base commit (a tmux window-relaunch liveness check in this sandbox) and the test file is unchanged by the diff, so it is pre-existing and unrelated. The only unmet part of the intent is the publish step: no upstream PR exists on kunchenguid/firstmate and origin is the upstream rather than the ChrisGutierrezNet fork — surfaced as an ask-user finding since performing the push/PR is outside testing scope. No visual/UI surface is involved (this is a shell CLI), so the reviewer-visible evidence is the captured CLI transcript rather than a screenshot. The working tree was left clean.

Evidence: Operator CLI transcript: full certification lifecycle + safety invariants

========================================================================
1. Two ship tasks enqueue for final certification (shared capacity = 1)
========================================================================
$ fm-certification.sh enqueue alpha <head> --intent-file .../alpha/intent
certification admitted: alpha at 8e23897d4ed408f54ab66124164c8b527b23ce27
$ fm-certification.sh enqueue beta  <head> --intent-file .../beta/intent

========================================================================
2. status: one admitted, one held queued behind the single slot
========================================================================
$ fm-certification.sh status   (cols: seq  state  task  branch  head  reason)
capacity=1
000000000001	admitted	alpha	fm/alpha	8e23897d4ed408f54ab66124164c8b527b23ce27	
000000000002	queued	beta	fm/beta	d2641696c599b64ad1489c647c6060d4907ac238	

========================================================================
3. The admitted worker received exactly one token-bound start steer
========================================================================
$ cat send.log
alpha	Certification slot admitted for alpha. Run exactly: FM_HOME='/tmp/cert-demo.7TmrqL/home' '/home/exedev/.no-mistakes/worktrees/579deb6a33c1/01KYA7YMYR2CYGGRT4C4MGRK6Y/bin/fm-certification.sh' start 'alpha' '2e2b1c2380ed1ec61c8a0a06'

========================================================================
4. Worker runs its own token-bound start (worker owns the No Mistakes gate)
========================================================================
$ FM_HOME=... fm-certification.sh start alpha <token>
would exec in /tmp/cert-demo.7TmrqL/alpha-wt at 8e23897d4ed408f54ab66124164c8b527b23ce27: /tmp/cert-demo.7TmrqL/bin/no-mistakes axi run --intent <stored-intent> (reattach=0)
-> record advanced to:
000000000001	launching	alpha	fm/alpha	8e23897d4ed408f54ab66124164c8b527b23ce27	
000000000002	queued	beta	fm/beta	d2641696c599b64ad1489c647c6060d4907ac238	

========================================================================
5. No Mistakes reports an authoritative terminal pass for alpha's run
========================================================================
$ fm-certification.sh reconcile --notify   (session-start / watch calls this)
certification terminal: alpha outcome=checks-passed
certification admitted: beta at d2641696c599b64ad1489c647c6060d4907ac238
-> queue after terminal advancement:
capacity=1
000000000001	terminal	alpha	fm/alpha	8e23897d4ed408f54ab66124164c8b527b23ce27	
000000000002	admitted	beta	fm/beta	d2641696c599b64ad1489c647c6060d4907ac238	

========================================================================
6. beta is now admitted and steered automatically — FIFO advanced, no manual nudge
========================================================================
$ cat send.log   (second line = beta's start steer)
alpha	Certification slot admitted for alpha. Run exactly: FM_HOME='/tmp/cert-demo.7TmrqL/home' '/home/exedev/.no-mistakes/worktrees/579deb6a33c1/01KYA7YMYR2CYGGRT4C4MGRK6Y/bin/fm-certification.sh' start 'alpha' '2e2b1c2380ed1ec61c8a0a06'
beta	Certification slot admitted for beta. Run exactly: FM_HOME='/tmp/cert-demo.7TmrqL/home' '/home/exedev/.no-mistakes/worktrees/579deb6a33c1/01KYA7YMYR2CYGGRT4C4MGRK6Y/bin/fm-certification.sh' start 'beta' '1105084bf310f1e1f53c5124'

========================================================================
7. SAFETY: a non-no-mistakes (direct-PR) task is refused at the boundary
========================================================================
$ fm-certification.sh enqueue pronly <head> ...   (delivery mode = direct-PR)
error: task pronly delivery mode is direct-PR; final certification admits only no-mistakes delivery, never direct-PR or local-only
(refused with exit 1, as required)

========================================================================
8. SAFETY: unpublished dirty work is preserved, never discarded
========================================================================
$ fm-certification.sh enqueue guarded <head> ...   (worktree is dirty)
certification blocked: guarded: isolated copy /tmp/cert-demo.7TmrqL/guarded-wt is not clean; preserve and resolve every unpublished change before certification
(refused, as required)
-> blocked reason:
000000000001	blocked	guarded	fm/guarded	2beb1eae4583b87fee2e20e5ef5748f406b57417	isolated copy /tmp/cert-demo.7TmrqL/guarded-wt is not clean; preserve and resolve every unpublished change before certification
-> dirty file still present & untouched:
precious unpublished change
-> HEAD unchanged: 2beb1eae4583b87fee2e20e5ef5748f406b57417

========================================================================
9. The coordinator never drove a No Mistakes gate/recovery/cancel command
========================================================================
$ cat nm-calls.log   (only read-only axi / status / runs inspections)
axi
runs --limit 200
axi
runs --limit 200
axi status
axi status
axi
runs --limit 200

CONFIRMED: no respond/abort/rerun/sync/recover/daemon/run command was issued

demo complete.
Evidence: Reproducible demo harness driving the real coordinator
#!/usr/bin/env bash
# End-to-end operator transcript for the durable certification coordinator.
# Drives the REAL bin/fm-certification.sh against real git worktrees, stubbing
# only the external no-mistakes and fm-send endpoints, and narrates each step the
# way an operator/watcher would experience it.
set -eu

ROOT=/home/exedev/.no-mistakes/worktrees/579deb6a33c1/01KYA7YMYR2CYGGRT4C4MGRK6Y
SCRIPT="$ROOT/bin/fm-certification.sh"
FIX=$(mktemp -d "${TMPDIR:-/tmp}/cert-demo.XXXXXX")
HOME_DIR="$FIX/home"
CERT="$FIX/cert"
SEND_LOG="$FIX/send.log"
mkdir -p "$HOME_DIR/state" "$HOME_DIR/data" "$FIX/bin"

# --- stub external endpoints (no-mistakes CLI + fm-send steer delivery) -------
cat > "$FIX/bin/no-mistakes" <<'SH'
#!/usr/bin/env bash
printf '%s\n' "$*" >> "${FM_CERT_TEST_NM_CALL_LOG:?}"
case "$*" in
  axi) printf 'current_branch: %s\nruns: 0 runs yet in this repository\n' "$(git symbolic-ref --quiet --short HEAD 2>/dev/null || true)" ;;
  "runs --limit 200") [ ! -f .nm-runs ] || cat .nm-runs ;;
  "axi status") if [ -f .nm-status ]; then cat .nm-status; else printf 'error: no runs yet\n'; fi ;;
  *) printf 'unexpected no-mistakes args: %s\n' "$*" >&2; exit 2 ;;
esac
SH
cat > "$FIX/bin/fm-send" <<'SH'
#!/usr/bin/env bash
printf '%s\t%s\n' "$1" "$2" >> "${FM_CERT_TEST_SEND_LOG:?}"
SH
chmod +x "$FIX/bin/no-mistakes" "$FIX/bin/fm-send"

export FM_CERTIFICATION_ROOT="$CERT"
export FM_CERTIFICATION_CAPACITY=1
export FM_CERT_NM_BIN="$FIX/bin/no-mistakes"
export FM_CERT_SEND_BIN="$FIX/bin/fm-send"
export FM_CERT_TEST_SEND_LOG="$SEND_LOG"
export FM_CERT_TEST_NM_CALL_LOG="$FIX/nm-calls.log"
export FM_CERT_NO_EXEC=1
: > "$SEND_LOG"; : > "$FIX/nm-calls.log"

gitc() { git -C "$1" -c user.name=demo -c user.email=demo@example.invalid "${@:2}"; }
mktask() { # <id>
  local id=$1
  local primary="$FIX/$id-primary"
  local wt="$FIX/$id-wt"
  mkdir -p "$primary"; git -C "$primary" init -q
  printf '# %s\n' "$id" > "$primary/README.md"
  gitc "$primary" add README.md; gitc "$primary" commit -qm initial
  git -C "$primary" worktree add --quiet -b "fm/$id" "$wt"
  mkdir -p "$HOME_DIR/data/$id"
  { printf 'window=fm-%s\nworktree=%s\nproject=%s\nharness=pi\nkind=ship\nmode=no-mistakes\nyolo=off\n' "$id" "$wt" "$primary"; } > "$HOME_DIR/state/$id.meta"
  printf 'Run the heavy No Mistakes final certification for %s safely.\n' "$id" > "$HOME_DIR/data/$id/intent"
  printf '%s\n' "$wt"
}
cert() { FM_HOME="$HOME_DIR" "$SCRIPT" "$@"; }
rule() { printf '\n========================================================================\n%s\n========================================================================\n' "$1"; }

wt_alpha=$(mktask alpha); h_alpha=$(git -C "$wt_alpha" rev-parse HEAD)
wt_beta=$(mktask beta);   h_beta=$(git -C "$wt_beta" rev-parse HEAD)

rule "1. Two ship tasks enqueue for final certification (shared capacity = 1)"
echo "\$ fm-certification.sh enqueue alpha <head> --intent-file .../alpha/intent"
cert enqueue alpha "$h_alpha" --intent-file "$HOME_DIR/data/alpha/intent"
echo "\$ fm-certification.sh enqueue beta  <head> --intent-file .../beta/intent"
cert enqueue beta "$h_beta" --intent-file "$HOME_DIR/data/beta/intent"

rule "2. status: one admitted, one held queued behind the single slot"
echo "\$ fm-certification.sh status   (cols: seq  state  task  branch  head  reason)"
cert status

rule "3. The admitted worker received exactly one token-bound start steer"
echo "\$ cat send.log"
cat "$SEND_LOG"

rule "4. Worker runs its own token-bound start (worker owns the No Mistakes gate)"
token=$(grep '^token=' "$(find "$CERT/queue" -name '*-alpha.record')" | cut -d= -f2-)
echo "\$ FM_HOME=... fm-certification.sh start alpha <token>"
cert start alpha "$token"
echo "-> record advanced to:"; cert status | grep -E 'alpha|beta' || true

rule "5. No Mistakes reports an authoritative terminal pass for alpha's run"
cat > "$wt_alpha/.nm-status" <<EOF
id: run-alpha
branch: fm/alpha
head: $h_alpha
status: completed
outcome: checks-passed
EOF
echo "\$ fm-certification.sh reconcile --notify   (session-start / watch calls this)"
cert reconcile --notify
echo "-> queue after terminal advancement:"; cert status

rule "6. beta is now admitted and steered automatically — FIFO advanced, no manual nudge"
echo "\$ cat send.log   (second line = beta's start steer)"
cat "$SEND_LOG"

rule "7. SAFETY: a non-no-mistakes (direct-PR) task is refused at the boundary"
wt_pr=$(mktask pronly)
{ printf 'window=fm-pronly\nworktree=%s\nproject=%s\nharness=pi\nkind=ship\nmode=direct-PR\nyolo=off\n' "$wt_pr" "$FIX/pronly-primary"; } > "$HOME_DIR/state/pronly.meta"
echo "\$ fm-certification.sh enqueue pronly <head> ...   (delivery mode = direct-PR)"
if cert enqueue pronly "$(git -C "$wt_pr" rev-parse HEAD)" --intent-file "$HOME_DIR/data/pronly/intent"; then
  echo "UNEXPECTED: direct-PR admitted"; else echo "(refused with exit $?, as required)"; fi

rule "8. SAFETY: unpublished dirty work is preserved, never discarded"
# Use a fresh coordinator root so this task has a free slot and preflight runs.
wt_g=$(mktask guarded); h_g=$(git -C "$wt_g" rev-parse HEAD)
printf 'precious unpublished change\n' > "$wt_g/unpublished.txt"
echo "\$ fm-certification.sh enqueue guarded <head> ...   (worktree is dirty)"
FM_CERTIFICATION_ROOT="$FIX/cert-b" cert enqueue guarded "$h_g" --intent-file "$HOME_DIR/data/guarded/intent" || echo "(refused, as required)"
echo "-> blocked reason:"; FM_CERTIFICATION_ROOT="$FIX/cert-b" cert status | grep guarded
echo "-> dirty file still present & untouched:"; cat "$wt_g/unpublished.txt"
[ "$(git -C "$wt_g" rev-parse HEAD)" = "$h_g" ] && echo "-> HEAD unchanged: $h_g"

rule "9. The coordinator never drove a No Mistakes gate/recovery/cancel command"
echo "\$ cat nm-calls.log   (only read-only axi / status / runs inspections)"
cat "$FIX/nm-calls.log"
echo
if grep -Eq 'respond|abort|rerun|sync|recover|--recover|daemon|axi run' "$FIX/nm-calls.log"; then
  echo "UNEXPECTED: coordinator invoked a gate-driving command"
else
  echo "CONFIRMED: no respond/abort/rerun/sync/recover/daemon/run command was issued"
fi

rm -rf "$FIX"
echo
echo "demo complete."
Evidence: fm-certification.test.sh — 18/18 focused regression tests pass
ok - capacity serializes concurrent requests and notification replay is idempotent
ok - a launch claim cannot create a duplicate certification run
ok - authoritative terminal reconciliation survives restart and advances FIFO
ok - detached, wrong, and changed branch identities stop without rewriting work
ok - obsolete custody blocks only that item and never performs recovery
ok - dirty unpublished work is preserved byte-for-byte
ok - an active validation outside coordinator ownership stops admission
ok - session restart and ordinary monitoring both invoke the state-machine owner
ok - notification ambiguity and no-mistakes read failures stop safely
ok - final certification refuses non-no-mistakes delivery modes
ok - withdraw frees the slot, advances the queue, and permits a corrected head
ok - withdrawal never cancels a matching active run
ok - a launch is graced while registering, surfaced past grace, and then withdrawable
ok - withdrawal waits out the launch grace before freeing a slot
ok - retry waits out the launch grace before re-driving a launch
ok - unsupported-version and malformed records are quarantined before accounting
ok - terminal retention keeps the newest records and prunes older intent
ok - quarantine and withdrawn archives are bounded by the retention count
- Outcome: ⚠️ 2 issues (1 warning, 1 info) across 1 run (6m59s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 infos
  • ℹ️ bin/fm-certification.sh:297 - no-mistakes CLI reads are deliberately time-bounded via run_bounded()/timeout because they can hang, but the surrounding git commands in preflight and inspect_record_run are not — notably git status --porcelain=v1 --untracked-files=all (line 297) and the rev-parse/merge-base calls (lines 289-296, 360-362). All of these execute while holding the single machine-global coordinator lock ($CERT_ROOT/.lock) that serializes every FM_HOME's certification work. A slow or wedged git in one worktree (huge untracked tree, stuck index/fs) therefore stalls the coordinator for all homes, undercutting the same availability property the timeout wrapping was added to protect. Consider running the git identity/cleanliness probes through the same bounded helper.
  • ℹ️ bin/fm-certification.sh:236 - record_field() (line 129) and meta_value() (line 236) are byte-for-byte identical grep|tail|cut helpers. They can be collapsed to a single helper (or one aliased to the other) to remove the duplication; the two names can be kept as thin wrappers if the call-site semantics are worth preserving.
⚠️ **Test** - 2 issues (1 warning, 1 info)
  • ⚠️ docs/certification-coordinator.md:1 - The intent's publish requirement — push the feature branch to the authorized ChrisGutierrezNet fork and open an upstream PR against kunchenguid/firstmate main — is not verifiably satisfied from the test environment. gh-axi search prs for a certification-coordinator PR and for author:ChrisGutierrezNet on kunchenguid/firstmate both return 0, and the worktree's origin points at kunchenguid/firstmate (the upstream), not a ChrisGutierrezNet fork. The target commit is not on any remote branch. Opening/pushing a PR is an outward-facing, hard-to-reverse action outside testing scope, so I did not perform it. The user needs to confirm the branch was pushed to the fork and the upstream PR opened (and left unmerged).
  • ℹ️ tests/fm-session-start.test.sh:1 - tests/fm-session-start.test.sh has one failing assertion ('the later fleet read did not confirm the relaunched window' — a tmux window-relaunch liveness check). It reproduces identically on the base commit 10ee779 (the test file is unchanged by this diff), so it is a pre-existing environmental/tmux-sandbox failure, not a regression from the certification change. No action needed for this change.
  • bash tests/fm-certification.test.sh — all 18 focused regression tests pass (concurrent admission, restart/FIFO advance, detached/wrong-branch/changed-head preflight, obsolete custody, dirty-work preservation, uncoordinated-run block, non-no-mistakes delivery refusal, malformed/version-skew quarantine, launch-grace stall/withdraw/retry, bounded terminal/quarantine/withdrawn retention)
  • Ran a custom operator CLI transcript (/tmp/no-mistakes-evidence/01KYA7YMYR2CYGGRT4C4MGRK6Y/demo-lifecycle.sh) driving the real bin/fm-certification.sh through enqueue → status → notify → start → reconcile terminal → FIFO advance, plus direct-PR refusal and dirty-work preservation, verifying only read-only no-mistakes inspections were issued
  • bash tests/fm-watch-triage.test.sh, bash tests/fm-watch-checkpoint.test.sh, bash tests/fm-brief.test.sh, bash tests/fm-test-run.test.sh — all pass (affected/adjacent scripts touched by the diff)
  • bash tests/fm-session-start.test.sh — 1 pre-existing tmux-liveness failure; confirmed identical failure on base commit 10ee779 via git checkout &lt;base&gt; -- bin/fm-session-start.sh tests/fm-session-start.test.sh (then restored), establishing it is environmental, not a regression
  • gh-axi search prs &#34;certification coordinator repo:kunchenguid/firstmate&#34; and author:ChrisGutierrezNet — 0 results; git remote -v and git branch -r --contains &lt;target&gt; confirm no upstream PR/push yet
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 127)
✅ **Push** - passed

✅ No issues found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1983adc0b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bin/fm-certification.sh
qroot=$(shell_quote "$fm_root/bin/fm-certification.sh")
qtask=$(shell_quote "$task")
qtoken=$(shell_quote "$token")
message="Certification slot admitted for $task. Run exactly: FM_HOME=$qhome $qroot start $qtask $qtoken"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve coordinator settings in admitted start commands

Captain, when the shared coordinator is initialized with non-default settings (for example FM_CERTIFICATION_ROOT/NO_MISTAKES_HOME or FM_CERTIFICATION_CAPACITY>1), this generated “run exactly” command only exports FM_HOME. The worker's start path recomputes the ledger root and calls capacity_ensure, so a worker that doesn't already have those env vars reads the wrong queue or refuses the durable capacity before invoking No Mistakes, leaving the admitted item occupying capacity until manual recovery. Include the coordinator root/capacity in the token-bound command or make start read the durable values without requiring matching ambient env.

AGENTS.md reference: AGENTS.md:L284-L286

Useful? React with 👍 / 👎.

ChrisGutierrezNet and others added 6 commits July 24, 2026 19:14
The rebase onto main picked up bin/fm-doc-audience-check.sh, which requires
every tracked prose surface to be classified. Register the coordinator design
doc and its skill so the audience inventory check passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ChrisGutierrezNet
ChrisGutierrezNet force-pushed the fm/certification-coordinator branch from e1983ad to b2bbe3d Compare July 24, 2026 19:17
@ChrisGutierrezNet

Copy link
Copy Markdown
Author

Queue status - 2026-07-24 UTC

This PR is open, non-draft, and green; the durable record says final validation passed at the published head. Next decision: maintainer review and merge approval.

@kunchenguid

kunchenguid commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#983 at b2bbe3d1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants