fix(ai): never wedge on a 0-token empty completion when thinking is off (RIG-2806) - #44
rigel-mintaka wants to merge 19 commits into
Conversation
…() (RIG-3255) (#37) * fix(eval): stream one coalesced agent progress event from handle.wait() (RIG-3255) `runEvalWait` emitted agent progress three ways: eagerly before waiting, on a 1-second interval, and once after settlement. The eager pre-wait emit surfaced whatever `latestDetails.progress` held at that instant, so a fast `agent()` that had already reported interim "running" progress before the wait ran would stream that interim snapshot as well as the final "completed" one: two `op:"agent"` events instead of the single coalesced "latest" the caller expects. The ordering is load-sensitive. In isolation the job has not scheduled its first progress callback by the time the eager emit runs (the snapshot guard returns), so only the final emit fires; under parallel test-bucket load the scheduling shifts and both leak through. That is the flake behind the intermittent `agent-bridge-policy` "streams the latest enriched agent progress" failure (`toHaveLength(1)` receiving 2). Drop the eager pre-wait emit. The interval covers heartbeats for a long wait and the post-settlement emit carries the final snapshot, so a wait that settles inside the first interval window emits exactly once (the settled state) and a long wait still streams periodic snapshots. Deterministic, and matches the test's stated "stream the latest" contract. This is upstream code (handle-bridge.ts is byte-identical to can1357/oh-my-pi's current tip); the same fix will be proposed upstream. Spec-impact: none. Refs RIG-3255 Co-authored-by: Matt Wilkinson <matt@rigel.build> * docs(eval): scope the handle.wait() single-emit comment to what holds The removed eager emit left a comment claiming a settling wait "emits exactly once (the settled state)". Neither half is unconditional: on the failure path the final progress report never runs, so the post-settlement emit carries the last interim snapshot, not the settled state; and a subagent that never reported progress emits nothing. Scope the claim to "at most once" and name both sub-paths so a maintainer does not build on an invariant the code does not hold. Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(eval): add deterministic guard for coalesced agent progress emit runEvalWait's single-emit contract had no red->green guard: the existing agent-bridge-policy assertion is load-dependent and passes unloaded whether or not the eager pre-wait emit is present, so a re-added eager emit would slip through local runs and most CI runs. Drive runEvalWait directly against a controlled AsyncJobManager job under fake timers, so the 1s heartbeat interval provably cannot tick. The job reports an interim "running" snapshot, parks until released, then reports the final "completed" snapshot; awaiting the interim signal guarantees latestDetails holds a snapshot before the wait runs. A re-added eager pre-wait emit fires with the interim snapshot (two agent events); the settle-only path emits once. Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <mattwilki17@gmail.com> Co-authored-by: Matt Wilkinson <matt@rigel.build>
…rt timeout (#38) The "resumes the long-lived GET stream with Last-Event-ID" test intermittently hung under CPU-starved CI oversubscription, tripping the pending-promise guard. Root cause is a timeout misconfiguration, not reconnect latency. The test built its transport through connectedTransport(), which sets timeout: 50ms (the value the negative-path timeout tests need). startSSEListener derives its GET startup budget as min(1000, floor(requestTimeout / 4)), so a 50ms request timeout yields a 12ms connect budget. Under load the initial GET crosses 12ms, the listener is aborted, and the second notification never arrives, an unbounded hang the guard then surfaces as a timeout. Give this positive-path test its own transport with a generous timeout (matching the sibling resume tests), lifting the SSE startup budget to the full 1000ms cap. Stress at 40-way oversubscription goes from intermittent hangs to 60/60 clean. The guard constant moves 500 to 4000 (staying below bun's 5000ms default per-test timeout so the labeled failure wins) and its comment is corrected to describe the measured behavior: a healthy localhost resume finishes in well under 100ms, so a trip indicates a stalled listener rather than a slow one. Co-authored-by: Matt Wilkinson <mattwilki17@gmail.com>
…4) (#36) * ci: run release and main-event jobs on GitHub-hosted runners (RIG-3144) The `omp-kata` self-hosted ARC runner scale set exists only in the upstream `can1357/oh-my-pi` org and was never registered in RigelBuild, so every non-pull_request job that selected it queues forever with no runner. Pull request runs already land on `ubuntu-22.04`; releases and main pushes did not, which is why the CI-gated npm publish never completed. Collapse the per-job runner selector to `ubuntu-22.04` at all eleven sites (`check`, `rust_validate`, `native_addons`, the six TS test buckets, and `install_methods`) so the GitHub-hosted branch is the only branch. No cache or native-build wiring changes: the `bazel-cache`, `bazel-natives`, and `bun-install` composite actions already auto-select the GitHub-hosted `actions/cache` backend whenever `BAZEL_REMOTE_USER`/`BAZEL_REMOTE_PASSWORD` are absent, which they are off cluster, and `bazel-cache-warm.yml` already seeds the hosted darwin scopes and the bun store. The step-level `github.event_name` branching that picks prebuilt-from-npm addons on pull requests versus a bazel build on main is unchanged. Delete `.github/actionlint.yaml`, whose only purpose was registering the now unused `omp-kata` label; actionlint passes clean without it, and the four `.github/actions/*` composites keep their `omp-kata` mentions only in descriptive comments (no `runs-on:`), so nothing else needs the registration. Re-laid onto upstream v18.1.7 as part of the fork re-sync (see docs/fork-resync.md, Task 2). Refs RIG-3144, RIG-2777 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: fetch fork-built natives on PRs instead of upstream's npm scope (RIG-3144) (#33) * ci: fetch fork-built natives on PRs instead of upstream's npm scope (RIG-3144) The native_addons PR path fetched @oh-my-pi/pi-natives-linux-x64@latest, which is upstream can1357's npm scope. Upstream advanced that package to 18.1.7, whose Linux binding surface no longer matches the fork's 18.0.x test tree, so every fork PR started reddening two required checks (Test TS native/integration, Test coding-agent native/unit) with "MacOSPowerAssertion.start is undefined". The main path was unaffected because it builds natives fresh from the fork commit. The fetch string was missed by the fork scope-rename (it is a hardcoded literal in the workflow, not a rewritten manifest), so fork PRs were validating against upstream's moving registry instead of the fork's own published addons. - Repoint the PR-path fetch and its explanatory comment to the fork's own @rigelbuild/omp-natives-linux-x64 leaf. Its tarball layout is identical (package/pi_natives.linux-x64-{baseline,modern}.node), so the copy steps are unchanged, and its version tracks the fork's own release cadence. - Guard the MacOSPowerAssertion test with a darwin-only early return, matching the platform-gated tests already in the file. A macOS-only power-assertion binding must not run on Linux, and this keeps the test green even if a future fork Linux build drops the macOS export the way upstream's 18.1.7 did. Verified: the fork's @rigelbuild/omp-natives-linux-x64@18.0.3 addon exports MacOSPowerAssertion.start on Linux (reproduced locally: upstream's build does not, fork's does), so the repoint alone turns the checks green; the guard is defense in depth. actionlint clean; the guarded test passes under bun:test. Spec-impact: none Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(natives): drop the darwin guard on the power-assertion test (RIG-3144) The prior commit added a `process.platform !== "darwin"` early-return to the MacOSPowerAssertion test on the premise that the binding is macOS-only. That premise is wrong: the class is compiled unconditionally and start() has an explicit non-macOS arm returning a no-op handle (crates/pi-natives/src/power.rs start/stop cfg arms), and the published contract documents the cross-platform no-op handle (packages/natives/native/index.d.ts). The addon exports it on Linux. Because every bun-test job runs on ubuntu, the guard was unconditional dead code that skipped the body on 100% of CI runs, deleting coverage rather than gating it, and would let a future fork Linux build that drops the export pass green instead of reddening. The scope repoint in the parent commit is the complete root-cause fix on its own; the unguarded body passes against the fork's published addon on Linux. Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: persist the linux bazel disk cache across main runs (RIG-3144) (#32) * ci: persist the linux bazel disk cache across main runs (RIG-3144) ## Problem The two main-event Rust/Bazel jobs, `rust_validate` (workspace tests + clippy + rustfmt over `//crates/...`) and `native_addons` (the six cross-compiled `//:natives-*` addon links), both requested the `linux` disk-cache scope but nothing ever **saved** it. The only Bazel cache-save step lives in the `bazel-natives` composite action, which only the `release-*` and warm scopes use. So the `linux` scope had a restore side with no producer: every main push rebuilt the entire Rust workspace and every native addon fully cold. Confirmed at source: the repo's Actions cache held zero `bazel-disk-v3-linux-*` archives, only `release-darwin-*` and `bun-*`. The cold native build is the roughly hour-long long pole on every main run, and every main push also recompiled the Rust workspace from scratch for `rust_validate`. ## Change Give each job its own save. The two jobs build under different Bazel configs (test + clippy vs release-codegen addon links), so their actions are keyed differently and cannot share one archive. Splitting the scope keeps each job's archive coherent: - `rust_validate` uses scope `linux-rust` and saves on `save-needed`. - `native_addons` uses scope `linux-natives` and saves on `save-needed` (guarded, like its build steps, to the non-PR path). Each save mirrors the existing `bazel-natives` save exactly: same `actions/cache/save` pin, same `~/.cache/omp-bazel-disk` + `~/.cache/omp-bazel-repo` path list, keyed by the `bazel-cache` action's `cache-key` output. The `save-needed` output is already false on an exact-key hit, so an unchanged run is a pure cache replay that saves nothing; a first run on a new key (or after a crates change) restores via the prefix/bare fallback, builds, and saves the refreshed archive. ## Effect Between upstream re-syncs `crates/**` does not change, which is the common case, so a later main run replays the cached Rust actions and the cross-compiled addon links instead of paying the multi-GiB rustc cost cold. This is the fork-side lever that makes frequent releases cheap: the Rust closure only rebuilds when it actually changes. Job names are unchanged, so the branch-protection required checks are unaffected. No warm workflow is needed for these scopes because both jobs run on every main push and self-seed continuously, unlike the darwin release scopes that only build at release time. Scope note: on the shared repo cache quota, the toolchain half of the repo cache now exists under two linux scopes rather than one. Eviction is graceful content-addressed LRU and self-heals, and a real remote cache (reachable from our own CI) removes the quota ceiling entirely as the durable follow-up. Spec-impact: none. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: document the linux bazel cache quota budget and path-mirror invariant (RIG-3144) Self-review follow-up on the linux bazel cache persistence change. - Add a quota note above both new save steps: four GiB-scale bazel scopes (linux-rust, linux-natives, release-darwin-x64, release-darwin-arm64) now share the 10 GiB repo cache, so a fifth scope needs an eviction plan first. GitHub evicts least-recently-used archives silently on overflow, and a quota-driven save failure only degrades to a warning line, so the symptom would be a silently slow CI rather than a red build. - Add the "must mirror the restore path list" warning above the path list in both new save steps, matching the note the bazel-cache restore action and the bazel-natives save already carry. The restore path list is now replicated at four sites; without the note the two new copies can silently drift and produce a partial cache, the same defect class this change fixes. Spec-impact: none Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <mattwilki17@gmail.com> Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <mattwilki17@gmail.com> Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(pi-shell): disable job-control pipeline kill test on hosted CI (#29) `kill_builtin_signals_every_process_in_a_jobspec_pipeline` self-suspends a two-process pipeline with `kill -STOP` and expects the shell's job-control to observe the stop within a 5s timeout. Its SIGSTOP/SIGCONT and process-group semantics do not hold under the bazel linux-sandbox on GitHub-hosted runners, so `run_string` never returns and the test times out (`pipeline did not stop`). It passed only on the previous self-hosted infrastructure; it has never completed on a hosted runner. Mark it `#[ignore]` so the `rust_validate` gate is green on hosted CI. The test stays compiled (no bitrot) and is trivially re-enabled with `--include-ignored` or by removing the attribute if the shell's job-control or the CI process model changes. The other 862 pi-shell tests are unaffected; primary coverage is intact. Co-authored-by: Matt Wilkinson <mattwilki17@gmail.com> Co-authored-by: Matt Wilkinson <matt@rigel.build> * docs: re-add the fork re-sync design record after the reset (RIG-3144) The reset to upstream v18.1.7 (Task 1) discards `docs/fork-resync.md`, which lived only on the pre-reset fork history: the record is absent from upstream `main` (the reset target) and from the pre-reset fork tip, so a bare reset leaves the frozen contract off `main`. Re-lay it here as the first thing on the reset base so Tasks 3 through 8 execute against the record on `main`, matching the design's Task 2 step 0 (T1b). The content is byte-for-byte the record frozen by PR #35; no edits. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: pin PR-path native fetch to the checkout's upstream base version (RIG-3144) The reset to upstream tip (v18.1.10) inverted the premise of the fork-scope natives repoint. The PR path fetched `@rigelbuild/omp-natives-linux-x64@latest`, the fork's own scope, whose only published addon is `18.0.3` (the fork's last release, roughly a hundred commits behind the reset tree). The v18.1.10 TS tree calls native functions that addon predates (`vcsGitDiscover`, `editDescription`, `extractInlineSloppyRegions`), so every native-backed TS test job died with "X is not a function" and the required `CI` check stayed red. Immediately after an upstream re-sync the fork's `crates/` tree equals upstream's (the reset base has zero `crates/` drift from the v18.1.10 tag), so upstream's own `@oh-my-pi/pi-natives-linux-x64@18.1.10` is the exact binding match and exports all three functions. Point the PR-path fetch back at the upstream scope, pinned to the checkout's own upstream base version rather than a floating `@latest`. The version is read from `packages/natives/package.json` and any `-rigel.N` fork suffix is stripped, so the pin resolves the upstream base for both a plain upstream version (`18.1.10` -> `18.1.10`) and a post-release fork version (`18.1.10-rigel.1` -> `18.1.10`). This tracks future re-syncs without another edit: the fork's own native line only diverges once it accrues `crates/` changes of its own, at which point the durable fix is publishing a fork addon at the fork version, tracked as the post-reset native-release follow-up. Verified: upstream's 18.1.10 addon exports the three functions (116 exports total); staged into the CI `bazel-bin/` layout it passes the job's own smoke check; the reset tree has zero `crates/` drift from the v18.1.10 tag. actionlint rc=0. Spec-impact: none. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: correct the addon-safety comment and bound the hosted bazel jobs (RIG-3144) Review follow-ups on the hosted-runner + natives-fetch changes. - Narrow the native_addons comment to the guarantee that actually holds. The workspace loader skips its version sentinel for workspace loads (packages/natives loader-state), so an upstream addon under a fork checkout is not validated against the fork's own bindings. Visible failure is guaranteed only for a newly-added export; a change to the behavior of an existing export tests silently against upstream semantics until the fork publishes its own @RigelBuild addon. The old wording claimed any fork-only native change "fails visibly", which overstates the protection. - Add timeout-minutes to the three bazel jobs relocated onto GitHub-hosted runners (check 30, rust_validate 90, native_addons 90). They were the only jobs in the workflow without a bound, and the diff's own premise is that a hosted bazel run can hang rather than fail (the job-control test times out under the linux-sandbox). rust_validate gates the release chain, so an unbounded hang would stall it for the GitHub default of 360 minutes; a bounded fail-loud is the house posture. Spec-impact: none. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(pi-shell): re-run the ignored kill test's body in the isolated child (RIG-3144) The re-lay disabled `kill_builtin_signals_every_process_in_a_jobspec_pipeline` with `#[ignore]` (it times out under the CI bazel sandbox, which lacks full job-control stop/continue semantics). That test is a self-re-exec harness: the outer invocation calls `run_isolated_kill_test`, which spawns the test binary with `--exact <name>` and a marker env var, and the real body only runs in that child. The child argv carried `--exact` but not `--include-ignored`, so once the outer test is `#[ignore]`d the child filters it out, runs zero tests, exits 0, and the outer `assert!(status.success())` passes vacuously. Anyone re-enabling the test by running with `--include-ignored` would get a green result while the assertions never execute, and a regression in `kill %1` pipeline signalling (the sole assertion of `process_ids()` in the crate) would ship undetected. Pass `--include-ignored` in `run_isolated_kill_test` so the re-exec child actually runs an ignored body. It is a no-op for the three non-ignored callers (`kill_builtin_refuses_own_pid`, `kill_builtin_refuses_own_process_group`, `pkill_builtin_excludes_own_pid`). This mirrors the existing precedent in this file: `process_test_sleeper` is `#[ignore]`d and its spawner already passes `--ignored`. Also cite RIG-3249 in the `#[ignore]` reason: that issue owns re-enabling the test (root-causing the sandbox job-control gap or making it deterministic), so the temporary ignore has a tracked owner rather than becoming permanent. Verified: the three non-ignored isolated kill tests still pass, and with the fix the ignored test's body executes and passes when run outside the sandbox with `--include-ignored` (it stays ignored inside the CI bazel sandbox). Spec-impact: none. Refs RIG-3144, RIG-3249 Co-authored-by: Matt Wilkinson <matt@rigel.build> * docs: reconcile the frozen re-sync record with the executed reset (RIG-3144) The record froze via PR #35 naming reset target v18.1.7 (c4da0d0). Upstream advanced before the reset ran, so main actually reset to v18.1.10 (ddde7db). Left unreconciled, the on-main contract that Tasks 3/7/8 execute from would instruct the opposite of what shipped and carry a live, copy-pasteable force-push command naming a three-tags-stale SHA. - Add a dated post-execution reconciliation note after the status line: reset target is v18.1.10 (ddde7db); the PR-path natives fetch ships from the upstream scope pinned to base version (the @RigelBuild scope has only 18.0.3, which lacks v18.1.10 bindings) and flips back to @RigelBuild once the fork cuts its first post-reset native release; version scheme is the Matt-ruled <upstream-version>-rigel.N, superseding the old npm-floor-18.1.8 + rigel-v* proposal. - Neutralize the Task-1 force-push runbook: it has already executed, so its commands are commented out and marked DO NOT RE-RUN, with the SHA corrected to the real target for provenance. - Title v18.1.7 -> v18.1.10; Status Draft -> Frozen (it merged as PR #35). Spec-impact: none. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: drop the dead kata clause from the warm-cache header comment (RIG-3144) The bazel-cache-warm.yml header still justified per-image warming with "a kata-produced disk cache misses every action on a hosted image". With no omp-kata runners in the fork, that clause describes nothing real. Keep the still-true constraint (hosted runners see only default-branch actions/cache entries, and bazel action keys do not transfer across runner images) and drop the kata reference, matching the same rewrite already applied to the warm_bun rationale lower in the file. Spec-impact: none. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: raise native_addons timeout above the measured cold build (RIG-3144) Review round 2 follow-ups. - `native_addons` timeout-minutes 90 -> 150. The prior 90 sat below this repo's measured 84-minute cold cross-compile (the six addon links, run 33789708655), and a job-level timeout kills the job before its `Save bazel disk cache` step runs, so a killed build persists nothing and every retry starts equally cold and times out again. 150 gives ~1.8x margin over the measured cold run while still bounding a genuine hang inside GitHub's 360-minute ceiling. The first post-reset main run is cold, so this is load-bearing for Task 2's acceptance gate. - Correct the ignored kill test's header comment: "Disabled on GitHub-hosted CI" -> "Disabled under the CI bazel sandbox", matching the actual mechanism (the bazel linux-sandbox, which applies on any runner bazel executes on) and the #[ignore] reason string. Spec-impact: none. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * docs: correct the reconciliation note's blast-radius and supersession scope (RIG-3144) Review round 2 follow-ups on the re-sync record's reconciliation note. - The runbook's `sha=` was corrected to the executed target in the prior commit, which falsified the note's own prose. Item 1 and the runbook preamble said the SHA was "stale" and that re-running "would roll main back three tags", both describing the old v18.1.7 value. With the corrected SHA, re-running would reset main to the reset point and discard every post-reset re-lay, a strictly larger blast radius. Reword both to say the SHA is the executed target and re-running discards the re-lays. - Broaden item 3's OQ1 supersession from the one Open-Questions entry to a blanket rewrite, matching item 1: the Global Constraints "Version + tag scheme" bullet and Task 3's "Depends on OQ1" paragraph both still stated a hard block on a question that has landed, so an agent picking up Task 3 would park work that is actually unblocked. State that Tasks 3/7 are no longer gated and the `--match v*` globs need no change under the `-rigel.N` scheme. - Add item 4: the pre-reset safety tags (OQ6) are pushed, so OQ6 no longer reads as outstanding. Spec-impact: none. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: re-add the aggregate CI gate the ruleset requires as its status context The branch ruleset requires a single status context named `CI`, but the re-sync's ci.yml carried only the per-job check-runs (their display names), not the aggregate job whose check-run posts that context. With nothing emitting `CI`, every PR sat mergeStateStatus=BLOCKED on a forever-pending required check and could only be merged by org-owner override. Re-add the `ci` job (name `CI`) that `needs` every PR-relevant job and fails closed on any non-success result. rust_validate is excluded because it is `if: github.event_name != 'pull_request'` and a skipped need would strand the gate; native_addons is excluded because every test job already needs it, so its failure propagates transitively. `if: !cancelled()` makes the gate run even when a need fails, so it posts `CI` = failure rather than a never-created, pending context. This is a fork-only gate: upstream defines no aggregate CI job, so it stays out of the upstream-bound branches. Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: bound the CI gate's runtime and scope its comment to what holds Two review nits on the aggregate CI gate. Add timeout-minutes: 5 so the job holding the sole required context cannot pin CI pending for the 360-minute default if a runner wedges (every other job in the file sets an explicit bound). Narrow the comment's "always posts a CI check-run" to "whenever the workflow runs it posts", since a run excluded by the pull_request paths filter posts nothing and a genuine cancel posts a cancelled conclusion, not a pass/fail. Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: always-trigger ci.yml with a change-detection gate (RIG-3144) The branch ruleset requires the `CI` rollup on every PR, but ci.yml carried a `pull_request.paths` filter, so a docs-only PR never triggered the workflow and the required `CI` check sat forever pending, unmergeable (PR #35 hit exactly this). Drop the trigger's `paths:` filter so the workflow always runs on PRs, and recreate the filter one level down as a per-job gate: a new `setup` job runs `scripts/ci-paths-affected.ts` against the PR diff and outputs `affected`; `check` and `native_addons` gate on it (the eight test/install jobs already cascade off `native_addons.result == 'success'`), so a docs-only PR skips the expensive matrix while the cheap rollup still posts `CI`. The change detector is fail-safe by construction: a push/dispatch event, an unresolvable base ref, or any git failure resolves to affected=true, so the only way to skip the heavy jobs is a clean diff that provably touches no code path. A bug can waste CI minutes; it can never let untested code merge. The rollup now treats a gated job's `skipped` as pass ONLY when `setup.outputs.affected == 'false'`; a failed setup leaves that output empty, so an excused skip is never granted and the rollup reds. Never a blanket skipped-is-green. Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: fix two change-detector fail-opens found in review (RIG-3144) Review of the always-trigger CI gate (commit c93871c) surfaced two ways the change detector could wrongly report `affected=false` and skip the matrix on a PR that does change code, which is the one failure mode the gate must never have. Both are fixed here as an additive commit on the reviewed head. ### `git diff` rename detection hid moved-out code (fail-open) `git diff --name-only` runs with rename detection on by default, so a rename prints only its destination path. A PR that moved code out of a gated directory (for example `packages/x.ts` to `docs/x.md`) reported only the docs destination and yielded `affected=false`, skipping every test while deleting code the workspace still imports. Verified against real history: commit 2e45297 moves 36 files out of `packages/` and reported 0 `packages/` hits under the old flags. Fix: pass `--no-renames` so every rename decomposes into its delete+add pair and the source path is seen. The same commit now reports 36 `packages/` hits. ### `CODE_PATHS` omitted config the `check` job reads (fail-open) The path list was a faithful mirror of the old `pull_request.paths` trigger, but that inherited list omitted root config the gated `check` job consumes (`tsconfig*.json`, `.oxlintrc.json`, `.oxfmtrc.json`, the `types/` root). As a trigger the gap was benign: such a PR simply never started CI, and the missing required check blocked the merge. As a job-level gate it became load-bearing. A config-only regression now yielded `affected=false`, skipped `check`, and the rollup excused the skip, so a merge-blocking gap turned into a silent green. Fix: add those inputs to `CODE_PATHS` and document that a gate must cover every gated job's inputs, not just mirror the old trigger. ### Also - Empty diff now fails safe to `affected=true`. An empty file list is doubt (a force-push race or botched merge-base), not proof nothing changed, so this restores the module's stated "any doubt runs the matrix" invariant. - `writeOutput` appends to `$GITHUB_OUTPUT` directly instead of read-modify-write. - Rollup comment: fix `reding` typo; document the three-edit sync a new gated job needs; note nix.yml deliberately keeps its own path filter (not a required context). - Tests now defend the fail-closed contract end to end. A temp-repo suite drives `main()` for the rename, docs-only, code, push-event, and unset-SHA cases (red-green on the rename guard), plus config-input assertions. Verified: bun test 14/14 (rename guard proven red without `--no-renames`); check:tools (oxlint + oxfmt) clean; actionlint rc=0; tsgo reports zero errors in both files; real-ref smokes (2e45297 to true, docs-only to false, code to true). Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: cover two more gated-job inputs in the change detector (RIG-3144) The confirming review of the fail-open fixes (commit 77fb62e) found the same class of bug on two more inputs the gated jobs read but the detector's path list did not cover, so a PR touching only those paths was classified affected=false, the job skipped, and the rollup excused it: CI green on a change that would have failed. ### `.cargo/config.toml` feeds the gated `check` job The `check` job runs `cargo fetch --locked` then `cargo deny --locked --offline` (ci.yml, gated on affected=='true'). Cargo reads `.cargo/config.toml` from the workspace root before doing anything, and a malformed or semantically-wrong one reds `cargo fetch` (verified: exit 101 on a broken table). The file pins the CMake policy version, the generator, and `embed-metadata`, and its own comments tie it to the rust-toolchain pin, so it is exactly the kind of file a toolchain PR edits. nix.yml already lists `.cargo/**` in its own paths filter, so the gate was strictly less careful than the workflow it is told not to align with. Fixed by adding a `.cargo/` prefix to CODE_PATHS. ### `python/robomp/web` is a real bun workspace member Root package.json declares it in `workspaces.packages` and bun.lock carries it, so every gated job's `bun install --frozen-lockfile` reads its manifest. A dependency edit to that manifest without a regenerated lockfile fails the frozen install (verified: exit 1, "lockfile had changes, but lockfile is frozen"), yet nothing under `python/` matched CODE_PATHS. Fixed with a `python/robomp/web/` prefix, scoped to the web member so the 149-file python tree (whose pytest suites this workflow never runs) does not force the matrix. ### The detector's own test now runs in CI `test_workspace` invokes a hardcoded test list, and ci-paths-affected.test.ts was not on it, so the red-green rename guard added last commit was never enforced on any future PR: a deletion of `--no-renames` would have stayed green in CI. Added the file to that list. It is fast (no native artifacts) and, because editing the detector matches the `scripts/` prefix, the gated job that runs it is never skipped on a PR that touches it. ### Test-suite integrity and fail-safe coverage - GITHUB_OUTPUT moved to a temp dir OUTSIDE the fixture repo; inside it, the harness's own output file was tracked by `git add -A` and showed up in the diff under test. - The fixture git env is now hermetic (GIT_CONFIG_GLOBAL/SYSTEM/NOSYSTEM), so a host `commit.gpgsign` or `diff.renames` config cannot skew a run. - `headSha` now checks the git exit code, so a rev-parse failure throws instead of silently returning an empty SHA that would mask a real result. - Added end-to-end cases for the highest-value fail-safe paths: a cargo-config change (affected=true), an unresolvable base SHA hitting the catch (affected=true), and an empty same-sha diff (affected=true). Verified: bun test 17/17 (the cargo-config guard is red-green: removing `.cargo/` from CODE_PATHS fails exactly the two cargo assertions); check:tools clean; actionlint rc=0; tsgo zero errors in both files; real-ref smokes correct (rename to true, docs-only to false, code to true). Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * ci: gate the last two cross-root fixtures the change detector missed (RIG-3144) A confirming review of the prior fold proved the same fail-open class one directory over: two gated test suites read repo-root fixtures outside their package, and neither path was in the detector's allowlist, so a fixture-only change was classified affected=false, its suite skipped, and the rollup excused the skip green. - docs/tools/: the tool-doc coverage test asserts a page exists for every builtin tool, so a docs-only rename reds test_coding_agent_runtime. - assets/: the terminal-image test reads assets/python.webp, so an assets-only change reds test_coding_agent_native. Both added to CODE_PATHS with matcher + end-to-end guards proven red-green. Also hardened the test fixture the detector relies on: the GITHUB_OUTPUT relocation now has a guarding assertion, runMain defaults the SHA/event vars so an ambient PR_BASE_SHA/PR_HEAD_SHA can't invert a fail-safe case, headSha shares the hermetic git env, and the unset-SHA fail-safe is split into its three shapes. Registered the suite in package.json test:scripts so the local aggregate matches CI. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> * test: name the F3 output-file guard and drop a vacuous diff assertion (RIG-3144) Round-3 review of the change-detector reworking flagged two test-quality lows on the M2 fixture guards: - The `diffNames(base, head)` assertion in the docs-only test ran before the first `runMain()` wrote GITHUB_OUTPUT, so it stayed green even when the output file was moved back inside the fixture worktree. It never defended F3 and its comment claimed otherwise. Removed the assertion and the now-unused helper; the test's own `affected=false` expectation already fails if a stray file leaks into the diff. - The real F3 defense lived in `beforeAll`, so a regression reported an unnamed failure. Extracted it into a named `it("keeps GITHUB_OUTPUT outside the fixture worktree (F3 guard)")` so a future break points at itself. Red-green: moving the output file back inside the repo now reds that named test specifically. Suite 22 pass / 0 fail; oxlint, oxfmt, actionlint, tsgo clean. Refs RIG-3144 Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <mattwilki17@gmail.com> Co-authored-by: Matt Wilkinson <matt@rigel.build>
…ng/tier on resume (RIG-3225) (#34) ## Problem On `--resume`, an explicit `--config`/`--profile` overlay's `modelRoles` were inert: the resumed session always restored the model, thinking level, and service tier baked into it at its original launch, so an updated profile never took effect. This is why the 2026-09-03 dream-team relaunch (`spawn-agent --resume <sid> --profile dream-team`) ran the fleet on each session's stale baked model instead of the profile's opus-5 stack. The restore is deliberate for a bare resume — a conversation should not be silently moved off the model it is running — so the fix is an explicit opt-in, not a behavior change to the default resume path. ## Change Add a `--reapply-config` CLI flag and a `reapplyConfig` SDK option (`CreateAgentSessionOptions`). When set, a resume adopts the config-resolved default **model**, its **thinking level** (including a `:high`/`:xhigh`/`:max` selector suffix on the default role), and its **service tier** instead of the values baked into the session. Default (unset) preserves the existing resume-restores behavior exactly. Adoption is **per-knob**: a knob is taken from config only when config actually specifies it, otherwise the session's own value is kept. So an overlay that only retunes a non-default role (or only a tier, or tombstones the default) never silently yanks a resumed conversation onto an arbitrary fallback model — config with nothing to say for a knob leaves that knob on the session's value. Locus is `createAgentSession` in `sdk.ts`, where both the CLI and SDK embedders (Compass) converge, so one option fixes both surfaces. Three restore sites are gated: - session-model restore array construction is always populated, and its early restore + the later reclaim loop are skipped only when `adoptConfigModel` (`reapplyConfig && !hasExplicitModel && config names a default role`); the session model therefore stays available as the fallback. "Config names a default" mirrors the resolver's own notion — a blank value, the bare `default` sentinel, or a tombstoned/absent `modelRoles.default` all count as no default, so an overlay that only retunes a non-default role (or only a tier) keeps the session model. When config names a default it cannot resolve (a typo), a post-`tryResolveDefaultRole` block restores the baked session model + thinking rather than dropping to an arbitrary `pickDefaultAvailableModel`. Extension-provided `litellm/*` defaults resolve on the post-extension path. - persisted `thinking_level_change` is skipped only when `adoptConfigThinking` (`adoptConfigModel && the default role carries an explicit selector`) — so `--model --reapply-config` keeps the session's level rather than dropping to a bare model default - session `service_tier_change` is merged per family when config specifies any `tier.*`: config's specified families override, families the config omits keep the session's baked tier. Unlike model/thinking, the tier is NOT gated on `--model` — `--model` pins only the model, and the dedicated `--service-tier` override still wins over the merge. A resume under `--reapply-config` is otherwise silent, so it now surfaces one notice via `modelFallbackMessage`: naming the adopted model when config swaps the session onto a different one; naming the unresolved config default (with its resolver warning) when a broken default falls back to the session's own model; and naming both the unresolved default and the unrestorable session model when neither resolves and the model comes from an arbitrary availability pick (the case a bare resume also warns about). The notice is gated on a baked session model existing, so a resume that adopts the same model the session already ran — or a fresh/no-model session — stays silent. Like `--model`, the adopted values are a per-run intent and are not persisted back as `*_change` entries, so a later bare resume still restores the session's own values; `spawn-agent` passes `--reapply-config` on every relaunch to keep the fleet on the profile. ## Tests `agent-session-model-persistence.test.ts`: model, thinking, and service tier each adopt the config value on resume with `reapplyConfig`, and each restore the baked session value on a bare resume without it; the session model/thinking are kept (silently) when config specifies no default, tombstones it (`null`), names an empty string, or names the bare `default` sentinel, and are kept with a notice when config names an unresolvable/typo'd default; a double failure (unresolvable config default AND unrestorable baked model) is reported naming both; a session with no `model_change` entry stays silent (no notice naming a nonexistent session model); `--model` + `reapplyConfig` keeps the session's thinking level; service tier merges per family (session `openai:priority` + config `google:flex` keeps both); the swap and broken-config notices are each asserted on their own branch (`resumed on` vs `kept the session's`, neither matching the double-failure `could not be restored`) so an inverted discrimination fails; and a `reapplyConfig` resume appends no new `model_change`/`thinking_level_change` (the per-run non-persistence contract). `flag-tables.test.ts`: `--reapply-config` parses to `reapplyConfig` without consuming the initial message. `profile-bootstrap.test.ts`: `--reapply-config` is exempted as a valueless flag so a trailing `--profile` still activates. Spec-impact: none. Refs RIG-3225 Co-authored-by: Matt Wilkinson <matt@rigel.build>
CI red is an unrelated flake, not this diffThe required Evidence — same SHA, same job, opposite verdicts (both runs attributed to this PR, both at
It launches a real Chromium and time-gates it at 30s, so it fails whenever the runner misses the window. This diff touches zero I have not re-run CI. A re-run against an unchanged tree is a retry, and Note on the PR numberThis supersedes #21, which GitHub auto-closed after a stray local |
Review loop closed — round 8 all clear, zero findings at any severityEight adversarial rounds. Head The round-8 delta was a comment reduction. That sounds trivial and was the highest-risk part of the diff: seven consecutive versions of that four-line comment were factually false, and four of those were introduced by corrections to earlier versions. The failure modes were (a) annotating a line that turned out to be deletable with zero errors while the load-bearing line sat bare, (b) an under-listed error set, and (c) confusing a diagnostic's cause with its reported position. So version 8 stopped restating the claim and deleted the claim class: no error codes, no line numbers, no counts — only symbolic referents that survive line drift. Comment clauses verified by a cell partition
B and C are complements and their diagnostic sets partition A's exactly (5 + 1 = 6), so neither clause rides on the other's evidence — a gap or overlap would itself have been a finding. That partition is the check the earlier wrong-line version lacked. The causation/position trap reproduced independently: TS2322 is reported at No regression from the earlier delete-then-restoreA wire-body test was deleted on a wrong CI diagnosis in an earlier round and later restored. Re-verified:
Totals and mechanism
The ollama fail-closed path was measured directly rather than inferred from red→green: Notes
|
…e-token auth (RIG-3358) (#48) Merges upstream `omp-authbroker-metrics` (`e336acb0`) into the fork's `main`, so the fork carries the auth-broker `/metrics` endpoint ahead of upstream PR can1357#10290 landing. Squashed to a single parent: `jj-vine submit` hangs indefinitely on a two-parent bookmark. The tree is byte-identical to the true merge tree (`fe20f3aa590c`), verified against a real two-parent merge of `main` and `e336acb0`. Upstream rewrote the branch to fix a config-layer defect this fork reported: `resolveMetricsEnabled` read settings with `cwd: process.cwd()`, and project settings merge over global, so a checked-in `.omp/config.yml` in the launch directory could expose the credential endpoint — and even override an explicit global `false`. The broker is a background daemon whose working directory is incidental, so it is now pinned to `getAgentDir()`. The same push also skips a plan-table row whose canonical plan is empty, which would otherwise emit `plan=""` series that no info series can join. A later push fixed a test-isolation defect this fork reported: two suites called `setAgentDir()` — which writes `PI_CODING_AGENT_DIR` — while restoring either a different variable or nothing at all, leaking a since-deleted temp dir into every later suite in the same worker. All three describes now use the shared `beginSettingsTest`/`restoreSettingsTestState` helper, which snapshots the whole env rather than needing to know which variable `setAgentDir` writes. Two conflicts against `main`, resolved as in the previous build: `mcp-http-transport.test.ts` takes main's side (its explicit `HttpTransport` construction supersedes the branch's `connectedTransport(0)`), and the CHANGELOG keeps both `### Added` entries under `[Unreleased]`. `/metrics` stays off unless enabled by `--enable-metrics`, `OMP_AUTH_BROKER_METRICS`, or `auth.broker.metrics` (flag beats env beats config), so this changes no behaviour for an existing broker. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…es tools (RIG-3073) (#26) * fix(ai): accept null strict/parameters/description on inbound Responses tools (RIG-3073) ## Problem LiteLLM converts chat-completions to Responses for the `gpt-5.6-*` backends and forwards `strict: null` on every function tool. The inbound Responses validator (`openai-responses-server-schema.ts` `toolSchema`) declared `strict?`, `parameters?`, and `description?` as non-nullable, so it fail-closed and 400'd every tool call: ```text openai-responses: tools[0] must be a valid bridged Responses tool (was {...,"strict":null,...}) or tools[0].strict must be boolean (was null) ``` This wedged every `codex-*` gateway model (a chat backend routed through the Responses bridge): every tool call bounced with a 400. ## Root cause The inbound schema was stricter than the SDK's own wire type. `openai-responses-wire.ts` `FunctionTool` declares all three fields nullable: `strict: boolean | null`, `parameters: {...} | null`, `description?: string | null`. And `buildTools` (`openai-responses-server.ts`) already coalesces null away before anything reaches a provider — `fn.description ?? ""`, `fn.parameters ?? {}`, and it drops a null `strict`. So the inbound `toolSchema` was the sole over-strict gate; nothing downstream ever saw the null. ## Fix Widen the three optional fields in `toolSchema` to match the wire type (`boolean | null`, `{...} | null`, `string | null`). Pure schema-type widening, no downstream behavior change since `buildTools` already null-coalesces all three. Invalid tools still fail closed via the union. ## Tests Two regressions in `auth-gateway-openai-responses.test.ts`, both confirmed red before the fix (they reproduce the exact 400) and green after: - `strict: null` on a function tool now parses and is dropped, so it never leaks to the provider. - `parameters: null` / `description: null` now parse and coalesce to `{}` / `""`; `strict: false` (the boolean path) stays valid. Target suite green (22 pass). The widened line uses only existing omptype API (`.or("null")`, the `null` keyword, string unions). Spec-impact: none Refs RIG-3073 Co-authored-by: Matt Wilkinson <matt@rigel.build> * style(ai): format the merged Responses tool tests Resolving the rebase conflict by hand left stray blank lines where the conflict markers were, which oxfmt --check rejects. Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(ai): keep rejecting a wrong-typed strict after the null widening The two null tests are both must-accept, so a future edit that blanket-loosened the tool schema would keep them green. A probe with only must-accept cases cannot detect that its own failure detection is broken. Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(ai): assert the rejected field, not arktype's union-arm text The must-reject test pinned the exact message arktype emits once the tool union has been exercised, so it only passed in committed file order: run it under -t, .only, or any reordering and the cold union reports a different arm. Assert the field name instead, and cover all three widened fields, so the guard survives the order it runs in. Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(ai): assert the diagnosis, not the echoed fixture The field-name regex had no anchor between `tools[0]` and the field, and the generic union message echoes the whole tool object -- so two of three arms were satisfied by the input we sent rather than by the validator's conclusion. They would have matched \\bname\\b or \\btype\\b equally. Assert each arm's real diagnostic contiguously, and carry the expected message in the case table so a new field cannot be added without stating what its rejection looks like. Co-authored-by: Matt Wilkinson <matt@rigel.build> * docs(ai): scope the warmth note to prior-parse count, not cold-vs-warm The gap note said 'cold, every field yields the same generic message'. Measured: warmth is a count of prior union parses in the module instance, with the field-level message appearing from the 2nd parse on -- so a filtered run catches a later case while its first case still escapes. PRIOR=0 tools[0] must be a valid bridged Responses tool (generic) PRIOR=1 tools[0] must be a valid bridged Responses tool (generic) PRIOR=2 tools[0].parameters must be { } or null (field-level) PRIOR=3 tools[0].parameters must be { } or null (field-level) Co-authored-by: Matt Wilkinson <matt@rigel.build> * docs(ai): the warming agent is any parse call, not a tool-union parse Permuting the case list shows position 3 is field-level whichever field sits there, so the threshold is ordinal rather than per-field. And two parseRequest calls carrying NO tools warm it just as well, so the note's 'times the union has been parsed' was one scope too narrow: prior calls valid tools invalid tools no tools at all 0 generic generic generic 1 generic generic generic 2 FIELD-LEVEL FIELD-LEVEL FIELD-LEVEL Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <matt@rigel.build>
…0.25 (RIG-3339) (#40) * fix(catalog): gate Fable/Mythos cacheRead on revision so 5.1+ bills $0.25 (RIG-3339) `claude-fable-5-1` and `claude-mythos-5-1` resolved `cost.cacheRead = 1`, but Anthropic cut 5.1's cache reads 75% to $0.25 — so every cache read on a 5.1 model was billed at 4x. The `dream-team.yml` designer role runs `fable:xhigh`, so this was live spend rather than a latent bug. `classes/anthropic.kdl` pinned the whole `fable` family unconditionally at `cache-read 1.0`, with `mythos` identical — correct for 5.0, wrong for 5.1+. Both families are affected; the mythos half had not been reported. The equivalent policy elsewhere expresses this as a version-keyed conditional (`semverGte(version, "5.1") ? 0.25 : 1`); the KDL form carried no revision gate, so that conditional had no expression here. ### Fix A `revision ">=5.1"` `cost-patch` block on each of `fable` and `mythos`, plus the `gen:compat` recompile. Uses the same shape as the sibling `opus revision "=4.5"` correction, so a future 5.2 inherits $0.25 rather than falling back. **The 5.1 blocks restate all four cost fields deliberately.** `cost-patch` is one cascade axis and `contest()` keeps a single winner per axis, taking the whole object off that winner — object axes replace, they do not merge. A minimal `cost-patch { cache-read 0.25 }` resolves to `{cacheRead: 0.25}` alone and silently drops `input`/`output`/`cacheWrite` to upstream values, which is worse than the original bug and still green under a cacheRead-only assertion. A comment records this so the next editor does not retry the minimal override. `limits-patch` is a separate axis and is unaffected — the 1M/128k pin still resolves on both branches. ### Why this layer `costPatch` is applied by `buildModel` at runtime, not only baked at generation: feeding a generated spec that already carries a corrected `cacheRead: 0.25` through `buildModel` returns `1`, because the KDL rule overwrites it. A generation-time correction is therefore a silent no-op, and the fix has to live in the rules. ### Verification Red/green against a pristine checkout at the same commit: with the original rules the new test fails on exactly `cacheRead 0.25` vs received `1`; with the fix the file is 30/30. Full catalog suite 872 pass / 0 fail. `bun run check` (oxlint + oxfmt + tsgo) clean. The regression test asserts the whole cost object on both 5.1 families, 5.0 staying at $1, 5.2 inheriting, and the limits pin surviving — so a partial patch cannot pass it. Spec-impact: none. Refs RIG-3339 Co-authored-by: Matt Wilkinson <matt@rigel.build> * fix(catalog): rebake the bundled Fable 5.1 cost and guard bake/rule cost parity (RIG-3339) Addresses the review's HIGH finding on this PR: the rule change alone left the compiled `models.json` bundle stale, so first-party Anthropic rows served **verbatim** still billed 5.1 cache reads at 4x — the exact defect the rule fix targets. The earlier reasoning held that no rebake was needed because `buildModel` re-applies the patch at runtime. That is true for consumers that rebuild, but not for the ones that don't: `getProviderModels` returns bundled rows verbatim to keep startup allocation-free, so `resolveIterationModel` (which feeds `calculateCost` for per-iteration and fallback cost) and the cold-start no-override path in the model registry both served the stale $1. Anthropic's discovery carries no pricing for this generation — that is why the pin exists — so the bundle was the only source for those paths. ### Changes **Corrected the one affected bundled row** to its cascade-resolved value: `anthropic/claude-fable-5-1` cacheRead $1 -> $0.25. Against the base bundle the file differs by exactly three bytes — `"cacheRead": 1` -> `"cacheRead": 0.25` — with provider order, key order, indentation and the absence of a trailing newline all preserved, so it stays byte-identical to canonical generator output apart from the intended value. The 5.0 rows correctly stay at $1 and no `mythos-5-1` row is bundled, so exactly one row moves. A full `gen:models` was deliberately not used: it refetches live upstream catalogs and pulled 905 changed / 42 added / 3 removed rows of unrelated third-party drift (`tps`, `contextWindow`, `name`, aggregator pricing) into what is a first-party pricing correction. The bundle stays in sync with the rule either way; the next scheduled regen will absorb the rest of that drift on its own. **Closed the gap that let this ship green.** The bake/runtime parity sweep compared `.compat` and `.thinking` but never `.cost`, so a `cost-patch` edit landed without a rebake was unguarded by any test. It now diffs the fields a `costPatch` actually pins (a patch may legitimately carry a subset), which is what makes the stale-bundle case fail CI rather than reach a release. ### Verification The new guard is a real gate, not a passing assertion: reverting the bundled row to $1 while keeping the fixed rule fails it with `anthropic/claude-fable-5-1.cost.cacheRead: baked=1 engine=0.25`, one diff across 4776 rows. It passes on the corrected bundle, and all 13 `costPatch`-governed rows across every provider are in sync. The verbatim path named in the finding was checked directly: `getBundledModel("anthropic", "claude-fable-5-1")` now returns `cacheRead: 0.25` with both 5.0 rows still $1. Full catalog suite 872 pass / 0 fail; `bun run check` clean. Spec-impact: none. Refs RIG-3339 Co-authored-by: Matt Wilkinson <matt@rigel.build> * test(catalog): document the cost-parity guard as the only bundle-staleness detector (RIG-3339) Neither `gen:models` nor `gen:compat` runs in CI and no workflow checks generated-file freshness, so the .cost sweep added in the parent commit is the only automated detector of a stale bundled cost. Say so at the callsite, with the two ways to break it: collapsing the subset compare into a whole-object compare (false-reds every legitimate partial cost-patch) or dropping it. Also record the guard's bound: the sweep iterates the baked bundle, so a model carrying a rule but no bundled row is never visited. Verified across all 67 providers that exactly one mythos row exists bundle-wide (anthropic/claude-mythos-5) and no claude-mythos-5-1 row exists anywhere, so that family rests entirely on the rule-layer assertions in test/generated-policies.test.ts. Comment-only; no behavior change. Spec-impact: none. Refs RIG-3339 Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <matt@rigel.build>
…ff (RIG-2806)
## Problem
On the `openai-completions` path, a mid-session switch to `thinking: off` (or any model/compat whose thinking-emit flags all resolve false — the live case is `claude-opus` over litellm) could serialize a **zero-body request**: when the only non-system history is a thinking-only assistant turn, `convertMessages` dropped it and returned just the system prompt. A valid `200` over an empty prompt tokenizes to `0` input tokens and the model returns an empty `stop` completion. The shared empty-completion retry wrapper then exhausted its cap and delivered the empty terminal as a benign `done`, so the agent loop accepted a 0-token no-op turn and idled — a silent wedge that survives `--resume` (the persisted `configured: "off"` re-wedges the resumed session). Tracked as RIG-2806.
Two independent defects compound into the unrecoverable wedge; this fixes both.
## Fix 1 — serializer never ships a zero-body request (`openai-completions.ts`)
`convertMessages` now records the reasoning of any assistant turn it fully drops (`lastDroppedThinkingText`) and where the message body begins (`bodyStartIndex`, after the system prompts). If every non-system message was dropped, it recovers the last dropped reasoning as a bare-prose assistant turn (`renderDemotedThinking`) so the request always carries real content. The recovery guards on the *rendered* output being non-empty (not just the input), so a dialect that renders to whitespace can never re-create a zero-content body. When any real history survives (a user or tool-result turn), the safety net does not fire.
## Fix 2 — empty completion fails closed and loud (`empty-completion-retry.ts`)
After the retry cap is exhausted with a still-degenerate empty stop, the wrapper now pushes a loud `error` terminal (`stopReason: "error"`, message naming the empty-completion failure) instead of the benign `done`. Gated on `isRetryableEmpty`, so `acceptEmptyResponse` callers (the passive Google advisor) are unaffected. This wrapper is shared across four providers (anthropic, ollama, openai-completions, openai-responses); the cross-provider retry suites stay green. `rule://no-retries`: a swallowed empty completion is the fail-open pattern to reject.
> **Open design fork (RIG-2831):** the error terminal is currently unclassified (`errorId 0`), which keeps it maximally loud but bypasses the existing `#isRecoverableProviderEmptyOutput` recovery machinery (keyed on `Flag.EmptyResponse`, emits "try switching models"). Whether to classify it — a resilience/UX tradeoff on the shared 4-provider path — is filed as RIG-2831 for Matt. This PR ships the literal fail-closed contract as the pending-ruling default.
## Tests
- `openai-completions-compat.test.ts`: a `claude-opus`/litellm thinking-only history now serializes a non-empty body (fails without Fix 1 — `nonSystem.length === 0`), and a history with a surviving user turn does not synthesize a spurious assistant turn.
- `empty-completion-retry.test.ts`: the exhausted-cap empty stop now surfaces as an `error` terminal (fails without Fix 2 — last event was `done`).
- `ollama-provider.test.ts` (packages/catalog): the tool-forcing test's stub was a bare `{done:true}` — a degenerate empty completion that now (correctly) fails closed after the retry cap. Replaced with a realistic forced tool-call response so the test exercises tool-forcing without tripping the empty-completion guard.
Both regressions were confirmed red before the fix and green after. `packages/ai`: edited-file suites all green; full suite has 2 pre-existing environmental flakes in the untouched `proxy.test.ts` / Bedrock-guardrails tests (both pass in isolation). `packages/catalog`: full suite green (680 pass). biome + tsgo clean.
Refs RIG-2806
Co-authored-by: Matt Wilkinson <matt@rigel.build>
… recovery (RIG-2806) Addresses upstream-omp's maintainer review (all-clear at the high+medium floor; these are the recommended lows). - **Abort race (low 1):** the fail-closed empty-completion terminal in `empty-completion-retry.ts` now also gates on `!signal?.aborted`, matching its sibling retry guard. On the narrow race where the caller aborts and the provider still returns a clean empty `done`, the aborted turn now delivers its benign terminal as-is instead of being relabeled a loud provider error. The exhausted-cap message derives its attempt count from `emptyAttempt + 1` rather than a hardcoded constant. - **Multi-turn recovery lock-in (low 2a):** new compat test asserts that when several consecutive thinking-only turns are the whole history, the recovered assistant turn carries the *last* dropped reasoning, not an earlier one. Shared `withEmptyCompletionRetry` regression suite green across all four providers; biome + tsgo clean. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…guards The fail-closed block referenced three identifiers that never existed in scope (`isRetryableEmpty`, `message`, `emptyAttempt`), so the module threw a ReferenceError on the unconditional post-loop path — every provider stream routes through it, not only empty ones. Split the existing predicate into a shape-only half and the cap check, and gate the block on the shape-only half: gating on the combined predicate would never fire, since its cap conjunct is false exactly when the cap is exhausted. The abort-gate test called a helper renamed to `withReplaySafeStreamRetry` and omitted the required policy argument, so it threw before reaching the gate it names — it had never executed. Fixed the call and passed `retryEmptyCompletion`, without which the assertion passes vacuously. Trim the recovered assistant turn: bare Anthropic-dialect demotion copies the thinking text verbatim, and the recovered turn is by construction the final message, which Anthropic rejects when it ends in whitespace. Matches the existing guard in transform-messages.ts. Corrected the comment claiming renderDemotedThinking returns "" for whitespace — the check is falsy-only. Added the trailing-whitespace regression (red-checked: fails without the trim) and an end-to-end assertion on the serialized request body, which is the contract the intermediate-array tests could not pin. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…mment
Records the user-visible half of the fix: an exhausted empty-completion
retry now surfaces a visible error instead of a benign terminal, and
demotable history recovers its dropped reasoning rather than serializing
an empty prompt.
The ollama forced-tool fixture claimed a bare `{done:true}` would now fail
closed after the retry cap. It would not: a single attempt never exhausts
the cap, so that branch is unreachable from the test either way. Reworded
to describe what the fixture actually models.
Co-authored-by: Matt Wilkinson <matt@rigel.build>
oxfmt --check gates CI; the added end-to-end test exceeded the line budget on the Response constructor and the some() predicate. Whitespace only. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…tionales
The previous comment claimed the forced-tool fixture was not load-bearing
and that a single attempt never exhausts the retry cap. Both halves are
false: streamOllama wraps streamOllamaOnce with retryEmptyCompletion, so a
bare {done:true} is retried to the cap and then fails closed with an error
terminal. Reverting the fixture makes the test fail with [start, error] --
verified by mutation. The comment would have invited exactly that revert.
Also moved the two guard rationales from the block comment onto the
conditions they defend, so a maintainer editing either one cannot skip the
sentence explaining why it is there, and retitled the compat describe block
to lead with prose like its neighbour.
Co-authored-by: Matt Wilkinson <matt@rigel.build>
…sertion The fetch-level test failed in CI while passing locally under CI's exact invocation (bun test --parallel=8 --timeout=30000, 4442 pass), and CI attributed the same test name to two different files -- it was not hermetic under the parallel runner. A test that only fails where it cannot be reproduced is a defect, not a signal, and retrying it is not an option. It also added no unique coverage: convertMessages' return value IS the serialized request body (openai-completions.ts:1808 builds messages = convertMessages(...) and sends it verbatim), so the four direct assertions already pin the zero-body contract -- including the red-checked trailing-whitespace regression. Documented that equivalence at the tests so the removal does not read as a gap. Co-authored-by: Matt Wilkinson <matt@rigel.build>
My removal of this test was justified by a diagnosis that was wrong. I called it non-hermetic under the parallel runner; the real cause is that it called streamOpenAICompletions with no apiKey, so openai-completions.ts:719 fell back to getEnvApiKey(). My shell exports LITELLM_API_KEY and CI does not, so the request failed the missing-key check before fetch was reached, captured stayed undefined, and body.length read 0 -- exactly CI's reported failure. It never depended on parallelism, and the name-collision claim was unsupported. Restored with apiKey: "test-key", matching the 28 other sites in this file. Verified as a four-cell differential rather than a single pass: the old form fails without a provider key and passes with one; the restored form passes both ways. Full package under CI's flags in a credential-free environment: 4442 pass / 332 skip / 0 fail. Also corrects two false claims in the comment that justified the removal. convertMessages' result is not sent verbatim -- maybeAddAnthropicCacheControl mutates it in place at :1809 before the assignment at :1810, and markLatestStableChatCompletionsCacheBreakpoint rewrites elements at :1639. Those steps only enrich the array (no step removes a message), which is the property the argument actually needs, so the conclusion held while its stated reasoning did not. The retry-timing rationale was also false: a first attempt carrying visible content commits immediately, so the wrapper never engages. Documents that the completedMessage re-check in the fail-closed guard is redundant at runtime but load-bearing for TypeScript's narrowing; removing it produces TS18048 + TS2345. Co-authored-by: Matt Wilkinson <matt@rigel.build>
My previous comment annotated the wrong line. It sat above the completedMessage re-check inside the fail-closed `if` and claimed that line was load-bearing for TypeScript's narrowing. An isolation matrix shows the opposite: deleting that line alone typechecks clean, while deleting the conjunct inside the `isRetryableEmpty` initializer alone produces TS18048 x4 plus TS2345. Narrowing does propagate through the intermediate boolean, so the comment's premise was false and its "do not simplify it away" directive pointed a maintainer at a removable line while leaving the real one bare -- the exact failure it was written to prevent. My earlier check removed both occurrences at once and attributed the errors to the wrong one. A differential that varies two things measures neither. Moves the annotation onto the conjunct that actually narrows, with the observed error codes, and deletes the genuinely dead re-check. Also widens the wire test's content assertion to accept both shapes the comment above it documents. It matched only a string body, so a compat row setting cacheControlFormat would have it report a zero body on a request carrying real content -- the very restructuring that comment describes. Red-checked: disabling the recovery path still fails it (4 fail / 74 pass). Verified at this commit: packages/ai 4442 pass / 332 skip / 0 fail under CI's flags in a credential-free environment, typecheck clean, format clean. Co-authored-by: Matt Wilkinson <matt@rigel.build>
The relocated comment listed TS18048 x4 + TS2345, which was measured on the file BEFORE this branch deleted the redundant re-check inside the fail-closed `if`. That re-check was suppressing TS2322 at the spread, so with it gone, removing the narrowing conjunct now produces six errors, not five. The omitted code is the one that most directly corroborates the comment's own claim: TS2322 lands on the `...completedMessage` spread the comment cites. Verified one variable at a time on the committed file: deleting the conjunct gives TS18048 x4 + TS2345 + TS2322; restoring it is clean. Co-authored-by: Matt Wilkinson <matt@rigel.build>
The parenthetical I added in the previous commit said TS2322 "lands on the
spread itself." It does not. Three compilers agree the diagnostic is reported
at column 11 of the `const errored: AssistantMessage = {` declaration, one
line above the spread -- TypeScript reports an object-literal assignability
failure at the declaration's name, not at the member that contributed the bad
property.
"Lands on" is a claim about reported position, and finding the error is the
only thing a maintainer would use that parenthetical for, so it sent them to
the wrong line.
Causation and position are different claims and only causation was verified:
non-null-asserting the spread alone makes TS2322 disappear (six errors to
five), so the spread is genuinely the cause. Reworded to say that, and to name
the line the compiler actually prints.
Verified both halves separately at this commit; packages/ai 4442 pass / 332
skip / 0 fail credential-free, typecheck clean, format clean.
Co-authored-by: Matt Wilkinson <matt@rigel.build>
Seven consecutive attempts at this four-line comment shipped a false claim, and three of those were specifically about error codes and reported line positions -- detail that is coupled to a compiler version and to line numbers that move with any edit above them. Restating it an eighth time would be betting on the same class of claim that has failed every time. Reduced to what the next maintainer needs and what stays true: the conjunct narrows completedMessage for the conjuncts below and for the spread in the fail-closed block, so deleting it does not typecheck, and it is not merely a runtime shape check. Anyone who wants the codes can delete the line and read them -- which is how every wrong version of this comment was caught. Every remaining clause is checkable and none depends on a version or a line number. Verified: deleting the conjunct yields 6 errors; restoring is clean. packages/ai 4442 pass / 332 skip / 0 fail credential-free (4 consecutive runs), typecheck clean, format clean. Co-authored-by: Matt Wilkinson <matt@rigel.build>
1fc8180 to
28d93f7
Compare
6d0108c to
71c5eec
Compare
My removal of this test was justified by a diagnosis that was wrong. I called
it non-hermetic under the parallel runner; the real cause is that it called
streamOpenAICompletions with no apiKey, so openai-completions.ts:719 fell back
to getEnvApiKey(). My shell exports LITELLM_API_KEY and CI does not, so the
request failed the missing-key check before fetch was reached, captured stayed
undefined, and body.length read 0 -- exactly CI's reported failure. It never
depended on parallelism, and the name-collision claim was unsupported.
Restored with apiKey: "test-key", matching the 28 other sites in this file.
Verified as a four-cell differential rather than a single pass: the old form
fails without a provider key and passes with one; the restored form passes
both ways. Full package under CI's flags in a credential-free environment:
4442 pass / 332 skip / 0 fail.
Also corrects two false claims in the comment that justified the removal.
convertMessages' result is not sent verbatim -- maybeAddAnthropicCacheControl
mutates it in place at :1809 before the assignment at :1810, and
markLatestStableChatCompletionsCacheBreakpoint rewrites elements at :1639.
Those steps only enrich the array (no step removes a message), which is the
property the argument actually needs, so the conclusion held while its stated
reasoning did not. The retry-timing rationale was also false: a first attempt
carrying visible content commits immediately, so the wrapper never engages.
Documents that the completedMessage re-check in the fail-closed guard is
redundant at runtime but load-bearing for TypeScript's narrowing; removing it
produces TS18048 + TS2345.
Co-authored-by: Matt Wilkinson matt@rigel.build