docs: record 2026-10-01 triage and roadmap - #104
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
|
|
||
| | Item | Disposition | | ||
| |---|---| | ||
| | #100 / PR #101 Mergify queue | Land it. `verify` and `Unfret` pass. Answer Greptile's P1 ("queues before live verdict"): AGENTS.md keeps a PR in draft until live evidence exists, and Mergify does not queue drafts. Record the first queue-merged PR on #100. Then close MASTRA-455, which duplicates it. | |
There was a problem hiding this comment.
The #101 instruction says to land the queue because PRs stay in draft until the live test. But the later finding says Unfret does not review drafts. Once a PR is ready for that review, #101 can queue it before the live test. Update this instruction to require the live-gate check rather than relying on draft status.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| - The installed `factory-ops.mjs`, factory-adjudicate, and babysit-pr match mastra-pilot `main`. factory-run's `SKILL.md` is one paragraph behind MASTRA-735 (the merge notice now comes from the deployment's `pullRequestMerged` override). Reinstall it from `main`. | ||
| - `begin` and `closeout` accept only Linear sources (MASTRA-737). This blocks every open-pstack run. | ||
| - #101 gap: Unfret doesn't review drafts, so a PR has to be non-draft to get `Unfret`. Once it's non-draft, #101 queues it when `verify` and `Unfret` pass, which can happen before the live test. Fix: a required live-gate commit status, posted by the #90 skill for the exact head, listed in `merge_conditions` and in the protection's `success_conditions`. A correction is posted on the #101 Greptile thread. | ||
| - Order: MASTRA-737, then the live-gate check in #101 and its merge, then #90. After that, Wave 1. |
There was a problem hiding this comment.
The plan merges #101 with a required live-gate status before building the #90 skill that must post it. If that status is required for #101 itself, #101 cannot merge. If it is deferred, PRs can queue without the live-test safeguard. Put a way to post the status before enabling the requirement, and explain how #101 passes its own check.
| 5. Update `CHANGES.md`, `NOTICE.md`, and `UPSTREAM.md`. | ||
| 6. Run the live gate. |
There was a problem hiding this comment.
Sync runbook skips required steps
The proposed sync runbook goes from updating three files straight to the live gate. The current UPSTREAM.md procedure also requires local CI-equivalent checks and an update to README-UPSTREAM.md when upstream changes it. Add those steps so a planner does not finish a sync with unchecked changes or a stale README mirror.
Knowledge Base Used: Upstream synchronization and maintenance
|
|
||
| Changes, in order: | ||
|
|
||
| 1. **Model registry (#103).** One port-owned table lists the families, defaults, efforts, optional entries, and native agent stems. It lives in `provider-dispatch.md`, with a machine-readable copy that the tests read. Upstream-derived files name roles ("`arena runners`; defaults in provider-dispatch.md") instead of slugs, and the tests derive counts from the registry. After this change, GPT-6.1 Sol is one row and a default change touches one file. |
There was a problem hiding this comment.
Roles lose their model choices
The registry plan replaces model names in skills with role names, but the listed registry fields do not map roles to models. Those assignments currently live in the skills and the setup example, not in provider-dispatch.md. Specify where they will move so a skill without a saved model sheet can still choose models.
Knowledge Base Used: Poteto runner and orchestration
✅ No blocking findingsWalkthroughAdds a single planning document, Since the last review
Details
Reviewed the whole pull request. @unfret review · @unfret status · @unfret help |
There was a problem hiding this comment.
🛑 1 blocking finding
🔴 High · Planned loop un-drafts PRs before live evidence, contrary to AGENTS.md
Check
| 6. Unfret reviews, and the builder fixes the findings. | ||
| 7. Claude runs the live gate on the Mac (A3), posts the evidence, and sets the `live-gate` check. |
There was a problem hiding this comment.
🔴 High · Planned loop un-drafts PRs before live evidence, contrary to AGENTS.md
AGENTS.md mandates: "A pull request without that evidence remains a draft." The planned loop puts Unfret review (step 6) before the live gate (step 7). Line 164 says Unfret only reviews non-draft PRs, and line 11 says "The draft convention can't hold the queue." A builder that follows this loop marks every factory PR ready with no live-gate evidence, which violates the mandate. A builder that obeys AGENTS.md keeps the PR as a draft, so Unfret never reviews it and the loop stalls at step 6. The doc plans no AGENTS.md amendment. Such a change may exist outside this diff, for example in #101, but none is visible here.
Withdrawn in the latest review.Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🛑 1 blocking finding
🔴 High · Factory loop opens PRs without the mandated pre-PR checks
Check
| 2. Claude moves the card to Planning with `factory-run`. | ||
| 3. The planner posts a plan. | ||
| 4. `factory-adjudicate` judges the plan with a different model. | ||
| 5. Build runs on the Factory branch, and the builder opens the PR as a draft. |
There was a problem hiding this comment.
🔴 High · Factory loop opens PRs without the mandated pre-PR checks
AGENTS.md requires running the Bun tests, strict typecheck, static invariants and plugin validation before opening any pull request. Step 5 has the Factory builder open the draft PR immediately after the build. Line 13 says Factory sandboxes have no bun or claude, and the plan moves only the live gate to the Mac. A Wave 1 build such as #25 that follows this loop therefore opens its PR without those checks; CI verify runs only after the PR exists. A builder that obeys AGENTS.md instead cannot open the PR and stalls at step 5. The sync runbook (line 122, then line 126, 'an ordinary factory ticket') puts 'Run the CI-equivalent checks locally' in that same sandbox. The sandbox may be provisioned outside this diff, but nothing visible here does it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🛑 1 blocking finding
🔴 High · Provide live-gate evidence for the queue-generated head
Check
|
|
||
| - The installed `factory-ops.mjs`, factory-adjudicate, and babysit-pr match mastra-pilot `main`. factory-run's `SKILL.md` is one paragraph behind MASTRA-735 (the merge notice now comes from the deployment's `pullRequestMerged` override). Reinstall it from `main`. | ||
| - `begin` and `closeout` accept only Linear sources (MASTRA-737). This blocks every open-pstack run. | ||
| - #101 gap: Unfret doesn't review drafts, so a PR has to be non-draft to get `Unfret`. Once it's non-draft, #101 queues it when `verify` and `Unfret` pass, which can happen before the live test. Fix: a required live-gate commit status, posted by the #90 skill for the exact head, listed in `merge_conditions` and in the protection's `success_conditions`. A correction is posted on the #101 Greptile thread. |
There was a problem hiding this comment.
🔴 High · Provide live-gate evidence for the queue-generated head
The referenced #101 configuration at 635d077 has empty queue_conditions and separate merge_conditions, so Mergify’s documented two-step behavior validates a new temporary batch head. Steps 6–8 post live-gate on the original PR head before queueing, while this line also requires it on the batch head. Once #90 exists, following the prescribed loop therefore enters the queue but cannot complete the required status on its generated candidate; #101’s checks_timeout: null leaves that wait unbounded. The plan needs an installed-test/status handoff for the queue-generated candidate as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🛑 3 blocking findings
🔴 High · Queue-head changes can merge without the live gate
🔴 High · Provide an Unfret review handoff for draft batch PRs
🔴 High · Wave 2 schedules work with no issue source
Check
|
|
||
| - The installed `factory-ops.mjs`, factory-adjudicate, and babysit-pr match mastra-pilot `main`. factory-run's `SKILL.md` is one paragraph behind MASTRA-735 (the merge notice now comes from the deployment's `pullRequestMerged` override). Reinstall it from `main`. | ||
| - `begin` and `closeout` accept only Linear sources (MASTRA-737). This blocks every open-pstack run. | ||
| - #101 gap: Unfret doesn't review drafts, so a PR has to be non-draft to get `Unfret`. Once it's non-draft, #101 queues it when `verify` and `Unfret` pass, which can happen before the live test. Fix: a required `live-gate` commit status, posted by the #90 skill on the exact PR head. It gates queue entry only: list it in the protection's `success_conditions` and the queue's `queue_conditions`, not in `merge_conditions`. Mergify validates a temporary queue head that the installed-harness test never sees, so `live-gate` there could never be satisfied (and `checks_timeout: null` would wait forever). `merge_conditions` keep `verify` and `Unfret` for the queue head. With `batch_size: 1`, the queue head is the tested PR head merged onto current `main`. A PR whose live run predates a `main` change that touches the plugin is rebased and re-tested before it queues. A correction is posted on the #101 Greptile thread. |
There was a problem hiding this comment.
🔴 High · Queue-head changes can merge without the live gate
This deliberately limits live-gate to queue entry even though Mergify may replace the tested PR head with a temporary head. If a plugin-changing PR ahead of this one lands after this PR enters the queue, Mergify rebuilds the candidate on the new main; verify and Unfret can pass and that candidate can merge without ever being installed or exercised. The stated rebase rule covers changes before queue entry only, so this violates the exact-candidate live-test gate on a reachable queue transition. Also reported for this defect: - Stale-live-run rule is unenforced; queued PRs merge onto a changed plugin untested (docs/plans/2026-10-01-triage-and-roadmap.md:166): Line 166 says a PR whose live run predates a plugin-touching main change is rebased and re-tested before it queues. The prescribed config only checks that live-gate exists on the PR head, which stays green when main moves, and merge_conditions hold no live check. Mergify auto-queues on those conditions, so nothing applies the rule, and the rule never covers PRs already queued. Scenario: Wave 3 runs #71 and #69 in parallel, both live-tested on M0. #71 merges and changes the runner and provider registry. #69 then queues, or is already queued. Its queue head (#69 on M1) passes verify and Unfret and merges with no installed-harness test of that plugin state. That breaks this rule and AGENTS.md's exact-candidate gate. - Preserve live verification for the generated merge candidate (docs/plans/2026-10-01-triage-and-roadmap.md:166): Moving live-gate out of merge_conditions removes the prior missing-status wait but violates AGENTS.md's exact-candidate installed-test gate. With #101's default speculative checks, even batch_size: 1 can combine a PR with earlier queued plugin changes absent from its live-tested head. Re-testing only before queue entry also misses later base updates. The prescribed queue can therefore approve changed candidate content without installed-harness evidence; it still needs a live-test/status handoff for the generated candidate.
|
|
||
| - The installed `factory-ops.mjs`, factory-adjudicate, and babysit-pr match mastra-pilot `main`. factory-run's `SKILL.md` is one paragraph behind MASTRA-735 (the merge notice now comes from the deployment's `pullRequestMerged` override). Reinstall it from `main`. | ||
| - `begin` and `closeout` accept only Linear sources (MASTRA-737). This blocks every open-pstack run. | ||
| - #101 gap: Unfret doesn't review drafts, so a PR has to be non-draft to get `Unfret`. Once it's non-draft, #101 queues it when `verify` and `Unfret` pass, which can happen before the live test. Fix: a required `live-gate` commit status, posted by the #90 skill on the exact PR head. It gates queue entry only: list it in the protection's `success_conditions` and the queue's `queue_conditions`, not in `merge_conditions`. Mergify validates a temporary queue head that the installed-harness test never sees, so `live-gate` there could never be satisfied (and `checks_timeout: null` would wait forever). `merge_conditions` keep `verify` and `Unfret` for the queue head. With `batch_size: 1`, the queue head is the tested PR head merged onto current `main`. A PR whose live run predates a `main` change that touches the plugin is rebased and re-tested before it queues. A correction is posted on the #101 Greptile thread. |
There was a problem hiding this comment.
🔴 High · Provide an Unfret review handoff for draft batch PRs
Keeping Unfret in merge_conditions requires that check on the temporary batch head. Mergify creates draft batch PRs, and separate merge checks disable in-place checking even with batch_size: 1; Unfret skips/cancels drafts, as this paragraph itself notes. A ready PR with successful admission checks therefore enters the queue but never receives Unfret on its batch head. With checks_timeout: null, the queue waits indefinitely. No step makes that generated candidate reviewable or publishes its review result. Also reported for this defect: - Queue head still requires Unfret, which never runs on Mergify's draft (docs/plans/2026-10-01-triage-and-roadmap.md:166): Line 166 keeps Unfret in merge_conditions "for the queue head". The same line, and line 11, say Unfret doesn't review drafts and that Mergify validates a temporary queue head. Mergify's temporary queue PRs are drafts, and checks in merge_conditions are evaluated on them; that is why live-gate was moved out. Scenario: a PR enters the queue with verify, Unfret and live-gate green. Mergify opens its draft candidate and Unfret skips it, so check-success=Unfret never matches. With checks_timeout: null the queue waits forever and blocks every later PR at batch_size 1. Moving live-gate to queue_conditions resolves the carried live-gate stall, but the same stall recurs on Unfret.
| - environment stripping | ||
|
|
||
| `PROVIDERS`, the route table, and the setup probe rows are derived from this registry, and the tests are table-driven. After this change, Devin needs one adapter, one fixture, and one registry row. | ||
| 3. **Port-patch ledger.** A checked-in file lists every intentional divergence with its file, anchor, reason, and issue. Examples: the `disable-model-invocation` flag removed from four skills, the dispatch-contract preface, the sheet-path substitution, and the excluded upstream hunks. `upstream-merge-probe.py` checks that each entry still holds after a merge, and the exclusion list in `UPSTREAM.md` is generated from the ledger. |
There was a problem hiding this comment.
🔴 High · Wave 2 schedules work with no issue source
The port-patch ledger here and the sync runbook at line 116 are durable repository work, but only the two registry items are assigned GitHub issue #103. The execution loop begins from an auto-intaken issue card and closes its GitHub issue, so these later Wave 2 items have no source/card from which factory-run can start or close them. This leaves the documented mainline plan unable to execute those tasks and violates the repository's GitHub-Issues-only queue rule.
…ook issues Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
What changed
Adds
docs/plans/2026-10-01-triage-and-roadmap.md, the 2026-10-01 triage of every open issue and recently closed PR. It records the maintainer decisions, the plan for making Cursor upstream syncs cheaper (model and provider registry #103, a port-patch ledger, a sync runbook), and the factory readiness gaps (MASTRA-737, the #101 live-gate check, #90). #102, #103, and MASTRA-737 reference this file.Verification
Documentation only. No skill, runner, manifest, or test file changes, so there's no installed-harness behavior to exercise.
verify).🤖 Generated with Claude Code