Skip to content

docs: record 2026-10-01 triage and roadmap - #104

Merged
ericlitman merged 6 commits into
mainfrom
claude/open-pstack-triage-de2546
Oct 1, 2026
Merged

ericlitman merged 6 commits into
mainfrom
claude/open-pstack-triage-de2546

Conversation

@ericlitman

Copy link
Copy Markdown
Owner

What changed

Adds docs/plans/2026-10-01-triage-and-roadmap.md, the 2026-10-01 triage of every open issue and recently closed PR. It records the maintainer decisions, the plan for making Cursor upstream syncs cheaper (model and provider registry #103, a port-patch ledger, a sync runbook), and the factory readiness gaps (MASTRA-737, the #101 live-gate check, #90). #102, #103, and MASTRA-737 reference this file.

Verification

Documentation only. No skill, runner, manifest, or test file changes, so there's no installed-harness behavior to exercise.

  • Bun tests, strict typecheck, static invariants, and plugin validation pass (CI verify).
  • No runtime behavior changed, so there's no live harness action to record.

🤖 Generated with Claude Code

ericlitman and others added 2 commits October 1, 2026 16:24
Buckets every open issue, records the closed-PR follow-ups and the
maintainer decisions, plans the model and provider registry (#103) and
upstream-sync changes, and lists the factory readiness gaps (MASTRA-737,
the #101 live-gate check, #90).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 3/5

[Low risk] Documents triage notes and roadmap planning.

The roadmap should not guide the #101 merge until its live-test instructions and check order agree.

Findings

  1. P1 Drafts cannot guard the queue ▶
  2. P1 Check arrives after its merge ▶
  3. P2 Sync runbook skips required steps ▶
  4. P2 Roles lose their model choices ▶

Summary

PR #104 adds an October 1 triage and roadmap document for Open Pstack. It records issue and PR decisions, factory readiness needs, and a proposed plan for upstream syncs and future work; runtime behavior is unchanged.

  • Groups issues and PR follow-ups into decisions, including what to keep, close, answer, or schedule.
  • Lays out registry, ledger, and runbook ideas for reducing repeated work during Cursor upstream syncs.
  • Describes the factory work sequence, from readiness gates through runner fixes and later work waves.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A["Build #90 status producer"] --> B["Post live-gate status for exact head"]
  B --> C["#101 required check can pass"]
  C --> D["Queue PR after live test"]
Loading

Reviews (1) · Last reviewed commit: "docs: record 2026-10-01 triage and roadm..."


| Item | Disposition |
|---|---|
| #100 / PR #101 Mergify queue | Land it. `verify` and `Unfret` pass. Answer Greptile's P1 ("queues before live verdict"): AGENTS.md keeps a PR in draft until live evidence exists, and Mergify does not queue drafts. Record the first queue-merged PR on #100. Then close MASTRA-455, which duplicates it. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Drafts cannot guard the queue

The #101 instruction says to land the queue because PRs stay in draft until the live test. But the later finding says Unfret does not review drafts. Once a PR is ready for that review, #101 can queue it before the live test. Update this instruction to require the live-gate check rather than relying on draft status.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

- The installed `factory-ops.mjs`, factory-adjudicate, and babysit-pr match mastra-pilot `main`. factory-run's `SKILL.md` is one paragraph behind MASTRA-735 (the merge notice now comes from the deployment's `pullRequestMerged` override). Reinstall it from `main`.
- `begin` and `closeout` accept only Linear sources (MASTRA-737). This blocks every open-pstack run.
- #101 gap: Unfret doesn't review drafts, so a PR has to be non-draft to get `Unfret`. Once it's non-draft, #101 queues it when `verify` and `Unfret` pass, which can happen before the live test. Fix: a required live-gate commit status, posted by the #90 skill for the exact head, listed in `merge_conditions` and in the protection's `success_conditions`. A correction is posted on the #101 Greptile thread.
- Order: MASTRA-737, then the live-gate check in #101 and its merge, then #90. After that, Wave 1.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Check arrives after its merge

The plan merges #101 with a required live-gate status before building the #90 skill that must post it. If that status is required for #101 itself, #101 cannot merge. If it is deferred, PRs can queue without the live-test safeguard. Put a way to post the status before enabling the requirement, and explain how #101 passes its own check.

Comment on lines +121 to +122
5. Update `CHANGES.md`, `NOTICE.md`, and `UPSTREAM.md`.
6. Run the live gate.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Sync runbook skips required steps

The proposed sync runbook goes from updating three files straight to the live gate. The current UPSTREAM.md procedure also requires local CI-equivalent checks and an update to README-UPSTREAM.md when upstream changes it. Add those steps so a planner does not finish a sync with unchecked changes or a stale README mirror.

Knowledge Base Used: Upstream synchronization and maintenance


Changes, in order:

1. **Model registry (#103).** One port-owned table lists the families, defaults, efforts, optional entries, and native agent stems. It lives in `provider-dispatch.md`, with a machine-readable copy that the tests read. Upstream-derived files name roles ("`arena runners`; defaults in provider-dispatch.md") instead of slugs, and the tests derive counts from the registry. After this change, GPT-6.1 Sol is one row and a default change touches one file.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Roles lose their model choices

The registry plan replaces model names in skills with role names, but the listed registry fields do not map roles to models. Those assignments currently live in the skills and the setup example, not in provider-dispatch.md. Specify where they will move so a skill without a saved model sheet can still choose models.

Knowledge Base Used: Poteto runner and orchestration

@unfret-eal

unfret-eal Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

✅ No blocking findings

Walkthrough

Adds a single planning document, `docs/plans/2026-10-01-triage-and-roadmap.md`, recording the 2026-10-01 triage of all open issues and recent PRs. It sorts the issues into buckets with dispositions and execution waves, and lists the replies owed to contributors whose PRs were closed. It also records the maintainer decisions D1–D6 and lays out a plan to make Cursor upstream syncs cheaper: a model and provider adapter registry (#103), a port-patch ledger (#105), and a sync runbook (#106). It closes with the factory readiness gaps blocking the pipeline, namely Linear-only `begin`/`closeout` (MASTRA-737), the missing `live-gate` check in the #101 Mergify queue, and the live-gate skill (#90). This is documentation only, with no code, skill, or test changes; the author left the CI `verify` checkbox unchecked.

Since the last review
  • Withdrawn: Queue-head changes can merge without the live gate. Why: The current head repairs the carried admission-only live-test rule. Line 166 now requires an up-to-date branch, explicitly states that any main change forces a rebase and fresh verify, Unfret, and live-gate checks, and requires the merged content to match the installed-tested content. The rule no longer ends at queue entry or permits the described changed-base candidate to merge without retesting. The incomplete in-place configuration alleged in finding-0000 remains a separate concern; it does not preserve this finding's assertion that the plan deliberately allows untested queue-head changes.
  • Withdrawn: Provide an Unfret review handoff for draft batch PRs. Why: The current head no longer keeps Unfret in merge_conditions. Line 166 explicitly prescribes empty merge_conditions, moves the required checks into queue_conditions, and targets in-place checking rather than a generated draft review handoff. This removes the particular separate-merge-check configuration responsible for the carried defect. Finding-0000 identifies a different possible reason that draft candidates could still be generated; correcting the old merge_conditions requirement does not establish that the newly proposed recipe is otherwise complete.
  • Withdrawn: Wave 2 schedules work with no issue source. Why: The current head supplies the missing GitHub issue sources: the port-patch ledger is assigned Add a checked port-patch ledger for upstream divergences #105 and the sync runbook Write a factory-runnable upstream sync runbook #106, both in their detailed definitions and in Wave 2. These tasks are no longer scheduled without issue identifiers, so the stated obstacle to issue-based intake and closeout has been repaired.
Details
Lane / stage Model Effort Outcome Duration
broad-gpt-5.6-sol-xhigh / review gpt-5.6-sol (reported model unavailable) xhigh succeeded 44 s
broad-opus-5.5-max / review anthropic/claude-opus-5-5 (reported model unavailable) max succeeded 8 min 7 s
focal-gpt-6.1-sol-xhigh / review openai/gpt-6.1-sol (reported model unavailable) xhigh succeeded 9 min 45 s
judge / judge openai/gpt-6-astra (reported model unavailable) high succeeded 1 min 3 s
confirmation / confirmation gpt-5.6-sol (reported model unavailable) xhigh succeeded 14 s

Reviewed the whole pull request.
3 reviewers finished.
Commit 2b78f64.
Check run.
Run: GET /unfret/run/panel:2b78f6450fe521ddd67c5a582de460bf2ff870d0:t9YaiHjNdLS:uHRvDA07RsM.

@unfret review · @unfret status · @unfret help

@unfret-eal unfret-eal Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 1 blocking finding
🔴 High · Planned loop un-drafts PRs before live evidence, contrary to AGENTS.md
Check

Comment on lines +135 to +136
6. Unfret reviews, and the builder fixes the findings.
7. Claude runs the live gate on the Mac (A3), posts the evidence, and sets the `live-gate` check.

@unfret-eal unfret-eal Bot Oct 1, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Planned loop un-drafts PRs before live evidence, contrary to AGENTS.md

AGENTS.md mandates: "A pull request without that evidence remains a draft." The planned loop puts Unfret review (step 6) before the live gate (step 7). Line 164 says Unfret only reviews non-draft PRs, and line 11 says "The draft convention can't hold the queue." A builder that follows this loop marks every factory PR ready with no live-gate evidence, which violates the mandate. A builder that obeys AGENTS.md keeps the PR as a draft, so Unfret never reviews it and the loop stalls at step 6. The doc plans no AGENTS.md amendment. Such a change may exist outside this diff, for example in #101, but none is visible here.

Withdrawn in the latest review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@unfret-eal unfret-eal Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 1 blocking finding
🔴 High · Factory loop opens PRs without the mandated pre-PR checks
Check

2. Claude moves the card to Planning with `factory-run`.
3. The planner posts a plan.
4. `factory-adjudicate` judges the plan with a different model.
5. Build runs on the Factory branch, and the builder opens the PR as a draft.

@unfret-eal unfret-eal Bot Oct 1, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Factory loop opens PRs without the mandated pre-PR checks

AGENTS.md requires running the Bun tests, strict typecheck, static invariants and plugin validation before opening any pull request. Step 5 has the Factory builder open the draft PR immediately after the build. Line 13 says Factory sandboxes have no bun or claude, and the plan moves only the live gate to the Mac. A Wave 1 build such as #25 that follows this loop therefore opens its PR without those checks; CI verify runs only after the PR exists. A builder that obeys AGENTS.md instead cannot open the PR and stalls at step 5. The sync runbook (line 122, then line 126, 'an ordinary factory ticket') puts 'Run the CI-equivalent checks locally' in that same sandbox. The sandbox may be provisioned outside this diff, but nothing visible here does it.

Withdrawn in the latest review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@unfret-eal unfret-eal Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 1 blocking finding
🔴 High · Provide live-gate evidence for the queue-generated head
Check


- The installed `factory-ops.mjs`, factory-adjudicate, and babysit-pr match mastra-pilot `main`. factory-run's `SKILL.md` is one paragraph behind MASTRA-735 (the merge notice now comes from the deployment's `pullRequestMerged` override). Reinstall it from `main`.
- `begin` and `closeout` accept only Linear sources (MASTRA-737). This blocks every open-pstack run.
- #101 gap: Unfret doesn't review drafts, so a PR has to be non-draft to get `Unfret`. Once it's non-draft, #101 queues it when `verify` and `Unfret` pass, which can happen before the live test. Fix: a required live-gate commit status, posted by the #90 skill for the exact head, listed in `merge_conditions` and in the protection's `success_conditions`. A correction is posted on the #101 Greptile thread.

@unfret-eal unfret-eal Bot Oct 1, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Provide live-gate evidence for the queue-generated head

The referenced #101 configuration at 635d077 has empty queue_conditions and separate merge_conditions, so Mergify’s documented two-step behavior validates a new temporary batch head. Steps 6–8 post live-gate on the original PR head before queueing, while this line also requires it on the batch head. Once #90 exists, following the prescribed loop therefore enters the queue but cannot complete the required status on its generated candidate; #101’s checks_timeout: null leaves that wait unbounded. The plan needs an installed-test/status handoff for the queue-generated candidate as well.

Withdrawn in the latest review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@unfret-eal unfret-eal Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 3 blocking findings
🔴 High · Queue-head changes can merge without the live gate
🔴 High · Provide an Unfret review handoff for draft batch PRs
🔴 High · Wave 2 schedules work with no issue source
Check


- The installed `factory-ops.mjs`, factory-adjudicate, and babysit-pr match mastra-pilot `main`. factory-run's `SKILL.md` is one paragraph behind MASTRA-735 (the merge notice now comes from the deployment's `pullRequestMerged` override). Reinstall it from `main`.
- `begin` and `closeout` accept only Linear sources (MASTRA-737). This blocks every open-pstack run.
- #101 gap: Unfret doesn't review drafts, so a PR has to be non-draft to get `Unfret`. Once it's non-draft, #101 queues it when `verify` and `Unfret` pass, which can happen before the live test. Fix: a required `live-gate` commit status, posted by the #90 skill on the exact PR head. It gates queue entry only: list it in the protection's `success_conditions` and the queue's `queue_conditions`, not in `merge_conditions`. Mergify validates a temporary queue head that the installed-harness test never sees, so `live-gate` there could never be satisfied (and `checks_timeout: null` would wait forever). `merge_conditions` keep `verify` and `Unfret` for the queue head. With `batch_size: 1`, the queue head is the tested PR head merged onto current `main`. A PR whose live run predates a `main` change that touches the plugin is rebased and re-tested before it queues. A correction is posted on the #101 Greptile thread.

@unfret-eal unfret-eal Bot Oct 1, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Queue-head changes can merge without the live gate

This deliberately limits live-gate to queue entry even though Mergify may replace the tested PR head with a temporary head. If a plugin-changing PR ahead of this one lands after this PR enters the queue, Mergify rebuilds the candidate on the new main; verify and Unfret can pass and that candidate can merge without ever being installed or exercised. The stated rebase rule covers changes before queue entry only, so this violates the exact-candidate live-test gate on a reachable queue transition. Also reported for this defect: - Stale-live-run rule is unenforced; queued PRs merge onto a changed plugin untested (docs/plans/2026-10-01-triage-and-roadmap.md:166): Line 166 says a PR whose live run predates a plugin-touching main change is rebased and re-tested before it queues. The prescribed config only checks that live-gate exists on the PR head, which stays green when main moves, and merge_conditions hold no live check. Mergify auto-queues on those conditions, so nothing applies the rule, and the rule never covers PRs already queued. Scenario: Wave 3 runs #71 and #69 in parallel, both live-tested on M0. #71 merges and changes the runner and provider registry. #69 then queues, or is already queued. Its queue head (#69 on M1) passes verify and Unfret and merges with no installed-harness test of that plugin state. That breaks this rule and AGENTS.md's exact-candidate gate. - Preserve live verification for the generated merge candidate (docs/plans/2026-10-01-triage-and-roadmap.md:166): Moving live-gate out of merge_conditions removes the prior missing-status wait but violates AGENTS.md's exact-candidate installed-test gate. With #101's default speculative checks, even batch_size: 1 can combine a PR with earlier queued plugin changes absent from its live-tested head. Re-testing only before queue entry also misses later base updates. The prescribed queue can therefore approve changed candidate content without installed-harness evidence; it still needs a live-test/status handoff for the generated candidate.

Withdrawn in the latest review.


- The installed `factory-ops.mjs`, factory-adjudicate, and babysit-pr match mastra-pilot `main`. factory-run's `SKILL.md` is one paragraph behind MASTRA-735 (the merge notice now comes from the deployment's `pullRequestMerged` override). Reinstall it from `main`.
- `begin` and `closeout` accept only Linear sources (MASTRA-737). This blocks every open-pstack run.
- #101 gap: Unfret doesn't review drafts, so a PR has to be non-draft to get `Unfret`. Once it's non-draft, #101 queues it when `verify` and `Unfret` pass, which can happen before the live test. Fix: a required `live-gate` commit status, posted by the #90 skill on the exact PR head. It gates queue entry only: list it in the protection's `success_conditions` and the queue's `queue_conditions`, not in `merge_conditions`. Mergify validates a temporary queue head that the installed-harness test never sees, so `live-gate` there could never be satisfied (and `checks_timeout: null` would wait forever). `merge_conditions` keep `verify` and `Unfret` for the queue head. With `batch_size: 1`, the queue head is the tested PR head merged onto current `main`. A PR whose live run predates a `main` change that touches the plugin is rebased and re-tested before it queues. A correction is posted on the #101 Greptile thread.

@unfret-eal unfret-eal Bot Oct 1, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Provide an Unfret review handoff for draft batch PRs

Keeping Unfret in merge_conditions requires that check on the temporary batch head. Mergify creates draft batch PRs, and separate merge checks disable in-place checking even with batch_size: 1; Unfret skips/cancels drafts, as this paragraph itself notes. A ready PR with successful admission checks therefore enters the queue but never receives Unfret on its batch head. With checks_timeout: null, the queue waits indefinitely. No step makes that generated candidate reviewable or publishes its review result. Also reported for this defect: - Queue head still requires Unfret, which never runs on Mergify's draft (docs/plans/2026-10-01-triage-and-roadmap.md:166): Line 166 keeps Unfret in merge_conditions "for the queue head". The same line, and line 11, say Unfret doesn't review drafts and that Mergify validates a temporary queue head. Mergify's temporary queue PRs are drafts, and checks in merge_conditions are evaluated on them; that is why live-gate was moved out. Scenario: a PR enters the queue with verify, Unfret and live-gate green. Mergify opens its draft candidate and Unfret skips it, so check-success=Unfret never matches. With checks_timeout: null the queue waits forever and blocks every later PR at batch_size 1. Moving live-gate to queue_conditions resolves the carried live-gate stall, but the same stall recurs on Unfret.

Withdrawn in the latest review.

- environment stripping

`PROVIDERS`, the route table, and the setup probe rows are derived from this registry, and the tests are table-driven. After this change, Devin needs one adapter, one fixture, and one registry row.
3. **Port-patch ledger.** A checked-in file lists every intentional divergence with its file, anchor, reason, and issue. Examples: the `disable-model-invocation` flag removed from four skills, the dispatch-contract preface, the sheet-path substitution, and the excluded upstream hunks. `upstream-merge-probe.py` checks that each entry still holds after a merge, and the exclusion list in `UPSTREAM.md` is generated from the ledger.

@unfret-eal unfret-eal Bot Oct 1, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Wave 2 schedules work with no issue source

The port-patch ledger here and the sync runbook at line 116 are durable repository work, but only the two registry items are assigned GitHub issue #103. The execution loop begins from an auto-intaken issue card and closes its GitHub issue, so these later Wave 2 items have no source/card from which factory-run can start or close them. This leaves the documented mainline plan unable to execute those tasks and violates the repository's GitHub-Issues-only queue rule.

Withdrawn in the latest review.

…ook issues

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@unfret-eal unfret-eal Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No blocking findings
Check

@mergify

mergify Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@ericlitman
ericlitman merged commit 77a91fd into main Oct 1, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant