fix(bot): harden automated candidate delivery - #2429
Conversation
|
Scope checkThis PR changes 2,289 lines across 35 files. Large PRs are harder to review and more likely to be closed without review. If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs. See CONTRIBUTING.md for contribution guidelines. |
github-actions
Bot
added
size/XL
review/needs-review
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-do | 94e6aa6 | Aug 12 2026, 06:46 AM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-cache | 94e6aa6 | Aug 12 2026, 06:46 AM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-playground | 94e6aa6 | Aug 12 2026, 06:48 AM |
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/auth-atproto
@emdash-cms/blocks
@emdash-cms/cloudflare
@emdash-cms/contentful-to-portable-text
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/plugin-cli
@emdash-cms/plugin-types
@emdash-cms/registry-client
@emdash-cms/registry-lexicons
@emdash-cms/registry-verification
@emdash-cms/sandbox-workerd
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-field-kit
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
![emdashbot[bot]](https://avatars.githubusercontent.com/in/3255304?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR hardens the bot's candidate-delivery path in the right direction: moving publication from the agent container to the trusted Worker via GitHub's Git Data API, adding fast-forward/idempotency checks, binding verification records to the candidate tree, and giving implement its own result contract. I read the diff, the full changed modules, the new tests, and traced the state-machine and proxy call sites.
The change is mostly clean and well-tested, but two issues need attention before merge.
-
renderAgentCommentfor the legacyawaiting_feedbackpath ignoresPREVIEW_PACKAGE. The newpreview_building→awaiting_reporterflow usesrenderPreviewReadyAskandrenderDraftPrBody, both of which respectPREVIEW_PACKAGE; the olderworking→awaiting_feedbackpath still returns a hard-codedemdash-cms/emdashURL.@emdashbot repro(which still lands inworking) will therefore advertise the production preview in a staging deployment, defeating the purpose of the new config. -
Container-side file modifications from verification commands are lost before publication.
snapshotCandidatematerializes the VFS onto the container before staging, so any edits made by a container command (e.g.pnpm format) are overwritten.run_checkrecords the container tree SHA after the command, sopublish_candidatethen fails the tree-match check and refuses to publish. The skill still instructs agents to run formatting throughrun_check, which sets up a legitimate workflow that cannot complete.
There was a problem hiding this comment.
Pull request overview
Hardens the infra/emdash-bot implementation + candidate-delivery pipeline by moving candidate publication into trusted Worker code (Git Data API), tightening what the sandbox can push, and recording/verifying run outputs against an exact published Git tree.
Changes:
- Adds a Worker-owned candidate publisher (
publish_candidate) with stale-head checks, fast-forward-only ref updates, and post-publication verification. - Restricts sandbox
git-receive-packto issue-scoped artifacts branches and parses the actual receive-pack command ref (not ref-like capability text). - Introduces an
implementlane/result contract (separate from bug reproduction fields), plus richer failure/run diagnostics, configurable preview package targeting, and an opt-in end-to-end implementation smoke script.
Reviewed changes
Copilot reviewed 33 out of 35 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| infra/emdash-bot/wrangler.jsonc | Adds PREVIEW_PACKAGE configuration for staging/prod preview targeting. |
| infra/emdash-bot/worker-configuration.d.ts | Regenerates Worker env typings to include PREVIEW_PACKAGE. |
| infra/emdash-bot/package.json | Adds evals:implementation smoke script entrypoint. |
| infra/emdash-bot/BOT_STATE_MACHINE.md | Updates state/event docs for the implement → fixing lane and “candidate change” wording. |
| infra/emdash-bot/.flue/skills/investigate/SKILL.md | Updates fix-loop guidance to route publication through publish_candidate. |
| infra/emdash-bot/.flue/skills/implement/SKILL.md | Adds a new skill defining the directed-implementation lane and its boundaries. |
| infra/emdash-bot/.flue/skills/fix/SKILL.md | Updates fix guidance to use run_check + publish_candidate and removes direct git push instructions. |
| infra/emdash-bot/.flue/lib/verification.ts | Adds verification-command hardening and “latest named checks must pass + match tree” enforcement. |
| infra/emdash-bot/.flue/lib/router.ts | Extends agent result with implemented and maps implement outcomes to agent.fix_ready/agent.failed. |
| infra/emdash-bot/.flue/lib/preview.ts | Adds preview-package parameterization + validation for pkg.pr.new URLs. |
| infra/emdash-bot/.flue/lib/comments.ts | Shifts copy from “fix” → “change”, supports staging preview packages, and adds failure stage/run-id in failed comments. |
| infra/emdash-bot/.flue/lib/github.ts | Adds Git Data API helpers (blob/tree/commit/ref) for Worker-owned publication. |
| infra/emdash-bot/.flue/lib/candidate-publisher.ts | Implements candidate snapshot publication via Git Data API with race/stale-head and tree-sha verification. |
| infra/emdash-bot/.flue/lib/github-proxy.ts | Tightens outbound gating: only artifacts pushes; parses receive-pack ref safely; surfaces structured gate diagnostics. |
| infra/emdash-bot/.flue/lib/exec-env.ts | Wraps container commands with bash -o pipefail, snapshots staged candidate content, and enforces publication bounds. |
| infra/emdash-bot/.flue/lib/machine.ts | Updates state descriptions and transitions so implement enters fixing and uses “candidate change” wording. |
| infra/emdash-bot/.flue/lib/machine.json | Regenerates machine JSON for updated transitions and descriptions. |
| infra/emdash-bot/.flue/lib/orchestrator.ts | Threads PREVIEW_PACKAGE through preview polling + comments/PR body; persists run diagnostics for failure reporting. |
| infra/emdash-bot/.flue/cloudflare.ts | Switches to structured GitHub request inspection and logs denial stage/refs/parse errors for ops visibility. |
| infra/emdash-bot/.flue/agents/investigate.ts | Adds run_check/publish_candidate, persists verification/publication metadata, and introduces report_implementation. |
| infra/emdash-bot/evals/src/types.ts | Extends reported result types with run/publication/verification details and implement-mode fields. |
| infra/emdash-bot/evals/src/client.ts | Updates reported-result detection logic and comments for new payload shape. |
| infra/emdash-bot/evals/README.md | Documents the new live implementation delivery smoke run and staging preview-package isolation. |
| infra/emdash-bot/evals/bin/implementation-smoke.ts | Adds an opt-in end-to-end smoke that exercises real delivery (branch, preview, confirm, draft PR). |
| infra/emdash-bot/tests/unit/router.test.ts | Updates router expectations for fixing state/label and implement-mode outcome semantics. |
| infra/emdash-bot/tests/unit/preview.test.ts | Adds coverage for previewPackage behavior and input validation. |
| infra/emdash-bot/tests/unit/orchestrator-comments.test.ts | Verifies new copy, staging preview package injection, and failed-stage/run diagnostics. |
| infra/emdash-bot/tests/unit/github-proxy.test.ts | Updates push gating expectations and adds tests for capability vs body parsing and command-ref parsing. |
| infra/emdash-bot/tests/unit/exec-env.test.ts | Updates expectations for pipefail-wrapped commands and adds candidate snapshot parsing/guard tests. |
| infra/emdash-bot/tests/unit/evals-scorer.test.ts | Updates eval harness expected reported payload shape. |
| infra/emdash-bot/tests/unit/evals-client.test.ts | Updates eval client fixture to include run/publication/verification fields. |
| infra/emdash-bot/tests/integration/orchestrator.test.ts | Updates integration expectations for implement lane, preview gating, and failed-comment diagnostics. |
| infra/emdash-bot/tests/unit/verification.test.ts | Adds direct unit coverage for verification-command and record selection/tree binding. |
| infra/emdash-bot/tests/unit/github.test.ts | Adds unit coverage for Git Data API request shapes and ref creation/update semantics. |
| infra/emdash-bot/tests/unit/candidate-publisher.test.ts | Adds unit coverage for candidate publication invariants (stale head, idempotency, tree match, fast-forward parent). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
github-actions
Bot
added
review/awaiting-author
github-actions
Bot
added
review/needs-rereview
There was a problem hiding this comment.
This PR hardens emdashbot's candidate-delivery path in the right direction: moving publication from the sandbox to the trusted Worker, adding fast-forward/idempotency checks, binding verification records to the candidate tree, and giving implement its own result contract.
On re-review, the two issues from the previous pass are resolved:
renderAgentCommentnow usespreviewInstallCommand(anchorNumber, previewPackage)for everyagent.fix_readybranch, so the legacyawaiting_feedbacklane no longer advertises the production package in staging deployments.- Verification commands are now read-only:
ExecEnv.runCheckdetects any working-tree mutation, restores the container to the VFS state, and rejects the command. The skill instructions have been updated to apply formatting viaedit_file/write_fileand run only check-only formatter commands.
I read the full changed modules, the new tests, and traced the state-machine and proxy call sites. The code is well-tested and the approach is sound, but one logic issue remains in the publication idempotency path.
| if (liveBefore) { | ||
| const liveCommit = await github.getCommit(liveBefore); | ||
| if (liveCommit.message.includes(runMarker)) { | ||
| return { branch: input.branch, commitSha: liveBefore, files }; |
There was a problem hiding this comment.
[needs fixing] The idempotency branch returns the live commit as soon as its message contains the run marker, without checking whether that commit's tree matches the current snapshot. If a single run calls publish_candidate more than once after making additional edits, the second snapshot will have a different treeSha than the commit already on the branch, but this branch silently returns the stale commit. The agent then reports implemented: true / fixed: true against a publication that does not contain the verified candidate tree.
| if (liveBefore) { | |
| const liveCommit = await github.getCommit(liveBefore); | |
| if (liveCommit.message.includes(runMarker)) { | |
| return { branch: input.branch, commitSha: liveBefore, files }; | |
| if (liveBefore) { | |
| const liveCommit = await github.getCommit(liveBefore); | |
| if (liveCommit.message.includes(runMarker)) { | |
| if (liveCommit.treeSha !== input.snapshot.treeSha) { | |
| throw new Error( | |
| "candidate branch already has a commit from this run with a different tree", | |
| ); | |
| } | |
| return { branch: input.branch, commitSha: liveBefore, files }; | |
| } | |
| } |
github-actions
Bot
added
review/awaiting-author
What does this PR do?
Hardens emdashbot's automated implementation and candidate-delivery path after its first real implementation attempt exposed weak verification and publication boundaries.
implementa dedicated result contract while retaining the preview, reporter-confirmation, and draft-PR gates.Follow-up hardening for #2383.
Type of change
Checklist
pnpm typecheckpassespnpm lintpassespnpm testpasses (or targeted tests for my change)pnpm formathas been runmessages.pochanges except in translation PRs — a workflow extracts catalogs on merge tomain. (N/A: no admin UI changes.)AI-generated code disclosure
Screenshots / test output
pnpm lintpnpm buildpnpm typecheckgit diff --checkThe live
evals:implementationsmoke was not run because it intentionally creates real comments, labels, a candidate branch, and a draft PR in a configured staging repository. The bot package's standalonetsc --noEmitstill reports its existing generated Flue/Wrangler RPC binding gaps; the repository package typecheck and production bot build pass.Try this PR
Open a fresh playground →
A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.
Tracks
codex/harden-emdash-bot-delivery. Updated automatically when the playground redeploys.