fix: make agent runs durable across worker restarts (#176)

## Summary

- Move the semantic agent loop out of a Worker-isolate promise and into
one deterministic Cloudflare Workflow per run.
- Checkpoint every model turn, tool invocation, transcript publication,
terminal transition, and cleanup so isolate eviction or deployment
resumes completed work.
- Reacquire provider credentials and sandbox capabilities inside each
active step, keeping plaintext secrets out of durable state.
- Reconcile the web client with the persisted terminal transcript when a
browser misses the end of a stream.

## What's Included

### Durable execution

- Add a model-only Mastra turn that exposes tool schemas without
executing them.
- Reconstruct each selected tool through Mastra's cross-process
execution API in a separate Workflow step.
- Replace lease epochs and the volatile stream driver with validated
JSON Workflow state.

### Idempotent publication and lifecycle

- Give Workflow events deterministic keys and atomically receipt
transcript fragments in Durable Object SQLite.
- Recover ambiguous admission, reconcile retained Workflow status, and
convert exhausted or externally terminated runs into explicit retryable
failures.
- Terminate the owning Workflow before cancellation or deletion and keep
cleanup independently retryable.

### Client convergence

- Poll the lightweight thread record only while its run is active and
the tab is visible.
- Stop a stale stream and refresh persisted messages and chat lists when
the authoritative run pointer clears.

## Architecture

```mermaid
sequenceDiagram
  participant G as Gateway
  participant D as AgentRun Durable Object
  participant W as Cloudflare Workflow
  participant M as Mastra
  participant S as Daytona / external tools

  G->>D: admit immutable run payload
  D->>W: create deterministic run instance
  W->>M: checkpoint one model-only turn
  M-->>W: text and tool calls
  W->>D: publish turn with idempotency key
  loop each tool call
    W->>M: reconstruct registered tool
    M->>S: execute with step-scoped credentials
    S-->>M: serializable result
    W->>D: publish result with idempotency key
  end
  W->>D: commit terminal status and transcript
  W->>S: retryable cleanup
```

## Decisions Made

| Decision | Choice | Alternatives considered | Reasoning |
|---|---|---|---|
| Execution owner | Cloudflare Workflow | In-isolate promise; Mastra
in-process durable agent | The existing promise disappears with isolate
eviction. Mastra's built-in durable agent is documented for
in-process/simple deployments and defaults to a hard step ceiling, while
Cloudflare already owns durable execution here. |
| Tool boundary | Model-only turn plus one Workflow step per tool |
Stream the whole Mastra loop in one step | Completed tool calls become
durable checkpoints and tools are reconstructed through Mastra's
intended cross-process API. |
| Transcript delivery | Deterministic event keys plus atomic SQLite
receipts | Best-effort append | A retried publication cannot duplicate
visible assistant or tool parts. |
| Failed retained Workflow | Visible retryable terminal failure |
Restart from the beginning | Blind restart can repeat external side
effects. |
| Secrets | Resolve inside each step | Serialize request context |
Plaintext BYOK credentials and capabilities never enter Workflow
storage. |

## Edge Cases Handled

| Scenario | Handling |
|---|---|
| Worker or Durable Object eviction | Workflow replays completed steps
and continues from the next checkpoint. |
| Ambiguous Workflow creation response | Alarm recovery reuses the
deterministic instance and validates the immutable input hash. |
| Repeated Workflow callback | Durable Object verifies instance
identity; transcript receipts suppress duplicates. |
| Transient tool failure | Tool error is returned to the model so it can
recover semantically. |
| Workflow API temporarily unavailable | Reconciliation is re-armed
without prematurely failing the run. |
| Workflow reports unknown | Three spaced observations are required
before declaring interruption. |
| User cancellation or deletion | Active queued/running/waiting/paused
Workflow is terminated before terminal state is committed. |
| Browser misses terminal stream chunks | Active-run polling stops the
stale stream and replaces transient state with the persisted transcript.
|

## How to Review

1. Start with `agent-run-workflow.ts` and
`agent-run-workflow-runtime.ts` for the checkpointed loop.
2. Review `durable-agent-step.ts` for the Mastra model/tool separation.
3. Review `agent-run-workflow-controller.ts` and `agent-run-output.ts`
for admission, reconciliation, and idempotency.
4. Finish with the web lifecycle hooks that converge a disconnected
client.

## Verification

- [x] `pnpm lint`
- [x] `pnpm typecheck`
- [x] `pnpm turbo build --force`
- [x] `pnpm deadcode`
- [x] `pnpm architecture:check`
- [x] `pnpm turbo skills:build`
- [x] Full documented Docker stack starts under Node 24.18.0
- [x] Gateway liveness returns HTTP 200
- [x] Web root returns HTTP 200 after cold compilation
- [x] Startup logs contain no runtime errors

Production browser acceptance will run after deployment so it exercises
the exact merged Worker and web release.

Author: iamjr15

apps/agent-worker/README.md

@@ -24,8 +24,8 @@ exact object before failing. A committed replay under a still-active run atomica
 retention before the output can be exposed through a fresh download capability. A terminal replay
 can acknowledge only the same unexpired output and never renews it; every committed replay verifies
 the exact R2 object again before returning. Terminal run
-persistence records upload quiescence only after its execution promise has settled, while deletion
-RPCs abort and join the same promise before returning.
+persistence records upload quiescence only after the Workflow-owned tool steps have settled, while
+deletion RPCs terminate the run's Workflow before removing its durable state.
 
 Artifact messages persist only the output UUID and presentation metadata. The authenticated
 `POST /v1/outputs/:outputId/download-url` path rechecks tenant ownership, retention, and R2
@@ -74,25 +74,25 @@ and clears the matching thread pointer in one transaction. Transport or reconnec
 leaves the pointer intact for the next idempotent replay. Active-run conflicts use the same
 reconciliation path instead of blindly returning a conflict.
 
-Each admitted semantic run has a deterministic chain of Cloudflare Workflow generations. A
-generation keeps the Durable Object execution request attached in four-minute ownership epochs,
-then checkpoints and renews the same in-memory coroutine. At 20,000 epochs (about 55 days), it
-atomically reserves `generation + 1` in the run-keyed Durable Object and creates the deterministic
-successor, leaving almost 5,000 steps below Cloudflare's configured 25,000-step platform ceiling.
-The successor's first execution atomically promotes its exact generation, input hash, and instance
-ID; any late callback from the predecessor becomes a no-op. A pending successor is recovered by
-the existing admission alarm if creation was ambiguous. `draining` permits that continuation,
-while `closed` fences it. Generations are an operational rollover mechanism, not a run-duration,
-step, token, or cost limit.
-
-A persisted execution-start fence makes retries at-most-once: a warm retry joins the exact
-promise, while a restart that lost that promise terminalizes the run instead of replaying model
-calls, tools, or other non-idempotent external side effects. An alarm-backed lease also
-terminalizes an admission or execution whose current Workflow owner stops renewing it.
-Before a Worker release, the closed gateway plus draining agent gate requires every retained
-`cheatcode-agent-runs` instance to be complete. Errored and terminated instances
-remain restartable on their pinned Worker version, so they block deployment until
-the exact retained instance has expired or been purged.
+Each admitted semantic run has one deterministic Cloudflare Workflow instance. The Workflow owns
+the agent loop and checkpoints preparation, every model turn, every tool invocation, transcript
+publication, completion, and cleanup as separate steps. Its state contains only validated JSON;
+provider keys and sandbox capabilities are reacquired inside the active step and never enter
+Workflow storage. A Worker isolate or Durable Object eviction therefore resumes from the last
+completed step instead of losing an in-memory coroutine. Transcript publication uses deterministic
+event keys and an atomic SQLite receipt, so Workflow step replay cannot duplicate visible parts.
+There is no application step, token, duration, or cost ceiling; semantic completion ends the loop,
+while per-operation timeouts and the platform Workflow limit remain operational safeguards.
+
+The run-keyed Durable Object is the authoritative status, cancellation, transcript, and stream
+store. It validates every Workflow callback against the stored input hash and deterministic
+instance ID, and late callbacks become terminal no-ops. Admission ambiguity is recovered by the
+existing alarm. Cancellation terminates the Workflow before committing terminal state. Before a
+Worker release, the closed gateway plus draining agent gate requires every retained
+`cheatcode-agent-runs` instance to be complete. Workflow retries resume individual model and tool
+steps from their durable checkpoints. An exhausted or externally terminated instance is never
+blindly restarted from the beginning because doing so could repeat an external tool side effect;
+the run object reconciles that terminal mismatch into a visible, retryable failure instead.
 
 Normal chat runs resolve provider credentials from Supabase Vault through `packages/byok`,
 pass only the request-scoped transport credential to Mastra, and execute tools against the
@@ -110,13 +110,12 @@ record with the canonical UI-message schema, and converts it with AI SDK
 `convertToModelMessages`. The current run's user message must be last and carry that run ID.
 Ephemeral app-builder context is appended only to that current model turn and is never stored.
 
-`AgentRun` is the Durable Object coordination shell rather than the implementation home for
-every concern. Its HTTP adapter owns bounded request parsing and route dispatch; the run
-lifecycle module owns progress, terminal persistence, and sandbox-lease cleanup; the run-path
-module selects general or app-builder execution; and the output component owns replay and
-answer segmentation. The Workflow controller owns admission, execution identity, the
-at-most-once fence, and ownership leases. The shell retains only Durable Object identity,
-cancellation, status, and dependency wiring.
+`AgentRun` is the Durable Object coordination shell rather than the implementation home for every
+concern. Its HTTP adapter owns bounded request parsing and route dispatch; the Workflow runtime
+owns model/tool step preparation and sandbox-lease cleanup; the app-builder path owns scaffold and
+preview setup; and the output component owns idempotent transcript publication plus resumable
+streams. The Workflow controller owns admission, callback identity, and cancellation. The shell
+retains only durable run identity, status, transcript, cancellation, and dependency wiring.
 
 An explicit app-builder mode remains authoritative. In a projectless chat, a narrowly
 matched imperative such as “build a website” or “create a mobile app” also enters the matching
@@ -240,9 +239,8 @@ as every segment's logical timestamp. PostgreSQL publishes a run only when its u
 segment exists; retries compare each segment's JSONB, final marker, timestamp, and tenant
 identity. Oversized structured parts use lossless bounded fragment envelopes rather than
 truncation, and there is no transcript-length, step, token, or cost ceiling.
-Mastra tool-call chunks also emit `step_started`, `step_completed`,
-`tool_invoked`, and `skill_invoked` events when those chunks are present in the
-live stream. If the last stream subscriber disconnects while a run is still
+Checkpointed tool steps emit `step_started`, `step_completed`, `tool_invoked`, and
+`skill_invoked` events independently of the live stream. If the last stream subscriber disconnects while a run is still
 running, AgentRun emits `run_abandoned` for the funnel trail.
 
 Project deletion first fences project/thread mutations, refuses an active run, records a

apps/agent-worker/src/durable-objects/abort-timeout.ts

@@ -4,7 +4,7 @@ import { pendingStatusRetryAt } from "./agent-run-status-persistence";
 import { getRunStateTimestamp, getRunStateValue } from "./agent-run-storage";
 import {
   AGENT_RUN_WORKFLOW_ADMITTED_KEY,
-  AGENT_RUN_WORKFLOW_LEASE_EXPIRES_AT_KEY,
+  AGENT_RUN_WORKFLOW_RECONCILE_AT_KEY,
   AGENT_RUN_WORKFLOW_RETRY_AT_KEY,
 } from "./agent-run-workflow-protocol";
 import { hasActiveRun } from "./run-state";
@@ -16,14 +16,14 @@ export async function armAgentRunAlarm(ctx: DurableObjectState): Promise<void> {
     return;
   }
   const isRunActive = hasActiveRun(getRunStateValue(ctx, "status"));
-  const executionLeaseAlarm = isRunActive
-    ? (getRunStateTimestamp(ctx, AGENT_RUN_WORKFLOW_LEASE_EXPIRES_AT_KEY) ??
-      Number.POSITIVE_INFINITY)
-    : Number.POSITIVE_INFINITY;
   const admissionRetryAlarm =
     isRunActive && getRunStateValue(ctx, AGENT_RUN_WORKFLOW_ADMITTED_KEY) !== "true"
       ? (getRunStateTimestamp(ctx, AGENT_RUN_WORKFLOW_RETRY_AT_KEY) ?? Number.POSITIVE_INFINITY)
       : Number.POSITIVE_INFINITY;
+  const workflowReconcileAlarm =
+    isRunActive && getRunStateValue(ctx, AGENT_RUN_WORKFLOW_ADMITTED_KEY) === "true"
+      ? (getRunStateTimestamp(ctx, AGENT_RUN_WORKFLOW_RECONCILE_AT_KEY) ?? Date.now())
+      : Number.POSITIVE_INFINITY;
   const assistantMessageRetryAlarm = pendingAssistantMessageRetryAt(ctx);
   const statusRetryAlarm =
     assistantMessageRetryAlarm === Number.POSITIVE_INFINITY
@@ -32,7 +32,7 @@ export async function armAgentRunAlarm(ctx: DurableObjectState): Promise<void> {
   await ctx.storage.setAlarm(
     Math.min(
       admissionRetryAlarm,
-      executionLeaseAlarm,
+      workflowReconcileAlarm,
       assistantMessageRetryAlarm,
       statusRetryAlarm,
       nextAgentRunAlarm(Date.now()),

apps/agent-worker/src/durable-objects/agent-run-alarm.ts

@@ -145,7 +145,7 @@ interface RunAppBuilderOptions {
   // Harness workspace setup runs before the model streams; its progress is status
   // chrome (run stage / Computer panel), never visible answer prose. Anything the
   // model needs to know is handed to it as agentContextNote.
-  setRunStage: (stage: string) => void;
+  setRunStage: (stage: string) => Promise<void>;
 }
 
 interface AppBuilderSetup {
@@ -219,10 +219,10 @@ async function prepareTemplateWorkspace(
 ): Promise<void> {
   const { input, logger, sandbox, setRunStage, shouldBootstrap, workspace } = options;
   const mobile = workspace.mobile;
-  setRunStage(mobile ? "Preparing the Expo workspace." : "Preparing the Next.js workspace.");
+  await setRunStage(mobile ? "Preparing the Expo workspace." : "Preparing the Next.js workspace.");
   await ensureAppBuilderRuntime(sandbox, mobile);
   if (!shouldBootstrap) {
-    setRunStage("Restoring the app workspace.");
+    await setRunStage("Restoring the app workspace.");
     if (!(await hasInstalledAppBuilderDependencies(sandbox, workspace))) {
       await installAppBuilderDependencies(sandbox, logger, workspace.dir, mobile);
     }
@@ -243,14 +243,14 @@ async function prepareTemplateWorkspace(
   if (mobile) {
     await ensureExpoWebSupport(sandbox, workspace.dir);
   } else {
-    setRunStage("Seeding the starter files.");
+    await setRunStage("Seeding the starter files.");
     await writeAppBuilderFiles(input, sandbox, workspace.dir);
   }
 }
 
 async function startTemplatePreview(options: WorkspaceOptions): Promise<void> {
   const { append, logger, sandbox, setRunStage, workspace } = options;
-  setRunStage("Starting the dev server.");
+  await setRunStage("Starting the dev server.");
   if (workspace.mobile) {
     await startExpoDevServer(sandbox, logger, workspace);
   } else {
@@ -274,7 +274,7 @@ export async function restartMobilePreview(
   options: Pick<RunAppBuilderOptions, "append" | "input" | "logger" | "sandbox" | "setRunStage">,
 ): Promise<void> {
   const { append, input, logger, sandbox, setRunStage } = options;
-  setRunStage("Reloading the preview.");
+  await setRunStage("Reloading the preview.");
   const workspace = await resolveAppWorkspace(sandbox, input, logger);
   await startExpoDevServer(sandbox, logger, workspace);
   await append({
@@ -299,7 +299,7 @@ async function importRepoWorkspace(
     throw repoImportError("The import URL must be a public https github.com repository.");
   }
   logger.info("repo_import_started", { repoHost: repoRef.host, repoPath: repoRef.path });
-  setRunStage(`Cloning ${repoRef.path}.`);
+  await setRunStage(`Cloning ${repoRef.path}.`);
   await resetTemplateAppBuilderDirectory(sandbox, workspace.dir);
   throwIfRunCanceled(options.abortSignal);
   const cloneDir = `${workspace.dir}/.cheatcode-import-${input.runId ?? crypto.randomUUID()}`;
@@ -350,7 +350,7 @@ async function restoreImportedWorkspace(
   options: WorkspaceOptions,
 ): Promise<{ agentContextNote: string }> {
   const { append, input, logger, sandbox, setRunStage, workspace } = options;
-  setRunStage("Restoring the imported workspace.");
+  await setRunStage("Restoring the imported workspace.");
   await installImportedDependencies(sandbox, logger, workspace.dir);
   throwIfRunCanceled(options.abortSignal);
   await clearBuildCache(sandbox, workspace.dir, workspace.mobile);