fix: make agent runs durable across worker restarts

[iamjr15]

Summary

• Move the semantic agent loop out of a Worker-isolate promise and into one deterministic Cloudflare Workflow per run.
• Checkpoint every model turn, tool invocation, transcript publication, terminal transition, and cleanup so isolate eviction or deployment resumes completed work.
• Reacquire provider credentials and sandbox capabilities inside each active step, keeping plaintext secrets out of durable state.
• Reconcile the web client with the persisted terminal transcript when a browser misses the end of a stream.

What's Included

Durable execution

• Add a model-only Mastra turn that exposes tool schemas without executing them.
• Reconstruct each selected tool through Mastra's cross-process execution API in a separate Workflow step.
• Replace lease epochs and the volatile stream driver with validated JSON Workflow state.

Idempotent publication and lifecycle

• Give Workflow events deterministic keys and atomically receipt transcript fragments in Durable Object SQLite.
• Recover ambiguous admission, reconcile retained Workflow status, and convert exhausted or externally terminated runs into explicit retryable failures.
• Terminate the owning Workflow before cancellation or deletion and keep cleanup independently retryable.

Client convergence

• Poll the lightweight thread record only while its run is active and the tab is visible.
• Stop a stale stream and refresh persisted messages and chat lists when the authoritative run pointer clears.

Architecture

sequenceDiagram
  participant G as Gateway
  participant D as AgentRun Durable Object
  participant W as Cloudflare Workflow
  participant M as Mastra
  participant S as Daytona / external tools

  G->>D: admit immutable run payload
  D->>W: create deterministic run instance
  W->>M: checkpoint one model-only turn
  M-->>W: text and tool calls
  W->>D: publish turn with idempotency key
  loop each tool call
    W->>M: reconstruct registered tool
    M->>S: execute with step-scoped credentials
    S-->>M: serializable result
    W->>D: publish result with idempotency key
  end
  W->>D: commit terminal status and transcript
  W->>S: retryable cleanup

Loading

Decisions Made

Decision	Choice	Alternatives considered	Reasoning
Execution owner	Cloudflare Workflow	In-isolate promise; Mastra in-process durable agent	The existing promise disappears with isolate eviction. Mastra's built-in durable agent is documented for in-process/simple deployments and defaults to a hard step ceiling, while Cloudflare already owns durable execution here.
Tool boundary	Model-only turn plus one Workflow step per tool	Stream the whole Mastra loop in one step	Completed tool calls become durable checkpoints and tools are reconstructed through Mastra's intended cross-process API.
Transcript delivery	Deterministic event keys plus atomic SQLite receipts	Best-effort append	A retried publication cannot duplicate visible assistant or tool parts.
Failed retained Workflow	Visible retryable terminal failure	Restart from the beginning	Blind restart can repeat external side effects.
Secrets	Resolve inside each step	Serialize request context	Plaintext BYOK credentials and capabilities never enter Workflow storage.

Edge Cases Handled

Scenario	Handling
Worker or Durable Object eviction	Workflow replays completed steps and continues from the next checkpoint.
Ambiguous Workflow creation response	Alarm recovery reuses the deterministic instance and validates the immutable input hash.
Repeated Workflow callback	Durable Object verifies instance identity; transcript receipts suppress duplicates.
Transient tool failure	Tool error is returned to the model so it can recover semantically.
Workflow API temporarily unavailable	Reconciliation is re-armed without prematurely failing the run.
Workflow reports unknown	Three spaced observations are required before declaring interruption.
User cancellation or deletion	Active queued/running/waiting/paused Workflow is terminated before terminal state is committed.
Browser misses terminal stream chunks	Active-run polling stops the stale stream and replaces transient state with the persisted transcript.

How to Review

1. Start with agent-run-workflow.ts and agent-run-workflow-runtime.ts for the checkpointed loop.
2. Review durable-agent-step.ts for the Mastra model/tool separation.
3. Review agent-run-workflow-controller.ts and agent-run-output.ts for admission, reconciliation, and idempotency.
4. Finish with the web lifecycle hooks that converge a disconnected client.

Verification

• pnpm lint
• pnpm typecheck
• pnpm turbo build --force
• pnpm deadcode
• pnpm architecture:check
• pnpm turbo skills:build
• Full documented Docker stack starts under Node 24.18.0
• Gateway liveness returns HTTP 200
• Web root returns HTTP 200 after cold compilation
• Startup logs contain no runtime errors

Production browser acceptance will run after deployment so it exercises the exact merged Worker and web release.