feat(sandbox): migrate sandbox backend from Blaxel to Daytona

- Daytona REST client (fetch, no SDK) in packages/tools-code/daytona-client.ts
- ProjectSandbox DO rewritten on Daytona: run-leases (beginRun/endRun) + keepalive
  alarm, DO-persisted process records, sandbox-disk persistence (autoDelete=-1),
  lifecycle-aware metering, get-or-create by label, runCode via execute
- New apps/preview-proxy worker: custom preview proxy + HMAC token contract
- Webhooks GDPR lifecycle on Daytona; env/secrets + wrangler bindings
- Remove @blaxel/core, BL_* code refs, blaxel.toml, sync-blaxel-local-token
- Dockerfile drops baked sandbox-api; published as Daytona snapshot (user: node)
- Docs: docs/plans/blaxel-to-daytona-migration.md + daytona-rest-reference.md
  + daytona-egress-broker.md; plan.md S9 + CLAUDE.md updated

Typecheck 41/41 and lint 41/41 green.

Author: iamjr15

CLAUDE.md

@@ -12,12 +12,12 @@ Direct competitors: Manus (generalist async agent), HappyCapy (GUI workstation +
 
 | Layer | Choice |
 |---|---|
-| Language | **TypeScript** everywhere. No Python in backend. Python lives only inside the Blaxel sandbox. |
+| Language | **TypeScript** everywhere. No Python in backend. Python lives only inside the Daytona sandbox. |
 | Backend runtime | **Cloudflare Workers + Durable Objects + Workflows** |
 | Frontend | **Next.js 16.2.6 + React 19.2.6 + Tailwind 4.3 + shadcn CLI 4.6 + AI Elements + Streamdown** on Cloudflare Workers via OpenNext |
 | Agent framework | **Mastra 1.35** on top of **Vercel AI SDK v6.0.182** |
-| Sandbox | Blaxel Sandboxes via `@blaxel/core@0.2.84` — one hosted persistent sandbox per project |
-| Browser automation | **Stagehand v3.2 LOCAL mode** inside the Blaxel sandbox image; noVNC for user takeover |
+| Sandbox | **Daytona Sandboxes** via REST-over-fetch (no SDK in Workers; `packages/tools-code/daytona-client.ts`) — one persistent sandbox per project (disk is the durable store) |
+| Browser automation | **Stagehand v3.2 LOCAL mode** inside the Daytona sandbox image; noVNC for user takeover (via the `preview-proxy` worker) |
 | Database | **Supabase Postgres via Cloudflare Hyperdrive** + **Drizzle 0.45.2** (no `service_role` from Workers — uses `app_worker` role) |
 | Auth | **Clerk 7.3.4** (Workers JWT verify) |
 | Billing | **Polar 0.46.4** (no fixed cost, rev-share only) |
@@ -32,7 +32,8 @@ Direct competitors: Manus (generalist async agent), HappyCapy (GUI workstation +
 apps/
   web/                    Next.js 16 (Cloudflare Workers/OpenNext)
   gateway-worker/         Public Hono router + Clerk JWT + rate limit
-  agent-worker/           Agent loop + AgentRun DO + ProjectSandbox DO + Blaxel adapter
+  agent-worker/           Agent loop + AgentRun DO + ProjectSandbox DO + Daytona adapter
+  preview-proxy/          Custom preview proxy (preview.trycheatcode.com) in front of Daytona previews
   webhooks-worker/        Clerk, Polar, Composio webhooks + internal ops workflows
 
 packages/
@@ -56,7 +57,7 @@ packages/
   biome-config/           Shared biome.jsonc
 
 skills/                   8 curated Anthropic SKILL.md skills
-infra/                    Wrangler configs, Supabase migrations, Blaxel sandbox Dockerfile
+infra/                    Wrangler configs, Supabase migrations, Daytona sandbox Dockerfile/snapshot
 scripts/                  Operational helpers only: build skills, secrets, deploy orchestration, migrations, audit archive
 ```
 
@@ -159,15 +160,15 @@ Full details in plan.md Section 12.
 
 Use Mastra Workflows for orchestration. Each subagent is a `ToolLoopAgent` with explicit `stopWhen: stepCountIs(N)` budget caps. The `deep-research-fanout` workflow shows the canonical fanout pattern (see plan.md Section 8.5).
 
-When using subagents for complex tasks, give each their own Blaxel-backed ProjectSandbox if they need isolated filesystem state. Otherwise share the parent's sandbox.
+When using subagents for complex tasks, give each their own Daytona-backed ProjectSandbox if they need isolated filesystem state. Otherwise share the parent's sandbox.
 
 ## Plan.md is source of truth
 
 If you're proposing a change that contradicts `plan.md`, **update plan.md first** in the same PR. Architectural drift between plan and code is the #1 cause of bit-rot.
 
 ## When stuck
 
-- Sandbox not working? `apps/agent-worker/src/durable-objects/project-sandbox.ts` — check Blaxel auth, sandbox name, image, region, and process/file normalization.
+- Sandbox not working? `apps/agent-worker/src/durable-objects/project-sandbox.ts` + `packages/tools-code/src/daytona-client.ts` — check Daytona auth (`DAYTONA_API_KEY`), sandbox name/snapshot, `DAYTONA_TARGET` region, and the toolbox/session paths. Reference: `docs/plans/daytona-rest-reference.md`.
 - Auth broken? `packages/auth/` — Clerk JWT verify pattern with `@clerk/backend`.
 - Skill not triggering? Inspect the bundled skill `description` first — it is
   the activation field, not the body. Use manual fixture review plus final UI

apps/agent-worker/package.json

@@ -21,7 +21,6 @@
     "@cheatcode/observability": "workspace:*",
     "@cheatcode/tools-code": "workspace:*",
     "@cheatcode/types": "workspace:*",
-    "@blaxel/core": "catalog:",
     "ai": "catalog:",
     "hono": "catalog:",
     "zod": "catalog:"

apps/agent-worker/src/durable-objects/mastra-stream-chunks.ts

@@ -92,8 +92,8 @@ export function usageFromMastraChunk(chunk: unknown): MastraUsageDelta | null {
   }
 
   const usage = usageRecord(record);
-  const tokensIn = tokenCount(usage, ["inputTokens", "promptTokens"]);
-  const tokensOut = tokenCount(usage, ["outputTokens", "completionTokens"]);
+  const tokensIn = tokenCount(usage, ["inputTokens", "promptTokens", "input_tokens"]);
+  const tokensOut = tokenCount(usage, ["outputTokens", "completionTokens", "output_tokens"]);
   if (tokensIn === 0 && tokensOut === 0) {
     return null;
   }
@@ -167,12 +167,26 @@ export function normalizeMastraStreamError(error: unknown): Error {
 
 function usageRecord(record: Record<string, unknown>): Record<string, unknown> {
   const payload = asRecord(record["payload"]);
-  for (const candidate of [
+  const metadata = asRecord(payload["metadata"]);
+  const candidates: Record<string, unknown>[] = [
     asRecord(record["usage"]),
     asRecord(record["totalUsage"]),
     asRecord(payload["usage"]),
     asRecord(payload["totalUsage"]),
+  ];
+  // Mastra 1.x finish chunks nest the raw provider usage under
+  // payload.metadata.providerMetadata.<provider>.usage (snake_case token fields),
+  // with no normalized top-level usage — so dig through every provider entry.
+  for (const providerMetadata of [
+    asRecord(record["providerMetadata"]),
+    asRecord(payload["providerMetadata"]),
+    asRecord(metadata["providerMetadata"]),
   ]) {
+    for (const providerValue of Object.values(providerMetadata)) {
+      candidates.push(asRecord(asRecord(providerValue)["usage"]));
+    }
+  }
+  for (const candidate of candidates) {
     if (Object.keys(candidate).length > 0) {
       return candidate;
     }

apps/agent-worker/src/durable-objects/project-sandbox-files.ts

@@ -1,145 +1,84 @@
-import type { SandboxInstance } from "@blaxel/core";
-import type { SandboxListFilesResult } from "@cheatcode/tools-code";
-import { z } from "zod";
+import type { DaytonaClient, DaytonaFileInfo, SandboxListFilesResult } from "@cheatcode/tools-code";
 
 const MAX_LIST_FILE_ENTRIES = 1_000;
 
-const FileListSchema = z
-  .object({
-    files: z.array(z.unknown()).default([]),
-    subdirectories: z.array(z.unknown()).default([]),
-  })
-  .passthrough();
-
-interface FileEntryCandidate {
-  lastModified?: string | undefined;
-  modifiedAt?: string | undefined;
-  name?: string | undefined;
-  path?: string | undefined;
-  size?: number | undefined;
-  type?: string | undefined;
-}
+type FileEntry = SandboxListFilesResult["files"][number];
 
+/**
+ * Recursive workspace listing over Daytona's single-level `listFiles` (the
+ * toolbox `GET /files` returns one level only). Walks `isDir` entries depth-first
+ * up to MAX_LIST_FILE_ENTRIES and maps Daytona FileInfo → the SandboxFileEntry
+ * contract callers already depend on.
+ */
 export async function listSandboxFiles({
+  client,
+  sandboxId,
   includeHidden,
   path,
   recursive,
-  sandbox,
 }: {
+  client: DaytonaClient;
+  sandboxId: string;
   includeHidden: boolean;
   path: string;
   recursive: boolean;
-  sandbox: SandboxInstance;
-}): Promise<SandboxListFilesResult["files"]> {
-  const entries = await listDirectory({
-    includeHidden,
-    parentPath: path,
-    recursive,
-    rootPath: path,
-    sandbox,
-  });
-  return entries.slice(0, MAX_LIST_FILE_ENTRIES);
+}): Promise<FileEntry[]> {
+  const root = stripTrailingSlash(path);
+  const out: FileEntry[] = [];
+  await walk(client, sandboxId, root, root, includeHidden, recursive, out);
+  return out.slice(0, MAX_LIST_FILE_ENTRIES);
 }
 
-async function listDirectory({
-  includeHidden,
-  parentPath,
-  recursive,
-  rootPath,
-  sandbox,
-}: {
-  includeHidden: boolean;
-  parentPath: string;
-  recursive: boolean;
-  rootPath: string;
-  sandbox: SandboxInstance;
-}): Promise<SandboxListFilesResult["files"]> {
-  const listing = FileListSchema.parse(await sandbox.fs.ls(parentPath));
-  const directories = listing.subdirectories
-    .map((entry) => toFileEntry(entry, parentPath, "directory", rootPath))
-    .filter((entry) => shouldIncludeEntry(entry, includeHidden));
-  const files = listing.files
-    .map((entry) => toFileEntry(entry, parentPath, "file", rootPath))
-    .filter((entry) => shouldIncludeEntry(entry, includeHidden));
+async function walk(
+  client: DaytonaClient,
+  sandboxId: string,
+  dir: string,
+  root: string,
+  includeHidden: boolean,
+  recursive: boolean,
+  out: FileEntry[],
+): Promise<void> {
+  if (out.length >= MAX_LIST_FILE_ENTRIES) {
+    return;
+  }
+  const infos = await client.listFiles(sandboxId, dir);
+  const entries = infos
+    .map((info) => toFileEntry(info, dir, root))
+    .filter((entry) => shouldInclude(entry, includeHidden));
+  for (const entry of entries) {
+    out.push(entry);
+  }
   if (!recursive) {
-    return [...directories, ...files];
+    return;
   }
-
-  const descendants: SandboxListFilesResult["files"] = [];
-  for (const directory of directories) {
-    if (descendants.length + directories.length + files.length >= MAX_LIST_FILE_ENTRIES) {
-      break;
+  for (const entry of entries) {
+    if (entry.type === "directory" && out.length < MAX_LIST_FILE_ENTRIES) {
+      await walk(client, sandboxId, entry.path, root, includeHidden, recursive, out);
     }
-    descendants.push(
-      ...(await listDirectory({
-        includeHidden,
-        parentPath: directory.path,
-        recursive,
-        rootPath,
-        sandbox,
-      })),
-    );
   }
-  return [...directories, ...files, ...descendants].slice(0, MAX_LIST_FILE_ENTRIES);
 }
 
-function toFileEntry(
-  value: unknown,
-  parentPath: string,
-  fallbackType: "file" | "directory",
-  rootPath: string,
-) {
-  const candidate: FileEntryCandidate = z
-    .object({
-      lastModified: z.string().optional(),
-      modifiedAt: z.string().optional(),
-      name: z.string().optional(),
-      path: z.string().optional(),
-      size: z.number().int().nonnegative().optional(),
-      type: z.string().optional(),
-    })
-    .passthrough()
-    .parse(value);
-  const name = candidate.name ?? basename(candidate.path ?? "");
-  const path = candidate.path ?? `${parentPath.replace(/\/$/, "")}/${name}`;
-  const relativePath = relativeSandboxPath(rootPath, path);
+function toFileEntry(info: DaytonaFileInfo, parentDir: string, root: string): FileEntry {
+  const path = `${stripTrailingSlash(parentDir)}/${info.name}`;
   return {
-    modifiedAt: candidate.modifiedAt ?? candidate.lastModified ?? new Date(0).toISOString(),
-    name,
+    name: info.name,
     path,
-    relativePath,
-    size: candidate.size ?? 0,
-    type: normalizeFileType(candidate.type, fallbackType),
+    relativePath: relativePath(root, path),
+    type: info.isDir ? "directory" : "file",
+    size: info.size,
+    modifiedAt: info.modifiedAt ?? info.modTime ?? new Date(0).toISOString(),
   };
 }
 
-function normalizeFileType(
-  value: string | undefined,
-  fallbackType: "file" | "directory",
-): "file" | "directory" | "symlink" | "other" {
-  if (value === "file" || value === "directory" || value === "symlink" || value === "other") {
-    return value;
-  }
-  return fallbackType;
-}
-
-function basename(path: string): string {
-  const index = path.lastIndexOf("/");
-  return index === -1 ? path : path.slice(index + 1);
-}
-
-function shouldIncludeEntry(
-  entry: { name: string; relativePath: string },
-  includeHidden: boolean,
-): boolean {
+function shouldInclude(entry: { relativePath: string }, includeHidden: boolean): boolean {
   if (includeHidden) {
     return true;
   }
   return !entry.relativePath.split("/").some((part) => part.startsWith("."));
 }
 
-function relativeSandboxPath(rootPath: string, path: string): string {
-  const normalizedRoot = rootPath.replace(/\/$/, "");
+function relativePath(root: string, path: string): string {
+  const normalizedRoot = stripTrailingSlash(root);
   if (path === normalizedRoot) {
     return basename(path);
   }
@@ -148,3 +87,12 @@ function relativeSandboxPath(rootPath: string, path: string): string {
   }
   return basename(path);
 }
+
+function basename(path: string): string {
+  const index = path.lastIndexOf("/");
+  return index === -1 ? path : path.slice(index + 1);
+}
+
+function stripTrailingSlash(value: string): string {
+  return value.length > 1 && value.endsWith("/") ? value.slice(0, -1) : value;
+}

apps/agent-worker/src/durable-objects/project-sandbox-metering.ts

@@ -19,14 +19,18 @@ export interface SandboxMeteringContext {
   storage: DurableObjectStorage;
 }
 
-/** Seed the metering checkpoint on first owner registration (idempotent). */
+/**
+ * Seed the metering checkpoint when a run lease opens (idempotent). Lifecycle-aware
+ * (Codex R1): the checkpoint exists ONLY while ≥1 run lease is active, so we accrue
+ * running-hours during agent runs and never bill idle/stopped time.
+ */
 export async function initSandboxMeterCheckpoint(storage: DurableObjectStorage): Promise<void> {
   if ((await meterCheckpointMs(storage)) === null) {
     await storage.put(SANDBOX_METER_CHECKPOINT_KEY, Date.now());
   }
 }
 
-/** Drop the metering checkpoint when the sandbox is destroyed. */
+/** Drop the metering checkpoint when the last run lease closes / sandbox destroyed. */
 export async function clearSandboxMeterCheckpoint(storage: DurableObjectStorage): Promise<void> {
   await storage.delete(SANDBOX_METER_CHECKPOINT_KEY);
 }
@@ -63,7 +67,7 @@ async function recordSandboxUsage(ctx: SandboxMeteringContext): Promise<void> {
   const previousCheckpointMs = await meterCheckpointMs(ctx.storage);
   const now = Date.now();
   if (previousCheckpointMs === null) {
-    await ctx.storage.put(SANDBOX_METER_CHECKPOINT_KEY, now);
+    // No active run lease → not running on our behalf → do not accrue (lifecycle-aware).
     return;
   }
   const hours = (now - previousCheckpointMs) / MILLIS_PER_HOUR;