fix(agent): bound deep research subrequests (#157)

## Why

Production deep-research runs could exhaust Cloudflare's per-invocation
subrequest budget. Each query launched a nested tool-using agent loop,
invalid structured output retried the workflow step three times, and the
outer agent could call the workflow again. The run then failed before it
could package the expected PDF deliverable.

## What changed

- collect a bounded provider evidence pack per query with one Exa search
and at most one best-effort Firecrawl extraction
- structure each evidence pack with one tool-free model call
- cap query concurrency at three and disable workflow-step retries
- turn invalid nested structured output into a stable upstream error
instead of leaking a Zod failure
- instruct the agent and deep-research skill to call the selected
research workflow once per request
- document the bounded provenance and retry boundary

The existing validated citation, PDF packaging, Daytona project-file
persistence, and R2 generated-output paths remain unchanged.

## Architecture and migration effects

- no database or migration changes
- no environment changes
- no new provider or dependency
- moves provider network I/O out of nested agent tool loops and into the
existing bounded research adapters

## Verification

- `pnpm lint`
- `pnpm typecheck`
- `pnpm turbo build --force`
- `pnpm deadcode`
- `pnpm architecture:check`
- `pnpm turbo skills:build`

Production acceptance will be run against the exact merged SHA after
Cloudflare deployment, covering natural-language research intent, one
workflow invocation, PDF delivery, project file persistence, download
validity, reload persistence, and `/` file recall.

Author: iamjr15

packages/agent-core/README.md

@@ -66,13 +66,16 @@ Nested research workflows bind the calling tool's abort signal idempotently to
 the Mastra workflow run, forward each workflow step signal through every nested
 `agent.generate`, and remove abort listeners before deleting the ephemeral run.
 Each concurrent research pass gets an isolated evidence collector populated only
-from parsed Exa result IDs/URLs and Firecrawl result URLs. Claim citations and the
-final synthesis are schema-validated against that evidence; prose URL scraping
-is not an accepted provenance boundary. Successful top-level deep-research and
-fan-out tools deterministically package that validated report, its claim-to-source
-map, and its source list as a PDF artifact. The project workspace is resolved only
-after remote research succeeds; the PDF is then stored both in the live project
-files and the durable generated-output store.
+from one bounded Exa discovery call and an optional Firecrawl extraction of its
+primary result. A single tool-free model pass structures each provider evidence
+pack, and workflow steps do not retry failed model output inside the same Worker
+invocation. Claim citations and the final synthesis are schema-validated against
+that evidence; prose URL scraping is not an accepted provenance boundary.
+Successful top-level deep-research and fan-out tools deterministically package
+that validated report, its claim-to-source map, and its source list as a PDF
+artifact. The project workspace is resolved only after remote research succeeds;
+the PDF is then stored both in the live project files and the durable
+generated-output store.
 
 Composio REST tool discovery and execution responses are byte-bounded before
 parsing, then projected into bounded, valid JSON before entering model context.

packages/agent-core/src/mastra/system-prompt.ts

@@ -232,7 +232,7 @@ Load the generate-media skill before creating or editing an image or generating
 
 const RESEARCH_MODULE = `## Research
 
-Gather sources with search_web / search_web_advanced / search_company / search_web_content, then use search_scrape or search_extract for source retrieval. For an explicit deep-research request, cited report, market analysis, due diligence, or comprehensive investigation of one topic, use research_deep with 3 queries for a concise/narrow report, 4 by default, and 5-6 only when the user explicitly asks for deeper coverage. Use research_fanout when the request compares many entities or independent angles. Those workflows produce the complete cited PDF deliverable automatically, so do not regenerate it with a document tool. Treat search snippets as leads, not sources — open the real pages and cross-check. Cite as you go: attribute each claim to its source inline with the page title and its URL, and make sure every citation resolves. End a research answer with a short Sources list of the URLs you actually used.`;
+Gather sources with search_web / search_web_advanced / search_company / search_web_content, then use search_scrape or search_extract for source retrieval. For an explicit deep-research request, cited report, market analysis, due diligence, or comprehensive investigation of one topic, use research_deep with 3 queries for a concise/narrow report, 4 by default, and 5-6 only when the user explicitly asks for deeper coverage. Use research_fanout when the request compares many entities or independent angles. Call the selected research workflow once per user request; if it fails, explain the failure instead of immediately rerunning it. Those workflows produce the complete cited PDF deliverable automatically, so do not regenerate it with a document tool. Treat search snippets as leads, not sources — open the real pages and cross-check. Cite as you go: attribute each claim to its source inline with the page title and its URL, and make sure every citation resolves. End a research answer with a short Sources list of the URLs you actually used.`;
 
 /** Compact all-domains pointer for an ambiguous general request — keeps the model aware without the full modules. */
 const GENERALIST_MODULE = `## Choosing your approach

packages/agent-core/src/mastra/workflows/deep-research-fanout.ts

@@ -10,10 +10,8 @@ export const deepResearchFanout = createDeepResearchWorkflow({
 
 function fanoutResearchPrompt(query: string): string {
   return [
-    "Run a breadth-first research pass for the query below.",
-    "Prefer search_web or search_company for discovery, then search_scrape for official pages.",
-    "Do not call research_deep or research_fanout from inside this workflow step.",
-    "Return structured claims only from provider results. Cite every claim with the exact Exa result ID and URL or exact Firecrawl URL returned by the tools.",
+    "Analyze the breadth-first provider evidence pack for the query below.",
+    "Return structured claims only from that evidence. Cite every claim with the exact Exa result ID and URL or exact Firecrawl URL present in the pack.",
     "Do not infer citation IDs from prose and do not cite a URL that no tool returned.",
     "",
     `Query: ${query}`,

packages/agent-core/src/mastra/workflows/deep-research-workflow.ts

@@ -1,11 +1,20 @@
+import { APIError } from "@cheatcode/observability";
 import { createStep, createWorkflow } from "@mastra/core/workflows";
-import { stepCountIs } from "ai";
 import { z } from "zod/v4";
+import {
+  executeExaSearch,
+  executeFirecrawlScrape,
+  ResearchRuntimeContextSchema,
+} from "../../tools/research";
+import { CONTEXT } from "../context";
 import {
   createResearchStepContext,
+  exaSource,
+  firecrawlSource,
   mergeResearchSources,
   ResearchPassDraftSchema,
   ResearchSynthesisDraftSchema,
+  registerResearchSources,
   validateResearchPass,
   validateSynthesisClaims,
 } from "./research-provenance";
@@ -18,14 +27,21 @@ import {
   ResearchReportSchema,
 } from "./research-schemas";
 
-const RESEARCH_CHILD_TOOLS = [
-  "search_extract",
-  "search_scrape",
-  "search_web_content",
-  "search_company",
-  "search_web",
-  "search_web_advanced",
-] as const;
+const RESEARCH_QUERY_CONCURRENCY = 3;
+const RESEARCH_RESULTS_PER_QUERY = 6;
+const RESEARCH_RESULT_TEXT_CHARACTERS = 2_500;
+const RESEARCH_SCRAPE_CHARACTERS = 12_000;
+
+interface ResearchEvidenceSource {
+  content: string;
+  provider: "exa" | "firecrawl";
+  providerResultId?: string | undefined;
+  sourceId: string;
+  title?: string | null | undefined;
+  url: string;
+}
+
+type RequestContextReader = { get(key: string): unknown };
 
 interface ResearchWorkflowPrompts {
   queryPrompt(query: string): string;
@@ -64,7 +80,9 @@ function buildDeepResearchWorkflow(config: DeepResearchWorkflowConfig) {
     outputSchema: ResearchReportSchema,
   })
     .then(createDeepPlanStep(config))
-    .foreach(createQueryStep("run-deep-research-query", config), { concurrency: 5 })
+    .foreach(createQueryStep("run-deep-research-query", config), {
+      concurrency: RESEARCH_QUERY_CONCURRENCY,
+    })
     .then(createSynthesisStep("synthesize-deep-research", config))
     .commit();
 }
@@ -77,7 +95,9 @@ function buildFanoutResearchWorkflow(config: FanoutResearchWorkflowConfig) {
     outputSchema: ResearchReportSchema,
   })
     .then(createFanoutPlanStep(config))
-    .foreach(createQueryStep("run-deep-research-fanout-query", config), { concurrency: 5 })
+    .foreach(createQueryStep("run-deep-research-fanout-query", config), {
+      concurrency: RESEARCH_QUERY_CONCURRENCY,
+    })
     .then(createSynthesisStep("synthesize-deep-research-fanout", config))
     .commit();
 }
@@ -111,18 +131,23 @@ function createQueryStep(id: string, config: ResearchWorkflowPrompts) {
     id,
     inputSchema: ResearchQuerySchema,
     outputSchema: ResearchFindingSchema,
+    retries: 0,
     execute: async ({ abortSignal, inputData, mastra, requestContext }) => {
       const agent = mastra.getAgent("general");
       const research = createResearchStepContext(requestContext);
-      const response = await agent.generate(config.queryPrompt(inputData.query), {
+      const evidence = await fetchResearchEvidence(
+        inputData.query,
+        research.requestContext,
+        abortSignal,
+      );
+      const response = await agent.generate(researchPassPrompt(config, inputData.query, evidence), {
         abortSignal,
         requestContext: research.requestContext,
-        activeTools: [...RESEARCH_CHILD_TOOLS],
-        stopWhen: stepCountIs(6),
         structuredOutput: { schema: ResearchPassDraftSchema },
+        toolChoice: "none",
       });
       return validateResearchPass(
-        ResearchPassDraftSchema.parse(response.object),
+        parseResearchPassDraft(response.object),
         inputData.query,
         research.collector,
       );
@@ -135,17 +160,17 @@ function createSynthesisStep(id: string, config: ResearchWorkflowPrompts) {
     id,
     inputSchema: z.array(ResearchFindingSchema),
     outputSchema: ResearchReportSchema,
+    retries: 0,
     execute: async ({ abortSignal, inputData, mastra, requestContext }) => {
       const agent = mastra.getAgent("general");
       const sources = mergeResearchSources(inputData);
       const response = await agent.generate(config.synthesisPrompt(inputData), {
         abortSignal,
         requestContext,
-        stopWhen: stepCountIs(6),
         structuredOutput: { schema: ResearchSynthesisDraftSchema },
         toolChoice: "none",
       });
-      const draft = ResearchSynthesisDraftSchema.parse(response.object);
+      const draft = parseResearchSynthesisDraft(response.object);
       return ResearchReportSchema.parse({
         claims: validateSynthesisClaims(draft.claims, sources),
         findings: inputData,
@@ -155,3 +180,140 @@ function createSynthesisStep(id: string, config: ResearchWorkflowPrompts) {
     },
   });
 }
+
+async function fetchResearchEvidence(
+  query: string,
+  requestContext: RequestContextReader,
+  abortSignal: AbortSignal,
+): Promise<ResearchEvidenceSource[]> {
+  abortSignal.throwIfAborted();
+  const runtime = ResearchRuntimeContextSchema.parse({
+    exaApiKey: requestContext.get(CONTEXT.exaApiKey),
+    firecrawlApiKey: requestContext.get(CONTEXT.firecrawlApiKey),
+  });
+  const search = await executeExaSearch(
+    {
+      highlightMaxCharacters: 800,
+      highlightQuery: query,
+      includeHighlights: true,
+      includeSummary: true,
+      numResults: RESEARCH_RESULTS_PER_QUERY,
+      query,
+      summaryQuery: query,
+      textMaxCharacters: RESEARCH_RESULT_TEXT_CHARACTERS,
+      type: "auto",
+    },
+    runtime,
+    abortSignal,
+  );
+  const primary = search.results[0];
+  if (!primary) {
+    throw new APIError(502, "upstream_provider_outage", "Research search returned no sources", {
+      retriable: true,
+    });
+  }
+  const evidence = search.results.map((result) => exaEvidenceSource(search.requestId, result));
+  registerResearchSources(
+    { requestContext },
+    search.results.map((result) => exaSource({ ...result, requestId: search.requestId })),
+  );
+  const scraped = await fetchPrimaryPageEvidence(primary, runtime, abortSignal);
+  if (scraped) {
+    evidence.push(scraped);
+    registerResearchSources({ requestContext }, [
+      firecrawlSource({ title: scraped.title ?? undefined, url: scraped.url }),
+    ]);
+  }
+  return evidence;
+}
+
+function exaEvidenceSource(
+  requestId: string,
+  result: Awaited<ReturnType<typeof executeExaSearch>>["results"][number],
+): ResearchEvidenceSource {
+  const content = [result.summary, ...result.highlights, result.text]
+    .filter((value): value is string => Boolean(value))
+    .join("\n\n");
+  return {
+    content,
+    provider: "exa",
+    providerResultId: result.id,
+    sourceId: exaSource({ ...result, requestId }).id,
+    title: result.title,
+    url: result.url,
+  };
+}
+
+async function fetchPrimaryPageEvidence(
+  primary: Awaited<ReturnType<typeof executeExaSearch>>["results"][number],
+  runtime: z.output<typeof ResearchRuntimeContextSchema>,
+  abortSignal: AbortSignal,
+): Promise<ResearchEvidenceSource | undefined> {
+  if (!runtime.firecrawlApiKey) {
+    return undefined;
+  }
+  try {
+    const page = await executeFirecrawlScrape(
+      { formats: ["markdown"], onlyMainContent: true, timeout: 30_000, url: primary.url },
+      runtime,
+      abortSignal,
+    );
+    if (!providerStatusIsSuccessful(page.metadata?.statusCode)) {
+      return undefined;
+    }
+    const content = (page.markdown ?? page.description ?? "").slice(0, RESEARCH_SCRAPE_CHARACTERS);
+    return {
+      content,
+      provider: "firecrawl",
+      sourceId: firecrawlSource({ url: page.url }).id,
+      title: page.title ?? page.metadata?.title,
+      url: page.url,
+    };
+  } catch {
+    abortSignal.throwIfAborted();
+    return undefined;
+  }
+}
+
+function researchPassPrompt(
+  config: ResearchWorkflowPrompts,
+  query: string,
+  evidence: ResearchEvidenceSource[],
+): string {
+  return [
+    config.queryPrompt(query),
+    "Use only the provider evidence below. For Exa citations, copy providerResultId and URL exactly. For Firecrawl citations, copy the URL exactly.",
+    "Do not cite sourceId directly and do not add sources that are absent from this evidence pack.",
+    "",
+    JSON.stringify(evidence, null, 2),
+  ].join("\n");
+}
+
+function parseResearchPassDraft(value: unknown) {
+  const parsed = ResearchPassDraftSchema.safeParse(value);
+  if (parsed.success) {
+    return parsed.data;
+  }
+  throw invalidStructuredResearchOutput("research pass");
+}
+
+function parseResearchSynthesisDraft(value: unknown) {
+  const parsed = ResearchSynthesisDraftSchema.safeParse(value);
+  if (parsed.success) {
+    return parsed.data;
+  }
+  throw invalidStructuredResearchOutput("research synthesis");
+}
+
+function invalidStructuredResearchOutput(stage: string): APIError {
+  return new APIError(
+    502,
+    "upstream_provider_outage",
+    `The ${stage} returned invalid structured output`,
+    { retriable: true },
+  );
+}
+
+function providerStatusIsSuccessful(statusCode: number | undefined): boolean {
+  return statusCode === undefined || (statusCode >= 200 && statusCode < 400);
+}

packages/agent-core/src/mastra/workflows/deep-research.ts

@@ -10,10 +10,8 @@ export const deepResearch = createDeepResearchWorkflow({
 
 function deepResearchPrompt(query: string): string {
   return [
-    "Run a focused research pass for the query below.",
-    "Use search_web_advanced for discovery and search_scrape for source pages that need extraction.",
-    "Do not call research_deep or research_fanout from inside this workflow step.",
-    "Return structured claims only from provider results. Cite every claim with the exact Exa result ID and URL or exact Firecrawl URL returned by the tools.",
+    "Analyze the focused provider evidence pack for the query below.",
+    "Return structured claims only from that evidence. Cite every claim with the exact Exa result ID and URL or exact Firecrawl URL present in the pack.",
     "Do not infer citation IDs from prose and do not cite a URL that no tool returned.",
     "",
     `Query: ${query}`,

skills/deep-research/SKILL.md

@@ -17,6 +17,7 @@ Answer complex questions with sourced synthesis. The output should read like an
 2. Run the request through `research_deep`; use 3 queries for concise or narrow reports, 4 by default, and 5-6 only when the user explicitly asks for deeper coverage. It validates citations and creates the PDF deliverable.
 3. Return the key conclusion and important caveats in chat, with a short source list.
 4. Refer to the PDF naturally as ready below. Do not call a separate document tool or recreate the report.
+5. Call the workflow once per user request. If it fails, explain the failure instead of immediately rerunning it.
 
 ## Fan-out Mode