Skip to content

Commit 56f112e

Browse files
authored
fix: make research chat and PDF use one canonical report (#178)
## Summary - Publishes a successful deep-research tool's validated Markdown directly as assistant text in the same durable Workflow step that publishes its artifact. - Treats successful `research_deep` and `research_fanout` calls as terminal response producers, so no second model turn can summarize or rewrite the report. - Suppresses model pre-tool narration for terminal research calls, making the visible report body and PDF input identical by construction. ## Why Production QA for PR #177 proved that the PDF was complete and valid, but the outer agent ignored its prompt and posted a 1,386-character summary instead of the canonical report. Prompt-only enforcement cannot guarantee content identity. This moves ownership to the durable runtime boundary. ## Architecture The research tool continues to return `{ artifact, report }`, with `report` already validated against collected citation provenance. The Workflow projects that strict output into both destinations: 1. `artifact` becomes the durable deliverable part. 2. `report` becomes the assistant text part. 3. The run completes immediately after the successful tool turn. Both parts share the deterministic `tool:<model-step>:<tool-index>` publication receipt, so Workflow replay cannot duplicate either one. ## Decisions Made | Decision | Choice | Alternatives considered | Reasoning | |---|---|---|---| | Response ownership | Durable Workflow publishes canonical report | Ask the outer model to copy tool output | The production model summarized despite explicit instructions. | | Completion | Successful research tool is terminal | Run another model turn | A second turn can rewrite, truncate, or append content. | | Progress prose | Suppress research-call model text | Keep preamble before the report | Ensures visible report content and PDF source are the same Markdown. | | Failure path | Continue agent loop on tool error | Treat every research attempt as terminal | The model still needs to explain provider or validation failures. | | Replay behavior | Report and artifact share tool publication | Separate non-durable append | Existing atomic receipts already guarantee exactly-once transcript publication. | ## Edge Cases Handled | Scenario | Handling | |---|---| | Research tool fails | No canonical response is projected; the next model turn explains the error. | | Non-terminal search tool runs | Existing model loop is unchanged. | | Research output shape is invalid | Strict schema parsing fails the run rather than publishing divergent content. | | Worker/DO replays publication | Deterministic event receipt prevents duplicate report or artifact parts. | | Model emits a preamble with the tool call | It stays in model context but is not published to the user. | ## How to Review 1. Review `agent-run-research-response-support.ts` for terminal-tool classification and strict output projection. 2. Review `agent-run-workflow.ts` for suppression, atomic report publication, and terminal completion. 3. Confirm the agent-worker README and system prompt document the same ownership boundary. ## Verification - `pnpm lint` - `pnpm typecheck` - `pnpm turbo build --force` - `pnpm deadcode` - `pnpm architecture:check` - `pnpm turbo skills:build` - Direct projection contract: successful report preserved byte-for-byte; failed, non-terminal, and invalid outputs handled as designed. - Production evidence before this fix: reload recovery completed one run with one tool and one artifact, but persisted assistant text was a short model summary rather than the PDF's report.
1 parent be85331 commit 56f112e

4 files changed

Lines changed: 57 additions & 4 deletions

File tree

apps/agent-worker/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,10 @@ completed step instead of losing an in-memory coroutine. Transcript publication
8383
event keys and an atomic SQLite receipt, so Workflow step replay cannot duplicate visible parts.
8484
There is no application step, token, duration, or cost ceiling; semantic completion ends the loop,
8585
while per-operation timeouts and the platform Workflow limit remain operational safeguards.
86+
Successful deep-research tools are terminal response producers: the Workflow suppresses the model's
87+
pre-tool narration, publishes the tool's validated canonical Markdown directly as the assistant text,
88+
and completes without asking a second model turn to copy or summarize it. The same Markdown is the
89+
input to the PDF renderer, so chat and deliverable content cannot diverge by model behavior.
8690

8791
The run-keyed Durable Object is the authoritative status, cancellation, transcript, and stream
8892
store. It validates every Workflow callback against the stored input hash and deterministic
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import { z } from "zod";
2+
3+
const RESEARCH_REPORT_TOOL_NAMES = new Set(["research_deep", "research_fanout"]);
4+
const ResearchReportToolOutputSchema = z.strictObject({
5+
artifact: z.unknown(),
6+
report: z.string().trim().min(1).max(20_000),
7+
});
8+
9+
interface ToolCallLike {
10+
toolName: string;
11+
}
12+
13+
interface ToolResultLike {
14+
error?: string | undefined;
15+
output?: unknown;
16+
toolCall: ToolCallLike;
17+
}
18+
19+
export function isResearchReportTool(toolName: string): boolean {
20+
return RESEARCH_REPORT_TOOL_NAMES.has(toolName);
21+
}
22+
23+
/** Returns the validated canonical response for a successful terminal research tool. */
24+
export function canonicalResearchReport(result: ToolResultLike): string | undefined {
25+
if (result.error || !isResearchReportTool(result.toolCall.toolName)) {
26+
return undefined;
27+
}
28+
return ResearchReportToolOutputSchema.parse(result.output).report;
29+
}

apps/agent-worker/src/durable-objects/agent-run-workflow.ts

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ import { z } from "zod";
1010
import type { AgentRun } from "./agent-run";
1111
import type { AgentRunEnv } from "./agent-run-env";
1212
import { toAgentRunStreamError } from "./agent-run-errors";
13+
import {
14+
canonicalResearchReport,
15+
isResearchReportTool,
16+
} from "./agent-run-research-response-support";
1317
import {
1418
agentToolCallUiChunks,
1519
agentToolErrorUiChunks,
@@ -164,6 +168,9 @@ async function runAgentLoop(
164168
...state,
165169
messages: [...state.messages, workflowJsonValue(toolResultMessage(toolTurn.results))],
166170
});
171+
if (toolTurn.results.some((result) => canonicalResearchReport(result) !== undefined)) {
172+
return;
173+
}
167174
if (
168175
state.input.runIntent === "skill-creator" &&
169176
toolTurn.results.some(
@@ -302,12 +309,15 @@ async function publishModelStep(
302309
if (model.fallback) {
303310
chunks.push({ data: { ...model.fallback, v: 1 }, type: "data-model-fallback" });
304311
}
305-
const text = model.step.text.trim();
312+
const modelText = model.step.toolCalls.some((toolCall) => isResearchReportTool(toolCall.toolName))
313+
? ""
314+
: model.step.text;
315+
const text = modelText.trim();
306316
if (text.length > 0) {
307317
const id = `answer-${stepIndex}`;
308318
chunks.push(
309319
{ id, type: "text-start" },
310-
{ delta: model.step.text, id, type: "text-delta" },
320+
{ delta: modelText, id, type: "text-delta" },
311321
{ id, type: "text-end" },
312322
);
313323
}
@@ -347,9 +357,18 @@ async function publishToolStep(
347357
toolCallId: result.toolCall.toolCallId,
348358
toolName: result.toolCall.toolName,
349359
};
350-
const chunks = result.error
360+
const report = canonicalResearchReport(result);
361+
const chunks: UIMessageChunk[] = result.error
351362
? agentToolErrorUiChunks({ ...payloadBase, error: result.error })
352363
: agentToolResultUiChunks({ ...payloadBase, result: result.output });
364+
if (report) {
365+
const id = `research-report-${stepIndex}-${toolIndex}`;
366+
chunks.push(
367+
{ id, type: "text-start" },
368+
{ delta: report, id, type: "text-delta" },
369+
{ id, type: "text-end" },
370+
);
371+
}
353372
await workflowStep.do(`publish tool ${stepIndex}.${toolIndex}`, STATE_STEP, async () => {
354373
emitToolCompletion(env, payload, result, toolStepIndex);
355374
await appendWorkflowEvent(
@@ -362,6 +381,7 @@ async function publishToolStep(
362381
return {
363382
...state,
364383
hasArtifact: state.hasArtifact || chunks.some((chunk) => chunk.type === "data-artifact"),
384+
hasVisibleText: state.hasVisibleText || report !== undefined,
365385
input: result.input,
366386
};
367387
}

packages/agent-core/src/mastra/system-prompt.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,7 @@ Load the generate-media skill before creating or editing an image or generating
233233

234234
const RESEARCH_MODULE = `## Research
235235
236-
Gather sources with search_web / search_web_advanced / search_company / search_web_content, then use search_scrape or search_extract for source retrieval. For an explicit deep-research request, cited report, market analysis, due diligence, or comprehensive investigation of one topic, use research_deep with 3 queries for a concise/narrow report, 4 by default, and 5-6 only when the user explicitly asks for deeper coverage. A comparison of a few sources, standards, or recommendations within one topic still uses research_deep. Reserve research_fanout for four or more independent entities or genuinely separate angles, and pass entities as an array of separate short names. Call the selected research workflow once per user request; if it fails, explain the failure instead of immediately rerunning it. Those workflows produce the complete cited PDF deliverable automatically, so do not regenerate it with a document tool. After a research workflow succeeds, present its returned report Markdown as the response body without summarizing, restructuring, or appending hidden claim-map data; the PDF is rendered from that same Markdown. Treat search snippets as leads, not sources — open the real pages and cross-check. Cite as you go: attribute each claim to its source inline with the page title and its URL, and make sure every citation resolves. End a research answer with a short Sources list of the URLs you actually used.`;
236+
Gather sources with search_web / search_web_advanced / search_company / search_web_content, then use search_scrape or search_extract for source retrieval. For an explicit deep-research request, cited report, market analysis, due diligence, or comprehensive investigation of one topic, use research_deep with 3 queries for a concise/narrow report, 4 by default, and 5-6 only when the user explicitly asks for deeper coverage. A comparison of a few sources, standards, or recommendations within one topic still uses research_deep. Reserve research_fanout for four or more independent entities or genuinely separate angles, and pass entities as an array of separate short names. Call the selected research workflow once per user request; if it fails, explain the failure instead of immediately rerunning it. Call research_deep or research_fanout without a prose preamble: a successful workflow is terminal and the runtime publishes its validated report Markdown directly as the response while rendering that exact Markdown into the PDF. Do not regenerate it with a document tool or add a summary afterward. Treat search snippets as leads, not sources — open the real pages and cross-check. Cite as you go: attribute each claim to its source inline with the page title and its URL, and make sure every citation resolves. End a research answer with a short Sources list of the URLs you actually used.`;
237237

238238
/** Compact all-domains pointer for an ambiguous general request — keeps the model aware without the full modules. */
239239
const GENERALIST_MODULE = `## Choosing your approach

0 commit comments

Comments
 (0)