[Don't merge] feat(derivatives): agentic compute mode (bounded tool loop over full sources)

[maciejwitowski]

What

A question registration may set "mode": "agentic" (default "completion", unchanged). The compute job then builds a per-compute, in-memory keyword (BM25/MiniSearch) index over the full source data — every item, not the newest maxSourceItems — and runs a bounded tool loop against the same stateless chat-completions relay, using the OpenAI tool-calling shape:

• search_data({query}) — top passages with refs, scope, and a snippet centered on the matched term
• read_data({ref}) — one source item's text, bounded

Both tools execute inside the Personal Server. The model sees the question, the snippets search returned, and the items it explicitly read — never the corpus, never the index. The index lives for one compute and is discarded: nothing new is stored, synced, or granted, and grant semantics are identical to completion mode (sources must be read-granted; re-checked per compute).

Spec first: docs/derivative-data-api.md → "Agentic mode".

Why

Newest-first trimming answers questions about recent data only. Measured on a real 36k-chunk ChatGPT corpus against a 25-question golden set (EN+PL):

Source selection	Right conversation retrieved
newest-first trim (maxSourceItems=50, favorable passage-granularity emulation)	8%
single-shot BM25 top-5	52%
single-shot hybrid (BM25+vectors)	64%
agentic loop over BM25 (this PR's shape, Haiku, ≤6 tool calls)	80%

avg 3.4 tool calls, ~8s, ~$0.015/question. The loop's reformulation + query translation recovers more than an embedding lane does, with no index build cost (BM25 over 53k chunks ≈ 1s) — so it runs identically on node PS, desktop, and PS-Lite. Latency is absorbed by the compute layer's already-async register→poll contract.

How

• core/derivatives/agentic.ts: corpus build (top-level array items → harvested text → 1.6k-char windows; text held once; 20M-char cap, truncation disclosed) + the loop (budget inference.maxToolCalls, default 6; over-budget calls refused inline; a cap-hit loop gets one forced tools-free answer turn so interim narration is never persisted; per-call retry hook so transient failures never replay the loop).
• core/derivatives/inference.ts: additive OpenAI tool-calling support (tools, tool_calls, role:"tool" messages). Reply tool_calls only count when the request offered tools (a hallucinated array can't defeat the empty-reply guard); tool calling with the E2EE encryption seam configured is refused explicitly (the protocol doesn't cover tool traffic).
• compute.ts: agentic branch; record gains mode + toolCalls; lineage unchanged. Binary sources are disclosed to the model via a context note; an all-binary source set fails with an accurate message.
• Stores: sqlite mode column via PRAGMA-guarded migration; PS-Lite state-store rehydrates pre-mode rows.

Validation

• 1,373 tests pass (22 new: corpus, loop budgets/forced-final-turn, wire shape, compute e2e, failure paths), tsc --build clean. packages/cli/src/index.test.ts fails identically on clean main (missing vendored browser_tls_rustls.js) — pre-existing.
• Independent high-effort review round completed; all correctness findings fixed in the second commit (narration-as-answer, hallucinated tool_calls, whole-loop retry replay, date-prefixed text dropped from the corpus, snippet/window mismatch, 3× text memory, swallow-all ALTER, silent binary omission).

Non-goals / follow-ups

• No persistent search-index derivative (this builds in-memory per compute); no relay changes; no new grant types; no precise citation lineage (answers keep scope-level lineage). Dedup of RESERVED_KEYS/system-prompt constants with prompt.ts left for a cleanup pass.

cc @volod-vana

🤖 Generated with Claude Code