feat(search): index document attachments with Mistral

[salmonumbrella]

What changed

• Add an opt-in Mistral OCR document-indexing lane with authenticated format capabilities, explicit retention/training consent, bounded processing, and private inline uploads.
• Classify attachment occurrences across importers with stable source-part provenance, then reconcile edits, replacements, deletions, retries, and rebuilds for SQLite and PostgreSQL.
• Store normalized source units and chunks locally, expose dedicated lexical document search and status through CLI, HTTP/OpenAPI, and MCP, and return exact attachment, message, and source provenance.
• Keep raw provider JSON and full provider Markdown transient, disclose derived plaintext in full backups, and leave attachment-document embeddings to Add document vectors and semantic retrieval for extracted attachments #617.

Why

Msgvault stores attachment bytes, but evidence inside documents is not searchable. This adds a consent-gated lexical foundation that can find document content without silently uploading unsupported formats or returning stale results after attachment lifecycle changes.

The authenticated Mistral capability matrix covers all 26 non-visual document formats supported by this lane:

• Documents: PDF, DOCX, DOC, ODT, and RTF
• Presentations: PPTX and PPT
• Spreadsheets: XLSX, XLS, ODS, Apple Numbers, and CSV
• Ebooks: EPUB
• Text and markup: TXT, Markdown, reStructuredText, and LaTeX
• Structured data: JSON, JSONL/NDJSON, XML, and YAML
• Source code: Go, Python, and JavaScript
• Email: EML and Outlook MSG

Each format still has to pass the authenticated private-fixture probe before Msgvault will send that format to Mistral or include it in the recorded consent profile.

Usage

Set MISTRAL_API_KEY, enable [attachments.documents], and choose explicit retention and training postures. Then probe a private synthetic format matrix and record consent for that exact capability manifest:

msgvault documents probe-mistral --fixtures <fixture-directory> > mistral-capabilities.json
msgvault documents consent-mistral --capabilities mistral-capabilities.json --yes
msgvault documents build --capabilities mistral-capabilities.json --yes
msgvault documents search "shipping damage"

Document embeddings remain unavailable in this PR; search is lexical and filename-ranked. The semantic/vector phase is tracked in #617.

Refs #608 and #617

[roborev-ci]

roborev: Combined Review (73d6c0b)

Code review found four medium-severity issues involving provider limits, stuck extraction candidates, unstable pagination, and reconciliation races.

Medium

• internal/documentindex/worker.go:201 — Provider-unit limits are enforceable before submission only for PDFs. Large DOCX/PPTX/XLSX files can exceed per-document and per-run limits—and incur unbounded cost—before post-processing rejection. Enforce provider-side limits for every supported format, determine and bound units locally, or exclude formats that cannot be bounded. Consent text should distinguish estimates from enforceable caps.

• internal/documentindex/worker.go:148 — Oversized candidates fail before receiving an extraction claim or durable terminal state, so they remain pending and repeatedly consume the fixed candidate window, potentially starving valid documents. Filter candidates by maximum file size or persist a terminal/suppressed preflight decision.

• internal/store/document_search.go:148 — Pagination expands the candidate window while reusing an offset from an earlier RRF ranking. Recomputed rankings can shift items across that offset, producing skipped or duplicated documents even at the same index revision. Keep a fixed ranking snapshot/window for the cursor lifetime or use stable keyset pagination over immutable ordering.

• internal/store/document_index.go:299 — Reconciliation reads attachment state outside the catalog mutation transaction and does not reject older source sequences. Concurrent reconcilers can overwrite newer state with stale state, after which cursor acknowledgement may leave no journal event to repair the catalog. Serialize reconciliation or atomically read and mutate state, and enforce monotonic source-sequence checks for upserts and deletions.

---

Reviewers: 2 done | Synthesis: codex, 16s | Total: 28m54s

[wesm]

I definitely want to get this in Docbank also, I wonder if we could build this functionality there and then pull in Docbank as a dependency (or it can start here and then migrate to Docbank)

[salmonumbrella]

We'd get SHA-256 ids and docbank's frontend ui for #622.

What if a user has an existing docbank installation? We'd have to rig docbank <> msgvault together in an clean way like kata in forge. I'll leave the decision to you.

[wesm]

Docbank is usable as an embedded library and does not have "one docbank to rule them all", so it's more a question of not having the implementation for this in multiple places. I'd like to pull docbank as a dependency into msgvault

[salmonumbrella]

Sounds good. I'm don't know my way around docbank, could you please take over file embeddings?

[wesm]

Sure, let me have a look at this and advise on the next steps