fix(rerank): support Voyage AI response format (data field + total_tokens)

[TurgutKural]

Problem

Voyage AI returns reranking results under "data" (per their OpenAPI RerankingObject schema) instead of "results" (the cohere/llama.cpp wire format the client was written for).

The Go JSON decoder silently left Results empty on every Voyage call, producing:

cross-encoder rerank: rerank: result count mismatch: got 0 scores for 50 documents

Production evidence: 6/6 real Voyage rerank calls failed with this error (both rerank-2.5 and rerank-2.5-lite). The fail-open path silently degraded every query to un-reranked RRF order. Local llama.cpp (bge-reranker-v2-m3) was unaffected — it speaks cohere format natively.

Additionally, Voyage reports usage as total_tokens while llama.cpp uses prompt_tokens, so the usage metering path also needed a fallback.

Fix

internal/rerank/rerank.go:

1. Data field added to rerankResponse (json:"data") — Voyage's OpenAPI field name
2. Fallback logic: after decode, if Results is empty and Data is non-empty, use Data
3. TotalTokens field added to the Usage struct; when PromptTokens is 0, fall back to TotalTokens

Precedence is preserved: "results" wins over "data" when both are present (cohere/llama.cpp path unchanged).

Tests

7 new tests added to rerank_test.go:

Test	Covers
TestScore_VoyageDataFallback	Core regression: "data" field decoded, scores + total_tokens returned
TestScore_VoyageDataReAlignsByIndex	Index re-alignment works through the "data" path
TestScore_PrefersResultsOverData	"results" takes precedence when both fields present
TestScore_VoyageTotalTokensFallback	total_tokens → promptTokens fallback
TestScore_VoyageDataRejectsCountMismatch	Count validation enforced on "data" path
TestScore_VoyageDataRejectsDuplicateIndex	Duplicate index validation on "data" path
TestScore_NeitherResultsNorData	Error when neither field present

=== RUN   TestScore_VoyageDataFallback
--- PASS
=== RUN   TestScore_VoyageDataReAlignsByIndex
--- PASS
=== RUN   TestScore_PrefersResultsOverData
--- PASS
=== RUN   TestScore_VoyageTotalTokensFallback
--- PASS
=== RUN   TestScore_VoyageDataRejectsCountMismatch
--- PASS
=== RUN   TestScore_VoyageDataRejectsDuplicateIndex
--- PASS
=== RUN   TestScore_NeitherResultsNorData
--- PASS
PASS (14/14 total, go vet clean)

Production verification

Post-deploy, 2/2 Voyage rerank calls succeeded (~690ms each, zero errors), confirmed via context_llm_log. Pre-fix: 0/6 succeeded.

[GottZ]

Thanks — this is exactly the kind of contribution we want: a real production failure, traced to the wire format, fixed minimally with the cohere path's precedence preserved, and brought with a proper test story (httptest against the real decode path, both validation gates re-covered through the new container). The diagnosis was correct and complete; your fallback approach ships as-is.

The review (4 finder dimensions + adversarial per-finding verification) surfaced one deeper issue your fix made reachable, plus hardening around it. All waves keep your prefer-results-fall-back-to-data structure:

• e328f9a — Voyage scores are calibrated [0,1] probabilities, not logits. Score()'s contract returns raw logits, and the single consumer sigmoids every score before blending. Voyage's already-calibrated values got sigmoided a second time, compressing the rerank signal into [0.5, 0.73] — and at blend_weight < 1 (the value our own validator recommends with graph expansion) RRF outvoted the reranker, with measured rank inversions against the identical relevance verdict expressed as logits. The wave maps data scores through logit(p) at the wire boundary, so the downstream sigmoid reconstructs your Voyage probabilities exactly. Your rebased tests now assert the logit values; a round-trip test pins the contract.
• 131f3f1 — absent relevance_score now fails open instead of scoring 0. Sibling data-shaped dialects name the score field score; that body used to pass every gate and neutralize the reranker silently (all-zero logits ⇒ uniform 0.5 ⇒ RRF order, telemetry green). Pointer decode + nil check turn it into the documented fail-open error — this also closes the same pre-existing gap on the results path.
• f9ba0ff — schema breaks are no longer reported as "count mismatch". Your PR body itself documents that message masking 6 production calls. Neither-container responses now get a dedicated error naming both fields plus a body snippet (the body is buffered, 1 MiB cap).
• 1f3b8e7 — data decodes lazily (json.RawMessage), so a backend serving valid results next to a non-array data field can't fail the whole decode; llama.cpp non-regression is pinned by test.
• 5b6a19a — test hardening: the prompt_tokens-over-total_tokens precedence was asserted but not exercised (no fixture set both fields — two mutations survived the suite); the both-set fixture now discriminates. Added the usage-absent contract and Authorization-header tests.
• 4970856 + a9cc340 — metering + docs: the total_tokens fallback silently flips total_tokens-only backends from "uncharged" to a real charge in the fairness meter — correct, but now visible (one-time INFO + operations.md note); architecture.md documents the Voyage dialect.

Deliberately not built here: a provider_class/score-domain slot on the backend pool model (the container-name⇒score-domain coupling is documented instead; a config-level discriminator is an architecture decision for a separate change), and persisting rerank token usage into context_llm_log (pre-existing gap, independent of this PR).

One logistical note: the PR branch carried your previous #12 commit from a stale base; it dropped out automatically as a duplicate during rebase onto current root — nothing of yours was lost. Full suite (36 packages) and lint are green on the rebased branch. Merging once CI confirms.

[GottZ]

I'll address that score-domain slot too. it will be configurable later.

[GottZ]

Merged and released in v4.23.0 — thanks again! Follow-up on top of your fix (post-merge, on root): the container⇒score-domain coupling is now a configurable per-backend slot (metadata.score_domain: auto | logit | probability, default auto = exactly the coupling this PR shipped), guarded by confirm_score_domain_change for dialect-mixing backends. Your Voyage setup needs no change — auto covers it.