fix(embed): surface Voyage AI total_tokens as prompt token count

[TurgutKural]

Problem

Voyage AI reports usage.total_tokens on the /v1/embeddings endpoint instead of the OpenAI-standard usage.prompt_tokens:

// Voyage AI response
{"data": [{"embedding": [...]}], "usage": {"total_tokens": 8}}

// OpenAI-standard response
{"data": [{"embedding": [...]}], "usage": {"prompt_tokens": 8, "total_tokens": 8}}

The Go decoder (openAIEmbedResponse.Usage) only had a prompt_tokens field, so PromptTokens was always 0 for Voyage. Downstream effects:

• llmlog rows carry no token accounting — every embed row shows —/— in the tok in/out column
• Dispatch usage metering broken — embed calls charge 0 tokens into the lease (MW22/C1)
• Status-page embed aggregation empty — no token data to aggregate
• Cost tracking impossible — cannot answer "how much does embed cost per month" from the DB

Same wire-format mismatch class as the rerank data/results fix (PR #13): Voyage follows their own OpenAPI schema, not the OpenAI wire format the client was written for.

Fix

internal/embed/embed.go:

1. TotalTokens field added to openAIEmbedResponse.Usage struct (json:"total_tokens")
2. Fallback in embedOpenAI: when PromptTokens == 0 && TotalTokens > 0, use TotalTokens

Precedence preserved: prompt_tokens wins when both are present (OpenAI sends both).

Tests

6 new tests in wire_test.go:

Test	Covers
TestEmbedOpenAI_VoyageTotalTokensFallback	Core regression: total_tokens surfaced as prompt count
TestEmbedOpenAI_PromptTokensPreferred	Both fields present → prompt_tokens wins
TestEmbedOpenAI_PromptTokensOnly	OpenAI-compat servers unchanged
TestEmbedOpenAI_NoUsage	No usage field → 0 (uncharged, C1 doctrine)
TestEmbedOpenAI_EmptyUsage	Both fields zero → 0
TestEmbedOpenAI_VectorStillCorrect	Token fallback does not disturb the embedding vector

=== RUN   TestEmbedOpenAI_VoyageTotalTokensFallback
--- PASS
=== RUN   TestEmbedOpenAI_PromptTokensPreferred
--- PASS
=== RUN   TestEmbedOpenAI_PromptTokensOnly
--- PASS
=== RUN   TestEmbedOpenAI_NoUsage
--- PASS
=== RUN   TestEmbedOpenAI_EmptyUsage
--- PASS
=== RUN   TestEmbedOpenAI_VectorStillCorrect
--- PASS
PASS (38/38 total in embed package)

Regression

Full go test ./internal/... run: all packages pass except 2 pre-existing environment failures unrelated to this change:

• internal/cli: git binary not found in golang:1.26-alpine test container
• internal/events: TestRebuildViaWorker_TimeoutKillsHangingChild flaky goroutine deadlock

go vet ./internal/embed/ clean.

[GottZ]

Reviewed with the same depth as #13 (4 finder dimensions + adversarial per-finding verification, 22 agents) — and this one came in noticeably cleaner: correct diagnosis, the precedence preserved, the godot fix already on board. The embed twin of the rerank dialect fix ships with your approach intact; the branch was rebased onto current root (v4.23.0), nothing of yours was lost.

Three hardening waves on top:

• b816487 — the precedence fixture is now discriminating. TestEmbedOpenAI_PromptTokensPreferred fixed both usage fields to the same value (10/10), so inverting the precedence guard survived all six tests — the exact lesson from fix(rerank): support Voyage AI response format (data field + total_tokens) #13's rerank waves. The fixture now sets total_tokens:99 against prompt_tokens:10 (the inversion mutation reds 4 assertions), and VectorStillCorrect asserts the token count too so it kills mutants on its own.
• abe2d21 — metering visibility, mirroring what the rerank fix got in v4.23.0. The fallback silently flips total_tokens-only backends from "uncharged" to a real charge in the fairness meter (the single production call site charges the dispatch lease; llmlog and the status rollup start carrying values where they were NULL). One-time INFO on first engagement + the operations.md meter paragraph now says "rerank and embed backends" — it had become false with this PR.
• 6d118dc — negative prompt_tokens is a sentinel, not a charge. == 0 blocked the fallback for a negative value and passed the negative through to the meter, while negative total_tokens was already discarded — both packages now guard with <= 0, applied symmetrically to embed and rerank so the twins stay byte-consistent.

Deliberately not built here: decode-tolerance for malformed usage counters (a non-integral total_tokens is now decode-fatal, but the adversarial pass showed the same strictness pre-existed for prompt_tokens on base — a symmetric json.Number follow-up for both packages is on the backlog, not this PR's scope), and Voyage's input_type parameter for the embed request (separate feature; note that external embed backends are gated behind metadata.embed_equivalence_verified regardless — foreign quantization corrupts the shared vector space).

Full suite (37 packages) and lint are green on the rebased branch. CI runs next; merge follows once it confirms.

[GottZ]

Merged and released in v4.23.1 — that makes five merged PRs from you across the whole Voyage dialect surface. Clean submission this round: precedence right, godot handled, no stale base. The only red on CI turned out to be a pre-existing conflict between two of our own hardening waves (B5 gate parity vs. the B6 body-cap fixture), fixed as part of this release — nothing in your change.