Repository navigation
feat: name a voice endpoint and add an OpenAI-compatible adapter (#86) - #122
Conversation
A voice role may now name the client that serves it instead of always using the one the harness built, so a local engine and a hosted one can sit in the same role table. The endpoint is carried per attempt rather than per role because a fallback can live on another engine, which is the case that makes it worth carrying: a free local speech server falling back to a paid one. An endpoint name that is not configured is refused while the attempts are built, so a typo costs no request, as with every other validation. The `voice` budget rises 540 to 600, measured 591, in C01 and ARCHITECTURE.md.
`voice.providers.<name>` is an OpenAI-compatible base URL and the credential it needs, if any: a local server needs none, so an empty key is not an error and sends no Authorization header. A role names one with `endpoint:`, and a speech fallback restates its own for the same reason it restates its voice — a local primary falling back to a hosted engine is the case this exists for. The role field is `endpoint:` rather than `provider:`, because D170 already gave `provider:` the OpenRouter routing pin and one key cannot mean both. A role that names nothing routes exactly as it did before. Config warns, and does not fail, on the three mistakes that would otherwise surface only on the wire: a role naming an endpoint that is not declared, an endpoint with no base_url, and an endpoint carrying both a key and a key env.
#86) One adapter covers OpenAI, any gateway, and the self-hosted stack: faster-whisper-server, whisper.cpp server, parakeet-mlx-fastapi, Kokoro-FastAPI and openedai-speech all answer `{base}/audio/speech` and `{base}/audio/transcriptions`. Three decisions worth recording: - A key is optional. An empty key sends no Authorization header at all, so a local engine needs no credential and nothing is invented on its behalf. - Transcription is multipart by default, because that is the shape every local server accepts, with the base64 `input_audio` body behind `TranscribeJSON` for a server that implements only that one. - There is no roster and no cost estimate. A local response carries no usage block, so a synthesis is zero cost with an empty CostSource: a free call is not a failed cost lookup. A transcription that does report usage is priced from it, with CostSource "provider". The routing pin is not sent. `provider` is OpenRouter's object and an OpenAI-compatible server has no such field. New package, so `voice/openai` gets a budget of 480 in C01 and ARCHITECTURE.md; measured 469.
Every entry in `voice.providers` becomes one OpenAI-compatible client, and a role's `endpoint:` picks it; a role that names none keeps the harness's own client, so an existing config is untouched. A local engine borrows no credential: only the endpoint's own `api_key` or the variable its `api_key_env` names is read, and neither means no Authorization header at all. An endpoint with no `base_url` is skipped rather than defaulted to OpenAI's API — config warns about it, and the role that names it should fail instead of quietly spending money at a vendor nobody asked for.
`provider:` is OpenRouter's routing object, and the OpenAI-compatible adapter does not send it: the field has no meaning to a local engine or to OpenAI itself. A config that sets both would otherwise look like it routed something. Say so at load, like the other three endpoint mistakes.
`VOICE.md` gains the section the config needs: the `voice.providers` table, why a role says `endpoint:` and not `provider:`, why a fallback restates it, why an empty key is not an error, and why the pin is not sent. It also records the Piper decision — its OpenAI-shaped server covers it, so no `os/exec` lifecycle enters the tree for one engine. D180 has the reasoning, and the changelog the user-visible half.
…#86) L08 asked for the leak check on both directions and on a cancelled call, which is the shape a long-lived `nova serve` makes one of per request. The test asserts the calls succeeded before it scans, so it cannot pass because the code under test failed early.
There was a problem hiding this comment.
The change adds the voice.providers table, endpoint: on speech/fallback/transcription roles, per-attempt client resolution in voice, the OpenAI-compatible voice/openai adapter, config warnings, and the supporting docs. I ran go test -race -count=1 ./voice/... ./mode/providers ./config, plus make importcheck and make linecount-all; all passed. I did not run the full make ci.
Two issues worth fixing before merge:
- A WAV response is advertised as supported, but it breaks the default multi-chunk
nova speakpath and there is no test for it. - The new routing tests reuse matrix IDs A33–A38, which already mean different rows in
docs/VOICE.mdand are already cited by existing tests invoice/openrouterandmode/providers.
Everything else looked consistent with the described behaviour: keyless endpoints send no Authorization, unknown endpoint names fail before a request, and the transcription/speech wire shapes match the OpenAI API.
| case "audio/wav", "audio/x-wav": | ||
| return voice.FormatWAV |
There was a problem hiding this comment.
Returning FormatWAV here makes the default multi-chunk nova speak path emit a corrupt file. nova speak chunks by default (voice.ChunkRunes is 300) and voice.Join byte-concatenates every non-PCM clip, so two audio/wav responses become RIFF…data(chunk1) followed by RIFF…data(chunk2) in one file, with the first header's data size covering only chunk 1. The docs explicitly advertise a Piper server (which answers audio/wav) as supported, so an ordinary long input hits this. Either decode the WAV payload to raw PCM before returning it, or make Join/speak refuse a multi-chunk WAV, and add a test with two audio/wav turns through Speak + Join.
|
|
||
| // TestSpeak_A33_EndpointNamesItsSpeaker: a role that names an endpoint is served | ||
| // by that endpoint's client, not the default one. | ||
| func TestSpeak_A33_EndpointNamesItsSpeaker(t *testing.T) { |
There was a problem hiding this comment.
These tests reuse matrix IDs A33–A38, which already describe the provider-pin and roster-cost rows in docs/VOICE.md and are already cited by TestSpeak_A33_RolePinReachesTheWire in mode/providers, and TestSpeak_A34_NonsenseProviderReachesTheServer, TestSpeak_A35_RosterIsReadOncePerClient, TestSpeak_A36_UnknownPriceLeavesCostUnknown, TestSpeak_A37_LookupOffMakesOneRequest, TestSpeak_A38_APinIsPricedAtItsEndpoint in voice/openrouter. The same row now names two different behaviours, so the matrix no longer identifies one test. Give the new endpoint-routing tests fresh rows (A41+) and add those rows to the docs/VOICE.md matrix.
|
pi-review reviewed |
A WAV file is PCM with a header, and a Piper-style server answers one whatever the request asked for. Mapping it to a wav-format clip was wrong downstream: the default `nova speak` chunks, and `voice.Join` concatenated whole files, leaving a header mid-stream and no header `voice.WAV` would write. The clip is now the samples without the header, with the layout from the `fmt` chunk, so Join and WAV stay correct; a body labelled wav that is not one is an error rather than a file a player cannot open. The new routing rows are L09 to L14 rather than A33 to A38, which `docs/VOICE.md` already assigns and other packages already cite. Every row this change adds is now in that matrix, including the second adapter's. `voice/openai`'s budget rises 480 to 560, measured 521, for the RIFF reader; in C01 and ARCHITECTURE.md.
|
Both points fixed in 87c0b16. 1. WAV support was a promise with no follow-through. It was worse than untested: a Tests: 2. Matrix IDs. Renamed: the routing rows are
Thanks — the WAV one would have shipped as a bug a user hit the first time they pointed at a Piper server with long text. |
There was a problem hiding this comment.
The follow-up addresses both findings from the previous review.
- A WAV response is now repackaged into a headerless PCM clip (
voice/openai/wav.go,speak.go), sovoice.Joinconcatenates samples and the finalvoice.WAVwrites a single header.TestSpeak_L15–L17cover the sample/layout extraction, the two-chunk join producing a validRIFF/WAVE, and a malformed wav body being an error. I confirmed these tests fail without the conversion (the old path returnedFormatWAVwith the whole file inBytes). - The new routing tests were renamed from the colliding
A33–A38toL09–L14and the endpoint-without-base_urltest toL18;docs/VOICE.mdlists the matchingL01–L18rows, and none of the new ids collide with existing matrix rows.
I re-ran go test -race -count=1 ./voice/... ./mode/providers ./config (all pass), the three new WAV tests specifically, gofmt, go vet ./voice/openai, and make matrixcheck-all linecount-all (exit 0). I did not run the full make ci.
The parser is bounded (all slices are checked against the body length, and chunk offsets advance by at least eight bytes), so I found no panic path on the shipped amd64/arm64 builds. Non-format-1 (WAVE_FORMAT_EXTENSIBLE) bodies are refused by design, which is documented. No blocking issues found.
Closes #86. Speech and transcription gain a second adapter, and a role can now address a self-hosted engine.
Every self-hosted audio server speaks the same OpenAI shape —
faster-whisper-server,whisper.cppserver,parakeet-mlx-fastapi,Kokoro-FastAPI,openedai-speech— so onevoice/openaiadapter covers them, OpenAI itself and any gateway, with no vendor code and no new dependency.Why
endpoint:and notprovider:—provider:is already the OpenRouter routing pin (D170), and one key cannot mean an engine and a route. A role that names nothing keeps the harness's own client, so existing configs are untouched.What lands
voiceroutes a role and each attempt to a named client, so a free local primary can fall back to a paid hosted one. An unconfigured name is refused while the attempts are built — a typo costs no request.configcarriesvoice.providersandendpoint:on speech roles, fallbacks and transcription roles, and warns on four mistakes: an endpoint not declared, nobase_url, both a key and a key env, and a routing pin on an endpoint.voice/openaispeaks{base}/audio/speechand{base}/audio/transcriptions: multipart by default, base64input_audioundertranscribe_json, format and PCM layout from the responseContent-Type(anaudio/wavanswer, as a Piper server gives, keeps WAV). No roster and no cost estimate.mode/providersbuilds one client per configured address. A keyless endpoint sends noAuthorizationheader — it never borrows the OpenRouter credential — and an endpoint with nobase_urlis skipped rather than silently defaulted toapi.openai.com.VOICE.mdsection on endpoints and the second adapter, D180, the changelog entry, and theARCHITECTURE.mdbudgets.Test plan
make cigreen,docker-free, no network. New coverage:voicerouting (A33–A38), config (TestLoad_VoiceEndpointsand three warning cases),voice/openai(L01–L05,L08, including atestkit.NoLeakspass over speak, transcribe and a cancelled call), andmode/providers(L02,L06).Also exercised the real binary against a local fake OpenAI-shaped server:
Upgrade notes
None breaking.
voice.providersandendpoint:are additive, every newvoicefield is a zero value that keeps today's behaviour, andvoice/openaiis a new package.Budgets, raised on purpose
voice540 → 600 (measured 591) for the endpoint fields and the per-attempt client resolution;voice/openaiadded at 480 (measured 469). Both indocs/archive/chunks/C01-foundations.mdanddocs/ARCHITECTURE.md.Not in this PR
The Piper process engine (text on stdin → WAV on stdout). Its OpenAI-shaped server already covers it, and an
os/execlifecycle, aPATHlookup and a stderr contract for one engine did not look worth it; D180 records the reasoning. Say the word and I will file it separately.