Skip to content

feat: name a voice endpoint and add an OpenAI-compatible adapter (#86) - #122

Merged
AutumnsGrove merged 8 commits into
mainfrom
feat/86-voice-endpoints
Nov 14, 2026
Merged

AutumnsGrove merged 8 commits into
mainfrom
feat/86-voice-endpoints

Conversation

@AutumnsGrove

Copy link
Copy Markdown
Owner

Closes #86. Speech and transcription gain a second adapter, and a role can now address a self-hosted engine.

Every self-hosted audio server speaks the same OpenAI shape — faster-whisper-server, whisper.cpp server, parakeet-mlx-fastapi, Kokoro-FastAPI, openedai-speech — so one voice/openai adapter covers them, OpenAI itself and any gateway, with no vendor code and no new dependency.

voice:
  providers:
    local:                              # a keyless engine on this machine
      base_url: http://127.0.0.1:8000/v1
    local-stt:
      base_url: http://127.0.0.1:8080/v1
      transcribe_json: true             # a server with no multipart shape
    openai:
      base_url: https://api.openai.com/v1
      api_key_env: OPENAI_API_KEY
  speech:
    roles:
      narrate:
        model: kokoro
        voice: af_heart
        endpoint: local
        fallback:
          model: gpt-4o-mini-tts
          voice: alloy
          endpoint: openai              # restated: a fallback may live elsewhere

Why endpoint: and not provider: — provider: is already the OpenRouter routing pin (D170), and one key cannot mean an engine and a route. A role that names nothing keeps the harness's own client, so existing configs are untouched.

What lands

  • voice routes a role and each attempt to a named client, so a free local primary can fall back to a paid hosted one. An unconfigured name is refused while the attempts are built — a typo costs no request.
  • config carries voice.providers and endpoint: on speech roles, fallbacks and transcription roles, and warns on four mistakes: an endpoint not declared, no base_url, both a key and a key env, and a routing pin on an endpoint.
  • voice/openai speaks {base}/audio/speech and {base}/audio/transcriptions: multipart by default, base64 input_audio under transcribe_json, format and PCM layout from the response Content-Type (an audio/wav answer, as a Piper server gives, keeps WAV). No roster and no cost estimate.
  • mode/providers builds one client per configured address. A keyless endpoint sends no Authorization header — it never borrows the OpenRouter credential — and an endpoint with no base_url is skipped rather than silently defaulted to api.openai.com.
  • Docs: VOICE.md section on endpoints and the second adapter, D180, the changelog entry, and the ARCHITECTURE.md budgets.

Test plan

make ci green, docker-free, no network. New coverage: voice routing (A33–A38), config (TestLoad_VoiceEndpoints and three warning cases), voice/openai (L01–L05, L08, including a testkit.NoLeaks pass over speak, transcribe and a cancelled call), and mode/providers (L02, L06).

Also exercised the real binary against a local fake OpenAI-shaped server:

$ nova speak --role narrate -o out.mp3 "hello there"
  → POST /v1/audio/speech  auth=None  {"model":"kokoro","input":"hello there","voice":"af_heart","response_format":"mp3"}
  → wrote out.mp3: 10 bytes, 1 chunk(s)
$ nova transcribe out.mp3 --role default --verbose --json
  → POST /v1/audio/transcriptions  auth=None  multipart/form-data
  → {"text":"hello from a local server", "usage":{...0...}}   ← a free call, not an unknown cost

Upgrade notes

None breaking. voice.providers and endpoint: are additive, every new voice field is a zero value that keeps today's behaviour, and voice/openai is a new package.

Budgets, raised on purpose

voice 540 → 600 (measured 591) for the endpoint fields and the per-attempt client resolution; voice/openai added at 480 (measured 469). Both in docs/archive/chunks/C01-foundations.md and docs/ARCHITECTURE.md.

Not in this PR

The Piper process engine (text on stdin → WAV on stdout). Its OpenAI-shaped server already covers it, and an os/exec lifecycle, a PATH lookup and a stderr contract for one engine did not look worth it; D180 records the reasoning. Say the word and I will file it separately.

A voice role may now name the client that serves it instead of always using
the one the harness built, so a local engine and a hosted one can sit in the
same role table. The endpoint is carried per attempt rather than per role
because a fallback can live on another engine, which is the case that makes
it worth carrying: a free local speech server falling back to a paid one.

An endpoint name that is not configured is refused while the attempts are
built, so a typo costs no request, as with every other validation.

The `voice` budget rises 540 to 600, measured 591, in C01 and ARCHITECTURE.md.
`voice.providers.<name>` is an OpenAI-compatible base URL and the credential it
needs, if any: a local server needs none, so an empty key is not an error and
sends no Authorization header. A role names one with `endpoint:`, and a speech
fallback restates its own for the same reason it restates its voice — a local
primary falling back to a hosted engine is the case this exists for.

The role field is `endpoint:` rather than `provider:`, because D170 already
gave `provider:` the OpenRouter routing pin and one key cannot mean both. A role
that names nothing routes exactly as it did before.

Config warns, and does not fail, on the three mistakes that would otherwise
surface only on the wire: a role naming an endpoint that is not declared, an
endpoint with no base_url, and an endpoint carrying both a key and a key env.
#86)

One adapter covers OpenAI, any gateway, and the self-hosted stack:
faster-whisper-server, whisper.cpp server, parakeet-mlx-fastapi,
Kokoro-FastAPI and openedai-speech all answer `{base}/audio/speech` and
`{base}/audio/transcriptions`.

Three decisions worth recording:

- A key is optional. An empty key sends no Authorization header at all, so a
  local engine needs no credential and nothing is invented on its behalf.
- Transcription is multipart by default, because that is the shape every local
  server accepts, with the base64 `input_audio` body behind `TranscribeJSON`
  for a server that implements only that one.
- There is no roster and no cost estimate. A local response carries no usage
  block, so a synthesis is zero cost with an empty CostSource: a free call is
  not a failed cost lookup. A transcription that does report usage is priced
  from it, with CostSource "provider".

The routing pin is not sent. `provider` is OpenRouter's object and an
OpenAI-compatible server has no such field.

New package, so `voice/openai` gets a budget of 480 in C01 and
ARCHITECTURE.md; measured 469.
Every entry in `voice.providers` becomes one OpenAI-compatible client, and a
role's `endpoint:` picks it; a role that names none keeps the harness's own
client, so an existing config is untouched.

A local engine borrows no credential: only the endpoint's own `api_key` or the
variable its `api_key_env` names is read, and neither means no Authorization
header at all. An endpoint with no `base_url` is skipped rather than defaulted
to OpenAI's API — config warns about it, and the role that names it should fail
instead of quietly spending money at a vendor nobody asked for.
`provider:` is OpenRouter's routing object, and the OpenAI-compatible adapter
does not send it: the field has no meaning to a local engine or to OpenAI
itself. A config that sets both would otherwise look like it routed something.
Say so at load, like the other three endpoint mistakes.
`VOICE.md` gains the section the config needs: the `voice.providers` table, why
a role says `endpoint:` and not `provider:`, why a fallback restates it, why an
empty key is not an error, and why the pin is not sent. It also records the
Piper decision — its OpenAI-shaped server covers it, so no `os/exec` lifecycle
enters the tree for one engine.

D180 has the reasoning, and the changelog the user-visible half.
…#86)

L08 asked for the leak check on both directions and on a cancelled call, which
is the shape a long-lived `nova serve` makes one of per request. The test
asserts the calls succeeded before it scans, so it cannot pass because the code
under test failed early.

@pi-review-autumnsgrove pi-review-autumnsgrove Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change adds the voice.providers table, endpoint: on speech/fallback/transcription roles, per-attempt client resolution in voice, the OpenAI-compatible voice/openai adapter, config warnings, and the supporting docs. I ran go test -race -count=1 ./voice/... ./mode/providers ./config, plus make importcheck and make linecount-all; all passed. I did not run the full make ci.

Two issues worth fixing before merge:

  1. A WAV response is advertised as supported, but it breaks the default multi-chunk nova speak path and there is no test for it.
  2. The new routing tests reuse matrix IDs A33–A38, which already mean different rows in docs/VOICE.md and are already cited by existing tests in voice/openrouter and mode/providers.

Everything else looked consistent with the described behaviour: keyless endpoints send no Authorization, unknown endpoint names fail before a request, and the transcription/speech wire shapes match the OpenAI API.

Comment thread voice/openai/wire.go
Comment on lines +134 to +135
case "audio/wav", "audio/x-wav":
return voice.FormatWAV

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning FormatWAV here makes the default multi-chunk nova speak path emit a corrupt file. nova speak chunks by default (voice.ChunkRunes is 300) and voice.Join byte-concatenates every non-PCM clip, so two audio/wav responses become RIFF…data(chunk1) followed by RIFF…data(chunk2) in one file, with the first header's data size covering only chunk 1. The docs explicitly advertise a Piper server (which answers audio/wav) as supported, so an ordinary long input hits this. Either decode the WAV payload to raw PCM before returning it, or make Join/speak refuse a multi-chunk WAV, and add a test with two audio/wav turns through Speak + Join.

Comment thread voice/roles_test.go Outdated

// TestSpeak_A33_EndpointNamesItsSpeaker: a role that names an endpoint is served
// by that endpoint's client, not the default one.
func TestSpeak_A33_EndpointNamesItsSpeaker(t *testing.T) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests reuse matrix IDs A33–A38, which already describe the provider-pin and roster-cost rows in docs/VOICE.md and are already cited by TestSpeak_A33_RolePinReachesTheWire in mode/providers, and TestSpeak_A34_NonsenseProviderReachesTheServer, TestSpeak_A35_RosterIsReadOncePerClient, TestSpeak_A36_UnknownPriceLeavesCostUnknown, TestSpeak_A37_LookupOffMakesOneRequest, TestSpeak_A38_APinIsPricedAtItsEndpoint in voice/openrouter. The same row now names two different behaviours, so the matrix no longer identifies one test. Give the new endpoint-routing tests fresh rows (A41+) and add those rows to the docs/VOICE.md matrix.

@pi-review-autumnsgrove

pi-review-autumnsgrove Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

pi-review reviewed 87c0b162ae7e — commented · 0 inline comments · details

A WAV file is PCM with a header, and a Piper-style server answers one whatever
the request asked for. Mapping it to a wav-format clip was wrong downstream: the
default `nova speak` chunks, and `voice.Join` concatenated whole files, leaving a
header mid-stream and no header `voice.WAV` would write. The clip is now the
samples without the header, with the layout from the `fmt` chunk, so Join and WAV
stay correct; a body labelled wav that is not one is an error rather than a file a
player cannot open.

The new routing rows are L09 to L14 rather than A33 to A38, which
`docs/VOICE.md` already assigns and other packages already cite. Every row this
change adds is now in that matrix, including the second adapter's.

`voice/openai`'s budget rises 480 to 560, measured 521, for the RIFF reader; in
C01 and ARCHITECTURE.md.
@AutumnsGrove

Copy link
Copy Markdown
Owner Author

Both points fixed in 87c0b16.

1. WAV support was a promise with no follow-through. It was worse than untested: a wav-format clip broke the default chunked nova speak, because voice.Join concatenated whole files and left a second RIFF header mid-stream, and voice.WAV returns nil for a non-pcm clip, so a --format pcm run produced nothing. A WAV file is PCM with a header, so the clip is now repackaged as the pcm clip it is — the samples without the header, and the layout read from the fmt chunk — which makes Join and WAV correct without touching either.

Tests: L15 (the repackaging), L16 (two WAV chunks joined into one playable file, asserting the single header and the concatenated samples), L17 (a body labelled wav that is not one is an error naming wav). A non-PCM WAV (format other than 1) is refused rather than misread.

2. Matrix IDs. Renamed: the routing rows are L09–L14 in voice/roles_test.go, and the mode-level empty-base_url row is L18. A33–A40 are untouched. Every row this change adds — L01–L06, L08–L18, with L07 recorded as deliberately absent for the declined Piper process route — is now in the docs/VOICE.md test matrix, so the ids are documented rather than invented in test names.

voice/openai's budget moved 480 to 560 (measured 521) for the RIFF reader, in C01 and ARCHITECTURE.md. Full make ci is green on the branch head, which covers the -race run from before.

Thanks — the WAV one would have shipped as a bug a user hit the first time they pointed at a Piper server with long text.

@pi-review-autumnsgrove pi-review-autumnsgrove Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The follow-up addresses both findings from the previous review.

  • A WAV response is now repackaged into a headerless PCM clip (voice/openai/wav.go, speak.go), so voice.Join concatenates samples and the final voice.WAV writes a single header. TestSpeak_L15–L17 cover the sample/layout extraction, the two-chunk join producing a valid RIFF/WAVE, and a malformed wav body being an error. I confirmed these tests fail without the conversion (the old path returned FormatWAV with the whole file in Bytes).
  • The new routing tests were renamed from the colliding A33–A38 to L09–L14 and the endpoint-without-base_url test to L18; docs/VOICE.md lists the matching L01–L18 rows, and none of the new ids collide with existing matrix rows.

I re-ran go test -race -count=1 ./voice/... ./mode/providers ./config (all pass), the three new WAV tests specifically, gofmt, go vet ./voice/openai, and make matrixcheck-all linecount-all (exit 0). I did not run the full make ci.

The parser is bounded (all slices are checked against the body length, and chunk offsets advance by at least eight bytes), so I found no panic path on the shipped amd64/arm64 builds. Non-format-1 (WAVE_FORMAT_EXTENSIBLE) bodies are refused by design, which is documented. No blocking issues found.

@AutumnsGrove
AutumnsGrove merged commit 13e3624 into main Nov 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local audio engines: an OpenAI-compatible adapter for self-hosted speech and transcription

1 participant