Modal Voice Assistant

A voice assistant POC that runs LLM inference and TTS on Modal. Speak into your mic, get a spoken response back. Speech-to-text runs locally for privacy — your voice never leaves your machine.

Speak → [Local: Silero VAD + Whisper STT] → text
                                              ↓
                                       [Modal A10G: Qwen3-8B → Kokoro TTS] → audio
                                              ↓
                                       [Local: playback]

Stack

Component	Tech	Where
Voice activity detection	Silero VAD	Local
Speech-to-text	faster-whisper (base, int8)	Local
LLM	Qwen3-8B via vLLM	Modal (A10G)
Text-to-speech	Kokoro (af_heart voice)	Modal (A10G)
Audio I/O	sounddevice	Local

Setup

Requires Python 3.12+ and uv.

# Install dependencies
uv sync

# Authenticate with Modal
uv run modal token set --token-id <id> --token-secret <secret>

# Deploy the Modal endpoint
uv run modal deploy src/modal_assistant/remote.py

Usage

Set the endpoint URLs from the deploy output, then run:

export MODAL_GENERATE_URL=https://<your-app>--modal-assistant-assistant-generate.modal.run
export MODAL_HEALTH_URL=https://<your-app>--modal-assistant-assistant-health.modal.run
uv run modal-assistant

Speak and pause — the assistant transcribes your speech, sends it to Modal for a response, and plays back the audio. Press Ctrl+C to quit.

How it works

1. Microphone audio is captured at 16kHz in 512-sample chunks
2. Silero VAD detects speech start/end boundaries
3. On speech start, a background health ping warms the Modal container
4. On speech end, the buffered audio is transcribed locally with faster-whisper
5. The transcription is POSTed to Modal, which generates a text response (Qwen3-8B) and synthesizes speech (Kokoro TTS)
6. The mic is paused during playback to prevent the assistant from hearing itself
7. Conversation history is maintained across turns
8. LLM responses are optimized for spoken output — no emojis, markdown, or special characters

Privacy

Voice transcription happens entirely on your machine via faster-whisper. Only the transcribed text is sent to the Modal endpoint — your audio never leaves your device.

Project structure

src/modal_assistant/
    remote.py       # Modal endpoint (vLLM + Kokoro on A10G)
    vad.py          # Silero VAD speech detection
    transcribe.py   # faster-whisper STT
    audio.py        # Mic capture + WAV playback
    client.py       # HTTP client for Modal endpoint
    main.py         # Main conversation loop

Cost

Uses Modal's free tier ($30/month). The A10G container scales to zero after 60 seconds of inactivity. Cold start takes ~2 minutes (model download + loading).