OpenAI Realtime Voice, but 100x Cheaper

I started Nomi [1], an open-source AI assistant that leans heavily on voice. OpenAI’s realtime services were too expensive to build on, so I had to find another way. This is the design I ended up with: a pipeline that reproduces the realtime voice loop at more than a hundred times lower cost. I used it for six months before OpenAI released its realtime API [2].

I’ve kept this accessible rather than exhaustive; the full implementation is in the repository [1].

The Cost Problem§

The realtime API is priced at $100 per million input tokens and $200 per million output tokens, making it one of the most expensive LLM interfaces available.

Voice processing is expensive, so the first question is what you can keep off the paid path. In its realtime documentation, OpenAI itself suggests handling Voice Activity Detection (VAD) locally for control and efficiency. VAD detects when someone is actually speaking, so the system processes only relevant audio segments. Not streaming every audio chunk to the API is the single largest cost lever, and it is the foundation of this design.

Language Selection: Go§

I chose Go for its concurrency model and cross-platform builds. Voice processing requires concurrent, low-level work, and Go handles both well. Cross-platform builds had their difficulties; the Dockerfiles in the repository record the targets, with darwin/intel and windows/arm64 in progress.

I still prefer Python for RL and ML work, but Go’s simplicity and constraints pay off when you ship a binary. Python (asyncio) and Rust (Tokio) have strong asynchronous primitives too; Go won on reliability and packaging, despite the low-latency limits I had already hit in bpfsnitch. Solomon Hykes’ rationale for choosing Go for Docker covers similar ground [3].

Some components of Nomi may remain in Python due to ML model constraints; Go simplifies maintenance, while deploying Python is more complex.

Transcription: Whisper, Local and Remote§

I wanted reproducible results on my laptop, with the option to outsource processing when needed. Whisper [4] satisfies both: an open-source speech recognition system with high accuracy across languages, runnable on-device via whisper.cpp or through OpenAI’s hosted Whisper API.

The hosted transcription API is accurate and modestly priced, but inference takes longer than the speech itself. So I run it in parallel to keep the rest of the system from blocking:

go func() {
    text, err := transcribeAsync(apiData)
    if err == nil {
        processText(text)
    }
}()

Optimizing the Transcription Loop§

Naive transcription calls were still too slow. The working configuration adds a few heuristics on top of the VAD: pause detection, periodic buffer flushes, and a double-buffer strategy that processes overlapping segments; the second buffer corrects first-buffer errors. Shorter chunks mean faster inference, so text processing keeps up with ongoing speech:

primaryBuffer := createBufferManager()
secondaryBuffer := createBufferManager()

I then prioritize longer segments, assuming they come from a higher-quality source. The same multi-buffer structure should eventually support speaker-change detection; I haven’t built that yet.

System Architecture§

Composing a VAD, a speech transcriber, and an LLM gives you a full voice round-trip comparable to the realtime API. The slowest component is the hosted Whisper call; total processing delay is typically 200–300 milliseconds, and effectively instant with on-device Whisper. The result is marginally slower than the realtime API but between 100 and 1,000 times cheaper. I tried text-to-speech and rejected it: the API latency was worse still.

In practice: the program listens continuously; once sufficient energy appears in the audio chunks, they are pushed to a buffer. After enough time elapses or a pause is detected, the buffer is released to the transcriber, which calls Whisper. The transcript, at end of speech or during pauses, is then passed to an LLM for the response.

Observations on Transcription Quality§

Telling Whisper the input language improves accuracy a lot. Even my English becomes acceptable, and it held up under heavy accents. On a Mac laptop, whisper.cpp is efficient enough that I could transcribe long recordings from work without difficulty.

Multi-speaker handling and speaker-change detection are still open problems here; some newer models ship this capability.

I left live transcription out of Nomi’s core: most people are happy to press a button, and that makes the whole flow synchronous and much simpler. The VAD and transcription code is reusable as-is if you are building in this space.

References§

1. Nomi — open-source AI assistant, GitHub.
2. Introducing the Realtime API, OpenAI, 2024.
3. Solomon Hykes on choosing Go for Docker, X, 2024.
4. whisper.cpp — port of OpenAI’s Whisper, G. Gerganov, GitHub.