Adding a real-time speech-to-text API to your app: streaming with Soniox
You want to add captions that appear as someone speaks. So you go looking for a real-time speech-to-text API — and find plenty of options, but surprisingly little written about what actually makes the implementation hold together. After six months building Hiroi, a solo iOS live-translation app, that gap is the one I kept hitting.
So here's an architecture-level walkthrough, using Soniox — the API I actually ship on — as the concrete example. Less "how to call the SDK," more "which structure underneath decides whether it works in the field." I build on iOS/SwiftUI, but the reasoning is platform-agnostic.
Why "after they finish talking" is too late
Speech-to-text APIs come in roughly two shapes. Batch (file) recognition takes a recorded clip, processes the whole thing, and hands back a result. Streaming STT takes the audio while the person is still talking and returns the transcript as it forms.
For captions or live translation, streaming is the only real choice. The reason is simple: batch can only produce a result after the utterance ends. At meeting tempo, if you wait for the sentence to finish, the conversation has already moved on. (I wrote about that felt difference in how a real-time meeting translation app changes a meeting.)
With streaming, unfinished, provisional text comes back while the speaker is mid-sentence. People are reassured less by a perfect caption than by something being there right now. Keeping the tempo intact matters more than raw accuracy — and a live-captions implementation is designed from that one fact outward.
WebSocket and PCM frames: the foundation
Streaming almost always rides on a WebSocket. In Hiroi's case, I open one WebSocket to Soniox and specify the model `stt-rt-v5`.
The audio format is plain and unglamorous: `pcm_s16le` (16-bit linear PCM), 16 kHz, mono. You slice that into small frames of roughly 120 ms each and push them down the socket in order. Capture from the mic, convert, send a frame every ~120 ms. When you want to close the stream, you send one empty frame — that's the "done talking" signal.
On connect, you pass a bit of JSON once (languages to recognize, translation settings, model name), and after that it's just a steady flow of audio frames. That's the whole foundation. Which also means: this dumb little loop — cut audio, send, receive result — is the premise for everything else. If it stalls, everything above it collapses.
Interim vs final tokens
The heart of streaming is that results arrive in two stages.
- Interim tokens: provisional text, still being written. As speech continues, it gets overwritten — it can change or disappear.
- Final tokens: text the engine has decided won't change anymore.
Your UI should follow that split honestly. Render interim text faintly, tentatively; when it turns final, settle it in as the real transcript. To the user it looks like words being born and then hardening. Blur the two together and the screen flickers, or a caption you thought was settled keeps twitching and gets hard to read. Interim is "a place still allowed to move"; final is "a place you never touch again." Deciding that line up front saves a lot of pain later.
Translating from audio, and routing by language
The interesting part of Soniox is that it can return translation directly from audio, not from text. Ordinary machine translation is two stages — audio → transcript → translate the transcript. Here, recognition and translation come back on a single stream.
Better still, each returned token (a word or chunk) carries a language tag. Hiroi uses that to route tokens by their normalized source language to the correct side of the screen. Original tokens sit on the side of their own language; translated tokens are placed by their source language. Being able to line up three languages — Japanese, English, Chinese — without them scattering per speaker is thanks entirely to that per-token language info.
Exactly how you use it will differ per product, but knowing those two properties — translation arrives straight from audio, and tokens are language-tagged — gives you far more freedom when you lay out the captions.
The unglamorous reliability work that actually decides it
Everything above is in every tutorial. But whether real-time STT works in the demo yet dies in the field comes down to something far less glamorous. Shipping alone, this is where most of my hours went.
1. Reconnect with exponential backoff. Mobile connections just drop. Don't reconnect instantly on every drop — back off, lengthening the wait a little each time. Spacing out failed attempts keeps you from hammering the endpoint forever in a dead zone.
2. Give each socket a "generation" counter. Dull, but it earned its keep. After a socket drops and you re-open, a **late async callback from the old socket** can reach in and touch the new connection. The fix is a generation counter, incremented per connection. At the top of each callback, check "is my generation still the current one?" and silently drop it if not — conceptually just `if generation != current { return }`. Without it, a dead socket's zombie callbacks corrupt a live one.
3. Re-pin the mic on every (re)start. This one bit me on macOS. Every time you (re)start recognition, explicitly re-assert which input device to use. Because if a Bluetooth earbud connects mid-session, the OS quietly switches the default mic to that earbud's mic. Pin the device only once at the start and, the moment the route changes, capture gets hijacked to a mic you never chose. So: not "once," but "every time."
4. API keys are region-scoped. One last operational trap. Soniox keys are bound to a region — a US key only works against the US endpoint. If the key's region and the endpoint disagree, you get an error that looks like "Incorrect API key," when the key is fine and the region is the real mismatch.
How you hand over the key belongs to this quiet layer too. Hiroi supports two paths through one seam. Managed: my backend mints a short-lived token server-side and the app connects with that, so no long-lived key sits on the device. BYO: the user brings their own Soniox key. Either way, one principle holds — the audio itself streams directly from device to Soniox; I never proxy it through my own server. The server only mints tokens; the heavy audio stream stays a direct device↔Soniox link. Lighter on privacy, cost, and latency all at once.
None of this is flashy. But whether the thing runs every day in someone's hand is decided almost entirely here.
Adding a real-time speech-to-text API is surprisingly easy up to the first WebSocket. The difference shows up after that — handling interim results, the generation guard, re-pinning the mic, the region trap. It isn't glamorous, but this quiet layer is exactly what separates a demo from a tool people use every day.
(As an aside: I eventually deleted the feature I'd spent the most time on — voice interpretation. I traced that call in killing the feature I spent the most time on.)
If you're curious what it added up to, take a look at hiroi.app. The first WebSocket is the easy part — may the quiet layer beneath it be the one you're quietly proud of.