Why voice AI still breaks: practical fixes for context and pipeline resilience

You're sitting in a hotel room, asking the in-room assistant for extra towels. It lights up, processes your request, and responds with the weather in Singapore. That's not a bug in the language model. That's a context pipeline that lost track of what mattered somewhere between wake word and response generation.

This is the gap that industry leaders keep pointing to. TechCrunch surveyed executives about voice AI's readiness, and the message was consistent: these execs think voice AI hasn't reached its ChatGPT moment yet TechCrunch. Conversational fluency exists in text models. Reasoning capability exists. What doesn't exist reliably is context continuity across the full voice pipeline, from audio capture to spoken response.

We've been building voice systems at Cropsly long enough to know the breaks almost never live in the model. They live in the pipeline.

Where Context Disappears

A voice AI pipeline is a chain of discrete stages. Audio comes in, gets chunked, gets transcribed, gets fed to an LLM, gets formatted as speech, gets played back. Every handoff is a place where context can leak.

Core issue: most pipelines treat transcription as pure signal conversion. Got words, now do something with them. But spoken language carries information that text doesn't preserve: prosody, timing, speaker identity, environmental context, the fact that someone paused mid-sentence. Flattening all of that into a transcript string loses the signals that disambiguate meaning.

Think about the phrase "no, not that one." In text it's ambiguous. In speech, the emphasis pattern tells you whether "that one" refers to a specific item the user previously discussed or is rejecting a suggestion the system just made. Strip audio to text before the LLM sees it, and the model has to guess. Sometimes it guesses wrong. That's not a hallucination. It's a context architecture problem.

The Latency Budget You Actually Have

Voice pipelines add latency that text pipelines don't deal with. You're not just generating tokens. You're capturing audio, running ASR, routing to an inference endpoint, generating tokens, routing to TTS, and playing back audio. Every stage has its own queue, its own network hop, its own failure mode.

Acceptable voice latency thresholds are unforgiving. Research on conversational systems suggests that delays beyond roughly 200ms start to feel unnatural, and anything past 500ms makes users think the system has stopped listening Google Research. Text chat tolerates multi-second responses because the user can see tokens appearing. Voice has no such affordance. Ask a question, silence, more silence, and the user starts wondering if the microphone is off.

That silence budget gets eaten fast. With ASR adding 80ms, LLM time to first token at 150ms, and TTS needing 100ms to produce the first audio chunk, you've already spent 330ms before the user hears anything NVIDIA TensorRT-LLM. Add network latency for a cloud-based model and you're easily past that 500ms threshold.

This is why we've moved toward on-device inference for voice products. Our RunHotel system runs voice AI entirely on-device for exactly this reason. Remove the network round-trip for inference, and you get a fixed latency budget that you can reason about. For teams weighing these tradeoffs, our AI cost estimator can model the on-device vs. cloud economics for your use case.

Speech Recognition Quality Is Not Solved

People assume ASR is solved because they've used Siri or Google Assistant. Those systems work reasonably well in quiet environments with native speakers. They degrade fast under real conditions: background noise, accents, code-switching, domain-specific vocabulary.

We tested a popular cloud ASR service against hotel room audio recordings and saw word error rates climb past 15% when the HVAC system was running Microsoft Azure Speech. One in seven words wrong. Errors landing on keywords mean the downstream LLM gets a garbled intent and produces a response that feels broken to the user.

Fixing this isn't just about better ASR. Fixing it means making your pipeline resilient to ASR errors. That means running a confidence score on the transcription and asking for clarification when it's low. It means maintaining a domain-specific vocabulary that gets injected into the ASR context. Using a model like Whisper, you can pass a prompt parameter with domain terms to bias the decoder toward correct spellings OpenAI Whisper API. It also means keeping the original audio available for re-processing when downstream stages flag an error. Preventing ASR errors entirely isn't possible. Designing your pipeline to catch them before they cascade is.

On-device SR adds another wrinkle. Smaller models mean lower quality, but they also mean lower latency and no network dependency. Model size versus error rate is something you have to measure explicitly for your domain. We've found that a well-tuned smaller model with domain vocabulary injection can match a larger generic model on in-domain utterances, even if its overall WER is higher.

Why Context Windows Break on Mobile

Here's where the mobile engineering perspective matters. Running voice AI on a device means you don't have unlimited memory to hold context. An Android mid-range phone ships with roughly 6GB of RAM, and the OS doesn't give your app all of it Android Developer Documentation. Apps might get 1.5GB for the process before the system starts killing background services.

Now consider what a voice pipeline needs in memory simultaneously: the ASR model, the LLM weights, the TTS model, the conversation buffer, and any multimodal inputs. Running a model like Qwen3-8B means the quantized weights alone consume around 4.5GB Qwen GitHub. Already over budget on a mid-range device before a single utterance is processed.

Naive approach: hold the entire conversation in the context window. But every token in that window adds compute time to each inference pass. On-device models already run at roughly 20 tokens per second on a mobile NPU Qualcomm AI Hub. Context windows growing from 500 tokens to 2000 tokens because you're keeping the full conversation history means latency per response grows with it.

Layered context is the fix. Keep a short rolling window of recent turns in full fidelity. Compress older turns into a summary that gets prepended to the active context. Maintain entity state separately so you don't need to re-derive it from the conversation every time. Say the user asks "what about the second one?" and your entity state layer should know what "the second one" refers to without needing the full transcript.

When Multimodal Sync Falls Apart

Voice AI doesn't exist in isolation. Building an assistant for a hotel room means it probably also has a screen. Screens mean the user expects visual responses to align with spoken responses. Misalignment makes the whole experience feel broken in a way that's hard to diagnose.

Synchronization is the problem: your audio pipeline and your visual pipeline run on different clocks. Your TTS engine produces audio chunks at a certain cadence. Your rendering pipeline targets 60 frames per second Android Graphics. LLM token generation runs at a variable rate depending on context length and model load. Getting these three to line up requires explicit coordination, not hope.

Handling this means timestamping every stage of the pipeline. Every token from the LLM gets a timestamp. Every audio chunk from TTS gets a timestamp. Rendering layer uses these timestamps to decide what to show and when. Audio ahead of visual means we buffer. Visual ahead of audio means we hold the frame. It sounds simple, but it requires building the timestamping into the pipeline from the start, not bolting it on later.

More insidious is the sync problem of interruption. Mid-response, the user says "actually, never mind." Your ASR needs to detect that interruption, signal the LLM to stop generating, signal the TTS to stop playing, and signal the visual layer to stop rendering. Any delayed signal means a response that keeps going after the user has already moved on. This is the moment where voice AI stops feeling like a conversation and starts feeling like a broken robot.

Building Error Recovery Into Every Stage

Pipelines break. Network connections drop. ASR returns garbage. LLMs generate something unhelpful. TTS fails on a particular phoneme. Architecture needs to handle all of these gracefully, and most don't.

Common failure mode we see is what we call the "committed response." Pipeline generates a response, commits to it, and has no fallback. Wrong responses mean the system either repeats it or starts the whole conversation over. Neither is acceptable.

What we recommend is a tiered response strategy. At low confidence, ask for clarification: "I heard 'extra towels for room 305', is that right?" This costs a turn but prevents cascading errors. At medium confidence, proceed with the response but keep a recovery path open. Should the user's next utterance suggest the previous response was wrong, the system should be able to rewind and re-process. At high confidence, respond directly and naturally.

Confidence scoring needs to happen at every stage, not just at the end. ASR confidence, LLM logprob analysis, and TTS prosody checking all contribute to the overall confidence of a response. ASR uncertainty paired with LLM confidence is a signal to check whether the LLM is confident about the wrong thing.

Our AI agents and on-device AI work builds this into the architecture from the start. Retrofitting error recovery onto a pipeline that was built as a straight-through path is painful. Having done it, we don't recommend it.

What We Actually Ship

After building several voice AI systems at Cropsly, here's the architecture we've settled on. Cropsly builds voice AI systems and custom models tuned to the constraints that matter, and we offer AI consulting for teams navigating these decisions. Reach out if you want to talk through a specific pipeline.

Run inference on-device when you can. Latency wins are worth the memory constraints. Can't run on-device? Use a streaming inference endpoint that returns tokens as they're generated, not in a batch. Difference between streaming and batch is the difference between a conversation and a walkie-talkie.

Keep context layered: a short active window, a compressed summary, and separate entity state. Don't try to hold the entire conversation in one context window. It's both expensive and unnecessary. Entity state layer is what makes "the second one" resolve correctly, and it's the piece most teams skip.

Entity state layer deserves more explanation. It's a structured representation of what the user is talking about, separate from the conversation history. Should the user ask about two rooms and then say "book the second one," the entity state tracks that there are two rooms, which one is "the second," and what "book" means in this context. Without this layer, the LLM has to infer all of that from the raw transcript every time, which burns tokens and introduces errors.

Build error recovery into every stage. ASR confidence, LLM confidence, TTS success. Every stage should be able to signal failure to the orchestrator, and the orchestrator should have a strategy for each failure type. Don't let any stage fail silently.

Timestamp everything. Audio chunks, tokens, frames. Synchronization is a data problem, not a hope problem.

Invest in SR quality before you invest in LLM quality. A mediocre LLM with excellent transcription will outperform a great LLM with poor transcription every time. LLMs can't reason about what they didn't receive. Our evaluation uses a combination of word error rate on domain-specific test sets and end-to-end conversation success rate to evaluate SR quality. Conversation success rate is the metric that actually matters, and it's the one most teams never measure NIST Speech Tools.

Sources

• TechCrunch: These Execs Think Voice AI Hasn't Reached Its ChatGPT Moment Yet
• Google Research
• NVIDIA TensorRT-LLM
• Microsoft Azure Speech Service
• OpenAI Whisper API
• Android Developer Documentation: Memory Management
• Qwen GitHub Repository
• Qualcomm AI Hub
• Android Graphics Documentation
• NIST Speech Tools