Stop LLM Outages and Cost Surprises: why system design matters
Hitesh Sondhi · July 13, 2026 · 9 min read
Your LLM feature works in staging. You deploy on a Friday. By Monday morning, the billing dashboard shows thousands of dollars in API costs, the p99 latency has climbed past ten seconds, and three enterprise customers have filed tickets because the assistant started returning empty responses. The model didn't change. The prompts didn't change. What broke was the system around the model.
This is why system design matters more than model selection. Teams spend weeks evaluating base models, fine-tuning strategies, and prompt engineering, then ship the whole thing behind a single API call with no routing, no caching, no fallback, and no observability. The model is one component. The architecture around it determines whether you survive production.
The Problem Isn't the Model, It's the Missing Layer
Traditional software systems have well-understood patterns for load balancing, caching, retries, and circuit breaking. When you move to LLM-based features, those patterns still apply but they need adaptation for non-deterministic outputs, variable token costs, and latency that swings wildly based on context length. A short request and a long one hit the same endpoint but behave like completely different workloads.
The core argument for treating LLM system design as a distinct discipline is that orchestration, context management, and failure modes don't map cleanly to traditional request-response architectures (Dev.to). We agree, and want to push it further with specific patterns we've deployed in production.
Route Requests Before They Touch the Model
The cheapest model that can handle a request is the one you should use. But most teams wire everything to a frontier model and call it done. That works until you see the invoice.
Prompt routing is the first architectural decision that pays for itself. You classify incoming requests by complexity and route them to different model tiers. A simple FAQ lookup doesn't need a frontier-scale model. A nuanced legal clause comparison probably does.
Here's a routing pattern we use:
flowchart TD
A[User Request] --> B{Complexity Classifier}
B -->|Simple| C[Small Model]
B -->|Medium| D[Mid Model]
B -->|Complex| E[Large Model]
C --> F[Response Cache Check]
D --> F
E --> F
F -->|Hit| G[Return Cached Response]
F -->|Miss| H[Generate Response]
H --> I[Cache and Return]
The classifier itself can be a small, fast model or even a rule-based system. We've used Qwen3-8B for classification on edge devices with under fifty milliseconds of added latency, which means the routing decision is nearly free in the request path. If you're building on-device AI, this pattern is even more valuable because you can handle most requests locally and only escalate to cloud models for the hard cases. See our work on [/services/on-device-ai] for more on that split.
The practical implication: measure the distribution of request complexity in your traffic. If most of your requests are simple, you're overspending by an order of magnitude on those requests if they're all hitting a frontier model.
Context Windows Are a Cost and Latency Problem
Every token you send to the model costs money and adds latency. Most teams treat context windows like a buffer they should fill. They shouldn't.
The instinct to stuff the prompt with everything the model might need is understandable but wrong. A large context window doesn't mean you should use all of it. Input token pricing scales linearly with context length on most providers (OpenAI), and latency follows a similar curve. We've seen requests go from under a second to more than ten seconds when context grows from roughly a thousand tokens to tens of thousands on the same model.
Context management means being deliberate about what goes in. Retrieve only relevant chunks. Summarize conversation history instead of replaying it. Trim system prompts to what's actually necessary. If you're building agents that maintain state across turns, compress earlier interactions into a running summary rather than appending every message.
For RAG pipelines specifically, the retrieval step should be doing the heavy lifting, not the model's context window. If you're sending a massive block of retrieved documents to the model and hoping it sorts through them, your retrieval is underperforming. Better retrieval means smaller context means lower cost and lower latency.
What Cost Controls Actually Look Like
Cost control in LLM systems isn't one lever. It's a stack of decisions that compound.
Caching is the highest-leverage one. If your LLM feature answers questions, a significant portion of queries are repeats or near-repeats. Semantic caching, which matches on embedding similarity rather than exact string match, can catch a substantial fraction of redundant requests in typical support and FAQ workloads. We've seen cache hit rates around a third of all traffic in production, which directly cuts API spend for those requests.
Token budgeting is the second lever. Set hard limits on input and output tokens per request, per user, and per tenant. Without caps, a single user sending very long requests can spike your costs and degrade performance for everyone else. We recommend per-tenant token budgets enforced at the gateway level, not at the application level.
The third lever is model tiering, which we covered under routing. If you want a quick estimate of what different architectures cost for your workload, our [/tools/ai-cost-estimator] can model the scenarios before you commit.
Here's how these controls fit together:
flowchart TD
A[Incoming Request] --> B[Rate Limiter]
B --> C[Token Budget Check]
C -->|Over Budget| D[Reject or Downgrade]
C -->|Within Budget| E[Semantic Cache]
E -->|Hit| F[Return Cached]
E -->|Miss| G[Route to Model Tier]
G --> H[Generate]
H --> I[Store in Cache]
I --> J[Log Cost and Latency]
The key metric to track here is cost per successful response, not cost per request. A request that returns a hallucinated answer and gets retried costs you twice. Quality and cost are coupled in ways that traditional systems don't deal with.
Observability for Non-Deterministic Systems
Standard APM tools tell you latency, error rate, and throughput. They don't tell you whether your model's outputs are degrading over time, whether certain prompt patterns trigger failures, or whether your routing classifier is misrouting complex queries to a small model.
LLM observability needs three layers.
The infrastructure layer is what your existing tools handle: p50 and p99 latency, error rates, token throughput. This catches the "is the API up" question.
The model layer tracks output quality metrics. Are response lengths growing over time, which is a sign of prompt drift? Are refusal rates climbing? Are certain input patterns producing shorter or lower-quality outputs? You need to log the full request-response pair, not just metadata, to answer these questions. That means storing prompts and completions, which has its own cost and privacy implications.
The business layer connects model behavior to user outcomes. Did the user accept the response? Did they retry? Did they escalate to a human? These signals tell you whether the model is actually doing its job, which is a different question from whether it's running without errors.
We log every LLM interaction with a correlation ID that ties it to the user session, the model tier used, token counts, latency, and a quality score where we can compute one. For our [/products/runhotel] voice assistant, we also log audio duration and transcription confidence because the LLM is downstream of a speech-to-text step that introduces its own failure modes. If you're building voice AI, that pipeline observability is essential: [/services/voice-ai].
An Architecture That Survives Production
Putting it together, a production LLM system needs these components in order: a gateway that handles auth, rate limiting, and token budgeting; a router that classifies request complexity and selects a model tier; a cache layer that handles both exact and semantic matching; the model endpoints themselves, which may include on-device models, self-hosted models, and API providers; an observability pipeline that logs every interaction; and a feedback loop that routes quality signals back into routing and retrieval improvements.
This isn't over-engineering. Each component addresses a failure mode we've seen in production. The gateway prevents cost spirals from runaway tenants. The router prevents overspending on simple requests. The cache prevents redundant compute. The observability pipeline prevents silent quality degradation. The feedback loop prevents the system from getting worse over time as user behavior shifts.
If you're early in your LLM adoption, you don't need all of this on day one. Start with routing and caching, add observability before you scale users, and add token budgeting before you open the system to multiple tenants. The order matters because cost surprises hit faster than quality surprises, and quality surprises are easier to debug with observability already in place.
For teams that want to skip the painful learning curve, we do this kind of architecture design as part of our [/services/ai-consulting] work, and we build production systems through [/services/custom-models] and [/services/ai-agents]. You can also reach us at [/contact] if you want to talk through a specific system you're building.
Back to Monday Morning
That invoice and those multi-second latencies we opened with? The root cause was a single API call to a frontier model with no routing, no caching, and no token limits. The team was paying frontier-model prices for every request, including the majority that could've been handled by a smaller model. They were re-generating identical responses because there was no cache. And they were letting users send unbounded context, which pushed latency past what any user would tolerate.
The fix wasn't a different model. It wasn't a better prompt. It was the system around the model: a router that sent simple queries to a small model, a semantic cache that caught repeat requests, and a gateway that capped token budgets per tenant. The invoice dropped by roughly eighty percent. Latency settled at just over a second for cached responses and under three seconds for cache misses. The tickets stopped.
The next step was adding observability so they'd catch the next drift before their customers did.
