Android Bench for LLMs: What Mobile Teams Must Change to Ship On-Device Models
Hitesh Sondhi · October 21, 2026 · 6 min read
Your API contract for an on-device LLM looks nothing like the one you wrote for your cloud inference endpoint. There's no GPU pool to scale, no latency SLA to negotiate with a vendor, and no option to swap in a bigger model when the current one feels slow. You get one shot at the right model, one quantization level, and one device thermal envelope.
Google's Android Bench, released this past July, is the first benchmarking framework built specifically to measure how LLMs perform on Android hardware. For teams evolving how llms are selected and deployed, it exposes a gap that server-side benchmarks have been hiding. You can find the full methodology in Google's announcement.
Why Server-Side Benchmarks Mislead Mobile Teams
MMLU scores tell you how well a model reasons. They don't tell you whether it fits in memory alongside your app's existing allocation, or whether sustained inference triggers thermal throttling on a mid-range device. Benchmarks that dominate model selection on HuggingFace assume a world where you have abundant VRAM and active cooling.
On mobile, you're working with shared memory between the CPU, GPU, and NPU, passive cooling, and a battery that drains faster under compute-heavy workloads. A model's accuracy score becomes almost irrelevant if it can't sustain generation speed for more than thirty seconds without the system throttling it down. We've seen this firsthand building RunHotel, our on-device voice AI for hotels: the model that scored best on paper was the first one we had to drop because it couldn't maintain response latency under sustained load.
What Android Bench Measures That You Were Missing
The framework runs models through a standardized pipeline on actual Pixel and Galaxy hardware, not simulated environments. It captures time-to-first-token, generation throughput, peak memory allocation, and thermal throttling behavior under sustained load. Full metric specification details are in Google's Android Developers blog post.
One critical insight from the framework is that memory allocation patterns matter as much as raw throughput. A model that generates tokens quickly but spikes memory unpredictably will trigger the Android low-memory killer, crashing your app mid-inference. Its sustained-load tests expose this behavior, which single-prompt benchmarks completely miss.
The Quantization Decision Is Not Obvious
Most mobile teams default to 4-bit quantization because it produces the smallest model file. Android Bench suggests this is often the wrong starting assumption.
It measures quality degradation alongside performance, and the tradeoff curve isn't linear. Dropping from 8-bit to 4-bit quantization can cut memory usage by roughly half, but the perplexity increase on smaller models is disproportionately large relative to the memory you save, according to HuggingFace's quantization documentation. You're trading a significant memory saving for a quality hit that may render the model unusable for tasks requiring nuance.
Your best approach is to benchmark your actual workload across multiple quantization levels and pick the knee of the curve. For our custom models work, we've found that 8-bit quantization on a 3B parameter model often outperforms 4-bit on a 7B model in real-world mobile conditions, because the smaller model sustains throughput without thermal throttling. Android Bench data supports this pattern, as documented in Google's benchmark results.
Memory Pressure Changes Your Model Architecture Choice
When you're selecting a model for cloud deployment, you think about parameter count and context window. On Android, your binding constraint is peak RSS during inference. A model with a smaller parameter count but larger context window can actually use more memory at runtime than a larger model with a short context window, because the KV cache grows linearly with sequence length.
This means model selection for mobile isn't just about picking the smallest model. You need to understand your actual input length distribution. If your app processes short prompts like voice commands or form autofill, a larger model with a constrained context window may fit better than a smaller model designed for long-context tasks. Android Bench metrics include memory profiling under different context lengths, which makes this tradeoff visible for the first time.
What to Change in Your Selection Process
Stop reading HuggingFace leaderboards for mobile model selection. They measure the wrong things. Instead, build your evaluation pipeline around three Android Bench-aligned metrics: sustained throughput under thermal load, peak memory allocation at your expected context length, and quality at your target quantization level.
If you're evolving how llms are integrated into your mobile app, the selection process should look like this: shortlist three models that fit your accuracy floor, benchmark all three on your target device with your actual prompts, and pick based on sustained performance rather than peak. Use our AI cost estimator to model the inference economics before you commit to a model size.
For teams building voice AI or AI agents that run on-device, the implication is clear: your model selection criteria need to change entirely. Cloud benchmark scores are noise on mobile. Only numbers measured on the device your users carry in their pocket matter.
The Integration Problem Nobody Talks About
Android Bench gives you the data. It doesn't solve the integration problem. Once you've selected a model, you still need to wire it into your app's lifecycle, handle the NPU delegation, manage memory during backgrounding, and deal with the fact that your model competes with every other process on the device for compute.
We've spent months on this integration work, and the benchmark is the easy part. What's hard is making the model behave well inside an app that also needs to render UI, handle touch events, and stay responsive. If you're hitting this wall, talk to us. We've been through it enough times to know which corners you can cut and which ones will bite you in production.
Download Android Bench, pick your top two candidate models, and run them against your actual prompt distribution on a mid-range Pixel device this week. Compare sustained throughput and peak memory, then pick the model that holds up under thermal load.
