Cropsly
Abstract smartphone silhouette with circuit lines, a battery meter, coins and a dollar sign on a balancing scale
← Back to BlogEdge AI

How to Scope On-Device Mobile AI Without Draining Battery or Budget

Shivam Kapoor · November 13, 2026 · 8 min read

You have a mobile app that needs offline transcription. Someone on the team says "just ship Whisper on-device." Six weeks later, the model eats over a gigabyte of storage, drains nearly a fifth of the battery per session, and mid-range Android phones crash on out-of-memory before the first token decodes. This is what happens when you don't scope on-device mobile AI features before you build them.

On-device AI isn't a checkbox you flip. Every feature you move to the edge trades against three hard constraints: memory, thermals, and battery. Get the scoping wrong and you ship a feature that works on three test devices and fails on everything your users actually carry.

Key Takeaways

  • Start with a hardware budget, not a model choice. Know your floor device's RAM, NPU throughput, and thermal envelope before picking architecture.
  • Quantization is mandatory, not optional. INT8 or INT4 is the difference between shipping and stalling.
  • Measure battery drain on real mid-range devices, not just flagships. What passes on a Pixel 9 Pro can double on a Galaxy A15.
  • Feature scope should be narrower on-device than in the cloud. One intent, one input modality, one output format.

Start With the Floor, Not the Ceiling

Most teams scope on-device AI against their dev phone. That's usually an iPhone 15 Pro or a Pixel 8, both with dedicated NPUs and generous memory. Your users are on something worse.

The first decision in any on-device AI project is: what's the minimum device we support? That answer defines everything downstream. A phone with limited RAM and no NPU can't run a seven-billion-parameter model. It doesn't matter how clever your quantization scheme is. A 7B model at INT4 still needs roughly 3.5 GB just for weights, which exceeds the usable app memory on most mid-range Android devices Hugging Face.

We've found that defining a "floor device" early prevents most late-stage rework. Pick the cheapest, lowest-spec device in your target market. If your feature works there, it works everywhere above it.

For Android, the Galaxy A series and Redmi Note series represent the actual global mid-range. Qualcomm's Snapdragon 6 Gen 1, common in these devices, delivers about 12 TOPS of AI throughput, compared to 45 TOPS on a Snapdragon 8 Gen 3 flagship Qualcomm. That gap is your reality.

On iOS, the iPhone 12 remains common in enterprise fleets. Its A14 Neural Engine handles 11 TOPS, while the A17 Pro pushes 35 TOPS Apple. Plan for the A14, not the A17.

Pick the Narrowest Feature That Ships Value

When you scope on-device mobile AI features, the temptation is to replicate the cloud experience. Don't. On-device means narrower scope, not the same scope with worse performance.

A cloud-based assistant might handle dozens of intents across text and voice. Your on-device version should handle five. Maybe three. The goal isn't feature parity, it's offline resilience for the interactions that matter most.

We ran into this building RunHotel, our on-device voice AI for hotels. The cloud version handles concierge questions, booking modifications, room service, and multi-turn dialogue. The on-device version does exactly one thing: wake word detection and basic room commands offline. Narrow scope, reliable execution, response times measured in tens of milliseconds.

The same principle applies to any on-device feature. Define the single highest-value interaction that must work without connectivity. Build that. Resist scope creep.

Memory Is Your Real Constraint

People talk about compute. They should talk about memory. A model that fits in compute but doesn't fit in RAM is worse than useless, it's a crash waiting for your next App Store review.

Your memory budget isn't the device's total RAM. It's the RAM available to your app after the OS and other processes take their cut. On a device with 4 GB of RAM, your app might get 300 to 400 MB before the system starts killing background processes Android Developers. On iOS, Apple's jetsam policy will terminate your app well before you hit the theoretical memory limit Apple Developer.

Here's how we think about the memory budget. Model weights (post-quantization) get 50 to 60% of your app's memory ceiling. KV cache and activation buffers take 20 to 25%. App overhead and UI need 15 to 20%. Leave a 10% safety margin. These ratios come from profiling mobile inference workloads across device tiers TensorFlow Lite.

If your floor device gives you 350 MB of usable RAM, your model weights need to land under 200 MB. A 4-bit quantized model in the one to two billion parameter range fits. A 3B model doesn't. That math is unforgiving Hugging Face.

Quantization Changes Everything

INT8 quantization roughly halves model size with minimal quality loss for most tasks. INT4 goes further, cutting size by 75% but with measurable degradation on complex reasoning NVIDIA.

For on-device mobile AI, we default to INT4 for models above a billion parameters and INT8 for smaller models. The quality tradeoff is acceptable for narrow, well-scoped features. It's unacceptable for general-purpose chat, which is another reason to keep your feature scope tight.

If you need custom quantization or model compression that goes beyond off-the-shelf tools, our /services/custom-models work covers exactly this. But try the standard path first: export to ONNX or TFLite, apply post-training quantization, profile on your floor device.

Battery and Thermals Will Kill Your Feature Quietly

A feature that drains significant battery per session will get uninstalled. Users don't read your release notes about "on-device inference." They just notice their phone is hot and their battery is dead by lunch.

Measure battery drain on real devices, not emulators. Run your feature for five minutes on your floor device and check the battery delta. If you're losing more than a couple percent per minute of active inference, you need to rethink your approach.

Thermal throttling is the silent killer. Sustained NPU usage causes the device to heat up, which triggers frequency scaling, which drops your throughput by 30 to 50% within minutes. Your benchmark of "fast first token" becomes "slow after 90 seconds of use" and nobody knows why. Arm documents this behavior thoroughly for mobile SoCs Arm.

What Belongs On-Device vs. in the Cloud

Not every AI feature belongs on the device. Keep on-device: wake word detection, offline transcription, on-device search, PII-sensitive inference, latency-critical responses, and features that must work without connectivity.

Keep in the cloud: multi-turn reasoning, large-context summarization, anything requiring a large model, features with frequent model updates, and tasks where you need full observability of inputs and outputs.

Hybrid is usually the right answer. Wake word on-device, intent recognition in the cloud. Local transcription with cloud-based correction. The split depends on your latency budget, privacy requirements, and connectivity assumptions. Our /services/on-device-ai and /services/ai-agents pages cover both sides of that split.

Cloud API latency for a simple LLM call typically lands between 800ms and 2000ms for first token Hugging Face. On-device inference can beat that by an order of magnitude, but only if you've scoped the feature correctly. Your on-device latency targets should be under 100ms for wake word detection, under 300ms for single-turn classification, and under 500ms for short generative responses. If you can't hit those numbers on your floor device, narrow the feature or move to hybrid.

If you're unsure where to draw the line, /services/ai-consulting can help you map the tradeoffs. You can also try our /tools/ai-cost-estimator to see what the cloud-side costs look like before you commit to an architecture. For voice-specific features, our /services/voice-ai work covers the full pipeline from wake word to response.

Five Questions Before You Write Inference Code

Before you build, answer these: What's your floor device and its RAM, NPU, and thermal profile? What's the single highest-value interaction this feature enables? What model size fits your memory budget at INT8 and INT4? What's your battery drain budget per session? Does this feature need to work fully offline, or is hybrid acceptable?

If you can't answer all five, you're not ready to build. If you can, /contact us and we'll help you validate the architecture before you spend engineering cycles on it.

This week, profile your floor device's available RAM using Android Studio's Memory Profiler or Xcode's Instruments, then write down the number. That single measurement determines what you can build.

Sources

ShareTwitterLinkedIn
aiengineering

Want AI that works offline, on your hardware?

On-device models and voice interfaces with real latency numbers. RunHotel runs ours in production hotels.

Get Weekly AI Insights

Join founders and CTOs getting our AI engineering newsletter.

By subscribing, you agree to our Privacy Policy. Unsubscribe anytime.