LLM Optimization
LLM Optimization for Mobile
Deploying language models on phones, tablets, and edge devices—TFLite, CoreML, quantization for mobile hardware, and the future of on-device AI.
- Frameworks — TFLite, CoreML, ONNX Runtime, MLC-LLM
- Hardware — NPUs, GPUs, and neural accelerators in mobile chips
- Techniques — INT4/INT8, pruning, speculative decoding for mobile
The future of AI is not in the cloud—it's in your pocket.
LLM Optimization for Mobile
Deploying language models on phones, tablets, and edge devices—TFLite, CoreML, quantization for mobile hardware, and the future of on-device AI.
Mobile Hardware Constraints
Resource Budgets
| Device Class | RAM | Power | Compute (TOPS) | Example |
|---|---|---|---|---|
| Budget phone | 2-4 GB | 1-2W | 1-2 | Snapdragon 6-series |
| Flagship phone | 8-12 GB | 3-5W | 10-15 | Snapdragon 8 Gen 3 |
| Tablet | 8-16 GB | 5-10W | 15-25 | Apple M2 iPad |
| Edge device | 4-16 GB | 5-20W | 20-50 | Jetson Orin |
Mobile Neural Processing Units
Modern mobile SoCs include dedicated NPUs:
| Chip | NPU | TOPS | INT8 Support |
|---|---|---|---|
| Snapdragon 8 Gen 3 | Hexagon | 45 | Yes |
| Apple A17 Pro | Neural Engine | 35 | Yes |
| MediaTek Dimensity 9300 | APU 790 | 46 | Yes |
| Google Tensor G4 | TPU | 15 | Yes |
Framework Comparison
TensorFlow Lite (TFLite)
Key features:
- Delegates: GPU, NNAPI, Core ML, Hexagon DSP
- Quantization: Post-training INT8, dynamic range quantization
- Optimization: Operator fusion, constant folding, pruning
- Deployment: Android, iOS, embedded Linux, microcontrollers
CoreML (Apple)
Key features:
- Automatic optimization: Neural Engine, GPU, CPU selection
- Quantization: INT8, FP16, and mixed-precision
- Tools: coremltools for conversion from PyTorch/TensorFlow
- Privacy: All computation on-device, no data leaves the device
MLC-LLM
Key features:
- Universal deployment: Android, iOS, web, GPUs, NPUs
- INT4 quantization: GPTQ and AWQ support
- Speculative decoding: For faster on-device generation
- Vulkan/Metal acceleration: Cross-platform GPU support
Quantization for Mobile
INT8 Quantization
The standard for mobile deployment:
INT4 Quantization
For maximum compression on memory-constrained devices:
| Quantization | Model Size (7B) | Memory | Quality (PPL) |
|---|---|---|---|
| FP16 | 14 GB | 16 GB | Baseline |
| INT8 | 7 GB | 8 GB | +0.1-0.3 |
| INT4 (GPTQ) | 3.5 GB | 4 GB | +0.5-1.5 |
| INT4 (AWQ) | 3.5 GB | 4 GB | +0.3-1.0 |
On-Device Inference Optimization
Speculative Decoding for Mobile
Use a small draft model on-device to accelerate generation:
On mobile, the draft model is typically a 100-500M parameter model running on the NPU, while the target model is a 1-7B model.
KV Cache Optimization
Mobile devices have limited memory for KV cache:
For a 7B model with 32 layers, 32 heads, d=128, and T=2048:
- FP16 KV cache: 2 × 32 × 32 × 128 × 2048 × 2 = 1.07 GB
- INT8 KV cache: 536 MB
Practical Deployment Guide
Model Selection for Mobile
| Use Case | Model Size | Quantization | Framework |
|---|---|---|---|
| Text completion | 1-3B | INT4 | TFLite/CoreML |
| Chat assistant | 3-7B | INT4 | MLC-LLM |
| On-device RAG | 1-3B + embeddings | INT8 | TFLite |
| Voice assistant | 1-3B + ASR/TTS | INT4 | CoreML |
Performance Benchmarks
| Model | Device | Framework | Tokens/sec | Memory |
|---|---|---|---|---|
| LLaMA 2 7B (INT4) | Snapdragon 8 Gen 3 | MLC-LLM | 12 | 4.2 GB |
| Phi-2 2.7B (INT4) | iPhone 15 Pro | CoreML | 18 | 1.8 GB |
| Gemma 2B (INT4) | Pixel 8 | TFLite | 15 | 1.5 GB |
| LLaMA 3 8B (INT4) | Snapdragon 8 Gen 3 | MLC-LLM | 10 | 5.1 GB |
Practice Exercises
-
Conceptual: Explain why INT4 quantization is preferred for mobile deployment over INT8. What are the tradeoffs in terms of quality, memory, and speed?
-
Mathematical: Compute the KV cache memory for a 3B model with 24 layers, 16 heads, d=64, and T=1024 using INT8 quantization. Can this fit in 512MB of available memory?
-
Practical: Convert a small LLM (e.g., Phi-2 2.7B) to TFLite format with INT4 quantization and benchmark inference speed on an Android device.
-
Research: Compare the energy efficiency (tokens per joule) of on-device inference vs. cloud inference for a 3B model. Under what conditions is on-device more efficient?
What to Learn Next
-> Model Compression Pipeline End-to-end compression: quantization + pruning + distillation.
-> Quantization Techniques Deep Dive Detailed quantization methods and theory.
-> Hardware-Aware LLM Design Co-designing models and hardware for efficiency.
-> LLM Inference Optimization General inference optimization techniques.
-> Pruning for LLMs Structured and unstructured pruning methods.
-> Environmental Impact Energy efficiency and sustainable AI practices.