Edge AI · On-Device ML · iOS
Edge AI on iOS: Why On-Device Beats the Cloud
Edge AI means running machine-learning inference where the data is created — on the phone, the watch, the camera — instead of shipping that data to a server. On iOS, that's not a compromise you settle for; for a huge class of features it's the better architecture. Here's the case, and how I reason about it in production.
What "edge AI" actually means
The "edge" is the device at the edge of the network. Edge AI is the practice of doing the inference there rather than in a data center. On Apple platforms you have three compute units to target — CPU, GPU and the Neural Engine — sharing one unified memory pool, plus two mature software paths: Core ML for ahead-of-time-compiled models and MLX for flexible, generative workloads. Together they make on-device ML genuinely practical, not just possible.
The four reasons it wins
1. Privacy
When inference runs on-device, user data never leaves the phone. There's no server log, no transit, no third-party processor. For health, finance, messaging and anything regulated, that's not a nice-to-have — it's often the only acceptable design. It also makes your privacy nutrition label honest.
2. Latency
A cloud call is a network round-trip plus queueing plus model time. On-device inference is just the model time, and on Apple silicon that's frequently single-digit milliseconds for vision and audio models. Features that need to feel instant — live camera effects, autocomplete, haptic-tight interactions — can only be built locally.
3. Cost
Cloud inference is a recurring per-request bill that scales with your success. On-device inference runs on hardware the user already paid for. At the scale of an app installed on millions of phones, moving inference to the edge can erase a six-figure server line item.
4. Offline reliability
Planes, subways, rural areas, dead zones. An on-device model works everywhere, every time, with deterministic behavior you control — no dependency on an upstream API's uptime or rate limits.
When the cloud still wins
Edge AI isn't a religion. Send it to the cloud when: the model is too large to quantize onto a phone; you need frequent retraining on aggregated data; or the task genuinely benefits from a frontier-scale model that won't fit in a 1–2 GB budget. A common pattern is hybrid: a small on-device model handles the common, latency-sensitive path and the cloud handles the rare, heavy one.
A decision checklist
- Is the data sensitive? → lean on-device.
- Does it need to feel instant? → on-device.
- Must it work offline? → on-device.
- Can the model quantize under ~2 GB? → on-device is viable (see running LLMs on iPhone with MLX).
- Do you need a frontier-scale model or constant retraining? → cloud, or hybrid.
Picking a stack
Once you've decided to stay on-device, the next fork is Core ML vs MLX. Core ML is the path of least resistance for vision/audio and ahead-of-time models with first-class Neural Engine support; MLX is the better tool for generative LLMs and anything that wants dynamic shapes and runtime flexibility. I break the choice down in MLX vs Core ML: choosing an on-device inference stack.
The right question isn't "cloud or edge?" — it's "what's the smallest model that can run where the data already lives?"