Put the Neural Engine to work: why the vision tower should not run on the GPU
2026-09-01
English · 中文版
Every Apple Silicon Mac contains two accelerators: a GPU and the Apple Neural Engine (ANE). On the M5 Pro, the latter has 16 cores and roughly 42 TOPS of int8 compute. Most local inference frameworks still make the same choice: put everything on the GPU and leave the ANE idle from start to finish.
For a text-only model, that choice is reasonable. LLM decoding is autoregressive, memory-bandwidth bound, and coupled to a dynamic KV cache. It is not a natural ANE workload. Vision-language models are different. Their extra component, the vision tower (ViT), is almost exactly the kind of computation the ANE wants.
A vision tower is one static forward pass
An image is split into patches, passed through 27 transformer layers, and converted into a sequence of visual embeddings. Then it is done. There is no autoregression, no KV cache, and no sequence that grows one token at a time. It is one large forward pass with a fixed shape, which Core ML can compile deeply and the ANE can execute efficiently.
On an M5 Pro, for a tower in the ViT-B/16 class:
| Latency | Energy | |
|---|---|---|
| GPU (MPS) | 15.9 ms | 612 mJ |
| ANE | 18.3 ms | 75 mJ |
Latency is nearly the same, while energy per image differs by 8×. The ANE uses about 2 W for the tower; the GPU uses about 8 W for the same work. On a desktop that becomes heat and fan noise. On a MacBook it becomes battery life.
The larger win is parallelism, not power
Energy is the single-image account. Serving has a different ledger: while the vision tower runs on the ANE, the GPU does not have to stop.
In Tempo9, an incoming request runs its chat template and tokenizer on the CPU, then sends the image through the tower on the ANE. During that entire interval, the GPU can decode another request. Once the visual embeddings are ready, the request joins the continuous batch and shares a forward pass with the decodes already in flight.
An agent repeatedly looking at screenshots, several sessions carrying different images, or video arriving one frame at a time all create two naturally overlapping streams: vision preprocessing and language decoding. Queue both on one GPU and that overlap disappears. Put them on separate processors and it is available by construction.
There is a third benefit that matters most on a 24 GB machine: the tower's weights do not consume the GPU wired-memory budget. When a 35B model is already near the limit, where a few hundred megabytes of tower weights live can decide whether another request's KV cache fits.
Fitting dynamic resolution into a static compiler
The complication is that Qwen-family vision towers use NaViT-style dynamic resolution. The number of patches changes with the input image, while Core ML wants shapes fixed at compile time.
We solve that with buckets. Each model is exported for a fixed patch budget. An image is scaled proportionally into the nearest bucket, padded, and masked with an additive mask. The release uses one small and one large bucket:
| Bucket | Pixel budget | Typical fit | LLM tokens |
|---|---|---|---|
| b1024 | 260K px | 683×384 / 512×512 | ≤ 256 |
| b4096 | 1.05M px | 1365×768 / 1024×1024 | ≤ 1024 |
Why two buckets instead of a ladder? Each bucket is an independently compiled model occupying several hundred megabytes on disk. More buckets make the download larger while the intermediate gains diminish. The small bucket serves thumbnails and agent screenshot streams; the large one serves documents and photos.
Moving processors must not change the answer
The largest risk in moving computation from GPU to ANE is not that it becomes slow. It is that it becomes quietly wrong. Core ML uses fp16 by default, its numeric path is different, and the tower output directly determines what the language model can see.
Every exported tower therefore passes three gates before release: residency inspection to confirm it really runs on the ANE rather than silently falling back to the CPU; per-bucket comparison with an fp32 reference (all current buckets have cosine similarity at least 0.98, with measured minimum 1.000000); and an end-to-end VQA smoke test. If any gate fails, the tower does not ship. A faster tower that sees the wrong image is not an optimization. It is a defect.
Closing
The silicon is already paid for. Vision on the ANE, language on the GPU, and scheduling on the CPU means all three parts of the chip are doing useful work at once. That is what Apple Silicon should mean in practice.