ThinkSpread

投机解码的经济学:同一个功能,9B 上 +41%,35B 上打平

2026-09-02T00:00:00+00:00

投机解码(我们用的是 MTP:模型自带的 next-n 草稿头)的原理一句话能讲完: 小代价猜 k 个 token,大模型一次 verify,猜中几个收几个。但"到底赚不赚" 取决于一个很少被拆开算的量:verify 的边际成本。

我们把它测出来了。同会话四个 k 值扫描,48 轮累计:

verify(m) ≈ 9 ms + 5.5 ms × m        (m = k+1 行)

固定项 9ms ≈ 一个普通 decode 步。为什么多 verify 一行只要 5.5ms、而不是再付一遍全部权重?因为 m 行 verify 走的是 batch GEMV:权重从显存读一次,m 行共享。稠密部分的带宽账在 m∈[2,8] 时几乎是免费的。

架构决定赚不赚

那 5.5ms 的边际里藏着关键:对 MoE 模型,每行激活的专家不同,专家权重不可共享 —— 每多一行就多读约 0.46GB 专家,约 3.2ms。这一项稠密模型没有。

后果直接写在结果里。同一套 MTP 实现:

模型	架构	结论
Qwen3.5-35B-A3B	MoE	accept ≈ 0.8 才打平,上限也只有 ~1.19×
Qwen3.5-9B	稠密	edit +41% · code +24% · chat +15% · en +6%,无负项

同一个功能,一边是"约等于不赚",一边是产品级收益 —— 差别就是每行边际里有没有那 3.2ms 的专家项。投机解码的性价比不是实现细节,是架构属性。

深度 k 的甜点:扫出来,不是猜出来

9B 上把 k 从 1 扫到 6(单并发,edit/en/code 三负载,基线 45.1/45.7/45.8 tok/s):

k=1   53.3 / 49.3 / 53.5
k=2   56.3 / 47.2 / 53.0
k=3   63.3 / 48.5 / 56.6    ← 合计最优,服务定值
k=4   65.1 / 46.5 / 48.4    (只利 edit,code 塌)
k=5+  全面衰退

acceptance 随深度单调下降(英文散文 k=1 时 0.85,k=3 时 0.59):猜得越深, 后面的 token 越接近瞎猜。不同负载的最优 k 不同 —— 散文偏浅、代码和编辑撑得住深 —— 但 k=3 是全局最优的固定值,这就是服务的默认。

门控:让它只在赚的地方开

最有意思的设计不是内核,是一个阈值:滚动 acceptance 低于 0.60 就自动退回普通解码(AS_SPEC_MIN_ACCEPT=0.60)。

这个门的效果超出它的本意。agent 负载 —— 改文件、发工具调用、生成 JSON —— acceptance 天然高(结构化文本可预测,35B 上 mid-edit 实测 0.92);中文散文天然低(0.28-0.5)。于是一个简单的阈值自动实现了"编辑场景开、散文场景关"的正确形态,不需要负载分类器,不需要用户配置。

35B 上它兜住下限:散文触发门、退回纯解码,mid-edit 负载 91 → 97.6 tok/s (+7%)照拿。9B 上它几乎不触发,全场景放行。

一笔要诚实写出来的账:快,但不省电

自制的 IOReport 采样器(免 root)量了解码窗的功耗,9B 英文思考负载:

	吞吐	功率	每 token 能量
k=0	45.8 tok/s	51.7 W	1.14 J
k=3	52.3 tok/s	64.3 W	1.26 J (+10%)

verify 的批量计算抬高了瞬时占用 —— MTP 用能量换时间,不是免费午餐。所以 App 形态里它对应两个档位:性能档 k=3,省电档 k=0。插电的 Mac mini 和电池上的 MacBook,答案本来就该不同。

收尾

一个功能从"听起来很快"到"知道什么时候开",中间隔着:一个成本模型 (9 + 5.5m)、一条架构分界线(专家可不可共享)、一次 k 扫描(甜点在 3)、一个自动门(0.60),和一笔功耗账(+10% J/tok)。这些都测完之后, "默认值该是什么"就不再是观点,是查表。

The economics of speculative decoding: +41% on 9B, break-even on 35B

2026-09-02T00:00:00+00:00

Speculative decoding can be summarized in one sentence. A cheap draft head guesses k tokens, the full model verifies them in one pass, and the runtime keeps however many were correct. We use MTP, the model's own next-n prediction heads. Whether this actually saves time depends on a quantity that is rarely broken out: the marginal cost of verification.

We measured it across four values of k, over 48 rounds in one session:

verify(m) ≈ 9 ms + 5.5 ms × m        (m = k+1 rows)

The fixed 9 ms is roughly one ordinary decode step. Why does each additional verification row cost only 5.5 ms instead of rereading every weight? Because the m rows run as a batched GEMV: the weights are read once and shared by the rows. For the dense part of the model, extra rows are nearly free over m ∈ [2,8].

Architecture decides whether it pays

The important part is hidden inside that 5.5 ms marginal cost. In a MoE model, each row can activate different experts, so their weights cannot be shared. Every additional row reads about 0.46 GB of expert weights, costing roughly 3.2 ms. Dense models do not have that term.

The consequence is visible in the results. This is the same MTP implementation on both models:

Model	Architecture	Result
Qwen3.5-35B-A3B	MoE	Break-even near 0.8 acceptance; ceiling about 1.19×
Qwen3.5-9B	Dense	edit +41% · code +24% · chat +15% · English +6%, no negative workload

The same feature is nearly a wash on one model and a product-level improvement on the other. The difference is the 3.2 ms expert term in every extra row. The economics of speculative decoding are an architectural property, not just an implementation detail.

The sweet spot for k is measured, not guessed

On the 9B model we swept k from 1 through 6 at single concurrency, using edit, English prose, and code workloads. Their baselines were 45.1 / 45.7 / 45.8 tok/s:

k=1   53.3 / 49.3 / 53.5
k=2   56.3 / 47.2 / 53.0
k=3   63.3 / 48.5 / 56.6    <- best overall; server default
k=4   65.1 / 46.5 / 48.4    (helps edit only; code collapses)
k=5+  regresses across the board

Acceptance falls monotonically with depth. On English prose it goes from 0.85 at k=1 to 0.59 at k=3: the farther ahead the draft predicts, the closer it gets to guessing. Different workloads prefer different depths. Prose prefers a shallow draft, while code and editing tolerate more. k=3 is the best fixed global value, so that is the server default for this model.

Gating: enable it only where it pays

The most interesting design is not a kernel. It is a threshold: when rolling acceptance falls below 0.60, fall back to ordinary decoding automatically (AS_SPEC_MIN_ACCEPT=0.60).

That gate does more than intended. Agent workloads such as editing files, emitting tool calls, and generating JSON are predictable and naturally have high acceptance (0.92 on the 35B mid-edit workload). Chinese prose is naturally low at 0.28-0.5. One threshold therefore produces the right behavior: speculate for editing and structured output; stop for prose, without a workload classifier or user configuration.

On the 35B model the gate protects the floor: prose drops back to ordinary decode, while the mid-edit workload still moves from 91 to 97.6 tok/s (+7%). On the 9B model it rarely fires, so speculation remains enabled across workloads.

One cost that belongs in the ledger: faster, not more efficient

A rootless IOReport sampler measured power during the decode window on the 9B English reasoning workload:

	Throughput	Power	Energy per token
k=0	45.8 tok/s	51.7 W	1.14 J
k=3	52.3 tok/s	64.3 W	1.26 J (+10%)

Batched verification raises instantaneous utilization. MTP trades energy for time; it is not a free lunch. In an application that naturally becomes two modes: k=3 for performance and k=0 for efficiency. A plugged-in Mac mini and a MacBook on battery should not have the same answer.

Closing the ledger

Between "this sounds fast" and "we know when to enable it" are a cost model (9 + 5.5m), an architectural boundary (whether expert weights can be shared), a sweep for k (the sweet spot is 3), an automatic gate (0.60), and a power measurement (+10% J/token). Once those are measured, the default is no longer an opinion. It is a lookup.

让 Neural Engine 干活:视觉塔为什么不该跑在 GPU 上

2026-09-01T00:00:00+00:00

每台 Apple Silicon Mac 里都有两个加速器:GPU,和一颗 Neural Engine(ANE, M5 Pro 上 16 核、约 42 TOPS int8)。跑本地大模型时,几乎所有框架的选择是一样的:所有东西都在 GPU 上,ANE 从头到尾闲着。

对纯文本模型,这个选择没什么可争的 —— LLM 解码是自回归的、访存主导、需要灵活的 KV cache,不是 ANE 的形状。但视觉语言模型不同:它多出来的那块计算 —— 视觉塔(ViT)—— 恰好是 ANE 最擅长的东西。

视觉塔是一次性的静态前向

一张图进来,patchify 之后过 27 层 transformer,输出一串视觉 embedding, 结束。没有自回归、没有 KV cache、没有逐步增长的序列 —— 就是一个形状固定的大前向。这正是 Core ML 编译器能吃透、ANE 能全速跑的负载。

数字说话。ViT-B/16 规模的塔,M5 Pro 实测:

	时延	能量
GPU (MPS)	15.9 ms	612 mJ
ANE	18.3 ms	75 mJ

时延几乎相同,每张图的能量差 8 倍。ANE 跑视觉塔约 2W,GPU 干同样的事约 8W。在插电的台式机上这是温度和风扇;在 MacBook 上这是电池。

但真正的收益不是省电,是并行

省电是单张图的账。serving 场景的账不一样:视觉塔在 ANE 上跑的时候, GPU 一毫秒都不用停。

Tempo9 的流水线是这样的:请求进来,聊天模板和 tokenizer 在 CPU 上跑,图像在 ANE 上过塔 —— 而这整段时间,GPU 在解码别的请求。等视觉 embedding 就绪,它才进入引擎的连续批处理,和在飞的解码共享同一个 forward。

一个 agent 循环反复看同一张截图、多个会话各带各的图、视频逐帧进来 —— 这些负载里,视觉预处理和语言解码是天然可重叠的两条流。用同一个 GPU 排队, 重叠就没了;分到两个处理器上,重叠是白拿的。

还有第三笔账,24GB 的机器上最要紧:塔的权重不占 GPU 的 wired 显存预算。 35B 模型贴着上限跑的时候,几百 MB 的塔权重放在哪一边,决定的是能不能再多放一个并发的 KV。

动态分辨率怎么塞进静态编译器

麻烦在于:Qwen 系的塔是 NaViT 式动态分辨率 —— patch 数随输入图变,而 Core ML 要的是编译时定死的形状。

我们的解法是分桶:按 patch 预算导出固定尺寸的模型,输入图等比缩放到最近的桶,不足的 pad 掉、用 additive mask 屏蔽。发布策略是「一大一小」:

桶	像素预算	典型适配	对应 LLM token
b1024	26 万 px	683×384 / 512×512	≤ 256
b4096	105 万 px	1365×768 / 1024×1024	≤ 1024

为什么只有两个桶、不是一个梯子?每个桶是一个独立编译的模型,磁盘上各占几百 MB —— 桶越多,下载越肥,而中间档位的收益递减。小桶伺候缩略图和 agent 截图流,大桶伺候文档和照片,够了。

换处理器,不能换答案

把计算从 GPU 挪到 ANE,最大的风险不是慢,是悄悄变错:Core ML 默认 fp16,数值路径完全不同,而视觉塔的输出会直接决定模型"看到"什么。

所以每个导出的塔要过三道门才能发布:算子驻留检查(确认真的在 ANE 上跑, 而不是静默回退 CPU)、逐桶和 fp32 参考比对(当前全部桶 cosine ≥ 0.98, 实测 min = 1.000000)、端到端 VQA 冒烟。任何一道不过,这个塔就不发布 —— 一个更快但看错图的塔,不是优化,是缺陷。

收尾

芯片上的硅是已经付过钱的。同一台 Mac,视觉走 ANE、语言走 GPU、调度走 CPU —— 三块硅同时在干活,这才对得起 Apple Silicon 这个名字。

Put the Neural Engine to work: why the vision tower should not run on the GPU

2026-09-01T00:00:00+00:00

Every Apple Silicon Mac contains two accelerators: a GPU and the Apple Neural Engine (ANE). On the M5 Pro, the latter has 16 cores and roughly 42 TOPS of int8 compute. Most local inference frameworks still make the same choice: put everything on the GPU and leave the ANE idle from start to finish.

For a text-only model, that choice is reasonable. LLM decoding is autoregressive, memory-bandwidth bound, and coupled to a dynamic KV cache. It is not a natural ANE workload. Vision-language models are different. Their extra component, the vision tower (ViT), is almost exactly the kind of computation the ANE wants.

A vision tower is one static forward pass

An image is split into patches, passed through 27 transformer layers, and converted into a sequence of visual embeddings. Then it is done. There is no autoregression, no KV cache, and no sequence that grows one token at a time. It is one large forward pass with a fixed shape, which Core ML can compile deeply and the ANE can execute efficiently.

On an M5 Pro, for a tower in the ViT-B/16 class:

	Latency	Energy
GPU (MPS)	15.9 ms	612 mJ
ANE	18.3 ms	75 mJ

Latency is nearly the same, while energy per image differs by 8×. The ANE uses about 2 W for the tower; the GPU uses about 8 W for the same work. On a desktop that becomes heat and fan noise. On a MacBook it becomes battery life.

The larger win is parallelism, not power

Energy is the single-image account. Serving has a different ledger: while the vision tower runs on the ANE, the GPU does not have to stop.

In Tempo9, an incoming request runs its chat template and tokenizer on the CPU, then sends the image through the tower on the ANE. During that entire interval, the GPU can decode another request. Once the visual embeddings are ready, the request joins the continuous batch and shares a forward pass with the decodes already in flight.

An agent repeatedly looking at screenshots, several sessions carrying different images, or video arriving one frame at a time all create two naturally overlapping streams: vision preprocessing and language decoding. Queue both on one GPU and that overlap disappears. Put them on separate processors and it is available by construction.

There is a third benefit that matters most on a 24 GB machine: the tower's weights do not consume the GPU wired-memory budget. When a 35B model is already near the limit, where a few hundred megabytes of tower weights live can decide whether another request's KV cache fits.

Fitting dynamic resolution into a static compiler

The complication is that Qwen-family vision towers use NaViT-style dynamic resolution. The number of patches changes with the input image, while Core ML wants shapes fixed at compile time.

We solve that with buckets. Each model is exported for a fixed patch budget. An image is scaled proportionally into the nearest bucket, padded, and masked with an additive mask. The release uses one small and one large bucket:

Bucket	Pixel budget	Typical fit	LLM tokens
b1024	260K px	683×384 / 512×512	≤ 256
b4096	1.05M px	1365×768 / 1024×1024	≤ 1024

Why two buckets instead of a ladder? Each bucket is an independently compiled model occupying several hundred megabytes on disk. More buckets make the download larger while the intermediate gains diminish. The small bucket serves thumbnails and agent screenshot streams; the large one serves documents and photos.

Moving processors must not change the answer

The largest risk in moving computation from GPU to ANE is not that it becomes slow. It is that it becomes quietly wrong. Core ML uses fp16 by default, its numeric path is different, and the tower output directly determines what the language model can see.

Every exported tower therefore passes three gates before release: residency inspection to confirm it really runs on the ANE rather than silently falling back to the CPU; per-bucket comparison with an fp32 reference (all current buckets have cosine similarity at least 0.98, with measured minimum 1.000000); and an end-to-end VQA smoke test. If any gate fails, the tower does not ship. A faster tower that sees the wrong image is not an optimization. It is a defect.

Closing

The silicon is already paid for. Vision on the ANE, language on the GPU, and scheduling on the CPU means all three parts of the chip are doing useful work at once. That is what Apple Silicon should mean in practice.

API reference

2026-08-31T00:00:00+00:00

One process serves four dialects on one port. Nothing needs to be enabled; they are all live.

OpenAI

Method	Route	Notes
`GET`	`/v1/models`	Reports the one loaded model.
`GET`	`/v1/models/{id}`	Single-model lookup. Answers for any id.
`POST`	`/v1/chat/completions`	Chat. `stream` defaults to false.
`POST`	`/v1/responses`	The Responses API, which Codex speaks.

Anthropic

Method	Route	Notes
`POST`	`/v1/messages`	What Claude Code speaks.
`POST`	`/v1/messages/count_tokens`	A fast estimate. Callers budget with it; nothing decodes from it.

Ollama native

For apps that integrate Ollama and offer no generic endpoint field.

Method	Route	Notes
`GET`	`/api/tags`	Model list. This is the discovery call.
`GET`	`/api/version`	Reports `0.12.0-tempo9` — semver-parseable, and tagged so it never claims to be an Ollama release.
`GET`	`/api/ps`	Loaded models. One, always resident.
`POST`	`/api/show`	Model detail. `capabilities` reflects this build: `tools` always, `vision` only with `--tower`.
`POST`	`/api/chat`	Chat.
`POST`	`/api/generate`	Bare-prompt completion. Replies use `response`, not `message`.
`POST`	`/api/embed`, `/api/embeddings`	501. We serve no embeddings, and say so in the caller's dialect rather than 404ing.

Three things differ from the OpenAI dialect, and each fails silently against a client that assumes otherwise:

Streaming is newline-delimited JSON. No data: prefix, no [DONE] sentinel; the last object carries done: true.
stream defaults to true. The opposite of OpenAI.
Tool-call arguments are a JSON object, not a JSON string.

Authentication

None. See Limits.

CLI reference

2026-08-31T00:00:00+00:00

tempo9 (--gguf <.gguf> | --ollama <model[:tag]>) [options]
tempo9 --list-ollama

Choosing a model

Flag	Default	Meaning
`--gguf <path>`	—	Model file. A bare `.gguf` path as the only argument works too.
`--ollama <model[:tag]>`	—	A model Ollama already pulled. A bare name means `:latest`. See Ollama models.
`--list-ollama`	—	Print what Ollama has pulled, then exit. Exits `1` if the store is empty or absent.
`--graph <path>`	built from the GGUF	Precomputed graph. Rarely needed — the engine builds and caches one.
`--name <string>`	the file's basename, or the Ollama name	What `/v1/models` and `/api/tags` report.

Serving

Flag	Default	Meaning
`--port <n>`	`11435`	Loopback port.
`--max-length <n>`	`32768`	Context length the engine is built for.
`--max-batch <n>`	`16`	Concurrent requests the engine schedules. `1` makes the server serial end to end.
`--speculation-k <n>`	`0`	MTP speculation depth (0 = off). The engine varies the effective k downward per request.
`--tower <dir>`	—	Core ML vision tower. Required for image input.

There is no --token flag: nothing is authenticated. See Limits.

Environment

Variable	Effect
`OLLAMA_MODELS`	Where to look for Ollama's store. Defaults to `~/.ollama/models`.
`AS_KV_OFFLOAD`	`0` disables the on-by-default KV disk cache. See Limits.
`AS_KV_OFFLOAD_GB`	Disk cap for that cache, default `16`.
`AS_KV_OFFLOAD_PATH`	Its location, default `~/Library/Caches/tempo9/kv`.

Connect an app

2026-08-31T00:00:00+00:00

Tempo9 speaks the protocols your tools already speak. In most cases connecting one is a change of address, not a change of code — and if you have already pulled models with Ollama, there is nothing to download either.

Start the server

tempo9 --gguf ~/models/qwen3-8b-q4ks.gguf

It listens on http://127.0.0.1:11435, loopback only — the wildcard is never bound, so nothing off this machine can reach it. Change the port with --port.

Your app has a "custom OpenAI endpoint" field

Most local-AI apps do. Fill it in:

Field	Value
Base URL	`http://127.0.0.1:11435/v1`
API key	anything, or leave blank
Model	whatever `/v1/models` reports

No key is required and none is checked, which is safe here only because the server is loopback-bound. Do not put it behind a tunnel or a reverse proxy and leave it that way.

Apps in this category include Cherry Studio, AnythingLLM, and anything else built on an OpenAI client — we have not tested each one, so treat this as "the field exists and this is what goes in it" rather than a certification.

Your app only knows how to talk to Ollama

Some apps do not offer a generic endpoint field at all; they have an Ollama integration and nothing else. Those work too. Point them at:

http://127.0.0.1:11435

Tempo9 serves Ollama's native routes alongside its own: /api/tags, /api/chat, /api/generate, /api/show, /api/version, /api/ps. Streaming is newline-delimited JSON and tool-call arguments are objects, exactly as those clients expect.

You already downloaded models with Ollama

Full detail, including what is never written: Ollama models.

Then you do not need to download them again. Ollama stores each model as a plain GGUF file, which is what Tempo9 loads, so it can read them where they sit:

tempo9 --list-ollama

qwen3:8b            4.68 GiB  /Users/you/.ollama/models/blobs/sha256-1f3d...
llama3.2:latest     1.88 GiB  /Users/you/.ollama/models/blobs/sha256-a41c...

tempo9 --ollama qwen3:8b

This is read-only. Nothing is copied, converted, moved, or written into Ollama's store — your models stay exactly as Ollama left them, and Ollama keeps working. If you moved the store, OLLAMA_MODELS is respected.

A bare name means :latest, the same as ollama run.

Agent tools

Claude Code, Codex and similar clients do not use the OpenAI chat protocol — they each want their own. Tempo9 serves all three from one process:

Protocol	Route	Spoken by
OpenAI Chat Completions	`/v1/chat/completions`	most apps and SDKs
Anthropic Messages	`/v1/messages`	Claude Code
OpenAI Responses	`/v1/responses`	Codex
Ollama native	`/api/chat`	apps with an Ollama integration

Agent workloads are the case Tempo9 is built for: many requests at once, long and growing context, heavy tool calling. That is where the architecture shows up rather than in a single stream — see the benchmarks.

What it does not do

Also: Limits.

Worth knowing before you wire something up:

No embeddings. /api/embed and /v1/embeddings are not served; the Ollama route answers 501 and says so rather than pretending.
One model per process. /v1/models and /api/tags report the one that is loaded. Run a second server on another port for a second model.
Vision needs a tower. Image input works with --tower <dir>. A GGUF vision projector inside an Ollama model is detected but not loaded; the server tells you when it sees one.
No authentication. By design, and only defensible because it is loopback-bound.

Getting started

2026-08-31T00:00:00+00:00

Run a model

tempo9 --gguf ~/models/qwen3-8b-q4ks.gguf

That is the whole setup. There is no configuration file, no daemon to install, and no model conversion step — Tempo9 loads GGUF directly.

The server listens on http://127.0.0.1:11435, loopback only. The wildcard address is never bound, so nothing off this machine can reach it.

First request

curl http://127.0.0.1:11435/v1/chat/completions \
  -d '{"model":"local","messages":[{"role":"user","content":"hello"}]}'

No API key is required and none is checked. That is safe only because of the loopback bind — see Limits before you put it behind a tunnel or a reverse proxy.

If you already use Ollama

You do not need to download anything:

tempo9 --list-ollama
tempo9 --ollama qwen3:8b

See Ollama models for how that works and what it does not touch.

Where to go next

Wiring up an app you already use: Connect an app
Every flag: CLI reference
Every route: API reference

Limits

2026-08-31T00:00:00+00:00

Things worth knowing before you design around them.

No embeddings

/v1/embeddings is not served and /api/embed answers 501 with a message saying so. If your pipeline needs embeddings, get them elsewhere.

One model per process

/v1/models and /api/tags report the single loaded model. To serve a second model, run a second tempo9 on another port.

Vision needs a Core ML tower

Image input works when the server is started with --tower <dir>, where dir is the directory containing tower_meta.json — the coreml/ subdirectory of an exported tower, not its parent:

--tower ~/models/qwen3vl_tower/coreml     # correct
--tower ~/models/qwen3vl_tower            # missingAsset: .../tower_meta.json
--tower ~/models/…/model-mmproj.gguf      # same error — a GGUF projector is not a tower

A GGUF mmproj is what a tower is exported from, offline; it is not a drop-in. A GGUF vision projector inside an Ollama model is detected but not loaded — the server prints a line saying so rather than leaving you to wonder why images do nothing.

No authentication

Nothing checks a key, and there is no flag to make it. This is defensible only because the listener binds 127.0.0.1 explicitly and never the wildcard, so the server is unreachable from other machines.

Do not put it behind a tunnel, a reverse proxy, or a port forward and leave it that way. That removes the only thing protecting it.

KV cache spills to disk, by default

Tempo9 keeps reusable KV prefixes in RAM and, when they overflow the RAM watermark, writes them to disk so a later session restores from SSD instead of re-computing the prompt. This is on by default.

What it does and does not buy you. Measured on a restarted engine with the same prompt, against the engine's own restored= counter and with the cache directory wiped as a control: with the cache preserved the first request after a restart reports restored=47 spans and the second request takes 0.61 s; with the directory wiped the same request reports restored=0 and takes 5.20 s. The restore is real and it is worth 8.5x — but the first request after a restart pays full prefill either way. An earlier version of this page implied that first request came back fast. It does not.

What that means for your disk:

Cache files live in ~/Library/Caches/tempo9/kv (AS_KV_OFFLOAD_PATH moves it), capped at 16 GB (AS_KV_OFFLOAD_GB changes the cap; oldest entries are evicted first).
The location is deliberate: macOS may purge it under disk pressure, and Time Machine does not back it up. Losing the cache costs a re-prefill, never data.
The cache survives restarts — the on-disk index reloads at startup.
A cache from a different model, dtype or KV layout is detected by a config hash and wiped rather than served back wrong.

To keep the disk untouched:

AS_KV_OFFLOAD=0 tempo9 --gguf model.gguf

An unwritable or full cache directory is not an error — the engine logs it and runs without offload.

Ollama's store is read, never written

Using --ollama opens Ollama's blobs in place. Nothing is copied, converted, moved, or written into their store. The one file Tempo9 creates is a symlink in its own cache — see Ollama models.

Models and quantizations

2026-08-31T00:00:00+00:00

The question before downloading anything is "will my file run". Two axes decide it, and neither is visible from the other: the quantization and the architecture. A third, the tokenizer, hides inside the second.

Nothing here is inferred from source. Every verified below means a model in that format answered prompts on this machine.

Quantizations

GGUF blocks are used as they are. Nothing is dequantized on load, so a Q4_K file costs Q4_K memory rather than expanding to FP16.

Verified — a real model in this format was served

GGUF type	Block	Seen in
`Q8_0`	32 / 34 B	Qwen3 0.6B, Qwen3.5 0.8B
`Q4_K` (`Q4_K_S`, `Q4_K_M`, `UD-Q4_K_XL`)	256 / 144 B	most of the table below
`Q5_K`	256 / 176 B	as the companion type inside `Q4_K_M` files
`Q6_K`	256 / 210 B	Qwen3.5 4B Q6_K
`Q3_K` (`Q3_K_M`)	256 / 110 B	Qwen3.5 35B-A3B
`MXFP4`	32 / 17 B	GPT-OSS 20B
`F32`	—	MoE routing gates, zero-copy

Q4_K_M and Q5_K_M files are mixed: some layers promote attn_v or ffn_down to Q6_K while the rest stay Q4_K. That is handled per segment, which is why Q5_K and Q6_K appear as companions above without a single-format file of their own.

Supported, but no model in this format has been run here

GGUF type	Note
`Q4_0`	The ARM edge tier. Code path present on all sides.
`F16` / `BF16`	Converted to FP32 at load. Fine for the KB–MB tensors that use it; a whole model in F16 is not the intended path.

Refused, with an error that says why

There is no silent fallback. A file in one of these formats fails at load with an actionable message rather than running slowly or badly:

IQ1 – IQ4 · Q2_K · Q4_1 · Q5_0 · Q5_1 · Q4_0_4_4 / 4_8 / 8_8 (removed from GGUF upstream) · TQ* · NVFP4 · Q8_K (an activation-only type — in a weight table it means the file is corrupt)

If your file is one of these, the usual fix is a Q4_K_M or Q8_0 build of the same model.

On picking a quantization for serving, Q4_K_S is generally preferred over Q4_K_M here: no faster Q6_K kernel exists, so the promoted tensors in a _M file cost more than they return.

Architectures

GGUF direct load accepts seven: qwen35, qwen35moe, gemma4, qwen2vl, qwen3, qwen3moe, and llama — the last covering the Llama line and most of Mistral.

The architecture field does not tell you whether a file will run, because the tokenizer is a second axis and invisible from there. Mistral 7B v0.3 and Llama 3.2 are both architecture=llama; one writes a SentencePiece tokenizer and the other writes gpt2.

Family	Arch	Tokenizer
Llama 3.1 / 3.2 / 3.3	`llama`	gpt2
Mistral v0.1 – v0.3	`llama`	SentencePiece
Llama 2, CodeLlama, Vicuna	`llama`	SentencePiece
Qwen3.5 / Qwen3 / Gemma 4	own	gpt2 / gemma4

Both sides of that split work. A dialect the engine does not implement is refused by name — never guessed — because a vocabulary read with the wrong algorithm still produces ids, and those ids decode into fluent text from the wrong distribution. A wrong answer that reads well is worse than an error.

Verified models

Produced by serving requests, not by reading a registry: bash harness/support_matrix.sh.

Model	Load	Text	10k context	Thinking	Image
gemma4-12b-it-q4ks	ok	PASS	PASS	—	PASS
gptoss-20b-mxfp4	ok	PASS	PASS	PASS	—
llama31-8b-q4km	ok	PASS	PASS	—	—
llama32-1b-q4km	ok	PASS	PASS	—	—
ministral3-3b-q4km	ok	PASS	PASS	—	—
mistral7b-v03-q4km	ok	PASS	PASS	—	—
qwen3-0.6b-q8_0	ok	PASS	PASS	PASS	—
qwen35-0.8b-q8_0	ok	PASS	PASS	PASS	—
qwen35-35b-a3b-q3km	ok	PASS	PASS	PASS	PASS
qwen35-4b-q4ks	ok	PASS	PASS	PASS	—
qwen35-4b-q6k	ok	PASS	PASS	PASS	—
qwen35-9b-imat-q4ks	ok	PASS	PASS	PASS	PASS
qwen38-27b-q4ks	ok	PASS	PASS	—	PASS
qwen3vl-4b-q4km	ok	PASS	PASS	—	PASS
qwen3vlmoe-30b-a3b-q4km	ok	PASS	PASS	—	PASS
tinyllama-1.1b-q4km	ok	PASS	N/A	—	—

A blank cell means nothing was run — no vision front end paired with that model, or no separate thinking mode to test. N/A means the model never claimed it: TinyLlama declares a 2048-token context, so a 10k prompt is outside what it offers. Neither is a failure, and both are printed rather than omitted, because the gap between "not tested" and "does not work" is the entire reason to publish a table instead of a claim.

The 10k context column exists because every check here was once a short prompt, and the table was green for two models that produced word salad past about 8k.

What the model registry is not

The engine's C++ registry lists roughly thirty model families — LLaMA v1–v4, ChatGLM, DeepSeek, GLM-5, MiniMax-M2, Baichuan, the Qwen line. Those reach the engine through a Python conversion path, and none of them is verified on Metal. Reading that registry as a support list would overstate this page by an order of magnitude, which is why the page is generated by running models instead.

Commonly recommended for a 24 GB Mac, and where they stand:

Model	Status
Llama 3.3 8B	yes — verified at 3.1 8B
Mistral 7B v0.3	yes
GPT-OSS 20B	yes — MXFP4
Mistral Small 3.2 24B	architecture yes, this size untested
Qwen3.6-27B	untested
Gemma 4 31B	architecture yes, this size untested
Phi-4	no

Ollama models

2026-08-31T00:00:00+00:00

If you already pulled models with Ollama, Tempo9 can run them where they sit — no download, no copy, no conversion.

tempo9 --list-ollama

qwen3:8b            4.68 GiB  /Users/you/.ollama/models/blobs/sha256-1f3d...
llama3.2:latest     1.88 GiB  /Users/you/.ollama/models/blobs/sha256-a41c...

tempo9 --ollama qwen3:8b

A bare name means :latest, the same as ollama run. If you moved the store, OLLAMA_MODELS is respected.

Ollama does not need to be running. This is not a protocol; nothing talks to the Ollama process. It is a read of its files.

How it works

Ollama has no model format of its own. It stores plain GGUF wrapped in an OCI-style content-addressed layout for deduplication and versioning:

~/.ollama/models/
├── manifests/registry.ollama.ai/library/qwen3/8b   small JSON — the path IS the name
└── blobs/sha256-1f3d...                            bare GGUF — the weights

The manifest lists layers:

{"schemaVersion":2,"layers":[
  {"mediaType":"application/vnd.ollama.image.model",   "digest":"sha256:1f3d…","size":4831838208},
  {"mediaType":"application/vnd.ollama.image.template","digest":"sha256:a7c1…","size":1482}
]}

The weights are the .model layer. Its digest sha256:1f3d… names the file blobs/sha256-1f3d… — colon becomes dash. That blob is byte-identical to the GGUF you would download yourself, and Tempo9 loads GGUF, so it simply opens it.

registry.ollama.ai/library is implicit in Ollama's display convention, so that manifest shows as qwen3:8b. Models from elsewhere keep their full host/namespace/model:tag name, and still answer to the short form.

What is deliberately not used

The manifest also carries template and params layers. Tempo9 ignores them and uses the chat template embedded in the GGUF — the one the model author shipped.

Ollama's template layer is written in Go's text/template; ours is Jinja. Honouring theirs would mean implementing a second template engine to obtain a transcription of something we already have in the original.

Why a symlink

Tempo9 creates one file, in its own cache, never in Ollama's store:

~/.cache/tempo9/ollama/qwen3-8b.gguf -> ~/.ollama/models/blobs/sha256-1f3d...

The engine currently decides GGUF-vs-graph by the file extension, and Ollama's blobs have none. Handed a blob path directly it refuses, and it refuses silently. The link works around that, and gives the engine's graph cache a legible key besides — that cache is keyed on basename, size and mtime, and qwen3-8b.gguf identifies itself where a digest would not.

A link left pointing at a blob that a re-pull replaced is detected and rewritten, so a re-pulled tag never serves last week's weights.

What it will not do

Never writes to Ollama's store. Read-only: directory listing, file reads, attribute lookups. Nothing else.
Skips a model whose blob is missing. Interrupted pulls leave manifests behind; listing one would turn a clean "not found" into a confusing load failure later.
Trusts the file over the manifest for size, because a truncated pull leaves the advertised size intact and the bytes short.
Does not load a GGUF vision projector. It is detected and reported; image input needs --tower. See Limits.

本地推理是为一个人打字设计的。Agent 不是那样跑的。

2026-08-30T00:00:00+00:00

llama.cpp 和 MLX 都很好,而且在一件事上比我们强:一个人、一个流、一个问题。这篇文章不打算争那一格 —— 我们在 M5 Pro 上单流 prefill 是对手的 0.90×,7k 上下文单流更是 0.64×,输得很清楚。

但那不是 agent 的形状。Agent 的形状是:好几个同时在跑、每轮重发一大段 system prompt、不停调工具、偶尔塞一张图进来。 一旦切到这个形状,排序会翻过来 —— 有几格甚至不是"慢",是开不起来。

Tempo9 是把服务器端那套东西(continuous batching、paged KV、异构执行) 搬到 Apple Silicon 上的一次尝试。

一、并发下,对手不是慢,是起不来

同一台 24 GB M5 Pro,同一个 GGUF 文件,同一个负载生成器。 effective tok/s(端到端,含 prefill):

上下文 · 并发	Tempo9	llama.cpp
~1k · 1 路	848 / 899	660 / 566
~1k · 8 路	1294 / 1429	410 / 368
~2.4k · 8 路	1661 / 1735	无法服务
~7k · 8 路	3755 / 3753	无法服务
~7k · 1 路	2505	3884(我们输)

最后一行留着,因为它是真的。但中间那两行是产品能不能用的分界: 2.4k 上下文 + 8 个并发,在这台机器上 llama-server 开不了服(加载期 Metal OOM)。

并发下引用用户真正感受到、且重复区间不重叠的 TTFT。四个 agent 共享 system prompt、各自做不同工作:

四个 Agent · 越短越好

四组测试里，Tempo9 最慢的一次都快于 llama.cpp 最快的一次

02467 秒

1.5K prompt · p50

Tempo90.56–0.89 秒

llama.cpp1.52–2.29 秒

1.5K prompt · p95

Tempo90.92–2.27 秒

llama.cpp4.94–6.52 秒

10K prompt · p50

Tempo90.94–0.95 秒

llama.cpp1.60 秒

10K prompt · p95

Tempo91.61–1.75 秒

llama.cpp6.32 秒

每格跑两到三轮，每轮 12 个请求；横轴为首 token 等待时间。

每轮 12 个请求,所以报多轮区间而不是倍数。10k 不是刻意构造的压力测试, 大致就是 OpenClaw 每轮重发的 system prompt 尺寸。

这里不比较并发 decode 吞吐:五轮重复实验里,Tempo9 每请求 decode 中位数仍有 18–33% 的离散,关掉投机也没有消失。在找到未控制变量之前,这个数字不适合拿来比较引擎。

二、声明 128K 上下文,不用按上限预付 KV

KV + 工作内存 · 越低越好

声明 128K 上下文，Tempo9 启动时仍只占约 0.55 GB

Tempo9557 MB

llama.cpp505–573 MB

32K

Tempo9547–558 MB

llama.cpp981–1051 MB

64K

Tempo9536–558 MB

llama.cpp2034–2041 MB

128K

Tempo9547–557 MB

llama.cpp3.6 GB · OOM

刻度为 0–4 GB；测的是填入上下文前的启动分配。

机制不是"我们更省",是KV 放在哪里:对手的 KV 是 Metal wired buffer, 和 16 GB 权重抢同一份 GPU wired 预算;我们的 KV 分页在 CPU 侧、按需触碰, 不占那份预算。

说清楚它不是什么:这量的是声明容量的预付成本,不是"填满 128K token 后内存也不涨"。真装进去的 token 当然要占内存 —— 分页只是让它按需付费,而不是开机就按最坏情况预留。

三、工具调用有一个可以核对的分数

这是我们最想强调的一格,因为本地推理圈几乎没人公布过。

用官方 BFCL v4 评测器,打我们自己的 OpenAI 端点(也就是说 HTTP、 chat template、工具调用解析这一整条真实链路都在测量范围内):

85.19%Non-Live AST

76.61%Live AST

查看分类明细

BFCL v4 分类	Tempo9
simple (Python)	95.25%
multiple	94.50%
parallel	89.00%
parallel multiple	83.50%
Non-Live irrelevance	84.17%

对 agent 开发者来说,"能不能稳定地把工具调对"比 tok/s 重要得多,而这个问题在本地模型圈基本靠感觉。

而且这一格是我们跑砸了才有的。 第一次跑官方评测器,multiple 只有 24%。不是模型不行 —— 报错逐条指着同一件事:

Incorrect type for parameter 'side2'. Expected type integer, got str.
Parameter value: '4'

模型的 XML 方言 <parameter=side1>5</parameter> 不携带类型,我们的服务端就把 "5" 原样发出去了。任何做严格校验的工具都会拒绝。修法是按工具 schema 里声明的类型转换 —— 不是猜:声明为 string 的参数即使长得像数字也必须保持字符串,否则订单号和电话号会被静默转成整数。

这次解析器修复把分数从 24% 拉到 93.50%；上表引用的后续全量重测为 94.50%。

更值得说的是为什么我们自己之前没发现:我们有一套自研判分器,它给字符串参数加了一层宽松的重解析,于是同一批输出它报 85.5%。那层重解析在做 A/B 时是无害的(两臂同等对待),但它把一个真实缺陷从视野里抹掉了, 抹掉的时长正好等于缺陷存在的时长。会把 bug 归一化掉的仪器,测的是另一个程序。

四、三套协议,改一行地址就能接

一个服务端同时说 OpenAI、Anthropic、Ollama。

tempo9 --gguf <model.gguf> --port 11435

# OpenAI 客户端    base_url = http://127.0.0.1:11435/v1
# Anthropic 客户端  base_url = http://127.0.0.1:11435    (/v1/messages)

原生 /v1/messages 的意思是:用 Anthropic SDK 写的 agent 可以直接指过来,不需要转接层。默认端口 11435 紧邻 Ollama 的 11434,方便并存切换。

五、看图的时候聊天不卡

不是"支持多模态",是视觉塔和 LLM 能同时跑而不互相饿死。

纯文本基线 · 同一张图 · 只改视觉计算位置

带一张图时，视觉计算放到 NPU，LLM 解码快 2.8×

不带图73.6 tok/s

带图 · NPU25.9 tok/s

带图 · GPU9.4 tok/s

不带图是纯文本基线。处理同一张图时，把视觉计算放到 NPU，可以把 GPU 留给 LLM 解码。

这里必须说反直觉的一半:塔自己在 NPU 上更慢,3931 vs 462 ms/图,慢 8.5 倍。价值不在塔快,在它不占 GPU —— 塔在 NPU 跑时 GPU 功率只有 0.35 W,约等于空载。

The NPU path is slower in isolation, but faster at the system level.

争用仍然存在(NPU 那条路要吃 CPU 的一半,和宿主侧的路由、采样撞车), 所以结论是"没那么糟",不是"免费"。

没做成的

发布页通常只写好消息。这一节写我们试过、测过、然后被数字否掉的东西 —— 因为在这个领域,"作者会公布自己被证伪的假设"本身就是一种可信度。

投机解码在散文上不赚钱。 这个 GGUF 自带 MTP 头,逐位置接受率也健康 (位置 1 = 0.818)。但最优深度只做到 11.03 ms/提交,而我们不开投机的基线就是 11.4 ms/步 —— 净收益约等于零。llama.cpp 开 MTP 有 +18%, 因为他们的基线比我们慢 30%。投机解码赚的是"把慢基线藏起来"的钱, 基线已经快的一方没有这笔钱可赚。

结构化输出上倒是有 +13%(k=4 到 104.9 tok/s,单流过 100),但它会让贪心输出随投机深度确定性地改变 —— 而"关掉温度就能复现"是很多人拿来做测试和缓存的性质。我们在 BFCL 2501 例上量过:开投机精度没有可测出的变化 (73.0% vs 73.2%,McNemar p = 0.557),但 1.9% 的调用逐字节不同。 所以它是一个显式开关,不是默认值。

MoE 专家分组是负的。 −10 ~ −17%。

tile 大小已经在最优点。 我们一度认为 MoE 的矩阵乘在 tile 填充上有浪费,扫了一遍:auto 480 ms、bm32 490、bm64 516、bm16 588。自动规则已经挑到最好的了,那个"看起来很有道理"的假设是错的。

归约优化让内核慢了 9%。 我们判断 GDN 的行内核受限于串行归约的延迟, 把两次归约折成一次深度 —— 结果慢了 9%。前提就是错的:它贴着的是发射吞吐,不是深度。但那次失败给归约网络定了价(占该内核 16%),从而把下一步准确指向了真正的地方:同一个 head 的 128 个 simdgroup 各自把 q/k 重读了一遍,占 51%。改成一个 simdgroup 管 4 行之后,内核快 25%,端到端 prefill 快 7%,而且输出逐字节一致。

冷启动的权重驻留只值一次 180 ms。 我们一度以为"Mac 上放一会儿再问就变慢"是个大问题,量下来:同进程内重复 prefill 是 2055 / 1875 / 1872 / 1871 ms,30 秒间隔不会让它变冷。

口径

机器:Apple M5 Pro,24 GB 统一内存,macOS 默认配置。 iogpu.wired_limit_mb = 0(未提高) —— 抬高它是系统级改动、非默认用户体验,而且对所有引擎一视同仁;本机实测抬到 22000 会触发 SoC watchdog 硬复位。
模型:Qwen3.5-35B-A3B q3km GGUF,15.98 GiB,两侧读同一个文件。视觉塔 Qwen3-VL-4B(Core ML,bucket 4096)。
llama.cpp:b10307 (fc3f10b),源码构建,Metal+BLAS。
BFCL:官方 bfcl-eval,single_turn 全套 3641 例,经 HTTP 打我们自己的 /v1/chat/completions。官方 overall 一栏含 multi-turn / web_search / memory 三组,我们只跑了 single_turn,所以只引用分组分数, 不引用 overall。
各图的负载定义、脚本、原始数据与未采用的假设,见 results_p4b_boundary.md / results_p6_mtp.md / results_p7_power.md / results_p13_serving_retest.md / results_p16_spec_variance.md / results_p8_effective.md。

Local inference was designed for one person typing. Agents do not work that way.

2026-08-30T00:00:00+00:00

llama.cpp and MLX are both very good at one shape of work: one person, one stream, one question. This article is not going to argue about that cell. On an M5 Pro, our single-stream prefill is 0.90× the other engine's, and at a 7K context it is 0.64×. We lose clearly.

But that is not the shape of an agent. Agents run several sessions at once, resend a large system prompt on every turn, call tools continuously, and occasionally add an image. Change the workload to that shape and the ordering changes. In a few cells, the difference is not "slow" but cannot start.

Tempo9 is an attempt to bring server-side machinery, including continuous batching, paged KV, and heterogeneous execution, to Apple Silicon.

1. Under concurrency, the other engine does not merely slow down

The same 24 GB M5 Pro, the same GGUF file, and the same load generator. These numbers are effective tok/s, end to end including prefill:

Context · concurrency	Tempo9	llama.cpp
~1K · 1 stream	848 / 899	660 / 566
~1K · 8 streams	1294 / 1429	410 / 368
~2.4K · 8 streams	1661 / 1735	cannot serve
~7K · 8 streams	3755 / 3753	cannot serve
~7K · 1 stream	2505	3884 (we lose)

The last row stays because it is true. The middle two are the boundary between a product that can run the workload and one that cannot: with a 2.4K context and eight concurrent sessions, llama-server cannot start on this machine. It runs out of Metal memory while loading.

For concurrency, the user-visible number with non-overlapping repeat ranges is time to first token. Four agents share one system prompt and perform different work:

Four agents · lower is better

Every Tempo9 range ends before the matching llama.cpp range

02467 s

1.5K prompt · p50

Tempo90.56–0.89 s

llama.cpp1.52–2.29 s

1.5K prompt · p95

Tempo90.92–2.27 s

llama.cpp4.94–6.52 s

10K prompt · p50

Tempo90.94–0.95 s

llama.cpp1.60 s

10K prompt · p95

Tempo91.61–1.75 s

llama.cpp6.32 s

Ranges across two to three runs per cell; twelve requests per run. Axis: seconds to first token.

Each run contains twelve requests, so the table reports ranges across runs rather than a multiplier. The 10K system prompt is not a synthetic stress test; it is approximately what OpenClaw resends on every turn.

Concurrent decode throughput is intentionally absent. Across five repeats, Tempo9's median per-request decode rate still varies by 18–33%, and disabling speculation does not remove the variation. Until the uncontrolled variable is identified, that number is not suitable for comparing engines.

2. Declaring 128K of context should not charge for all of it up front

KV + working memory · lower is better

Declare 128K. Tempo9 still starts at about 0.55 GB

Tempo9557 MB

llama.cpp505–573 MB

32K

Tempo9547–558 MB

llama.cpp981–1051 MB

64K

Tempo9536–558 MB

llama.cpp2034–2041 MB

128K

Tempo9547–557 MB

llama.cpp3.6 GB · OOM

Scale: 0–4 GB. Startup allocation before the declared context is filled.

The mechanism is not simply "we use less memory." It is where the KV cache lives. The other engine stores KV in a Metal wired buffer that competes with 16 GB of weights for the same GPU wired-memory budget. Tempo9 pages KV on the CPU side and touches pages on demand, outside that budget.

It is equally important to say what this measurement is not. It measures the up-front cost of declared capacity, not a claim that memory stays flat after 128K tokens are actually resident. Real tokens consume real memory. Paging lets the runtime pay as they arrive instead of reserving the worst case at startup.

3. Tool calling has a score that can be checked

This is the result we most want to emphasize because local inference runtimes rarely publish it.

We use the official BFCL v4 evaluator against Tempo9's own OpenAI endpoint. That means the measurement includes the real HTTP path, chat template, and tool call parser rather than an isolated model invocation:

85.19%Non-Live AST

76.61%Live AST

Category breakdown

BFCL v4 category	Tempo9
simple (Python)	95.25%
multiple	94.50%
parallel	89.00%
parallel multiple	83.50%
Non-Live irrelevance	84.17%

For an agent developer, "does it call the right tool reliably?" matters more than tok/s. Local model stacks have mostly answered that question by intuition.

This result also exists because our first run failed badly. The official evaluator gave multiple only 24%. The model was not the problem; the errors all pointed at the same thing:

Incorrect type for parameter 'side2'. Expected type integer, got str.
Parameter value: '4'

The model's XML dialect, such as <parameter=side1>5</parameter>, carries no type information. Our server emitted "5" unchanged, so any tool with strict validation rejected it. The fix is to convert according to the declared tool schema, not to guess. A parameter declared as string must remain a string even when it looks numeric, or order numbers and phone numbers will be silently turned into integers.

That parser fix moved the score from 24% to 93.50%. The later full retest shown above measured 94.50%.

The more useful lesson is why we had not found it ourselves. Our evaluator had a permissive reparsing layer for string parameters and scored the same output at 85.5%. That normalization was harmless for A/B tests because both arms were treated equally, but it erased a real defect for exactly as long as the defect existed. An instrument that normalizes a bug away is measuring a different program.

4. Three protocols, one address change

One server speaks OpenAI, Anthropic, and Ollama:

tempo9 --gguf <model.gguf> --port 11435

# OpenAI client       base_url = http://127.0.0.1:11435/v1
# Anthropic client    base_url = http://127.0.0.1:11435    (/v1/messages)

A native /v1/messages endpoint means an agent written against the Anthropic SDK can point to Tempo9 directly, without a translation layer. Port 11435 sits next to Ollama's 11434 so both servers can run side by side.

5. Looking at an image should not stop the conversation

The useful property is not merely "supports multimodal." It is that the vision tower and the LLM can run at the same time without starving each other.

Text baseline · same image · placement only

With an image, LLM decode is 2.8× faster when vision runs on the NPU

No image73.6 tok/s

1 image · NPU25.9 tok/s

1 image · GPU9.4 tok/s

No image is the text-only baseline. With the same image, putting vision on the NPU keeps the GPU available for LLM decoding.

The counterintuitive half matters: the tower itself is slower on the NPU, 3931 versus 462 ms per image, an 8.5× difference. The value is not a faster tower; it is leaving the GPU available. While the tower runs on the NPU, GPU power is only 0.35 W, roughly idle.

The NPU path is slower in isolation, but faster at the system level.

Contention does not vanish. The NPU path still uses part of the CPU and collides with host-side routing and sampling. The conclusion is "less bad," not "free."

What did not work

Release pages usually contain only good news. This section records what we tried, measured, and then rejected, because in this field publishing the hypotheses that the numbers disproved is itself evidence of reliability.

Speculative decoding does not pay on prose. This GGUF includes an MTP head, and first-position acceptance is healthy at 0.818. The best depth reaches 11.03 ms per committed step, while ordinary decoding is already 11.4 ms per step. The net gain is approximately zero. llama.cpp gains 18% from MTP because its baseline is 30% slower. Speculation earns its return by hiding a slower baseline; a fast baseline has less to hide.

Structured output is different: at k=4, single-stream decode reaches 104.9 tok/s, a 13% gain and above 100 tok/s. Across 2501 BFCL cases, speculation causes no measurable accuracy change (73.0% versus 73.2%, McNemar p = 0.557), although 1.9% of calls differ byte for byte. It also changes greedy output deterministically with speculation depth, while many tests and caches rely on temperature zero being reproducible. That is why speculation remains an explicit switch instead of the default.

Grouping MoE experts regressed by 10–17%.

The tile size was already optimal. We suspected wasted padding in the MoE matrix multiplies, then swept it: auto 480 ms, bm32 490, bm64 516, bm16 588. The automatic rule had already chosen the best point. The plausible hypothesis was wrong.

A reduction optimization made the kernel 9% slower. We thought the GDN row kernel was limited by serial reduction latency and folded two reductions into one deeper reduction. The result regressed by 9%. The premise was wrong: the kernel sat against issue throughput, not depth. The failure still priced the reduction network at 16% of the kernel and pointed to the actual target: 128 simdgroups for one head reread q/k independently, accounting for 51%. Assigning four rows to each simdgroup made the kernel 25% faster and end-to-end prefill 7% faster, with byte-identical output.

Keeping cold-start weights resident is worth only one 180 ms event. We suspected that a Mac slowed down after sitting idle. Measured in one process, repeated prefill took 2055 / 1875 / 1872 / 1871 ms. A 30-second pause did not make it cold again.

Method

Machine: Apple M5 Pro, 24 GB unified memory, stock macOS settings. iogpu.wired_limit_mb = 0 (not raised). Raising it is a system-level modification rather than the default experience, and it would benefit every engine. On this machine, setting it to 22000 triggers an SoC watchdog hard reset.
Model: Qwen3.5-35B-A3B q3km GGUF, 15.98 GiB. Both engines read the same file. The vision tower is Qwen3-VL-4B (Core ML, bucket 4096).
llama.cpp: b10307 (fc3f10b), built from source with Metal and BLAS.
BFCL: the official bfcl-eval, all 3641 single_turn cases, sent over HTTP to Tempo9's own /v1/chat/completions. The official overall column also includes multi-turn, web search, and memory groups that we did not run, so we publish group scores rather than overall.
Workload definitions, scripts, raw data, and rejected hypotheses are in results_p4b_boundary.md, results_p6_mtp.md, results_p7_power.md, results_p13_serving_retest.md, results_p16_spec_variance.md, and results_p8_effective.md.