Local inference was designed for one person typing. Agents do not work that way.

2026-08-30

English · 中文版

llama.cpp and MLX are both very good at one shape of work: one person, one stream, one question. This article is not going to argue about that cell. On an M5 Pro, our single-stream prefill is 0.90× the other engine's, and at a 7K context it is 0.64×. We lose clearly.

But that is not the shape of an agent. Agents run several sessions at once, resend a large system prompt on every turn, call tools continuously, and occasionally add an image. Change the workload to that shape and the ordering changes. In a few cells, the difference is not "slow" but cannot start.

Tempo9 is an attempt to bring server-side machinery, including continuous batching, paged KV, and heterogeneous execution, to Apple Silicon.

1. Under concurrency, the other engine does not merely slow down

The same 24 GB M5 Pro, the same GGUF file, and the same load generator. These numbers are effective tok/s, end to end including prefill:

The last row stays because it is true. The middle two are the boundary between a product that can run the workload and one that cannot: with a 2.4K context and eight concurrent sessions, llama-server cannot start on this machine. It runs out of Metal memory while loading.

For concurrency, the user-visible number with non-overlapping repeat ranges is time to first token. Four agents share one system prompt and perform different work:

Each run contains twelve requests, so the table reports ranges across runs rather than a multiplier. The 10K system prompt is not a synthetic stress test; it is approximately what OpenClaw resends on every turn.

Concurrent decode throughput is intentionally absent. Across five repeats, Tempo9's median per-request decode rate still varies by 18–33%, and disabling speculation does not remove the variation. Until the uncontrolled variable is identified, that number is not suitable for comparing engines.

2. Declaring 128K of context should not charge for all of it up front

The mechanism is not simply "we use less memory." It is where the KV cache lives. The other engine stores KV in a Metal wired buffer that competes with 16 GB of weights for the same GPU wired-memory budget. Tempo9 pages KV on the CPU side and touches pages on demand, outside that budget.

It is equally important to say what this measurement is not. It measures the up-front cost of declared capacity, not a claim that memory stays flat after 128K tokens are actually resident. Real tokens consume real memory. Paging lets the runtime pay as they arrive instead of reserving the worst case at startup.

3. Tool calling has a score that can be checked

This is the result we most want to emphasize because local inference runtimes rarely publish it.

We use the official BFCL v4 evaluator against Tempo9's own OpenAI endpoint. That means the measurement includes the real HTTP path, chat template, and tool call parser rather than an isolated model invocation:

For an agent developer, "does it call the right tool reliably?" matters more than tok/s. Local model stacks have mostly answered that question by intuition.

This result also exists because our first run failed badly. The official evaluator gave multiple only 24%. The model was not the problem; the errors all pointed at the same thing:

Incorrect type for parameter 'side2'. Expected type integer, got str.
Parameter value: '4'

The model's XML dialect, such as <parameter=side1>5</parameter>, carries no type information. Our server emitted "5" unchanged, so any tool with strict validation rejected it. The fix is to convert according to the declared tool schema, not to guess. A parameter declared as string must remain a string even when it looks numeric, or order numbers and phone numbers will be silently turned into integers.

That parser fix moved the score from 24% to 93.50%. The later full retest shown above measured 94.50%.

The more useful lesson is why we had not found it ourselves. Our evaluator had a permissive reparsing layer for string parameters and scored the same output at 85.5%. That normalization was harmless for A/B tests because both arms were treated equally, but it erased a real defect for exactly as long as the defect existed. An instrument that normalizes a bug away is measuring a different program.

4. Three protocols, one address change

One server speaks OpenAI, Anthropic, and Ollama:

tempo9 --gguf <model.gguf> --port 11435

# OpenAI client       base_url = http://127.0.0.1:11435/v1
# Anthropic client    base_url = http://127.0.0.1:11435    (/v1/messages)

A native /v1/messages endpoint means an agent written against the Anthropic SDK can point to Tempo9 directly, without a translation layer. Port 11435 sits next to Ollama's 11434 so both servers can run side by side.

5. Looking at an image should not stop the conversation

The useful property is not merely "supports multimodal." It is that the vision tower and the LLM can run at the same time without starving each other.

The counterintuitive half matters: the tower itself is slower on the NPU, 3931 versus 462 ms per image, an 8.5× difference. The value is not a faster tower; it is leaving the GPU available. While the tower runs on the NPU, GPU power is only 0.35 W, roughly idle.

The NPU path is slower in isolation, but faster at the system level.

Contention does not vanish. The NPU path still uses part of the CPU and collides with host-side routing and sampling. The conclusion is "less bad," not "free."

What did not work

Release pages usually contain only good news. This section records what we tried, measured, and then rejected, because in this field publishing the hypotheses that the numbers disproved is itself evidence of reliability.

Speculative decoding does not pay on prose. This GGUF includes an MTP head, and first-position acceptance is healthy at 0.818. The best depth reaches 11.03 ms per committed step, while ordinary decoding is already 11.4 ms per step. The net gain is approximately zero. llama.cpp gains 18% from MTP because its baseline is 30% slower. Speculation earns its return by hiding a slower baseline; a fast baseline has less to hide.

Structured output is different: at k=4, single-stream decode reaches 104.9 tok/s, a 13% gain and above 100 tok/s. Across 2501 BFCL cases, speculation causes no measurable accuracy change (73.0% versus 73.2%, McNemar p = 0.557), although 1.9% of calls differ byte for byte. It also changes greedy output deterministically with speculation depth, while many tests and caches rely on temperature zero being reproducible. That is why speculation remains an explicit switch instead of the default.

Grouping MoE experts regressed by 10–17%.

The tile size was already optimal. We suspected wasted padding in the MoE matrix multiplies, then swept it: auto 480 ms, bm32 490, bm64 516, bm16 588. The automatic rule had already chosen the best point. The plausible hypothesis was wrong.

A reduction optimization made the kernel 9% slower. We thought the GDN row kernel was limited by serial reduction latency and folded two reductions into one deeper reduction. The result regressed by 9%. The premise was wrong: the kernel sat against issue throughput, not depth. The failure still priced the reduction network at 16% of the kernel and pointed to the actual target: 128 simdgroups for one head reread q/k independently, accounting for 51%. Assigning four rows to each simdgroup made the kernel 25% faster and end-to-end prefill 7% faster, with byte-identical output.

Keeping cold-start weights resident is worth only one 180 ms event. We suspected that a Mac slowed down after sitting idle. Measured in one process, repeated prefill took 2055 / 1875 / 1872 / 1871 ms. A 30-second pause did not make it cold again.

Method

• Machine: Apple M5 Pro, 24 GB unified memory, stock macOS settings. iogpu.wired_limit_mb = 0 (not raised). Raising it is a system-level modification rather than the default experience, and it would benefit every engine. On this machine, setting it to 22000 triggers an SoC watchdog hard reset.
• Model: Qwen3.5-35B-A3B q3km GGUF, 15.98 GiB. Both engines read the same file. The vision tower is Qwen3-VL-4B (Core ML, bucket 4096).
• llama.cpp: b10307 (fc3f10b), built from source with Metal and BLAS.
• BFCL: the official bfcl-eval, all 3641 single_turn cases, sent over HTTP to Tempo9's own /v1/chat/completions. The official overall column also includes multi-turn, web search, and memory groups that we did not run, so we publish group scores rather than overall.
• Workload definitions, scripts, raw data, and rejected hypotheses are in results_p4b_boundary.md, results_p6_mtp.md, results_p7_power.md, results_p13_serving_retest.md, results_p16_spec_variance.md, and results_p8_effective.md.