The economics of speculative decoding: +41% on 9B, break-even on 35B

2026-09-02

English · 中文版

Speculative decoding can be summarized in one sentence. A cheap draft head guesses k tokens, the full model verifies them in one pass, and the runtime keeps however many were correct. We use MTP, the model's own next-n prediction heads. Whether this actually saves time depends on a quantity that is rarely broken out: the marginal cost of verification.

We measured it across four values of k, over 48 rounds in one session:

verify(m) ≈ 9 ms + 5.5 ms × m        (m = k+1 rows)

The fixed 9 ms is roughly one ordinary decode step. Why does each additional verification row cost only 5.5 ms instead of rereading every weight? Because the m rows run as a batched GEMV: the weights are read once and shared by the rows. For the dense part of the model, extra rows are nearly free over m ∈ [2,8].

Architecture decides whether it pays

The important part is hidden inside that 5.5 ms marginal cost. In a MoE model, each row can activate different experts, so their weights cannot be shared. Every additional row reads about 0.46 GB of expert weights, costing roughly 3.2 ms. Dense models do not have that term.

The consequence is visible in the results. This is the same MTP implementation on both models:

The same feature is nearly a wash on one model and a product-level improvement on the other. The difference is the 3.2 ms expert term in every extra row. The economics of speculative decoding are an architectural property, not just an implementation detail.

The sweet spot for k is measured, not guessed

On the 9B model we swept k from 1 through 6 at single concurrency, using edit, English prose, and code workloads. Their baselines were 45.1 / 45.7 / 45.8 tok/s:

k=1   53.3 / 49.3 / 53.5
k=2   56.3 / 47.2 / 53.0
k=3   63.3 / 48.5 / 56.6    <- best overall; server default
k=4   65.1 / 46.5 / 48.4    (helps edit only; code collapses)
k=5+  regresses across the board

Acceptance falls monotonically with depth. On English prose it goes from 0.85 at k=1 to 0.59 at k=3: the farther ahead the draft predicts, the closer it gets to guessing. Different workloads prefer different depths. Prose prefers a shallow draft, while code and editing tolerate more. k=3 is the best fixed global value, so that is the server default for this model.

Gating: enable it only where it pays

The most interesting design is not a kernel. It is a threshold: when rolling acceptance falls below 0.60, fall back to ordinary decoding automatically (AS_SPEC_MIN_ACCEPT=0.60).

That gate does more than intended. Agent workloads such as editing files, emitting tool calls, and generating JSON are predictable and naturally have high acceptance (0.92 on the 35B mid-edit workload). Chinese prose is naturally low at 0.28-0.5. One threshold therefore produces the right behavior: speculate for editing and structured output; stop for prose, without a workload classifier or user configuration.

On the 35B model the gate protects the floor: prose drops back to ordinary decode, while the mid-edit workload still moves from 91 to 97.6 tok/s (+7%). On the 9B model it rarely fires, so speculation remains enabled across workloads.

One cost that belongs in the ledger: faster, not more efficient

A rootless IOReport sampler measured power during the decode window on the 9B English reasoning workload:

Batched verification raises instantaneous utilization. MTP trades energy for time; it is not a free lunch. In an application that naturally becomes two modes: k=3 for performance and k=0 for efficiency. A plugged-in Mac mini and a MacBook on battery should not have the same answer.

Closing the ledger

Between "this sounds fast" and "we know when to enable it" are a cost model (9 + 5.5m), an architectural boundary (whether expert weights can be shared), a sweep for k (the sweet spot is 3), an automatic gate (0.60), and a power measurement (+10% J/token). Once those are measured, the default is no longer an opinion. It is a lookup.