Engineering notes on running a local inference engine, kept for the bench harness.

1. Model loading. A GGUF file carries the tensors, the tokenizer vocabulary, the merge table, and the chat template in one container. The loader maps the file instead of reading it, so the operating system pages weights in on first touch and shares the pages between processes. On Apple Silicon the mapped pages are the same physical memory the GPU reads, which is why a second engine on the same machine does not double the resident set until one of them writes. The loader records the quantization type per tensor, because a single file may mix formats: the embedding and output projections are often kept at a higher precision than the expert matrices, and the attention projections may differ again.

2. Graph construction. The engine walks the architecture name in the file header, looks up the layer recipe, and builds a static graph of operators: embedding, per-layer attention with its norms and rotary positions, the feed-forward block or the mixture-of-experts router and experts, and the final norm with the output projection. Each operator chooses a kernel at construction time from the batch size, the data type, and the weight format. Decode at batch one takes a matrix-vector path; prefill and batched decode take a tiled matrix-matrix path. The choice is revisited when the batch changes, which happens on every scheduling step under continuous batching.

3. Prefill. The prompt is tokenized once and pushed through the graph in chunks so that the key-value cache fills without a single oversized allocation. The chunk size is a tradeoff: large chunks keep the matrix-matrix kernels busy, small chunks let other requests interleave and keep time to first token predictable for everyone else. The prefix cache sits in front of prefill. It hashes the prompt in fixed-size spans and reuses any span whose keys and values already exist, so a repeated system prompt or a long document that appears in consecutive turns costs almost nothing after the first time.

4. Decode. One token per request per step. The scheduler assembles the batch from every request that has a token to produce, runs the graph once, samples, and appends to each request's cache. Throughput at this stage is bounded by how fast the weights can be streamed from memory, so the work is to touch each byte once per step and to share that touch across as many requests as possible. The gain from batching comes almost entirely from this sharing; the arithmetic is cheap and the bytes are not.

5. Memory accounting. Weights are fixed. The key-value cache grows with context length times batch size times the number of layers that keep a cache. Workspace for attention and the expert matmuls is allocated at the maximum shape before the first request so that a later, larger batch never triggers a reallocation in the middle of a step. The idle budget matters as well: an engine that holds a model but serves no request should not keep a core busy polling, and the GPU should be allowed to drop its clocks.

6. Measurement. Rates are measured by the client from the streamed response, with the server's reported token usage checked against the number of content events, so that an implementation that coalesces several tokens into one event is not mistaken for a slower one. Time to first token is measured from the request being sent to the first content event, with no prefix cached, after a short warm-up request so that the first-ever kernel compilation does not land in the figure. Two engines are never run at the same time on one machine; whichever runs second runs on a warmer machine, so the order is alternated across rounds and both orders are reported.

7. Known traps. A benchmark that reads tokens per second from a log line written by the engine itself measures what the engine chose to count. A benchmark that runs while a build or a continuous-integration job is active measures the build. A benchmark on a laptop that has just woken from sleep measures the clock ramp. A benchmark with a fresh 2,000-token prompt that happens to share a prefix with the previous one measures the cache. Each of these has produced a confident wrong number at least once, which is why the harness refuses to run when another engine is alive, prints the engine versions in its first line, and keeps the raw per-request lines so that the summary can be recomputed later.

8. What the numbers do and do not say. A single-stream rate describes one person typing to one model. An aggregate rate at sixteen clients describes a small team or a set of agents sharing one machine. Time to first token on a long prompt describes the pause before an answer begins when a document is pasted in. None of these is the whole story; a serving engine is judged by all three at once, and by what happens to the cache when the conversation changes.

9. Quantization and bytes per token. The fair comparison between two engines is not the same nominal bit width but the same number of weight bytes read per generated token. A four-bit K-quant with six-bit output projections and a three-bit expert format with four-bit attention can land within a few percent of each other in bytes per token while looking nothing alike on paper. When the files differ, the bytes are reported next to the rate; when the files are identical, as in the Gemma 4 runs, the rate alone is the comparison.

10. Concurrency and the cache. Sixteen clients each holding a 4,096-token context need sixteen caches, and the scheduler must reserve them before admitting the request rather than discover the shortage on the step that fills the last page. Admission control is what keeps a loaded server from thrashing: a request that cannot be given its full context at the moment it arrives waits in the queue, and the client sees a longer time to first token rather than a failed generation in the middle. The prefix cache complicates the accounting, because a span that two requests share is counted once but freed only when the last holder releases it.

11. Reproduction checklist. Record the engine versions from the binaries, not from a release note. Record the exact file hash of the weights. Stop every other process that can touch the GPU, including continuous-integration runners that start on a push. Run each engine alone, alternate the order, and keep every raw line. Publish the prompt text, the scripts, and the launch flags next to the figures, so that a reader with the same machine can get the same numbers and a reader with a different machine can get comparable ones.