[eugenehp]

Two bugs of the same shape: a value that could not express "unspecified", so the code answered a question nobody had asked.

1. auto_runner silently chose CPU, and a 128-token context

LmBuildOptions.device was a bare Device defaulting to Device::Cpu, so "the caller did not choose" and "the caller chose CPU" were the same value. Every auto_runner* convenience builds LmBuildOptions::default() — so auto_runner, auto_runner_with_mmproj and auto_runner_servable* all ran on CPU on machines with a GPU, from functions named auto_.

LmBuilderCommon had already drawn exactly this distinction, for exactly this reason (see its module docs); the registry's options type had not. Two tests asserted device == Device::Cpu as "the documented default" — they locked the bug in rather than catching it, which is worse than no test, because a passing assertion answers the question before anyone thinks to ask it.

It also had no context field at all. Families size their bucketed decode compile cache from max_seq, whose shared default is 128 — a ladder covering ~256 tokens of past. One token beyond that the bucket lookup misses, and a native-packed (K-quant) runner has no F32 decode fallback, so generation dies with weight not found: model.embed_tokens.weight. That reads like a GGUF-vs-safetensors naming problem and is really the context cap; it cost a bug report and a long hunt through tensor names downstream.

Between them, these two gaps are why callers wanting a GPU or a real context had to bypass the registry and drive family builders by hand.

• device: Option<Device> + device_or_auto(), resolved once per build so one auto_runner call cannot split a model across devices. An explicit Device::Cpu is still honoured verbatim.
• max_seq: Option<usize> + max_seq_opt() on the shared builder macro and the hand-written qwen35 builders, forwarded to qwen3/qwen35/gemma/llama32. RLX_LM_MAX_SEQ covers families whose arm is not threaded yet.

2. The in-place KV append was never reachable

bind_decode_inputs_with_capacity had zero callers workspace-wide, so Op::KvAppend was never emitted and every decode concat-copied the whole O(context) cache per token — 30% of decode GPU time at ctx 1024 on qwen3-0.6B/Metal, growing with context. See MIT-RLX/rlx#4 for the shared predicate this builds on.

qwen3, llama32 and gemma now declare the spare row and bind the contract, derived from the same rlx_flow::blocks::kv_spare_rows the layer decides with. batch > 1 deliberately stays on concat: a [batch, seq, kv_dim] cache at axis 1 has a strided prefix that cannot alias, and declaring the spare row there corrupted every sequence after the first. Additive _spare feeder variants in autoregressive.rs pad to upper + spare; the 13 existing callers keep the 0-spare originals.

Measured, token-identical throughout: +37–45% decode at ctx 1024, +56% at 2048 (qwen3 native-packed); Metal 1.38x over an 80-token window through the public LmRunner.

Verification

Guard tests per family (decode_graph_appends_kv_in_place, batched_decode_graph_keeps_concat), each confirmed to fail under RLX_NO_INPLACE_KV=1. Gemma additionally passes gemma4_decode_backend_parity 4/4 with the append on and off. Suites green: qwen3 95, llama32 58+6, gemma 89+others, rlx-models-core 281.

Two family-agnostic harnesses driven through auto_runner, so a family is covered the moment it migrates and an unmigrated one shows up as a ~1.00x row rather than as silence:

• examples/inplace_kv_ab.rs — best-of-N A/B, asserts token identity.
• examples/inplace_kv_matrix.rs — checkpoint × device × arm, reporting weights / latency / speed / precision / agreement.

All 8 matrix cells (Qwen3-0.6B-Q4_K_M and BitNet ternary Bonsai-8B-Q1_0, CPU and Metal) are token-identical with max|dlogit| 0.0 and cosine 1.000000.

Both take PROMPT_TEXT: synthetic filler ids drive a model into near-tied logits where a 1e-7 difference flips greedy argmax, and two apparent cross-device divergences turned out to be that artifact — on natural text the same cells agree exactly.

Not included

Still on concat (layer side is ready, flow + host pad not wired): llama32's hand-built packed decode graph (builder.rs, a Graph-level concat with its own scratch pad), rlx-voxtral-tts, rlx-unlimited-ocr, rlx-vibevoice-asr, and rlx-qwen3/pipeline_decode.rs. Each needs the spare row, the contract, and the host pad moved together, then one inplace_kv_matrix run to confirm.

🤖 Generated with Claude Code