[chrishayuk]

The CPU generation fallback streams, samples and reports failure

What was wrong

generate_streaming hands a request to the CPU Q4K path when the backend has no fused Q4 pipeline: the CPU backend, and per-layer-embedding models (Gemma 4 E2B) unless LARQL_METAL_PLE=1. The hand-off passed six arguments and dropped two:

• the token callback, so a stream delivered nothing until generation had finished;
• the sampling settings, so every request was decoded greedily whatever temperature, top-p, top-k or seed it carried.

Inside the CPU loops, a decode step that produced nothing did break, and the result was returned with error: None: a truncated generation reported as a success.

What changes

• generate_via_cpu_q4k takes the sampling settings and the callback.
• A greedy request keeps the KV-cached (or legacy uncached) loop and calls back once per token, in order.
• A sampled request runs the existing sampling-aware CPU decoder (generate_constrained_via_cpu_q4k_streaming_sampled) with a mask that changes nothing.
• A step that produces no token sets the result's error (GenerateError::Other, naming the step and how many tokens preceded it) and keeps the tokens already produced.

Evidence

Two tests through the public generate_streaming with CpuBackend on the Q4K fixture:

Test	On main (df20de1e)	With the fix
streaming on the CPU backend reports each token	fail: the callback never fires	pass
sampling on the CPU backend is not decoded greedily (32 seeds at temperature 5)	fail: every seed reproduces the greedy output	pass

Four more tests on the CPU path directly cover the greedy and sampled callbacks and seed determinism. Full larql-inference suite: 1,978 passed, 0 failed. cargo fmt --check, cargo clippy -p larql-inference --lib --tests --benches --no-deps -D warnings and cargo check -p larql-server -p larql-cli --lib --bins --tests are clean.

Server: no delta event for an empty token

With the callback firing, tokens that decode to no text (a special token, or the first half of a multi-byte character) reached the OpenAI routes, and the responses stream sent an output_text.delta with an empty delta for each. CI caught it in responses_streaming_emits_typed_events_and_done. The second commit makes the server's token tap buffer such a token and emit nothing for it. Full larql-server suite with it: 1,324 passed, 0 failed.

Costs and limits

• A sampled CPU request is slower than before, because before it was not sampled. The sampling-aware decoder is not the KV-cached loop; I have not measured the difference. Greedy CPU generation is unchanged apart from the callback.
• The mid-generation failure path has no test: I found no way to make a decode step return nothing on the fixture without a fault hook.
• The Metal decode loop's own handling of a failed step was not changed here.