Dual-SSD streaming: read the model from two drives at once (COLI_MODEL_MIRROR)

[steve-m]

Summary

Decode on most boxes is disk-bound — a cold GLM-5.2 token streams ~11 GB of experts — and the weights are read-only. So if a machine has a second SSD, a plain copy of the model on it is untapped read bandwidth. This PR lets the engine stream experts from two copies at once:

COLI_MODEL=/fast/glm52_i4 COLI_MODEL_MIRROR=/second/glm52_i4 ./coli chat
COLI_DISK_WEIGHTS=9,3 ...   # optional primary,mirror ratio; default = startup bandwidth probe

Each expert is routed to one drive by a deterministic (layer,eid) hash, weighted either by COLI_DISK_WEIGHTS or by a startup probe that measures both drives with the engine's own access pattern (parallel ~19 MB reads, O_DIRECT). Deterministic routing means the readahead/PILOT prefetch and the demand read always hit the same drive, and no expert is ever cached twice.

Design notes

• Validation, not trust: a mirror file is accepted only if its size and safetensors header are byte-identical to the primary (identical data_offsets by construction, so every pread is valid on either copy). Divergent or missing files are skipped per-file with a warning — a partial mirror works (a smaller second SSD holding only some shards still helps).
• The mirror is never written: .coli_usage, .coli_kv, and all sidecars stay on the primary.
• Degrades, never dies: a mirror read error falls back to the primary (one warning); unplugging the second drive mid-run slows the server instead of killing it.
• Routing covers every expert read path: the coalesced slab pread, O_DIRECT, mmap/Metal and scale reads, and the OMP-parallel pin/autopin warmup (which then streams from both drives). Mirror reads go through pread_full, inheriting the short-read/EINTR handling.
• Per-drive traffic is reported in a MIRROR: stats line (GB + read counts per drive).

Correctness

• tests/test_st_mirror (in TEST_BINS) covers mirror validation, read equality on both replicas, and the rejection paths (divergent header, size mismatch, missing file).
• Routing is value-preserving by construction — both copies are byte-identical, so output is byte-identical with the mirror on or off (verified on greedy decodes).
• With MIRROR unset, no code path changes.

Measured performance

Hardware: Ryzen 9 3900X, 64 GB DDR4-3200, two NVMe drives (Samsung 990 PRO 4 TB + WD SN850X), each ~6 GB/s under the engine's parallel-19 MB access pattern (spec sequential numbers don't survive this pattern). GLM-5.2 744B int4, DIRECT=1, drafts off.

• Original cold-ish measurement (the commit's numbers): 0.42 → 0.57 tok/s (+36%), expert-disk service 15.2 s → 10.3 s per 24 tokens, probe chose a 48/52 split, byte-identical output.
• On a tuned warm config (OMP pinned to physical cores, GPU dense/attention offload): +8% decode, −27% prefill (12.9 → 9.4 s) — prefill is the most streaming-heavy phase and benefits most.
• The split holds at 49–51% of expert bytes from the mirror across every run and workload we've thrown at it over a week of daily use (COLI_DISK_WEIGHTS=1,1 on this symmetric pair).
• Page-cache interaction: with DIRECT=1 (recommended together with the mirror), both copies bypass the page cache for expert reads and the measured cache footprint is the same as single-disk — the "two identical files fight for cache" failure mode doesn't occur. In buffered mode a warm, cache-bound workload can lose a few percent from the split cache; the docs say so.

For context: every number in PR #418 (Vulkan backend) was measured on top of exactly this commit, mirror active for all configs — the two features compose.

Integration surface

st.h (+87: replica registration/validation, routed pread), glm.c (+164: routing at every expert read site, probe, stats), tests/test_st_mirror.c (+109), README section, docs/ENVIRONMENT.md table, small Makefile/coli/gitignore bits. +370/−15 over dev, two commits (feature + docs). Feature is entirely opt-in via COLI_MODEL_MIRROR.

Limits

• The mirror must be a byte-identical copy of the same quantization (differing files are rejected per-file, which is what makes partial mirrors safe).
• Same-machine drives only — this is a read replica for bandwidth, not RAID and not network distribution.
• Warm, fully cache-resident workloads gain little by definition (the win scales with how disk-bound the run is).

[JustVugg]

Rebase needed: today's merges into dev (#298 in particular) moved the expert-load surroundings in colibri.c and this now conflicts. Your dual-SSD striping logic is orthogonal to what changed, so it's likely mechanical — if you don't get to it, I can push the rebase to your branch; just say which you prefer.

[JustVugg]

Update: I took care of the rebase myself (maintainer edits) — pushed a merge of current dev into dual-ssd-mirror resolving the #298/#192-era conflicts. Resolution notes: mir_pread/st_prefetch_rep now carry dev's DISK-CLASS accounting unwind and the O_DIRECT prefetch skip; the direct path keeps dc_direct=1 plus your mirror read counters. Verified locally: clean build, token-exact tiny models unchanged, test_st_mirror + test_st_pread pass. @steve-m please sanity-check the resolution — if CI is green and it looks right to you, this merges.

[JustVugg]

Merged into dev 🎉 Thanks @steve-m — dual-SSD streaming is in.

Call for data — if you (or anyone with two drives) can run this on dev, we'd love numbers:

Setup: put a second copy of the model on a separate physical drive, then

COLI_MODEL=/drive1/glm52_i4 COLI_MODEL_MIRROR=/drive2/glm52_i4 ./coli chat

What we'd like to see:

1. tok/s with vs without COLI_MODEL_MIRROR (same prompt, same machine) — the headline number: does it actually ~2× on a disk-bound box?
2. Per-replica balance from the mirror counters (g_mir_bytes/g_mir_nread) — are both drives sharing the load evenly, or is one starved?
3. A real chat transcript — confirm output is unchanged (mirror is a bandwidth trick, must be bit-identical to single-drive).
4. Drive types (NVMe/SATA, same or mixed) — mixed-speed mirrors are the interesting edge case.

Anyone running on a slow single disk today: this is the cheapest real speedup available — a second budget NVMe roughly halves your streaming wait. Please report back here.