Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84) - #418
Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84)#418steve-m wants to merge 38 commits into
Conversation
|
Heads-up: |
|
Rebased onto current dev — the split turned out painless: git tracked |
|
pp512/tg128-style numbers, for comparability with the llama.cpp benchmark conventions (same box and config as the tables above; dual-SSD mirror active for both backends; fresh process per run; two runs each):
Notes: prefill here is expert-streaming-bound (a 512-token batch-union touches nearly every expert of the 744B model), so pp measures the whole engine + disks more than the GPU. The accounting slightly favors HIP — Vulkan uploads its dense weights during the first prefill while the CUDA/HIP tier uploads at load, before the prefill timer — and Vulkan leads regardless (+13% pp, +4–5% tg). These standard workloads also reproduce much more tightly than free-running generations (≤2% spread), so we'll use them for future comparisons. |
|
Rebase needed: #298 just merged into |
|
Rebased onto current dev ( Grouped scales in the Vulkan kernels (the #298 ask). All three kernels (dense GEMV, fused gate+up, absorb attention) now decode fmt=4 grouped int4 with one scale per Validation. The Also in the refresh, from production use on our box:
One measured caveat worth documenting: on our RX 9070/RADV, growing the resident expert tier past ~6 GB slows the other GPU paths' per-dispatch execution (~1.3 s per tier-GB over a 64-token decode; profiled to the fence wait, not the submit call, with eviction/allocation-count/free-memory all ruled out) — so the sweep-derived Numbers on the box are unchanged from the tables above (the refresh is format coverage + hygiene, not a perf change): 64-tok tg on the int3-g64 container now reaches 1.73–1.75 tok/s with the engine-side improvements that landed in dev (#421 mirror + the |
|
Thanks @steve-m — this is a big, welcome piece of work, and I want to get it in. Two things before merge, one mechanical and one that only you can provide (I don't have a Vulkan GPU to verify it myself). 1. CI is red for a mechanical reason — 9 committed test binaries. The PR added compiled binaries under 2. Because it's a new hardware backend I can't run, I'd like to read validation data before merging — this is the bar we hold every backend to. Could you post, from your Vulkan GPU:
Plan: once CI is green (step 1) and the data checks out (coherent output + tok/s in step 2), I'll review and merge. I've verified the mechanical side is the only CI blocker; the rest is your GPU's numbers. Happy to fix the committed-binary bit for you and push to your branch if you'd rather — just say the word. |
|
Both items addressed — plus one more commit that was in flight. 1. CI / committed binaries: removed from the entire history, not just the tip. All nine entered through one commit (an over-broad 2. Validation from the Vulkan GPU. Hardware: AMD Radeon RX 9070 (RADV GFX1201), Mesa 26.1.4, Vulkan 1.4.354, Linux 6.19. Model: GLM-5.2 744B, per-row int4 container, dual-SSD mirror (#421), full Vulkan stack = dense + absorb attention + 320-expert VRAM tier. Correctness. The tiny-model TF oracle fixture isn't bundled, so here is the equivalent evidence on the real model:
Performance — standard workloads (pp526 = 526-token prompt, tg128; fresh process per run, two runs each, same box/prompt/config, only
+21% prefill, +32–35% decode. Expert hit rate ~80% both (VK serves 15.2–15.5% of lookups from the VRAM tier with zero disk I/O; CPU-only covers the same hit rate from RAM alone — the delta is the GPU offload, not cache coverage). 3. New commit since the refresh: single-submit q-prep chain ( |
|
Rebased onto current dev ( Two new commits since the last push, both from profiling decode on the
Standard workloads, same box/prompt/config/convention as the tables above
Decode attention dropped 26.6 → 15.8 s per tg128 run; what remains of decode Validation on this exact branch build (RX 9070, RADV, Mesa 26.1.4): Follow-up we're preparing as a separate PR: a second-device Vulkan expert |
|
Tested this on RDNA4 (RX 9070 XT / gfx1201, RADV, Mesa 26.1.3) since @JustVugg mentioned not having a Vulkan GPU to verify against — it came up and ran correctly first try on a box it had never seen. Nice work. Two things that should be useful for the merge:
Also a small one: Full numbers, methodology and a ROCm/HIP comparison are in #523 rather than here, to keep this thread focused. There's an open question there about the |
…nse + MLA absorb, on top of main
|
Rebased onto current dev (
Also reworded the On the substantive #523 items: the |
Records the full Vulkan investigation as RESEARCH.md §7, plus updated benchmark table and resume points. Findings: - Vulkan is the fastest config on this box: 0.40 tok/s, +18.9% over HIP at ~12 GB and +23.7% at ~6 GB (10 runs/arm, no overlap between arms). - Resizable BAR is mandatory and fails silently without it: the tier lands in system RAM while logging "320 hot experts resident (6.04 GB VRAM)" with the card at 82 MB. 0.11 -> 0.24 tok/s once enabled. - MTP is an I/O amplifier: positions per emitted token = 4/(1+3a). At our 33% acceptance that is 2.0, matching measured ereq/token 330.1 -> 674.5 (2.04x). Costs ~35-40% on BOTH backends, so DRAFT=0 is the single biggest knob here. - Shrinking the CPU LRU 20x barely moves either backend: the VRAM tiers were doing the work, RAM was never masking residency. Also records the confound that made the first 50-run battery report the opposite conclusion (COLI_CUDA=1 trips the g_draft guard, so the HIP arms ran without speculation while the Vulkan arms drafted), and the misuse of experts/token, which is a router-side counter that cannot show tier behaviour. Both corrected upstream in JustVugg#523.
|
Rebased onto current dev ( One thing worth noting from it: dev picked up the fmt=6 E8/IQ3 container (#465, #501) in the meantime. The Vulkan paths are unaffected by construction — every VK entry point is gated on the Validation on this exact branch build (RX 9070, RADV, Mesa 26.1.4):
|
…A blocks Takes Qwen3.5-35B-A3B GPTQ-Int4 decode on the Arc B390 from 0.82 to 11.71 tok/s (all experts + attention + GDN on the iGPU). - fmt=6 expert-group bug: the shared expert is dense-quantized (fmt=2) but was folded into the fmt=6 routed-expert group; the fused gate_up shader decodes the whole batch with one fmt, so the mismatch rejected the ENTIRE group and dropped every routed expert to CPU. moe_token now folds the shared expert in only when its fmt matches (l->gsg.fmt == c->expert_fmt). 0.82 -> 7.09 tok/s. - shared expert rides the routed group: vk_dense quantizes it as fmt=6 when the container is fmt=6 (VkDense gained gs; quant_rows/vkd_make/mm_dense handle the grouped-asym format), removing ~80 mm_dense submits/token. 7.09 -> 7.59. - fused output->projection: coli_vk_gqa_project / coli_vk_gdn_project record the attention/GDN kernel + its o/out-projection matmul in ONE submit via a barrier; the intermediate ctx/y stays in GPU-only scratch (mirrors JustVugg#418 absorb_project). 7.59 -> 9.79. - fused whole GDN input block: gdn_conv.comp (causal conv1d + SiLU with a device- resident ring updated in-kernel) + gdn_delta_cv.comp (delta rule reading the raw conv output, q/k L2-norm folded in-shader). coli_vk_gdn_full chains gqkv/gz matmul -> conv1d -> qknorm+delta -> out-proj as 5 dispatches in one submit; only the tiny b/a->decay/beta stay on the CPU. COLI_VK_GDN_FUSE=0 keeps the staged path. 9.79 -> 11.71. All stages validated token-identical to the CPU/staged path. Microbenches: test_vk_gdn_fused.c (conv relerr 2e-8, delta y 3.6e-7). Diagnostics: qwen.c prints "[VK] routed experts: tier-served N", VK_EG_DBG=1 prints expert-group reject reasons.
|
Thank you a lot for your hard work! It's so exciting to see all the progress on local AI inference! 😃 Does this also help with iGPUs like the AI MAX 395? What's the correct/best setup for such a box? Set GPU RAM to max possible (e.g. 96 GB)? |
… RX 9070 backend_vulkan.c/.h + shaders/qmatmul.comp: opt-in Vulkan int4/int8 quantized GEMV via RADV, mirroring coli_cuda_matmul. Shader decodes int4 as nibble-8 (offset-binary), numerically consistent with the CPU path. Validated on the RX 9070 (RADV GFX1201, Mesa 26.1): the built-in VK_TEST harness passes all int4/int8 cases vs the CPU reference (maxrel ~1e-4). Bypasses ROCm's dropped Polaris support -> also targets the RX 580. Head-to-head naive int4 GEMV on the RX 9070 (same shapes, synchronous per-call): ROCm 0.115ms vs Vulkan 0.306ms (6144->1536, S=1) -- ROCm ~2.7x faster as-is (naive shader + host-visible memory + heavy submit overhead; not the tuned coopmat path that wins in llama.cpp). Next: optimize the shader/backend to reach or beat ROCm. Not yet wired into glm.c (Makefile/hooks pending).
Optimizations toward matching ROCm on the RX 9070: - backend: cache descriptor set + resubmit the prerecorded command buffer when tensor/shape/scratch are unchanged (single int4 GEMV 6144->1536: 0.306->0.214 ms). - shader: llama.cpp-style mul_mat_vec (MIT techniques, per-row int4/int8): x staged once in shared memory, one subgroup per output row + subgroupAdd (no barrier tree), grid-stride rows. Needs SPIR-V 1.3 (glslc --target-env=vulkan1.2). Correct (maxrel ~1e-4). - VK_TEST: batched throughput probe (N dispatches / one submit). Honest result (noise-controlled, vs colibri's production coli_cuda_expert_group): production ROCm ~0.179 ms/expert (fused dual gate+up + down, batched); our unfused VK ~0.260 ms/expert -- ~45% slower. VK wins the short-reduction down-proj, loses the long-reduction gate/up. Reaching ROCm needs fused dual gate+up + expert batching + a shape-adaptive reduction.
…pert New qmatmul_gate_up.comp + coli_vk_gate_up: computes hidden=silu(gate(x))*up(x) in ONE dispatch, reading x once for both projections (VK equivalent of colibri's grouped_hidden_w4_dual). Second 6-binding compute pipeline; build_pipeline() helper refactors the pipeline setup for both. Correct vs CPU ref (maxrel ~9e-5). The fusion is the win: gate+up drops from ~0.13 ms (two separate matmuls) to ~0.080 ms/expert (fair, 8 DISTINCT experts cycled so weights come from VRAM not L2 — added bench_experts_fair to control for the caching artifact). Full fused expert: VK gate_up ~0.080 + down ~0.050 = ~0.13 ms/expert ROCm production coli_cuda_expert_group = 0.179 ms/expert (stable) -> VK ~25-30% FASTER, holding under cache-controlled measurement (some run variance 0.07-0.10 on gate_up). Reaching this needed the fused dual projection, exactly the optimization the production HIP path already had and our unfused VK lacked.
…one submit) The real engine primitive: K experts, fused gate+up+silu then down, hidden staying on-device, all in one submit. Per-expert descriptor sets (gate_up: 6-binding, down: 4-binding) sliced into packed x/hidden/y via descriptor offsets; one phase barrier between gate_up and down. Mirrors coli_cuda_expert_group; correct vs CPU ref (maxrel ~7e-4 at K<=8). Honest throughput (RX 9070, K=8, distinct experts): GPU-only (resubmit recorded cmd buffer): 0.113 ms/expert -- BEATS ROCm 0.179 (~37%) per-call (as-is API): 1.03 ms/expert -- 5.6x SLOWER than ROCm The GPU compute is genuinely faster (fused kernel delivers), but the per-call HOST setup (80 descriptor updates + recording 16 dispatches every call) dominates. The earlier microbench 0.13 was the GPU-only number; it did not capture per-call cost. Fix (next): cache descriptor sets + command buffer across calls (hot experts are reused across tokens), or a bindless/BDA single-dispatch-multi-expert design like ROCm's grouped kernels. Either drops per-call toward the 0.11 GPU-only floor. (K=32 maxrel 2e-3, slightly over 1e-3 threshold -- fp accumulation, worth a look.)
Both projections read the same x row(s): coli_vk_matmul_pair stages x once,
records both dispatches, and waits one fence — replacing two full submit+wait
roundtrips per layer per token. Harness-validated (maxrel 4.4e-5, 0.123 ms/pair
vs ~0.35 for two singles); engine greedy still matches pure CPU ('Paris.').
Also: expert_group harness threshold 3e-3 at any K (the fp32 accumulation
lives in the 6144-length reduction chain, not in the expert count).
…ch overlapped with CPU rows Replaces the LRU-slot-tied tier (which uploaded on the decode critical path, churned with evictions, and computed ALL routed experts on the GPU while 12 cores idled — measured slower than CPU-only experts). - vk_registry_fill(): at startup, upload the top-COLI_VK_EXPERTS experts by persistent usage history into a (layer,eid) registry, decoupled from cache slots — stable residency like the HIP VRAM tier (RX 9070: 256 experts, 4.83 GB, 1.0s; pinned RAM slots feed uploads directly, the rest stream through one transient slot). - coli_vk_expert_group_issue/_take: the group gets its own command buffer + fence and splits into submit-and-return / join, so moe() ISSUES the GPU batch, computes the CPU share concurrently, drains the pipe + loads the GPU-side slots (cache invariants unchanged), then takes and accumulates. Sync coli_vk_expert_group (shared expert, harness) wraps the same path. - vk_active now keys on a non-empty registry; empty (no history/budget 0) falls back to the normal parallel CPU loop. Harness: issue/take validated bit-identical to the sync path; full PASS. Engine: 'Paris.' correct with the tier active.
…etch entirely The resolve phase now classifies registry-resident experts FIRST (before pin/LRU): they take no cache slot, dispatch no load, and skip the LRU recency bump — so they age out of RAM and free capacity for the CPU-served experts, the same effect CUDA_RELEASE_HOST gives the HIP tier. The pilot worker, cross-layer lookahead, and next-block readahead treat registry residency like RAM residency (decode only; prefill still loads normally since the tier serves S<=4). Counted as a new 'vk' bucket in the hit-rate split. The group block computes GPU entries straight from the registry (no ESlot), attributes its CPU share to t_ecpu/rows and only the take-wait to t_egpu, and on device-loss falls back via a transient ws slot load.
…ath was inverted Without a RAM pin every candidate loads transiently, and the inverted check skipped every SUCCESSFUL load (pin-fed fills masked it: the pins covered the whole top-256). PIN_GB=0 + COLI_VK_EXPERTS=256 now fills from disk in ~1.6s. Also: bail after 64 consecutive load failures, diagnostics on the first few.
256-384 measured flat within noise (1.73-1.78 tok/s); 320 is the best median with ~2.3 GB VRAM headroom left for the long-context KV mirror.
Upstream PR JustVugg#399 (KV8 fp8 / TQ4 quantized latent KV) leaves Lc/Rc NULL when a quantized tier is active — the mirror sync would deref NULL. Guard on the arrays themselves (env-independent), falling back to the CPU attention path; an fp8-aware VK mirror (upload Lc8+scale, decode e4m3 in the shader, 4x less mirror traffic) is the follow-up once JustVugg#399 merges.
Upstream JustVugg#168 (merged as 5e42e70) now carries the engine int3-g64 support this commit used to port, textually near-identical — including the uring_finalize_load qt_resolve_fmt fix we carried separately. What remains ours is the Vulkan side: vk_matmul_qt/pair accept fmt 5, the absorb kv_b/o and shared-expert paths feed fmt-5 tensors, and vk_registry_fill passes the true fmt through xf/d.fmt instead of assuming int4.
qmatmul/qmatmul_gate_up/attention_absorb learn the 24B-per-64-group two-plane layout: lanes stride the 16-value low-plane words, each word's partial dot is multiplied by its group scale before the subgroup reduction, and the per-row scale multiply is skipped. The absorb query pass stops pre-folding the kv_b row scale into q for fmt=5 (scales vary along K) and applies the (row,group) scale per element instead. upload_tensor accepts fmt=5 (rows are already word-aligned: ceil(I/64)*24), sizes the scale buffer O*ceil(I/64) via scale_floats() mirrored in tensor_free's accounting, and the expert-group prep validates up/down fmts (down may differ per-projection; phase 2 pushes downs[0]->fmt). Engine gates opened: vk_matmul_qt/pair, absorb kv_b+o, shared expert, and the pinned tier registry (int4/int3 accepted, fmt passed through). VK_TEST: fmt=5 cases across dense (incl. tail group + unstaged o_proj shape), fused gate_up, expert group K=8/32, absorb (+causal window), int3-vs-int4 fair throughput lines. Shaders validated against glslangValidator vulkan1.2.
- sample.h: COLI_LOGIT_DUMP=1 prints top-5 (id:logit) per pick_tok step — the tool that separated backend error from fp tie-flips (VK matched CPU logits to ~1e-4 at every matched step; run-to-run token divergence turned out to be engine-wide threading jitter, present on int4 CPU-only too). - harness: fmt=5 matmul_pair case (the q_a+kv_a decode path) and count=1 expert group (the shared-expert shape) — the two production paths the first harness round left uncovered. - tier messages made format-neutral (int4/int3-g64).
… — classify the VK-path call sites demand=1 on the moe CPU-share miss load (same semantics as the upstream miss path: routing-driven, FASE A snapshot valid); demand=0 on the device-lost VK-fallback reload (bookkeeping for vk-served experts diverges from the snapshot's assumptions — DISK-CLASS leaves it unclassified, per its own doc) and on the startup tier fill (matches the PIN-load sites).
pipe_ready() (non-blocking peek of the load-done flag; URING reports not-ready and defers to the wait) lets the block's CPU share compute loaded/resident experts FIRST, so a still-loading expert's I/O hides behind their matmuls instead of head-of-line blocking. Ordering is a hint only — every entry still pipe_waits/loads in its body. Measured neutral on a mirror config (waits already near their floor) and aimed at wait-heavy setups (single disk: felt waits 20-37s). t_ecpu now counts kernel time only, like the default path (it used to swallow the in-loop waits); waits land in t_ewait.
classify / issue / take+accumulate wall plus avg nvk/ncpu per block, printed under PROF next to t_ecpu/t_ewait/t_egpu — enough to see where the big-tier regression (420: 1.44 vs 950: 1.20 with LOWER felt wait) actually spends its extra 9 s: submit overhead growth, take waits, or outside the block entirely (dense/absorb under VRAM pressure).
…t-capped tier fill The big-tier regression was never the tier mechanics (VK-BLOCK: classify+issue+ take <=0.3s/run) — it was decode attention degrading with VRAM occupancy (7.8s at 7.6 GB resident -> 15.1s at 11.6 -> 17.8s at 15.2, identical workload): once expert weights crowd the heap, the kernel evicts and re-migrates buffers under the per-token attention submits. Two runtime-detected extensions fix the two halves: - VK_EXT_memory_priority: allocations now carry an eviction class — scratches and the KV mirror pin at 1.0 (they ride every submit), dense weights 0.75, and the engine brackets the expert-tier fill at 0.4. An oversubscribed heap sheds cold tier experts instead of thrashing attention. - VK_EXT_memory_budget: vk_registry_fill checks the device-local budget every 8 uploads and stops while COLI_VK_RESERVE_GB (default 3) is still free for the lazily-allocated dense + KV mirror + staging, logging the stop. Reserve 0 disables the stop; without the extension the count cap applies as before. Both are no-ops on drivers without the extensions.
… submit tax Isolation matrix + harness probes pinned the big-tier regression precisely: per-submit driver cost grows ~linearly with the number of distinct device-memory objects the queue actively references (~0.35 ms/submit extra at a 950-expert tier's ~5.7k allocations), hitting every synchronous fence-waited path — the dense projections (4-5 submits/layer, accounted in t_attn) most of all. Decode attention: 7.9s @420 experts -> 17.6s @950; flat when dense+attn run on CPU; IDLE allocations are free (ballast probe: absorb and dense GEMV unchanged under 5000 dummy allocations); eviction exonerated (2.9 GB free showed the same tax, scaling continuous in tier size, no budget-edge cliff). upload_tensor now binds wbuf/sbuf into shared 256 MB arena blocks (alignment- respecting bump allocator, memory-priority tag carried over) instead of one vkAllocateMemory per buffer: a 950-expert tier + dense drops from ~7k memory objects to ~60. Arena slices are not reclaimed per-tensor (registry/dense live for the process; the rare fill-failure free leaks its slice — tensor mem handles stay VK_NULL_HANDLE, vkFreeMemory's documented no-op). Arenas are unmapped/freed at shutdown. Scratches and the KV mirror stay standalone (few, constant count).
…inear submit tax)
… semantics The merged JustVugg#298 gave the CUDA kernels per-group scale handling for fmt=4 — without the same semantics a g64 container on Vulkan would decode with per-row scales (the exact bug JustVugg#298 fixed). All three shaders gain a fmt==4 branch: int4 nibble decode with one scale per gs inputs, applied to the packed-word partial (host gates gs to multiples of 8 so a word never straddles a group; per-row scaling is skipped like fmt=5). Group size flows as an explicit parameter through the upload-triggering entries into ColiVkTensor and the push constants; engine call sites pass QT.gs and the VK gates accept fmt=4 via VK_FMT_OK (word-aligned gs only — anything else stays on the CPU path, which JustVugg#298 already fixed). Harness: ref helpers generalized to runtime group size (g_ref_gs); fmt=4 cases at gs=64 across the real shapes (dense, o-proj, batch, expert_group, matmul_pair, absorb incl. S=2 causal + window) plus a gs=32 sanity case.
The attention section paid three synchronous fence roundtrips per layer per token (pair, q_b, absorb+o); the middle one existed only because the q-latent RMS norm ran on the CPU between q_a and q_b. A 40-line rmsnorm.comp (one workgroup per row; fp32 vs the CPU's f64 accumulate differs ~1e-7 at D=1536) lets coli_vk_attn_qprep record the whole chain in one command buffer with compute barriers: [q_a + kv_a] -> norm -> q_b, one fence. q, the kv latent, and the NORMED q latent return to the host (the DSA indexer consumes the latter from QR; RoPE + the canonical KV append stay CPU-side). Norm weights upload once per layer (KV-mirror pattern). The engine tries the chain first and falls back whenever rmsnorm.spv is absent, formats mismatch, or the call fails; COLI_VK_QPREP=0 forces the split path, =2 runs both and prints per-layer chain-vs-split divergence. Validation: harness chain cases vs a matmul->rmsnorm->matmul CPU reference (int8/int4, S=1/2/11, real GLM shapes; threshold 1e-2 — two quantized reductions + the norm through cancellation-heavy random rows compound ~10x a single GEMV, same reasoning as the expert_group 3e-3). Engine COLI_VK_QPREP=2 on the real container: all 78 layers within 1.8e-4 of the split path, kv bit-exact; decode-context logits match to 3e-5. Step-1 logits shift O(1) with the chain on — measured to be the container's sensitivity to ANY fp-order change, not a chain defect: pure-CPU vs split-VK step-1 logits differ ~4.0 on the same prompt with the same stable argmax.
The ~8 GB of dense weights (attention projections, absorb kv_b, o-proj, shared expert) uploaded lazily on the first forwards — after the expert tier had already filled to its budget. On a 16 GB card the late arrivals overflowed to GTT (measured 2.1 GB spilled with a 320-expert int4 tier) and every per-token attention submit paid PCIe latency: absorb core 1.69 -> 2.26 ms/call. Claiming the dense set at startup lets the budget-capped tier fill self-size to what actually fits: zero spill, decode attention 13.1 -> 9.4 s per tg64, +7% decode.
Every sync submit's vkWaitForFences pays a scheduler wake (~50-150 us) on signal; the engine fences ~2 submits per layer per token. Poll vkGetFenceStatus for a short budget first (default 300 us, the common decode dispatch completes in 0.5-2 ms), then fall back to the blocking wait. The spinning thread is stalled on the GPU result anyway. COLI_VK_SPIN_US=0 restores the pure blocking wait.
… to the binary Field report JustVugg#523 (RX 9070 XT): with Resizable BAR disabled the HOST_VISIBLE|DEVICE_LOCAL combination exists only in a ~256 MB BAR window; the driver then places tier allocations in system RAM while the resident- experts log looks healthy, and the run comes out slower than CPU-only with nothing indicating why. Compare the chosen memory type's heap against the largest DEVICE_LOCAL heap at init and say so up front. Also from the same report: the COLI_VK_SHADERS fallback was CWD-relative, so the engine only started from inside c/. Resolve the default next to the binary (/proc/self/exe), accept a directory as well as the .spv path, and name what was tried in the startup error. Makefile note on the ROCm comparison now states what was measured (the VRAM-resident expert-group primitive, per-call) instead of implying end-to-end tok/s.
…hmarking notes The benchmarking section captures the two defaults that silently skew any Vulkan-vs-CUDA/HIP comparison (DRAFT auto-resolves to 0 under COLI_CUDA per JustVugg#163 but stays 3 elsewhere — the arms then run different decode loops; and DPM never ramps for microsecond-burst dispatches), plus what the experts-loaded/token stat actually counts.
The eroute readahead (last forward's routing, hinted at layer entry) was the one prefetch path without a vk_reg_served check. Tier-served experts are never demand-read, so their pages never stay warm: on a RAM-tight streaming box the hint re-fetches the same hot experts from disk each time cache pressure evicts them — pure wasted bandwidth on the box class JustVugg#523 measured on. Prefill (S>4) keeps the hint: the tier only serves decode, so those bytes are still needed.
The README stated "measured faster than ROCm on RDNA4" unqualified; a careful third party (JustVugg#523) measured the opposite end-to-end on a storage-bound box (the gap turned out to be a DRAFT/speculation confound, but the unqualified claim was the problem). Lead with the unassailable point — Vulkan is the only backend for cards ROCm dropped — and call the RDNA4 speed "competitive", pointing at the docs' benchmarking notes for the honest, config-dependent picture.
The fmt=5 branches decoded one value at a time into a single accumulator: 16 products chained through one FMA dependency, a per-value `i < I` bounds test, and (in qmatmul) a per-value staged/unstaged select on a uniform condition. That left the branch ALU-latency-bound rather than bandwidth-bound -- int3 moved 12.5% fewer bytes than int4 yet ran 1.41x SLOWER per expert. Every container the converter emits pads I to whole 64-groups, so the common path now drops the bounds test, hoists the uniform select out of the loop (the int8/int4 branches already did this), pre-shifts the high-plane word so its bits start at bit 0, and accumulates into four independent lanes. The bounds-checked loop stays for ragged dims. RX 9070 (RADV, Mesa 26.1.4), VK_TEST harness, medians of 3 runs: expert_group fmt=5, 32 experts 0.1450 -> 0.0667 ms/expert 2.17x fused gate_up int3, 8 experts 0.0929 -> 0.0486 ms/expert 1.91x matmul_pair fmt=5 6144->(2048,576) 0.217 -> 0.149 ms/call 1.46x absorb fmt=5 K=512 T=300 0.791 -> 0.688 ms/call 1.15x int3 now decodes FASTER than int4 on the same shapes (int4 expert_group/32 is 0.1038 ms/expert), so the container's byte saving finally converts to speed on the GPU path instead of being eaten by the unpack. Exactness harness PASS. Four-way accumulation is also slightly more accurate than the 16-deep serial chain (expert_group maxrel 1.07e-4 -> 7.83e-5). End-to-end on this box the win is invisible, and the profile says why: the tier is issued ASYNC and overlaps the CPU expert share, so `routed GPU critical` is 0.027s of an 8.71s M2.7 decode (0.3%), tier straggler 89.69x. M2.7 stays I/O-bound (31% of decode waiting on expert reads; 4.79/4.57 -> 4.76/4.86 tok/s, inside run-to-run spread). The kernel win lands where the GPU is the critical path -- a larger tier share -- and it removes decode cost as an argument against fitting more experts per VRAM byte.
…each VRAM CUDA gained an fmt=6 kernel in JustVugg#627; on Vulkan the format was still refused at upload, so every E8 tensor stayed CPU-side. That inverts what the container is for: E8 costs 3.0625 bpw against int3-g64's real 3.5 (3.00 weights + 0.5 scale), so it exists to fit ~12% MORE experts in the same VRAM — but as shipped it put all of them on the CPU tier instead. Decode strides 32-weight SUB-blocks rather than elements: that is the granularity at which the sign/scale word changes, and the two 4-dim grid rows a lane touches stay in L1 across its four 8-weight lanes. Every field inside a 98-byte super-block is 4-aligned relative to the block START, but 98 % 4 == 2, so blocks alternate 4- and 2-aligned within the row — hence the byte-wise word assembly, with the straddle shift only ever 0 or 16. The fp16 super-scale is read 16 bits wide on purpose: its 32-bit window can reach past the last row of the slab, its 2 bytes never do. The four lanes are expanded and consumed as two vec4 chains for the same reason dot16_* splits its accumulator. The rotation splits exactly as on the CPU and CUDA paths. All routed experts of a layer share the gate/up input, so the CALLER rotates it once per layer; the down projection's input is the per-expert silu product, produced and consumed on the GPU, so e8_rot.comp rotates it in place between the two phases of the same expert-group submit. It runs over the whole hidden buffer at once — the transform is per row, so every expert's slice rides one set of dispatches instead of paying the per-expert cost. Blocks tile block-diagonally as e8_rot_rows does, each staged in shared memory. Only the ROUTED expert projections are ever fmt=6: --ebits/--io-bits are integer bit widths and only --xbits/--{up,gate,down}-bits take "e8", so attention and dense tensors cannot reach the format and VK_FMT_OK is left alone. The converter also refuses any input dim that is not a whole number of 256-weight blocks, which upload_tensor re-checks — so the shader has no ragged tail. moe() fed x straight to the tier at three sites inside the Vulkan block; all three now take the rotated copy. The registry serves an expert with NO RAM slot, so its format is only knowable from the resident tensor — hence coli_vk_tensor_fmt. The block's two CPU fallbacks (the CPU share and the device-lost recompute) were also missing the down-input rotation the main CPU path does; unreachable before, since fmt=6 could not be uploaded at all. shaders/e8_grid.glsl carries a second copy of quant.h's codebook because GLSL cannot include the C header — the same arrangement tools/iq3xxs_grid.json has for the Python codec. Without shaders/e8_rot.spv, coli_vk_e8_ok() reports 0 and fmt=6 stays on the CPU rather than decoding against an unrotated input. Tests: make vk-test checks the shader decode against matmul_e8 and the full expert MLP against the CPU path, with int4 on one side to isolate the fused gate_up branch from the rotation. Reported as relative L2, which is the meaningful metric for these sign-cancelling dot products; fmt=6 lands at 3e-7..5e-7, at or below the int4 baseline's 1e-6. Green on RADV GFX1201 (RX 9070) and llvmpipe — both subgroup widths. make check 220, and the existing VK_TEST harness is unchanged for fmt 1/2/4/5 on both. Refs JustVugg#452, JustVugg#627
|
Rebased onto current dev ( That is a change of position from my last comment, where I said the VK paths were unaffected by the E8 container "by construction" because every entry point is gated on What it doesDecode in The rotation, which is what the expert tier actually needs. fmt=6 stores Scope is narrower than it might look:
If ValidationNew Errors are relative L2 on purpose — per-element relative error is the wrong metric for these long, sign-cancelling dot products, where individual outputs land near zero by cancellation and their relative error explodes on summation order alone. fmt=6 sits at or below the int4 baseline. Green on RADV GFX1201 (RX 9070) and llvmpipe, so both subgroup widths.
MeasuredGLM-5.2 (744B, E8 container converted with
+24%, generated text character-identical between the two. 440 experts resident in 6.36 GB = 14.45 MB/expert, exactly 3 × (98 B per 256 weights) — about 14% more experts per GB of VRAM than int3-g64, which is the capacity argument for the format. Two caveats so these are not read as more than they are. They come from a container whose CPU-side One thing worth flagging to maintainers
|
4 participants
Summary
An opt-in Vulkan compute backend (
make glm VK=1, runtimeCOLI_VULKAN=1) that runs the GLM decode compute path on any GPU with a Vulkan 1.2 driver — no CUDA, no ROCm. Three tiers, each independently switchable and each falling back to the CPU path on any failure:COLI_VK_EXPERTS, default 320): the top-N routed experts by.coli_usageheat are uploaded once at startup and served from VRAM with no RAM slot, no disk read, no prefetch — issued as one fused async batch per layer (gate+up+silu→down, hidden on-device) that overlaps with the CPU computing the remaining experts. Appears as a newvkbucket in the hit-rate line.COLI_VK_DENSE=1): q_a+kv_a fused into one submit, q_b, the o-projection, and the shared expert as a single fused expert-group submit.COLI_VK_ATTN=1): one dispatch per layer covering absorbed query → scores → softmax → weighted latent → value rows, fused with the o-projection so the context vector never leaves the GPU. The latent/rope KV lives in a persistent per-layer device mirror appended at ~2.3 KB/token/layer, with the same invalidation points as the CUDA KV shadow (row rewrite, rebind, resize).This is a continuation of @kryptt's #84 (the original backend skeleton, tensor-resident
coli_vk_matmul, and the int4 nibble-8 shader decode are his design — thank you!). It addresses the two things asked for there: it is rebased onto currentdev, and it ships a CPU-reference exactness harness covering every primitive.Why
libvulkan+ a Vulkan 1.2 ICD with subgroup arithmetic, andglslcat build time.Correctness
gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk shaders/qmatmul.spv): CPU-reference exactness for every primitive — int4/int8 GEMV across shapes including the long-row o-projection (I=16384), fused gate+up, the full expert group (sync and async issue/take, verified bit-identical to each other), the q_a+kv_a pair, and the absorb attention core including causal S=2,kv_startwindows, int8, and long-context (T=2000) cases. Typical maxrel 1e-5…2e-3 (fp32 reduction order on 6144-long dots).#ifdef COLI_VULKAN;make glmwithoutVK=1compiles the same code as before (plus three always-false locals).make test-cfully green on the branch.Measured performance
Hardware/software: AMD RX 9070 16 GB (RDNA4/gfx1201, PCIe 4.0 x16 effective), Ryzen 9 3900X (12C/24T Zen2,
OMP_NUM_THREADS=12, threads pinned), 64 GB DDR4-3200 dual-rank, GLM-5.2 744B int4 streamed from NVMe (~19 MB/expert). Mesa/RADV 26.1.4, Vulkan 1.4 ICD; ROCm 7.2.4 + HIP for the comparison backend. Model: 78 layers, 256 experts/layer, kv_lora 512, MLA/DSA.Primitive microbenchmarks (per-call including readback, K-expert int4 MLP 6144→2048→6144):
End-to-end GLM-5.2 decode (same box, same engine settings, expert-routing top-p 0.7, drafts off, interleaved/batched A/B runs; our fork additionally carries an unrelated dual-SSD-mirror patch on the same base — active for both backends in every comparison, so the deltas are backend-attributable):
Run-to-run variance on this box is ±0.1 tok/s; the 64-token advantage (~+13%) narrows to ~+5% at long form as the attention window and expert-I/O share grow for both backends.
The two memory-type rules that made it work (documented in
docs/vulkan.md, learned the hard way):HOST_CACHED— reading write-combined ReBAR VRAM from the CPU runs at ~40 MB/s and was a hidden 5.6× per-call penalty.HOST_VISIBLE|DEVICE_LOCAL, so uploads are plain memcpys and the GPU reads at VRAM speed.Integration surface
c/backend_vulkan.c(~1.3k lines, plain C99 + libvulkan),c/backend_vulkan.h,c/shaders/qmatmul.comp,c/shaders/qmatmul_gate_up.comp,c/shaders/attention_absorb.comp,docs/vulkan.md.c/glm.c(hooks under#ifdef COLI_VULKAN),c/Makefile(VK=1block +glslcrule),docs/ENVIRONMENT.md,README.md.COLI_VULKAN,COLI_VK_SHADERS,COLI_VK_EXPERTS,COLI_VK_DENSE,COLI_VK_ATTN.One operational note worth flagging beyond this PR: the engine's OMP self-tune (
OMP_WAIT_POLICY=active+ spin) is skipped underCOLI_CUDA/COLI_METALbut not under other configurations. With 12 pinned spinning threads the async I/O pool starves (measured 28 → 5 GB/s CPU expert bandwidth, ~2.8× end-to-end).docs/vulkan.mdtells Vulkan users to setCOLI_NO_OMP_TUNE=1; extending the self-tune skip to any GPU backend (or toPIPE=1runs) may be worth a separate look.Limits / future work
S≤4; prefill keeps the CPU/batched paths (dense projections do run on VK at prefill, which is why prefill got faster).KV8=1(already implemented and validated againstkv_fp8.hon our side — the absorb shader decodes e4m3 in place, 4× less mirror traffic; this PR just cleanly falls back to CPU attention when the f32 cache is absent), cooperative-matrix prefill kernels, and a fully resident-layer pipeline.How to run