[BBaditi]

Description

Adds ep.cuda.moe_gemv_max_expanded_rows, a CUDA session option that caps the routed row count, tokens x k, for which QMoE runs its INT2/INT4/INT8 MoE GEMV. Larger counts take the prefill path, which is the grouped GEMM for INT4/INT8. The value is an integer in [0, 80]: 0 never uses the GEMV, and the default of 80 (moe_gemv::kMaxProfiledExpandedRows, the GEMV's own limit) keeps today's routing. QMoE reads it once at kernel construction, and any other value fails session creation.

The value reaches the runner through QuantParams and joins the existing FC1 and FC2 GEMV disable terms (MoeGemvFc1Disabled and fc2_gemv_disabled), so the skip-expand prediction in runMoe and the FC1 launch in gemm1 keep making the same decision. QMoE's packed INT2/mixed-width GEMV checks it as well. The INT3 and FP4 decode kernels have their own selectors and are not affected.

This also adds the guard suggested in #34168 (comment): gemm1 hands the split-K SwiGLU partials to the GEMV only up to kMoeGemvSplitKSwiGLUMaxRows, the constant getWorkspaceDeviceBufferSizes now uses to size them, so the buffer and its consumer cannot drift apart. The buffer stays sized for 80 rows whatever the option says.

Motivation and Context

Follow-up to #34168, which capped the split-K partials at the GEMV's 80-row limit: should that limit be configurable, does prefill use the GEMV, and should prefill go to the GEMM instead?

Prefill uses the GEMV only when tokens x k <= 80. For Qwen3-30B-A3B (top-8) that means calls of at most 10 tokens, such as very short prompts, small decode batches or speculative verification; 11 or more tokens already run the grouped GEMM.

Measured on A100 80GB (SM80) with layer 0 of Qwen3-30B-A3B-Instruct-2507 as a standalone QMoE model (FP16, INT4 block 128, 128 experts, router included), IOBinding, sessions in A-B-B-A order with 200 timed calls per token count each. Median microseconds per call, limit 80 (current routing) vs limit 0 (grouped GEMM everywhere):

tokens x k	top-8	top-8, every token on the same 8 experts	top-4 (same layer, k=4)	top-2 (same layer, k=2)
8	84 vs 122	85 vs 122	88 vs 129	87 vs 129
16	108 vs 150	101 vs 133	107 vs 141	102 vs 142
32	139 vs 161	130 vs 138	127 vs 160	120 vs 151
48	150 vs 194	130 vs 138	143 vs 189	140 vs 188
64	165 vs 197	147 vs 132	159 vs 196	159 vs 196
80	179 vs 220	164 vs 131	175 vs 198	176 vs 218

• With routing spread over experts the GEMV is faster at every row count up to 80, by 11% to 39%, so the default stays at 80.
• When every token picks the same 8 experts, the GEMV is still faster at 8, 16, 32 and 48 rows but the grouped GEMM wins at 64 and 80 rows (the GEMV is 12% and 25% slower). Deployments with such concentrated routing, or GPUs other than SM80, can lower the limit with this option. Concentrated routing was measured for top-8 only.
• With ORT_MOE_GEMV_FP32_ACCUM=1 (the split-K two-pass GEMV) the GEMV is still faster at every row count, by 4% to 22%.
• Above 80 rows both settings run the grouped GEMM and the outputs are bitwise identical. GEMV and grouped GEMM outputs differ by at most 1.7e-3 relative L2 per token.
• Nsight Systems shows the expected kernels at each limit. For example, with a limit of 8 one token runs moe_gemv_interleaved_swiglu_kernel and moe_gemv_fused_finalize while four tokens run the CUTLASS grouped GEMM, and with a limit of 48, 6 tokens run the GEMV and 7 tokens the grouped GEMM.

End to end with the full Qwen3-30B-A3B-Instruct-2507 through ORT GenAI on the same GPU (30 requests per prompt length and setting, A-B-B-A processes), prefill of 2, 4, 8 and 10-token prompts takes 8.05, 9.44, 11.74 and 14.26 ms with the current limit vs 9.29, 10.34, 12.31 and 14.77 ms with limit 0, while 11 and 16-token prompts take the same time with both (15.05 vs 15.05 ms, 16.01 vs 15.97 ms). That run set the limit through an environment variable that an earlier revision of this branch also read; this PR only has the session option, and for a given limit the routing code is the same. The limit applies to every call, so a value below 8 also moves batch-1 decode (8 rows at top-8) to the grouped GEMM.

Tests:

• MoETest.QMoETest_Bf16Int4GemvMaxExpandedRows: 12 tokens x top-4 = 48 routes with limits 0, 47 and 48 must all match the CPU reference, and -1, 81 and abc fail session creation. With the skip-expand prediction in runMoe changed to ignore the limit, the limit 0 and 47 cases fail at the gemm1 check "FC1 activations were not expanded but the MoE GEMV did not run". It passes 10 of 10 with --gtest_repeat=10.
• MoETest.* passes on A100 (125 passed, 14 skipped), and so do the other MoE/QMoE provider tests and the CUDA EP internal MoE tests, including QMoEWorkspaceTest.

Limits:

• Measured on SM80 only; CI covers SM86, Windows and the plugin EP build.
• The FP4 (64-row) and INT3 decode kernels are not covered by the option.