[Qiming Bao (14H034160212)]

Description

On x64, MatMulNBits with accuracy_level=4 (CompInt8) returns zeros when scales or zero_points is not a constant initializer.

The x64 CompInt8 kernels read the scales and the block sums (derived from the scales and zero points) from packed B, and ignore QuantBScale. PrePack(B) can only fold in constant scales and zero points. When they are runtime inputs, B is packed without them, they are never added later, and that part of the packed buffer stays zero.

ARM64 already handles this case for KleidiAI: RequiresDynamicQuantizationParameterPrepackFallback declines prepacking B, so the node computes from unpacked B. This change makes that check platform-aware and applies it on x64 to CompInt8 with 4- and 8-bit weights. The ARM64 condition is unchanged. The three call sites (both PrePacks and UseSharedPrePackedBuffers) are no longer ARM64-only. Nodes with constant scales and zero points are not affected.

The x64 test for dynamic metadata (Float32_4b_Accuracy4_ForceFp32DynamicMetadata) forces CompFp32, so CompInt8 with runtime scales was not covered.

Motivation and Context

Fixes #34216.

That issue is one way to hit this: an FP16 DequantizeLinear -> MatMul model at ORT_ENABLE_EXTENDED. The QDQ fusion produces MatMulNBits with accuracy level 4 (the default), and the CPU EP inserts a Cast on the fp16 scales, which makes them a runtime input. With ORT_ENABLE_ALL the Cast is constant-folded, so that level is not affected.

The same happens for float models whose scales are graph inputs or computed in the graph. With onnxruntime 1.31.1 and a small MatMulNBits (K=32, N=4, accuracy level 4), runtime scales give an output error equal to the largest output for both 4-bit and 8-bit weights, while constant scales are correct. 1.29.0, 1.30.0 and 1.31.0 behave the same.

The fallback path dequantizes B instead of using the int8 kernel, so these nodes are slower than with constant scales, but correct.

#34197 is a separate issue (accuracy level 1 and FP16 rounding of the dequantized weight) and is not changed here.

Tests

• New MatMulNBits.Accuracy4_4b_RuntimeScalesOrZeroPoints covers runtime scales and, separately, runtime zero points.
• New MatMulNBits.Accuracy4_8b_RuntimeScales covers runtime scales. It needed a scales_are_initializers option in the 8-bit test helper.
• Both use float and MLFloat16, with block sizes 16 and 32.
• On main both fail on every case with the output all zero (for example cur_expected[i] evaluates to -0.2037, difference 0.2037). With this change they pass.
• onnxruntime_provider_test --gtest_filter='MatMulNBits*:*MatMulNBits*:*QDQ*MatMul*:*DQMatMul*': 78 passed, 9 skipped (ARM64/KleidiAI/WebGPU-only), 0 failed (Linux x64, CPU, Release).
• End to end with the model from [CPU EP] FP16 DQ + MatMul fusion returns zero with MatMulNBits accuracy_level=4 #34216, onnx_test_runner -e cpu -o 2:
  • On main: expected 0.25, got 0.
  • With this change: passes.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

The new test fails with zero outputs on ARM64 CPUs with INT8 dot-product support because the 8-bit path reads unpopulated packed scales; it must be restricted to x86/x64 or the ARM64 fallback extended to handle 8-bit runtime scales.

1 open issue

• Restrict the INT8 regression test to x86 or add ARM64 scale handling · New

What changed in this PR

Fixes #34216 by routing x64 CPU MatMulNBits CompInt8 operations with runtime scales or zero points through unpacked computation instead of incomplete packed weights.

Changes:

• Makes the prepacking fallback platform-aware for 4- and 8-bit x64 weights while preserving ARM64 behavior.
• Applies the check to both prepacking implementations and shared-buffer reuse.
• Adds float and FP16 regression cases for runtime quantization parameters.

File	Description
onnxruntime/​test/​contrib_ops/​matmul_8bits_test.cc	Enables runtime scales in the helper and adds 8-bit regression coverage.
onnxruntime/​test/​contrib_ops/​matmul_4bits_test.cc	Adds regression coverage for runtime scales and zero points.
onnxruntime/​contrib_ops/​cpu/​quantization/​matmul_nbits.cc	Extends prepacking safeguards to x64 CompInt8 with runtime metadata.

_{Review effort: Balanced}