[Afloat (Afloat16)]

Description

Packing INT4/UINT4 weights currently runs a Python iteration for every pair of elements. Add a NumPy bitwise path for bytes and bytearray inputs, writing the low and high nibbles into the destination bytearray.

The layout, odd-length zero padding, return type and input ownership are unchanged. Other sequences retain the original element-wise path, including noncontiguous and non-byte memoryviews.

Validation

• Seven new unittest methods cover all signed/unsigned nibble pairs, empty/odd/even lengths, bytearray ownership, strided/reversed/signed/wider buffer inputs, explicit nibble order and persisted INT4/UINT4 initializer bytes.
• Native CPU DequantizeLinear followed by MatMul checks actual quantized initializers against independently reconstructed float16/float32 weights.
• New and existing quantization utility tests: 16 methods passed, 163 subtests passed.
• The same new tests pass on the original helper: 7 methods, 32 subtests. This is a performance improvement; the original packing outputs are already correct.
• Ruff lint/format and git diff --check passed.

Exact-source native validation and benchmark: https://github.com/Afloat16/onnxruntime/actions/runs/43401492131

Measured helper performance

One warmup and five timed calls per case, with median time reported. Input construction is outside the measured region. Peak Python/NumPy allocations are measured separately with tracemalloc; packed output hashes match in every case.

Input elements	Original median	Patched median	Original peak	Patched peak
32	0.002233 ms	0.005738 ms	73 B	978 B
4,096	0.298420 ms	0.006619 ms	2,261 B	5,134 B
1,000,000	74.687110 ms	0.519886 ms	500,213 B	1,001,038 B
8,000,000	571.160013 ms	7.104263 ms	4,000,213 B	8,001,038 B

The shifted odd-element array adds temporary storage of roughly half the input length. Tiny inputs are slower, as shown above. These results concern this packing helper only, not end-to-end quantization throughput or model accuracy.

The native runtime comes from the repository's official CPU Release build.py/CMake build at 8dea119. Its C++ sources match base a8246c0. The validation installs that built wheel and overlays the complete quantization package from this exact patch, verifying every Python file byte-for-byte and recording a SHA-256 manifest. Tests use normal native imports. This is targeted Linux CPU validation, not GPU execution or the full C++ suite.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟢 Approval recommended

No active review issues were identified for the INT4/UINT4 weight-packing optimization.

No open issues 🎉

What changed in this PR

This PR accelerates INT4/UINT4 weight packing in the quantization utilities while preserving nibble order, odd-length padding, return type, and input ownership.

Changes:

• Adds a NumPy fast path for bytes and bytearray, retaining element-wise packing for other sequences.
• Adds seven tests covering packing correctness, buffer compatibility, serialized initializers, and native CPU dequantization with MatMul.

No actionable findings were identified.

File	Description
onnxruntime/​test/​python/​quantization/​test_four_bit_packing.py	Adds packing regression tests and native CPU integration coverage.
onnxruntime/​python/​tools/​quantization/​quant_utils.py	Vectorizes byte-input packing with NumPy bitwise operations.

_{Review effort: Balanced}