[Afloat (Afloat16)]

Description

Compute distribution-calibration histogram moments in float64 before converting the resulting mean and standard deviation back to the histogram dtype:

• Form bin midpoints after promotion, so finite float16 endpoints do not overflow during addition.
• Apply the existing signed-power transform in the promoted dtype.
• Compute the second central moment directly, avoiding the overflow, underflow, and cancellation in E[x²] - E[x]².

The change preserves the histogram's output dtype, signed-power convention, empty-histogram behavior, and zero variance of a single occupied bin.

Motivation and Context

Finite float16/float32 histograms can currently produce infinite, zero, or NaN standard deviations even when the correct result is representable. This propagates into distribution calibration ranges and FP8 quantization scales.

The regression tests compare histogram moments with independently expanded observations and exercise real CPU MatMul calibration, persisted calibration data, and static FP8 quantization. They cover midpoint overflow, squared-moment overflow/underflow, cancellation, signed powers, and control cases.

Validation

• Built the Release CPU runtime from this exact source commit with the official tools/ci_build/build.py entry point on Ubuntu 24.04 / Python 3.12. The source-build job completed compilation and produced the wheel.
• The successful native validation downloads only that source-built wheel, verifies its source commit and installed calibrate.py bytes, and runs the new tests together with test_calibration.py: 31 passed, 15 subtests passed.
• Restoring the original calibrate.py into the same native runtime reproduces 7 failing test methods and 6 failing subtests; the remaining controls pass. The initial build job's test-collection path was corrected in the linked follow-up validation.
• Ruff lint, Ruff formatting, and git diff --check pass.

Validation is limited to the CPU calibration/quantization paths above. This does not claim support for arbitrary extreme float64 powers or a general model-accuracy improvement; the full C++ and GPU suites were not run.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟢 Approval recommended

No active review issues block the numerical-stability improvements to distribution-calibration moments.

No open issues 🎉

What changed in this PR

Improves numerical stability of distribution-calibration moments, helping prevent invalid calibration ranges and FP8 quantization scales for float16 and float32 histograms.

Changes:

• Promotes bin edges to float64 before computing midpoints, signed powers, and moments, then restores the output dtype.
• Computes variance directly from centered values to avoid cancellation and intermediate overflow or underflow.
• Adds numerical regression tests, CPU calibration checks, and static FP8 quantization coverage.

File	Description
onnxruntime/​test/​python/​quantization/​test_distribution_moments.py	Adds moment regression tests and CPU calibration and FP8 integration checks.
onnxruntime/​python/​tools/​quantization/​calibrate.py	Uses float64 intermediates and centered variance for stable histogram moments.

_{Review effort: Balanced}