Repository navigation
Preserve histogram probability mass in entropy calibration - #34180
Open
Afloat (Afloat16) wants to merge 2 commits into
Open
Afloat (Afloat16) wants to merge 2 commits into
Afloat (Afloat16) wants to merge 2 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified in the entropy-calibration histogram reconstruction changes.
No open issues 🎉
What changed in this PR
This PR corrects histogram reconstruction in entropy calibration so KL-divergence comparisons preserve bucket probability mass and histogram support.
Changes:
- Use floating-point reconstructed counts to avoid truncating fractional averages.
- Include remainder bins in the final bucket and distribute mass only over occupied bins.
- Add regression tests covering threshold selection, CPU calibration, persisted ranges, static quantization, and inference.
No actionable findings were identified.
| File | Description |
|---|---|
onnxruntime/test/python/quantization/test_entropy_reconstruction.py |
Adds reference-based regression tests and CPU calibration-to-inference coverage. |
onnxruntime/python/tools/quantization/calibrate.py |
Fixes reconstruction precision, final-bucket coverage, and support masking. |
Review effort: Balanced
mirounga
self-requested a review
November 13, 2026 16:12
| from pathlib import Path | ||
|
|
||
| import numpy as np | ||
| import onnx |
Remove the mixed import forms flagged by CodeQL without changing the test behavior.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Correct reconstruction of a quantized histogram when evaluating entropy-calibration thresholds:
These changes make the reconstructed distribution preserve the quantized bucket mass and original support. Candidate threshold enumeration, outlier handling, smoothing, KL selection, and threshold dtypes retain their existing behavior.
Motivation and Context
Integer truncation and incomplete expansion can change the distribution compared by KL divergence, selecting an incorrect clipping interval. The regression cases use independent Python bucket reconstruction to check mass conservation and support, then exercise real CPU calibration, persisted calibration data, static quantization, and quantized inference.
For the fixed integration fixture, correcting histogram reconstruction changes the stored UINT8 activation scale from
16/255to the independently expected13/255. This is a correctness result for that fixture, not a claim of a general model-accuracy gain.Validation
tools/ci_build/build.pyon Ubuntu 24.04 / Python 3.12. The source-build job completed compilation and produced the wheel.git diff --checkpass.Validation covers the CPU calibration/quantization paths above; the full C++ and GPU suites were not run.