Model weights compression / effective 4-5 bit quant#1630
Open
dxqb wants to merge 5 commits into
Open
Conversation
Squashed import of PR Nerogar#1623 (which contains Nerogar#1617), the dependency this feature is stacked on.
Quantized weights (fp8, W8A8, GGUF-A8) can be stored nvCOMP-compressed in VRAM and decoded on the fly during forward/backward, trading a small decode cost for lower resident VRAM. Compression is an opt-in per-component "Compress" toggle on the model tab, gated on ModelType.supports_compression() and wired generically through the centralized _setup_model_part call. quantize_layers stamps the flag onto the layers, guards against enabling it for a non-compressible dtype or without nvCOMP, and prints a per-model "% saved" line. Two decode backends are selected by import: a low-level batched-ANS path (via ctypes, sync-free decode; preferred) and a high-level nvCOMP Codec fallback. AutogradFunctionWrapper decodes the blob in both passes so backward re-derives the weight instead of holding it resident. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add nvidia-nvcomp-cu13 (and the cu12 legacy build) at 5.3.0.16; the feature silently no-ops without it. nvCOMP ships linux and win_amd64 wheels, so no platform marker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GGUF k-quants already normalize per 32-value block and bit-pack tightly, so the packed bytes sit at ~7.6-8.0 bits/byte entropy. Measured across all nvCOMP algorithms (ANS, Zstd, GDeflate, Deflate, Gzip, LZ4, Snappy, Bitcomp, Cascaded), real Q4_K_S and Q8_0 files top out at ~2-4% saved (ANS gives ~0% on Q4_K, ~3% on Q8_0) - not worth the decode cost. The int8 wins come from tensorwise scaling where outliers collapse most weights toward zero (low entropy); GGUF's per-block scale is the opposite by design. Revert LinearGGUFA8 to plain GGUFLinear (no CompressedWeightMixin/QuantizedModuleMixin) so its layers are never compressed. Toggling Compress on a GGUF dtype now raises in quantize_layers via the empty-compressible-layers guard. Drop GGUF from the UI tooltip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Delegate locating the shared library to nvidia.libnvcomp's own load_library(), which resolves libnvcomp.so.5 on Linux and nvcomp64_5.dll on Windows, instead of the Linux-only lib64/libnvcomp.so* glob. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR is the result of an experiment a while ago. After NVIDIA has fixed a roadblock in their API that I have reported (NVIDIA/CUDALibrarySamples#300) it turns out to be very useful:
It implements model weights compression. Transformer weights are compressed during quantization, and decompressed on each forward and backward. What's non-intuitive is how high the compression ratios are on int8 quantized weights, even using a very efficient compression codec:
This turns int8 effectively into a 4-5 bit quant without quality loss compared to int8.
Raw performance impact is between 5 and 7% of steptime on 512px batch size 2 for the decompression; less on higher batch sizes
However, the effective performance impact on a 16 GB card:
As to why this works so well on int8, even though model weights commonly have high entropy. It works much less on float8 (~ 15%). The quantization to int8 removes entropy due to its linear scale and rounding - outliers stretch the scale, pushing the many smaller weights toward 0:
diff looks larger than it is, because it includes #1623
diff of this PR only: 5ecdb24
Test plan
pre-commit run --all-filespassesAI assistance