Multi-model build? #377
-
|
The readme includes these build instructions: Lines 193 to 199 in 404980e Does that mean that we have to recompile BitNet for each model? Is there no way to get a single binary that can run all 1.58 bit models? Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
bitnet.cpp is based on the llama.cpp framework and is optimized specifically for text-only inference with specialized kernels for 1-bit matrix operations |
Beta Was this translation helpful? Give feedback.
-
|
OK yes, but do we have to recompile it for each model? |
Beta Was this translation helpful? Give feedback.
-
|
You don’t need to recompile BitNet for each model. The binary is compiled for the quantization type / kernel configuration, not for a specific model file. In the example: The -q i2_s flag determines which optimized kernels are built (for 1.58-bit format). As long as multiple models use the same quantization format and architecture assumptions, the same compiled binary can run them. You would only need to rebuild if: The quantization type changes The kernel configuration changes There are architecture-specific compile flags Otherwise, a single binary can run all compatible 1.58-bit models. |
Beta Was this translation helpful? Give feedback.
You don’t need to recompile BitNet for each model.
The binary is compiled for the quantization type / kernel configuration, not for a specific model file.
In the example:
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
The -q i2_s flag determines which optimized kernels are built (for 1.58-bit format). As long as multiple models use the same quantization format and architecture assumptions, the same compiled binary can run them.
You would only need to rebuild if:
The quantization type changes
The kernel configuration changes
There are architecture-specific compile flags
Otherwise, a single binary can run all compatible 1.58-bit models.