[feature request]Support quant type q6_0 and q6_0 kv from ik llama

[FNsi]

I noticed q6_0 is much faster than q8 and sometimes faster than q4;

[Pento95]

It is often recommended with Hadamard transform. ~30% less VRAM is not bad and, thanks to the new attn-rot, quality should be very good.

But this issue is "duplicate" with #2113 which is asking for Q5_0.
Both Q6_0 and Q5_0 are not "from ik_llama.cpp", mainline llama.cpp supports them, but compiling llama.cpp binaries including the support for those 2 KV Cache quantization types was not that meaningfull, until now.

[FNsi]

But this issue is "duplicate" with #2113 which is asking for Q5_0.

I agree...but q5_0 is slow than q4 and q8 in most case...

Both Q6_0 and Q5_0 are not "from ik_llama.cpp",

Emm, I think mainland only support q5_0 and q5_1; q6 is build by ik thus mainland not support it yet..

I agree its so much work to cross files...add a new quant type........but tbh its faster and stable...

Btw...I mean quant type q6_0, kv cache is okay to use q4 kv outside llm or short ctx one, like the music or audio, embeddings etc...q6 kv is better when running in llm with large context.

[Pento95]

Emm, I think mainland only support q5_0 and q5_1; q6 is build by ik thus mainland not support it yet..
You are completely right, my apologies for the confusion.
I double-checked and mainline llama.cpp currently supports f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, and q5_1 for the KV cache. q6_0 is indeed an ik_llama exclusive as you pointed out.

That means it's unlikely that koboldcpp is gonna support this KV Cache quant type.

[FNsi]

That means it's unlikely that koboldcpp is gonna support this KV Cache quant type.

You just used a GitHub bug? Woo, in my iOS app showed you were the owner of the project!

[LostRuins]

I am the owner of the koboldcpp repo

[FNsi]

I am the owner of the koboldcpp repo

I see, so pentos95? In my app showed that's your account 'lostruins' I said 'showed' cause now show it's 'pentos 95'

[Pento95]

I'm a random guy on the internet, who uses koboldcpp and follows the project. i'm not even a python developer.
Probably an iOS app UI bug, LostRuins is the boss here ;)

[FNsi]

I'm a random guy on the internet, who uses koboldcpp and follows the project. i'm not even a python developer.
Probably an iOS app UI bug, LostRuins is the boss here ;)

I agree it's the most possible case😂😂😂 sorry for possible misleadings

[LostRuins]

Btw next version will support q5_1 kv cache type

[FNsi]

Btw next version will support q5_1 kv cache type

Nice!

[johnlabor]

Will q6 kv cache type be supported in later versions? Seems interesting.

I understand bf16 is great on gemma4 but my vram32gb is bumping deployment of a gemma4 31b q6 quants to 31.4/32gb which is not usable.

Thanks!