InfLLM-v2 attention kernel

[maxin9966]

It seems that vllm does not support the InfLLM-v2 attention kernel. Could you tell me how much of a performance improvement InfLLM-v2 offers, and what the trade-offs are? For example, any performance loss? How can CPM.cu be set up to run an OpenAI-compatible API service like vllm?

[Achazwl]

1. The vLLM support requires maintaining an extra semantic k cache (compressed k cache), adding this to the original page attention management is a bit challenging.
2. You can see Figure 1 in our Tech Report, most of the efficiency improvement compared to other models in 128k context length comes from the InfLLM v2. You can also run our example in CPM.cu and add --no-apply-sparse to disable InfLLM v2 and compare the speed yourself.
3. For task performance, you can see Section 5.3 of our Tech Report.
4. The Python interface of CPM.cu at link provides a generate interface. We'll see how to wrap it into the API service, it should take a few days, and I'll notify you once it's done.

[JFLFY2255]

We have already supported a lightweight OpenAI API implementation, you can refer to the README.