Will Vllm Lora support SGMV in handling multi-Lora request?

[chenqianfzh]

In the multi-lora feature, vllm refer to the BGMV in punica.

Yet in the punica project(https://github.com/punica-ai/punica), the authors said SGMV (Segmented Gather Matrix-Vector Multiplication) is more flexible. Is there a plan to support SGMV in the community?

[Darinochka]

There is an answer to your question here #1804

We are using BGMV kernels instead of new SGMV kernels from punica. The BGMV kernel is not efficient for prefill, but the current SGMV CUTLASS-based kernel is not configurable enough and suffers from accuracy drops due to the intermediate output being stored in half-precision. Once punica updates with custom, non-CUTLASS SGMV kernels, I will update the code to make use of them.

[chenqianfzh]

Thanks for reply!