anyone tests lora's throughput

[white-wolf-tech]

I installed vllm from the lastest code.

found it supports Qwen2 series model.

I test Qwen1.8B with 16 concurrency. got the following result:

I merge the lora weight to Qwen1.8B. latency(ms)：
min: 222, average: 400, max:418

without merging lora weight to Qwen1.8B, using lora dynamic calling through query.
min: 307, average: 780, max: 874

vllm lora way is much more slower than merging version? Is this okay?

[Nipi64310]

It's normal for the speed to increase after merging because there is no need to perform matrix multiplication between the input and lora weights in each lora layer.

[changun]

Thanks @Nipi64310 for explanation. We are seeing similar slow-down as well. It is a trade-off between convivence and speed now.

[FerranAgulloLopez]

When merging there is no overhead in memory neither in computation. However, if willing to serve multiple subtasks, you will need to replicate the full model.