You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's normal for the speed to increase after merging because there is no need to perform matrix multiplication between the input and lora weights in each lora layer.
When merging there is no overhead in memory neither in computation. However, if willing to serve multiple subtasks, you will need to replicate the full model.
I installed vllm from the lastest code.
found it supports Qwen2 series model.
I test Qwen1.8B with 16 concurrency. got the following result:
I merge the lora weight to Qwen1.8B. latency(ms):
min: 222, average: 400, max:418
without merging lora weight to Qwen1.8B, using lora dynamic calling through query.
min: 307, average: 780, max: 874
vllm lora way is much more slower than merging version? Is this okay?
The text was updated successfully, but these errors were encountered: