Compatibility of lora and radix attention.

[walker-ai]

I tried to use LoRA Linear from sglang to implement a feature in my own code which is about inferencing with LoRA, and I have already incorporated the RadixAttn. But I found that the output is always <eos> which means nothing. For example:

# prompt: "who are you"
# answer: ""

Until I saw the

assert (
    self.max_loras_per_batch > 0
    # FIXME
    and (self.lora_paths is None or self.disable_cuda_graph)
    and (self.lora_paths is None or self.disable_radix_cache)
), "compatibility of lora and cuda graph and radix attention is in progress"
assert self.base_gpu_id >= 0, "base_gpu_id must be non-negative"

I'm afraid that the question I met is caused by this? I m not sure about that, could you give me some advice?

@Ying1123 Thx for your time!

[Galaxy-Husky]

I think you should disable cuda_graph and radix_cache when loading lora adapters like:

sglang/test/srt/models/test_lora.py

Lines 85 to 94 in f8b0326

	with SRTRunner(
	base_path,
	torch_dtype=torch_dtype,
	model_type="generation",
	tp_size=tp_size,
	lora_paths=all_lora_paths,
	max_loras_per_batch=3,
	disable_cuda_graph=True,
	disable_radix_cache=True,
	) as srt_runner:

[walker-ai]

Thank you for your reply! I noticed the update about apply S-LoRA method to adapt multi-lora switch, the code comments says that Not compatible with RadixCache currently. I wanna know if this is fundamentally infeasible, or since the intermediate computation results generated by different LoRA adapters are all different, making it difficult to reuse KV between requests (similar to Radix Cache), is this technically challenging to implement?

[Galaxy-Husky]

Thank you for your reply! I noticed the update about apply S-LoRA method to adapt multi-lora switch, the code comments says that Not compatible with RadixCache currently. I wanna know if this is fundamentally infeasible, or since the intermediate computation results generated by different LoRA adapters are all different, making it difficult to reuse KV between requests (similar to Radix Cache), is this technically challenging to implement?

Sorry, I'm not a developer and I'm not sure about your question. You might get a faster official response by submitting an issue. Good luck!

[walker-ai]

thx anyway, I'll ask in SGL slack