Exporting LoRA to HF format without merging

[M1TR]

I am using litgpt to finetune LLMs and the results are better than finetuning with HF TRL (which has far more levers to tweak I guess). I understand that litgpt forces you to merge the LoRA with the base model before exporting to HF format. Can someone guide me on how to export only the LoRA to HF format as I would like to experiment with multi-LoRA in vLLM?

[Andrei-Aksionov]

Hello @M1TR

Yes, the current code requires to merge first.
If you want to convert only LoRA weights to HF format, then you need to do it manually.
Depending on the type of the model it might be either not hard, or quite tricky 🙂: perhaps splitting the combined QKV matrix in a correct way, in some cases splitting a layer in MLP block, and of course layer names renaming.

I would recommend to start with the Lit -> HF conversion script for the type of the target model and to manipulate the weight map.

Another way to do this is to convert the merged model and then subtract from the model that you trained weights of the base model.
Because for LoRA the final merged weights are original weights + LoRA weight updates.

Of course these manipulations need to be done only on layers where LoRA was applied.

---

Perhaps it's something that could be added on LitGPT side.
cc @rasbt

[M1TR]

Hi @Andrei-Aksionov

I did spend some time trying to manually convert the LoRA file. I am working with Phi-3 so QKV needed to be recombined along with the layer renaming from the conversion script. I am still starting out with this so I am not yet super confident in messing around with direct row-level manipulation of the weight tensors. I am happy to spend some time on if you can point me in the direction of some reading materials on the model architecture and terminology.

Also, just from a curiosity standpoint, what might be the reasons why the litgpt finetune performs better than the HF TRL finetune even when using the same configuration from the YAML file in the config_hub directory? I also noticed that the LoRA LTH file is much smaller (~20MB) than the equivalent TRL safetensors file (~200MB).