SDXL lora on inf2

[MrD005]

want to implement LoRA on SDXL model on inf2. Is there any support or process to implement this ?

[JingyaHuang]

Hi @MrD005,

Thanks for opening the issue, and the contribution is very welcomed!

To support LoRA, the first thing we need to check is whether the neuron compiler support the compilation of text encoder / unet loaded with LoRA weight (theorectically yes).

To do so, we can do a quick hack:

• Load the LoRA weight to component models in the pipeline before the compilation before this line
  

  optimum-neuron/optimum/exporters/neuron/utils.py

  Line 163 in e9c2a9f

  models_for_export = _get_submodels_for_export_stable_diffusion(pipeline=pipeline, task=task)

with for example

pipeline.load_lora_weights("ostris/super-cereal-sdxl-lora", weight_name="cereal_box_sdxl_v1.safetensors")

If the model compiled successfully, and doesn't meet any issue during the inference, then we can add offically support to the exporter:

• Add a flag like "--lora_weight" in optimum/commands/export/neuronx.py for supporting it in optimum CLI
• Add lora info to main_export throught the argument submodels
• Load the LoRA weight through the function replace_stable_diffusion_submodels
  

  optimum-neuron/optimum/exporters/neuron/utils.py

  Line 329 in e9c2a9f

  def replace_stable_diffusion_submodels(pipeline, submodels):

Then we are normally all set, the compiled artifacts should contain the LoRA weights and there is nothing else to add during the inference.

Much appreciation if you would like to work on it 🙏 . Feel free to ping me for review or any further question!

[MrD005]

Thanks @JingyaHuang i will implement this and revert back if its works or not

[Dev-hestabit]

@JingyaHuang i have one more query if you can help into that also
when i am running SDXL model it is truncating my prompt upto 77 length and i found different solution for GPU and try to implement them on inf2 also but stuck on tensor calculation. if you can help in this also it will be very helpful for me

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[khurramkhalil]

Hi there,
Just wanting to know if there is any update on adding LORAs support on inf2?
Thanks

[JingyaHuang]

Hi @khurramkhalil, it's already supported through this PR: #483.

You could also find an example here: https://huggingface.co/docs/optimum-neuron/en/tutorials/stable_diffusion#load-adapters

[khurramkhalil]

Thank you very much.