AttributeError: 'Linear' object has no attribute 'set_lora_layer'

[aihao2000]

Describe the bug

I use (ttps://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py to train lora and raise an error AttributeError: Linear' object has no attribute 'set_lora_layer.

Reproduction

training examples/text_to_image/train_text_to_image_lora.py

Logs

No response

System Info

• diffusers version: 0.24.0.dev0
• Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.27
• Python version: 3.10.13
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Huggingface_hub version: 0.19.4
• Transformers version: 4.35.2
• Accelerate version: 0.24.1
• xFormers version: 0.0.22.post7
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@sayakpaul @patrickvonplaten

[aihao2000]

Solved after uninstalling peft.

[johnowhitaker]

Likewise.

[patrickvonplaten]

Hmm we should not have to uninstall PEFT in order to run our example scripts. cc @sayakpaul @younesbelkada could you take a look here?

[BenjaminBossan]

I did a quick search and indeed, there are 3 examples which call set_lora_layer unconditionally:

• examples/text_to_image/train_text_to_image_lora.py (reported here)
• examples/dreambooth/train_dreambooth_lora.py
• examples/dreambooth/train_dreambooth_lora_sdxl.py

The package code itself looks good though, all instances of set_lora_layer that I could find were guarded.

[sayakpaul]

@patrickvonplaten I had a chance to investigate the problem. We have:

https://github.com/huggingface/diffusers/blob/50a749e90990932d49556ee54a333278559722f3/src/diffusers/models/attention_processor.py#L176-#L179

In case peft is present we're relying on nn.Linear, instead of LoRACompatibleLinear (which is the right thing to do). However, our training scripts rely on set_lora_layer(). I am not sure there's any other way around for this other than just using peft. #5419 should solve all of these problems quite nicely.

[Johnson-yue]

how to fixed this bug , during training lora with script must uninstall peft but during inference , dependence peft ??? It is a big problem for usage

[aihao2000]

@johnowhitaker You can try torch.load lora_weights file. Then use pipe.unet.load_attn_procs

[patrickvonplaten]

Can we add if-else statements to the example scripts just to make sure?

[sayakpaul]

Can you elaborate what you mean?

[patrickvonplaten]

Like can we make sure that we don't use PEFT in the unet if the example script doesn't use PEFT (even if peft is installed)

[sayakpaul]

Honestly, I don't think we need to do that because we know that #5388 is going to be merged very very soon. So, any changes will soon become redundant and will need to be removed.

[patrickvonplaten]

The old scripts will still be used quite a bit I think. I'm sure that people will go back to use them as they don't want to update all their training scripts to PEFT yet. As it's an easy change I'd be in favor of adding it even if it might be there only for a week

[yashveer08]

Hii, I am facing the same issue, It was working good for me till yesterday that is 5/12/2023, and since today it is showing 'set_lora_layer' issue. Any update on this or how to resolve this issue?

[sayakpaul]

#6045

Does the above PR work?

[yashveer08]

I checked onto your branch from which the PR is raised and tried running through there. I got the same error.

AttributeError: 'Linear' object has no attribute 'set_lora_layer'

this is my launch command,

!accelerate launch train_text_to_image_lora_sdxl.py \ --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \ --train_data_dir="/kaggle/input/training-dataset-612" --caption_column="text" \ --resolution=1024 --random_flip \ --train_batch_size=1 \ --checkpointing_steps=400 \ --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \ --seed=42 \ --output_dir="/kaggle/working/sd-lora-sdxl" \ --gradient_checkpointing \ --mixed_precision="fp16" \ --num_train_epochs=100 \ --push_to_hub \ --hub_token $HF_TOKEN \ --hub_model_id $REPO_ID \

Am I missing something becuase it was perfectly fine till yesterday.

[sayakpaul]

Can you shed some light on your environment? Versions, etc.? Do you have peft installed?

[yashveer08]

I am running the code on Kaggle notebook with GPU P100
I have these library installed:
peft 0.6.2
accelerate 0.24.0
diffusers 0.25.0.dev0

I explicitly too tried installing peft. then too it shows the same error.

[sayakpaul]

Will investigate. Sorry for the trouble.

[sayakpaul]

@yashveer08 sorry about the delay. I think you would need to propagate the changes introduced in #6045 to the train_text_to_image_lora_sdxl.py script to be able to see if they resolve the problem. Could you try it and report once?

Also note that with #5388, this won't actually be needed.

But we will just keep the PR open for respecting our users as @patrickvonplaten pointed out here:
#5919 (comment).

Does that make sense?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[satvikkk]

Hi, I am trying to run https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py and getting the same error: AttributeError: 'Linear' object has no attribute 'set_lora_layer'. Since a lot of changes have been made since then, can you help me with what versions of diffusers, torch, xformers, torchvision, etc. should I use? I am mostly facing compatibility issues. If there are any changes to be made in the training script, can you mention that as well? I have tried running it with and without PEFT. Thanks!
@sayakpaul @patrickvonplaten

[sayakpaul]

You should make sure you are using the latest version of diffusers pulled from the main and then try again.