[`PEFT`] Adapt example scripts to use PEFT #5388

younesbelkada · 2023-10-13T16:03:13Z

What does this PR do?

Wandb logs: https://wandb.ai/younesbelkada/text2image-fine-tune/runs/tj5k8xtm?workspace=user-younesbelkada

I have fine-tuned it on Pokemon blip captions and the model trained after 1200 steps can be found here: https://huggingface.co/ybelkada/sd-1.5-pokemon-lora-peft

cc @sayakpaul @patrickvonplaten

TODO:

Update the SDXL LoRA script + add peft dependency on example tests workflow

examples/text_to_image/train_text_to_image_lora.py

HuggingFaceDocBuilderDev · 2023-10-13T16:12:10Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

examples/text_to_image/train_text_to_image_lora.py

sayakpaul · 2023-10-15T03:16:05Z

Left some comments. Thanks for initiating this!

patrickvonplaten · 2023-10-16T13:14:12Z

Before doing this change it would be good to have both a PEFT and a Transformers release to not force the user to install these packages from main

younesbelkada · 2023-10-16T17:03:28Z

Before doing this change it would be good to have both a PEFT and a Transformers release to not force the user to install these packages from main

OK I will push the changes for SDXL and wait until the next release of PEFT before merging this PR

younesbelkada · 2023-10-23T16:19:21Z

I can confirm the scripts work fine under peft + diffusers main! I'll mark the PR as ready for review once do a release on PEFT

younesbelkada · 2023-11-14T16:00:10Z

This PR is ready for review! Let me know if you want me to work on saving adapter configs here, otherwise i can do it as a follow up PR. Please also see the attached wandb logs in the PR description above

examples/dreambooth/requirements.txt

sayakpaul · 2023-11-29T02:53:18Z

@patrickvonplaten this should be more or less ready to be merged, especially with huggingface/peft#1189.

@BenjaminBossan if would like to review the peft related changes introduced in this PR, that would be awesome.

Here's my Colab Notebook: https://colab.research.google.com/gist/sayakpaul/e112cc40288c5f0563efa4034dd50b72/scratchpad.ipynb.

sayakpaul · 2023-11-29T02:54:05Z

examples/dreambooth/train_dreambooth_lora.py

+    unet_lora_config = LoraConfig(
+        r=args.rank,
+        init_lora_weights="gaussian",
+        target_modules=["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"],


@younesbelkada in your previous iterations, we were missing "to_out.0" in the target_module for some scripts. I have added that.

BenjaminBossan · 2023-11-29T13:24:04Z

From the PEFT-side of things, this looks pretty good, thanks Sayak.

Just a consideration: Right now, the requirements include a source install of PEFT. This works for the moment but could be dangerous in the future, in case we add something to PEFT main which breaks the examples (of course, that wouldn't be intentional, but these things can happen). So I wonder if it would be more cautious to fix the hash of the PEFT install and add a comment that we should change the dependency to a proper PEFT version once the next PEFT release is out (most likely, that would be >=0.7.0).

sayakpaul · 2023-11-29T13:42:38Z

So I wonder if it would be more cautious to fix the hash of the PEFT install and add a comment that we should change the dependency to a proper PEFT version once the next PEFT release is out (most likely, that would be >=0.7.0).

@BenjaminBossan good point. Could you get me the required hash here? I will resolve ASAP.

patrickvonplaten · 2023-11-29T13:59:23Z

I'd like to wait a bit until PEFT has done a release to make it easier for users (think it doesn't hurt us to wait a tad more here no?)

Does the training now match more or less (e.g. is the new init used?).

BenjaminBossan · 2023-11-29T14:03:06Z

Could you get me the required hash here? I will resolve ASAP.

Latest hash is 8298f1a3668604ac9bc3f6e28b24e8eb554891a1, so adding @ + the hash to the requirement should do the trick. This is of course not required if we do:

I'd like to wait a bit until PEFT has done a release to make it easier for users

sayakpaul · 2023-11-29T14:22:04Z

Does the training now match more or less (e.g. is the new init used?).

@patrickvonplaten yes it does. Have verified it. This PR uses the new init method from peft too. I don't mind waiting either.

…/diffusers into adapt-example-peft

younesbelkada · 2023-12-06T16:28:52Z

PEFT 0.7.0 has been released ! https://github.com/huggingface/peft/releases/tag/v0.7.0
I have updated the requirements file to reflect the correct PEFT version - requesting a final review cc @patrickvonplaten @sayakpaul

patrickvonplaten

Ok for me to add once @sayakpaul gives the green light

sayakpaul

Simply amazing here!

* adapt example scripts to use PEFT * Update examples/text_to_image/train_text_to_image_lora.py * fix * add for SDXL * oops * make sure to install peft * fix * fix * fix dreambooth and lora * more fixes * add peft to requirements.txt * fix * final fix * add peft version in requirements * remove comment * change variable names * add few lines in readme * add to reqs * style * fix issues * fix lora dreambooth xl tests * init_lora_weights to gaussian and add out proj where missing * ammend requirements. * ammend requirements.txt * add correct peft versions --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

adapt example scripts to use PEFT

2cbc52a

younesbelkada commented Oct 13, 2023

View reviewed changes

examples/text_to_image/train_text_to_image_lora.py Show resolved Hide resolved

Update examples/text_to_image/train_text_to_image_lora.py

b86543f

sayakpaul reviewed Oct 15, 2023

View reviewed changes

examples/text_to_image/train_text_to_image_lora.py Show resolved Hide resolved

sayakpaul reviewed Oct 15, 2023

View reviewed changes

examples/text_to_image/train_text_to_image_lora.py Outdated Show resolved Hide resolved

younesbelkada added 4 commits October 16, 2023 21:46

fix

af99c12

Merge remote-tracking branch 'upstream/main' into adapt-example-peft

cee4255

add for SDXL

89d4bed

oops

4281913

younesbelkada added 10 commits November 5, 2023 07:17

Merge remote-tracking branch 'upstream/main' into adapt-example-peft

b18d0e8

make sure to install peft

6a48ad0

fix

069a929

Merge remote-tracking branch 'upstream/main' into adapt-example-peft

11fd6f5

fix

e4b0f1d

fix dreambooth and lora

62c33c0

more fixes

a1e1cdf

add peft to requirements.txt

c3d3002

fix

340150b

final fix

dff2995

sayakpaul mentioned this pull request Nov 14, 2023

[Training] Add datasets version of LCM LoRA SDXL #5778

Merged

3 tasks

younesbelkada marked this pull request as ready for review November 14, 2023 15:44

younesbelkada requested review from sayakpaul and patrickvonplaten November 14, 2023 15:57

sayakpaul reviewed Nov 16, 2023

View reviewed changes

examples/dreambooth/requirements.txt Outdated Show resolved Hide resolved

sayakpaul added 2 commits November 29, 2023 07:59

ammend requirements.

252dcda

ammend requirements.txt

90b760a

sayakpaul reviewed Nov 29, 2023

View reviewed changes

sayakpaul mentioned this pull request Nov 29, 2023

Do not save unet.unet keys when training LoRAs #5977

Closed

Merge branch 'main' into adapt-example-peft

0692a13

Merge branch 'main' into adapt-example-peft

f27fb29

younesbelkada mentioned this pull request Dec 4, 2023

[PEFT] Update the example scripts to reflect PEFT API #5341

Closed

younesbelkada added 3 commits December 6, 2023 16:25

Merge remote-tracking branch 'upstream/main' into adapt-example-peft

99df659

Merge branch 'adapt-example-peft' of https://github.com/younesbelkada…

32ffcf1

…/diffusers into adapt-example-peft

add correct peft versions

57516ed

younesbelkada requested a review from sayakpaul December 6, 2023 16:28

patrickvonplaten reviewed Dec 6, 2023

View reviewed changes

sayakpaul approved these changes Dec 7, 2023

View reviewed changes

sayakpaul merged commit c271731 into huggingface:main Dec 7, 2023
14 checks passed

sayakpaul mentioned this pull request Dec 7, 2023

Assign device to unet. Resolves #5897 #6061

Closed

6 tasks

younesbelkada deleted the adapt-example-peft branch December 7, 2023 07:20

sayakpaul mentioned this pull request Dec 10, 2023

[Advanced Diffusion Training] Adapt train_dreambooth_lora_sdxl_advanced.py to use peft #6118

Closed

younesbelkada mentioned this pull request Dec 18, 2023

Training "train_dreambooth_lora_sdxl.py" on multi-gpu using accelerate #6146

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[`PEFT`] Adapt example scripts to use PEFT #5388

[`PEFT`] Adapt example scripts to use PEFT #5388

younesbelkada commented Oct 13, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 13, 2023

sayakpaul commented Oct 15, 2023

patrickvonplaten commented Oct 16, 2023

younesbelkada commented Oct 16, 2023

younesbelkada commented Oct 23, 2023

younesbelkada commented Nov 14, 2023

sayakpaul commented Nov 29, 2023

sayakpaul Nov 29, 2023

BenjaminBossan commented Nov 29, 2023

sayakpaul commented Nov 29, 2023

patrickvonplaten commented Nov 29, 2023

BenjaminBossan commented Nov 29, 2023

sayakpaul commented Nov 29, 2023

younesbelkada commented Dec 6, 2023

patrickvonplaten left a comment

sayakpaul left a comment

[PEFT] Adapt example scripts to use PEFT #5388

[PEFT] Adapt example scripts to use PEFT #5388

Conversation

younesbelkada commented Oct 13, 2023 • edited Loading

What does this PR do?

HuggingFaceDocBuilderDev commented Oct 13, 2023

sayakpaul commented Oct 15, 2023

patrickvonplaten commented Oct 16, 2023

younesbelkada commented Oct 16, 2023

younesbelkada commented Oct 23, 2023

younesbelkada commented Nov 14, 2023

sayakpaul commented Nov 29, 2023

sayakpaul Nov 29, 2023

Choose a reason for hiding this comment

BenjaminBossan commented Nov 29, 2023

sayakpaul commented Nov 29, 2023

patrickvonplaten commented Nov 29, 2023

BenjaminBossan commented Nov 29, 2023

sayakpaul commented Nov 29, 2023

younesbelkada commented Dec 6, 2023

patrickvonplaten left a comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

[`PEFT`] Adapt example scripts to use PEFT #5388

[`PEFT`] Adapt example scripts to use PEFT #5388

younesbelkada commented Oct 13, 2023 •

edited

Loading