Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Wuerstchen] text to image training script #5052

Merged
merged 80 commits into from
Oct 16, 2023
Merged

[Wuerstchen] text to image training script #5052

merged 80 commits into from
Oct 16, 2023

Conversation

kashif
Copy link
Contributor

@kashif kashif commented Sep 15, 2023 •
edited
Loading

What does this PR do?

For issue #5043

cc @sayakpaul

@kashif kashif marked this pull request as draft September 15, 2023 10:25
pcuenca
pcuenca reviewed Oct 9, 2023
View reviewed changes
Copy link
Member

@pcuenca pcuenca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool!

examples/wuerstchen/text_to_image/README.md Outdated Show resolved Hide resolved
examples/wuerstchen/text_to_image/README.md Outdated Show resolved Hide resolved
examples/wuerstchen/text_to_image/README.md Outdated Show resolved Hide resolved
examples/wuerstchen/text_to_image/README.md Outdated Show resolved Hide resolved
examples/wuerstchen/text_to_image/README.md Outdated Show resolved Hide resolved
examples/wuerstchen/text_to_image/train_text_to_image_prior.py Outdated Show resolved Hide resolved
examples/wuerstchen/text_to_image/train_text_to_image_lora_prior.py Outdated Show resolved Hide resolved
examples/wuerstchen/text_to_image/requirements.txt
accelerate>=0.16.0
torchvision
transformers>=4.25.1
wandb
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no its not technically required but the sample snippet in the README has the --report_to="wandb" option

src/diffusers/loaders.py Outdated Show resolved Hide resolved
src/diffusers/schedulers/scheduling_ddpm_wuerstchen.py Show resolved Hide resolved
kashif and others added 8 commits October 9, 2023 13:46
@kashif @pcuenca
Update examples/wuerstchen/text_to_image/README.md
4de3fbe 
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
@kashif @pcuenca
Update examples/wuerstchen/text_to_image/README.md
162500e 
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
@kashif @pcuenca
Update examples/wuerstchen/text_to_image/README.md
b3e54cb 
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
@kashif @pcuenca
Update examples/wuerstchen/text_to_image/README.md
12209ef 
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
@kashif @pcuenca
Update examples/wuerstchen/text_to_image/README.md
a1527b2 
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
@kashif @pcuenca
Update examples/wuerstchen/text_to_image/train_text_to_image_lora_pri…
a28f5c0 
…or.py

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
@kashif @pcuenca
Update src/diffusers/loaders.py
cda5de4 
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
@kashif @pcuenca
Update examples/wuerstchen/text_to_image/train_text_to_image_prior.py
d9964e2 
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
pcuenca
pcuenca approved these changes Oct 10, 2023
View reviewed changes
Copy link
Member

@pcuenca pcuenca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick attr fix!

@patrickvonplaten
Copy link
Contributor

@kashif this test:

FAILED tests/pipelines/wuerstchen/test_wuerstchen_prior.py::WuerstchenPriorPipelineFastTests::test_save_load_local - AssertionError: 0.4609375 not less than 0.0005

seems to fail quite persistently

@kashif
Copy link
Contributor Author

kashif commented Oct 11, 2023

@patrickvonplaten I wanted to ask you about it, all the seeds etc. are set and I don't get why the values are different each time... is it the lora-adaptors in the prior?

I have checked that all is in eval mode as well...

@patrickvonplaten
Copy link
Contributor

Hey @kashif,

I'm not sure why it fails either, but it doesn't seem to fail on current main so it does look like this PR introduces the bug. Think we have to investigate/debug a bit to figure out what's going on

@kashif
Copy link
Contributor Author

kashif commented Oct 13, 2023

agree! so i will check too!

@kashif
Copy link
Contributor Author

kashif commented Oct 13, 2023

@patrickvonplaten so the test passes if I comment out the set_default_attn_processor method in the WuerstchenPrior model...

@patrickvonplaten
Copy link
Contributor

Very nice!

@patrickvonplaten patrickvonplaten merged commit d03c909 into huggingface:main Oct 16, 2023
11 checks passed
@kashif kashif deleted the wuerstchen-train branch October 16, 2023 13:01
yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request Dec 25, 2023
@kashif @patrickvonplaten @pcuenca
[Wuerstchen] text to image training script (huggingface#5052)
129c25b 
* initial script

* formatting

* prior trainer wip

* add efficient_net_encoder

* add CLIPTextModel

* add prior ema support

* optimizer

* fix typo

* add dataloader

* prompt_embeds and image_embeds

* intial training loop

* fix output_dir

* fix add_noise

* accelerator check

* make effnet_transforms dynamic

* fix training loop

* add validation logging

* use loaded text_encoder

* use PreTrainedTokenizerFast

* load weigth from pickle

* save_model_card

* remove unused file

* fix typos

* save prior pipeilne in its own folder

* fix imports

* fix pipe_t2i

* scale image_embeds

* remove snr_gamma

* format

* initial lora prior training

* log_validation and save

* initial gradient working

* remove save/load hooks

* set set_attn_processor on prior_prior

* add lora script

* typos

* use LoraLoaderMixin for prior pipeline

* fix usage

* make fix-copies

* yse repo_id

* write_lora_layers is a staitcmethod

* use defualts

* fix defaults

* undo

* Update src/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_prior.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/diffusers/pipelines/wuerstchen/modeling_wuerstchen_prior.py

* Update src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* add graident checkpoint support to prior

* gradient_checkpointing

* formatting

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/train_text_to_image_lora_prior.py

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update src/diffusers/loaders.py

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/train_text_to_image_prior.py

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* use default unet and text_encoder

* fix test

---------

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
@kashif @patrickvonplaten @pcuenca
[Wuerstchen] text to image training script (huggingface#5052)
1b2c871 
* initial script

* formatting

* prior trainer wip

* add efficient_net_encoder

* add CLIPTextModel

* add prior ema support

* optimizer

* fix typo

* add dataloader

* prompt_embeds and image_embeds

* intial training loop

* fix output_dir

* fix add_noise

* accelerator check

* make effnet_transforms dynamic

* fix training loop

* add validation logging

* use loaded text_encoder

* use PreTrainedTokenizerFast

* load weigth from pickle

* save_model_card

* remove unused file

* fix typos

* save prior pipeilne in its own folder

* fix imports

* fix pipe_t2i

* scale image_embeds

* remove snr_gamma

* format

* initial lora prior training

* log_validation and save

* initial gradient working

* remove save/load hooks

* set set_attn_processor on prior_prior

* add lora script

* typos

* use LoraLoaderMixin for prior pipeline

* fix usage

* make fix-copies

* yse repo_id

* write_lora_layers is a staitcmethod

* use defualts

* fix defaults

* undo

* Update src/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_prior.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/diffusers/pipelines/wuerstchen/modeling_wuerstchen_prior.py

* Update src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* add graident checkpoint support to prior

* gradient_checkpointing

* formatting

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/README.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/train_text_to_image_lora_prior.py

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update src/diffusers/loaders.py

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/wuerstchen/text_to_image/train_text_to_image_prior.py

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* use default unet and text_encoder

* fix test

---------

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@younesbelkada younesbelkada younesbelkada left review comments

@sayakpaul sayakpaul sayakpaul left review comments

@patrickvonplaten patrickvonplaten patrickvonplaten approved these changes

@yiyixuxu yiyixuxu yiyixuxu approved these changes

@pcuenca pcuenca pcuenca approved these changes

@DN6 DN6 Awaiting requested review from DN6

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@kashif @HuggingFaceDocBuilderDev @patrickvonplaten @pcuenca @yiyixuxu @sayakpaul @younesbelkada