Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add (Scheduled) Pseudo-Huber Loss training scripts to research projects #7527

Merged
merged 21 commits into from
Apr 13, 2024

Conversation

kabachuha
Copy link
Contributor

This PR adds an option to use (Scheduled) Pseudo-Huber loss from https://arxiv.org/abs/2403.16728 for text2image and Dreambooth trainings.

Because of this feature being experimental, it's firstly needed to add it to the research projects.

See #7488 for detailed description, plots and examples

@sayakpaul @kashif @patil-suraj

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@kabachuha kabachuha marked this pull request as draft April 1, 2024 19:30
@kabachuha
Copy link
Contributor Author

The kohya-ss scripts have been merged after throughout testing in kohya-ss/sd-scripts#1228. Will update the rest of this Pull Request soon and prepare for merging in Diffusers

@sayakpaul
Copy link
Member

Sure.

@kabachuha
Copy link
Contributor Author

@sayakpaul updated all scripts, ready for merging

@kabachuha kabachuha marked this pull request as ready for review April 8, 2024 18:16
@kabachuha
Copy link
Contributor Author

Should I do anything about the code quality?

@kashif
Copy link
Contributor

kashif commented Apr 9, 2024

you can run make style in the root folder to fix

@kabachuha
Copy link
Contributor Author

kabachuha commented Apr 9, 2024

@kashif what is make: ruff: No such file or directory?

nvm

@sayakpaul
Copy link
Member

Please ping me once the CI run is complete.

@kabachuha
Copy link
Contributor Author

@sayakpaul CI check ran, but failed.

I'm getting exhaused 😢

@sayakpaul
Copy link
Member

If you check https://github.com/huggingface/diffusers/actions/runs/8641181505/job/23690277573?pr=7527, it clearly says the solution:

Run echo "Quality check failed. Please ensure the right dependency versions are installed with 'pip install -e .[quality]' and run 'make style && make quality'" >> $GITHUB_STEP_SUMMARY

Is it not helpful?

@kabachuha
Copy link
Contributor Author

@sayakpaul Thanks for the instruction! Now I'll cross my fingers and hope it'll finally work🤞

Sorry for all these misunderstandings and extra commits 😓 (please, squash them when merging)

@sayakpaul sayakpaul merged commit 2f23437 into huggingface:main Apr 13, 2024
8 checks passed
@sayakpaul
Copy link
Member

Thank you for your contributions.

@kabachuha kabachuha deleted the huber-losses branch April 13, 2024 09:48
sayakpaul added a commit that referenced this pull request Dec 23, 2024
…ts (#7527)

* add scheduled pseudo-huber loss training scripts

See #7488

* add reduction modes to huber loss

* [DB Lora] *2 multiplier to huber loss cause of 1/2 a^2 conv.

pairing of kohya-ss/sd-scripts@c6495de

* [DB Lora] add option for smooth l1 (huber / delta)

Pairing of kohya-ss/sd-scripts@dd22958

* [DB Lora] unify huber scheduling

Pairing of kohya-ss/sd-scripts@19a834c

* [DB Lora] add snr huber scheduler

Pairing of kohya-ss/sd-scripts@47fb1a6

* fixup examples link

* use snr schedule by default in DB

* update all huber scripts with snr

* code quality

* huber: make style && make quality

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants