v0.9.8 - flux 24 gig training has entered the chat

Flux

It's here! Runs on 24G cards using Quanto's 8bit quantisation, or 25.7G on a Macbook system (slowly)!

If you're after accuracy, a 40G card will do Just Fine, with 80G cards being somewhat of a sweet spot for larger training efforts.

What you get:

• LoRA, full tuning (but probably just don't do that)
• Documentation to get you started fast
• Probably better for just square crop training for now - might artifact for weird resolutions
• Quantised base model unlocking the ability to safely use Adafactor, Prodigy, and other neat optimisers as a consolation prize for losing access to full bf16 training (AdamWBF16 just won't work with Quanto)

What's Changed

• trainer: simplify check by @bghira in #592
• documentation updates, apple pytorch 2.4 by @bghira in #595
• staged storage for image embed support by @bghira in #596
• fix: loading default image embed backend by @bghira in #597
• fix: loading default image embed backend by @bghira in #598
• multi-gpu console output improvements by @bghira in #599
• vae cache: hash_filenames option for image sets by @bghira in #601
• multi-gpu console output reduction by @bghira in #602
• fix for relative cache directories with NoneType being unsubscriptable by @bghira in #603
• multigpu / relative path fixes for caching by @bghira in #604
• backend for csv based datasets by @bghira in #600
• CSV data backend by @bghira in #605
• config file versioning to allow updating defaults without breaking backwards compat by @bghira in #607
• config file versioning for backwards compat by @bghira in #608
• experiment: small DiT model by @bghira in #609
• merge by @bghira in #610
• Fix crash when using jsonl files by @swkang-rp in #611
• merge by @bghira in #612
• flux training by @bghira in #614
• update base_dir to output_dir by @bghira in #615
• merge by @bghira in #616
• flux: validations should ignore any custom schedulers by @bghira in #618
• release: flux by @bghira in #617
• bugfix: correctly set hash_filenames to true or false for an initial dataset creation by @bghira in #620
• release: minor follow-up fixes by @bghira in #628
• Flux: Fix random validation errors due to some tensors being on the cpu by @mhirki in #629
• Improve config support for transformers with accelerate by @touchwolf in #630
• quanto: exploring low-precision training. by @bghira in #622
• remove all text encoders from memory correctly by @bghira in #637

New Contributors

• @swkang-rp made their first contribution in #611
• @touchwolf made their first contribution in #630

Full Changelog: v0.9.7.8...v0.9.8