FLUX LoRA training time

[primecai]

Hi all,

I'm just wondering how many seconds/iteration is somewhat reasonable to see for FLUX LoRA training. I am on two RTX A6000s, and using batch size 8 with 4 steps of gradient accumulation for resolution 1024x1024, I can get 30 seconds/iter. Is this somewhat reasonable?
I think A6000s should support bf16 well, but a bit scared because of my failures at SD3 LoRA training here (even though I upgraded from RTX8000s to A6000s).

Thanks in advance.

[bghira]

no it should be more like 7 seconds / step on an A6000 but 4 gradient accumulations is probably too many. set it to just 1 or 2. it linearly increases the runtime.

[primecai]

I see, thanks! May I ask what batch size and resolution were you using on an A6000?

[bghira]

t'wasn't me 😄 just a random report I saw.

[primecai]

I see...I just checked again and I was using 1 gradient accumulation already, not 4. So this logically transits to 30sec/iter for a batch size of 8, across 2 GPUs (an effective batch size 16) on resolution 1024x1024.
I also did a sanity check on SD3 results, and I can easily get 2~3 seconds per iteration results using the identical configuration, which suggests my setup should be correct.
I am guessing the 7 seconds/step performance is obtained using 512x512 resolution, while I used 1024x1024 because it's exactly a x4 efficiency...

[bghira]

no it should be 7 seconds per step period, even with 2 GPUs

edit: ah, 1024px. yes. that

[bghira]

i think 512px is the more efficient training option. same quality as 1024px, maybe better, and costs less.