Initial Delay in Image Generation with Flux Schnell on H100

[uayodev]

Thank you very much for your incredible work @aredden!

I wanted to ask you about something I've noticed when using Flux Schnell with an H100. (when using compile_extras and compile_blocks) After running the three warmups of Flux Schnell, the first image I generate takes about 45 seconds to start the first iteration, but the subsequent images generate quickly. Is this normal? Is there any way to avoid this initial delay?

I appreciate your help in advance.

[aredden]

The slowdown is due to the torch.compile compilation, it should speed up after that, but the initial generation may take a while, and also may take a while for each new requested image shape. The initial slowdown is much more reasonable with torch nightly, or just torch > 2.4.x, since I believe they made it quite a bit faster, or at least it is faster on my machine. I barely notice compilation speed anymore, though I have a beefy computer so there is that.

[uayodev]

Thanks so much for your reply! I really appreciate it.

[Muawizodux]

I have tested this slowdown on h100 and rtx4090. The slowdown is around 1 minute for just torch and for torch nightly its around 3-7 seconds

[aredden]

Yeah- so essentially using nightly is significantly better.