Asking for some clarification regarding the iterations

[FirestName]

Hello,

As I've understood; one iteration equals processing of one batch?
If so why is the training progress calculated accordingly?
In my naivety just looking at the number of samples processed would seem more intuitive.

I mean as for stylegan training the batch size is variable but the training progress is still counted per (kilo)images presented to the discriminator. Thus the training progress is substantially impacted by the available Vram since more images can be processed per batch. Obviously the batch size used also affects the end result as that controls the gradient accumulated before back propagation (or something).

For me this begs the question: is there something intrinsically different going on here so that it's indeed more appropriate to track the training according to iterations vs samples processed?

yeah, obviously the best would be to just monitor them tensorboards and assess from there, but this just piqued my intrigue.

Best regards
Firest

[FirestName]

Tested this.
Started training with batch size of 32 and the training progress started slowing by 500k iterations. I proceeded to change the batch size to 8 and observed accelerated further improvement of the losses.

Seems indeed that the batch size of 8 is better than 32.
I'm still somewhat intriqued as to if the second phase of the training would react similarily to a larger batch size.

Best regards
Firest

[becker929]

Hi Firest, I'm curious if you did any more experimentation on this point, or if there is an official answer on this? Otherwise, it seems that the best idea is to increase the batch size until the GPU is saturated, right?