GNS Calculation at Transformer Model

[HariSeldon11988]

Dear all,

I run your transformer code in gns_workloads to have a look at the calculation of gns. I haven't found the part that calculates the gns. Have I overlooked it, or does the code not calculate gns?

Would help me a lot if you could give me some support or feedback.

[ruipeterpan]

Hi, apologies for the inconvenience -- it seems that, indeed, the current scripts for the translation workloads do not calculate GNS. It's been a few years since I touched this codebase and I no longer have access to my original dev setup, so I unfortunately won't be able to provide a script that works out of the box, but I believe that the GNS calculation scripts are model-agnostic as they are only accessing param.grad.data. For reference, please refer to how get_GNS() is used and implemented here.

cc'ing @DartingMelody as she is more familiar with the GNS implementations in case :)

[HariSeldon11988]

@DartingMelody
Is your gns implementation based on appendix A1 from An Empirical Model of Large-Batch Training?