[FEATURE] ImageNet1k weights for ViT Huge?

[NightMachinery]

Looking through timm/models/vision_transformer.py, the only huge variants fine-tuned on ImageNet1K are pretrained by the CLIP loss. Are the normal variants (augreg2 etc.) not available for the bigger sizes such as Huge and Giant?

[rwightman]

There are some ImageNet-22k H/14 weights without a valid head from the original google ViT weights. Not very good though.

Best non CLIP option on ImageNet-22k/1-k is the Deit-III H/14. Although it was pretty clearly running out of steam vs the L. I don't feel ImageNet-22k is enough to train H optimally so best to stick with big dataset pretrain w/ image-text or semi/self-supervised learning.

The SO400M ViT would probably be a better bet but don't think anyone has done a supervised ImageNet-22k - 1k fine-tune with that...