Pretraining question

[noahcjones]

Hi, does the SwinTransformer v2 do SimMiM pretraining? This is shown in the paper:
https://arxiv.org/pdf/2111.09883.pdf

If not, any plans to add/how difficult would it be to port?

[shkarupa-alex]

No, provided weights converted from original V1 and V2 models trained on ImageNet (22k when available, 1k otherwise).
If you want to try simmim weights you can modify this script https://github.com/shkarupa-alex/tfswin/blob/master/convert_weights.py to port required weights

[noahcjones]

Oh okay, I ask because I'd like to pretrain on my own, not use the converted weights. I see the example here vit-tensorflow and would like to do something similar with the swinv2.