Training CLIP-ViT

[Meteorix]

@jongwook Thanks for this great work!

I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT?

In the paper: We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.

[yuxulingche]

Did you write the training code yourself？

[Meteorix]

Did you write the training code yourself？

yes, it's not hard.

[jongwook]

Are you comparing ViT-based and ResNet50-based models trained with the same dataset? Vision transformers tend to underperform ResNet-based models unless they're trained on a huge dataset, so I'd suspect that could have been the reason, rather than the initialization scheme.

[Meteorix]

Are you comparing ViT-based and ResNet50-based models trained with the same dataset? Vision transformers tend to underperform ResNet-based models unless they're trained on a huge dataset, so I'd suspect that could have been the reason, rather than the initialization scheme.

Thanks for your reply. We have trained on 100M text-image pairs. It turned out that ViTB32 model outperformed ResNet50 model on some of benchmarks.

[ValerioB88]

@Meteorix what dataset did you use?

[dragen1860]

@jongwook Thanks for this great work!

I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT?

In the paper: We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.

Hi, Meteorix:
I am also working on this exciting project. I found you have implemented your own training code. That's pretty cool. May i ask for sharing your training code? it's would be good if you can kindly send the code to me. liangqu.long@gmail.com . Thank you very much.

[jongwook]

@dragen1860 see #83 and also other third-party implementations by @KeremTurgutlu and @lucidrains.

[JACKHAHA363]

Hi, I am also running ViT-B/32 and with the open-clip code on CC3M. I also notice a discrepancy between ResNet and ViT. You can see my curves at mlfoundations/open_clip#14. I am interested to know if you solve this performance gap @jongwook @Meteorix

[Shelton-Zhou]

@jongwook Thanks for this great work!

I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT?

In the paper: We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.

Hi, Meteorix:
I am also working on this exciting project. I found you have implemented your own training code. May i ask for sharing your training code? it's would be good if you can kindly send the code to me at x888632157@163.com . Thanks a lot!