🚀Fast ConvMAE🚀

Fast ConvMAE: Fast Pretraining of ConvMAE

This repo is the faster implementation of ConvMAE: Masked Convolution Meets Masked Autoencoders

Updates

17/June/2022

Released the pre-training codes for ImageNet-1K.

Introduction

Fast ConvMAE framework is a superiorly fast masked modeling scheme via complementary masking and mixture of reconstrunctors based on the ConvMAE.

Pretrain on ImageNet-1K

The following table provides pretrained checkpoints and logs used in the paper.

	Fast ConvMAE-Base
50epoch pretrained checkpoints	N/A
logs	N/A

Main Results on COCO & ImageNet-1K

Models	Masking	Tokenizer	Backbone	PT Epochs	PT Hours	COCO FT Epochs	$AP^{Box}$	$AP^{Mask}$	ImageNet Finetune Epochs	Finetune acc@1(%)	ADE 20K mIoU
ConvMAE	25 %	RGB	ConvViT-B	200	512	25	50.8	45.4	100	84.4	48.5
ConvMAE	25 %	RGB	ConvViT-B	1600	4000	25	53.2	47.1	100	85.0	51.7
MAE	25 %	RGB	ViT-B	1600	2069	100	50.3	44.9	100	83.6	48.1
SimMIM	100 %	RGB	Swin-B	800	1609	36	50.4	44.4	100	84.0	-
GreenMIM	25 %	RGB	Swin-B	800	887	36	50.0	44.1	100	85.1	-
ConvMAE	100 %	RGB	ConvViT-B	50	266	25	51.0	45.4	100	84.4	48.3
ConvMAE	100 %	C+T	ConvViT-B	50	333	25	52.8	46.9	100	85.0	52.7
ConvMAE	100 %	C+T	ConvViT-B	100	666	25	53.3	47.3	100	85.2	52.8
ConvMAE	100 %	C+T	ConvViT-L	200	N/A	25	N/A	N/A	50	86.7	54.5

Visualizations

NOTE: Grey patches are masked and colored ones are kept.

Getting Started

Prerequisites

• Linux
• Python 3.7+
• CUDA 10.2+
• GCC 5+

Training and evaluation

• See PRETRAIN.md for pretraining.

Acknowledgement

The pretraining and finetuning of our project are based on DeiT, MAE, and ConvMAE. Thanks for their wonderful work.

License

FastConvMAE is released under the MIT License.