Skip to content

Latest commit

 

History

History

BEiT

README.md

BEiT: BERT Pre-Training of Image Transformers, arxiv

PaddlePaddle training/validation code and pretrained models for BEiT.

The official and 3rd party pytorch implementation are here.

This implementation is developed by PaddleViT.

drawing

BEiT Model Overview

Update

  • Update (2022-03-24): Code is refactored and bugs are fixed.
  • Update (2021-10-19): Bug fix and weights links are updated.
  • Update (2021-09-27): Code is released and ported weights are uploaded.

Models Zoo

Model Acc@1 Acc@5 #Params FLOPs Image Size Crop_pct Interpolation Link
beit_base_patch16_224 85.21 97.66 87M 12.7G 224 0.9 bicubic google/baidu
beit_base_patch16_384 86.81 98.14 87M 37.3G 384 1.0 bicubic google/baidu
beit_large_patch16_224 87.48 98.30 304M 45.0G 224 0.9 bicubic google/baidu
beit_large_patch16_384 88.40 98.60 304M 131.7G 384 1.0 bicubic google/baidu
beit_large_patch16_512 88.60 98.66 304M 234.0G 512 1.0 bicubic google/baidu

Note:

The results are evaluated on ImageNet2012 validation set.

These models have been fine-tuned (ImageNet 22k -> 1k), weights are ported from here

Note : We are developing the pretraining using DALL-E, which is not supported right now.

Data Preparation

ImageNet2012 dataset is used in the following file structure:

│imagenet/
├──train_list.txt
├──val_list.txt
├──train/
│  ├── n01440764
│  │   ├── n01440764_10026.JPEG
│  │   ├── n01440764_10027.JPEG
│  │   ├── ......
│  ├── ......
├──val/
│  ├── n01440764
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......
│  ├── ......
  • train_list.txt: list of relative paths and labels of training images. You can download it from: google/baidu
  • val_list.txt: list of relative paths and labels of validation images. You can download it from: google/baidu

Usage

To use the model with pretrained weights, download the .pdparam weight file and change related file paths in the following python scripts. The model config files are located in ./configs/.

For example, assume weight file is downloaded in ./beit_base_patch16_224.pdparams, to use the beit_base_patch16_224 model in python:

from config import get_config
from beit import build_beit as build_model
# config files in ./configs/
config = get_config('./configs/beit_base_patch16_224.yaml')
# build model
model = build_model(config)
# load pretrained weights
model_state_dict = paddle.load('./beit_base_patch16_224.pdparams')
model.set_state_dict(model_state_dict)

Evaluation

To evaluate model performance on ImageNet2012, run the following script using command line:

sh run_eval_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
python main_multi_gpu.py \
-cfg='./configs/beit_base_patch16_224.yaml' \
-dataset='imagenet2012' \
-batch_size=256 \
-data_path='/dataset/imagenet' \
-eval \
-pretrained='./beit_base_patch16_224.pdparams' \
-amp

Note: if you have only 1 GPU, change device number to CUDA_VISIBLE_DEVICES=0 would run the evaluation on single GPU.

Training

To train the model on ImageNet2012, run the following script using command line:

sh run_train_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
python main_multi_gpu.py \
-cfg='./configs/beit_base_patch16_224.yaml' \
-dataset='imagenet2012' \
-batch_size=256 \
-data_path='/dataset/imagenet' \
-amp

Note: it is highly recommanded to run the training using multiple GPUs / multi-node GPUs.

Reference

@article{beit,
      title={{BEiT}: {BERT} Pre-Training of Image Transformers}, 
      author={Hangbo Bao and Li Dong and Furu Wei},
      year={2021},
      eprint={2106.08254},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}