[Feature]: CAE Supported #284

YuanLiuuuuuu · 2022-04-20T03:54:11Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.

Checklist

Before PR:

Pre-commit or other linting tools are used to fix the potential lint issues.
Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
CLA has been signed and all committers have signed the CLA in this PR.

liming-ai · 2022-04-21T07:32:44Z

Good job, could you reproduce the performance in CAE paper (especially for downstream tasks)?

YuanLiuuuuuu · 2022-04-21T07:37:45Z

Good job, could you reproduce the performance in CAE paper (especially for downstream tasks)?

Currently, we pre-train ViT-B on ImageNeg1K for 300 epochs. And the fine-tuning accuracy is 83.2, which is only 0.1 lower than that reported in the original paper.

liming-ai · 2022-04-21T07:50:30Z

Good job, could you reproduce the performance in CAE paper (especially for downstream tasks)?

Currently, we pre-train ViT-B on ImageNeg1K for 300 epochs. And the fine-tuning accuracy is 83.2, which is only 0.1 lower than that reported in the original paper.

How about the performance of detection and segmentation?

YuanLiuuuuuu · 2022-04-21T08:25:17Z

Good job, could you reproduce the performance in CAE paper (especially for downstream tasks)?

Currently, we pre-train ViT-B on ImageNeg1K for 300 epochs. And the fine-tuning accuracy is 83.2, which is only 0.1 lower than that reported in the original paper.

How about the performance of detection and segmentation?

Because of the limited time, we did not run seg and det. We encourage developers from community to run seg and det.

fangyixiao18

Please check docstrings of class and function, add docstrings as much as possible

configs/benchmarks/classification/_base_/datasets/imagenet.py

configs/benchmarks/classification/imagenet/swin-base_ft-8xb256-coslr-100e_in1k-224.py

fangyixiao18 · 2022-04-21T07:26:48Z

configs/benchmarks/classification/imagenet/vit-base-p16_ft-8xb128-coslr-100e-rpe_in1k.py

@@ -0,0 +1,38 @@
+_base_ = 'vit-b-p16_ft-8xb128-coslr-100e_in1k.py'


rename it to vit-base-p16 ? I have created a PR to rename some config files

Sounds good

change the 'base' file name?

configs/selfsup/cae/cae_vit-base-p16_32xb64-coslr-300e_in1k.py

mmselfsup/core/hooks/cosineAnnealing_hook.py

mmselfsup/models/algorithms/classification.py

mmselfsup/models/utils/network_blocks.py

liming-ai · 2022-04-21T13:22:21Z

Please check docstrings of class and function, add docstrings as much as possible

Could you please check this PR ?

mmselfsup/models/heads/cae_head.py

configs/selfsup/_base_/datasets/imagenet_cae.py

mmselfsup/datasets/pipelines/transforms.py

configs/benchmarks/classification/imagenet/swin-base_ft-8xb256-coslr-100e_in1k-224.py

mmselfsup/datasets/pipelines/transforms.py

mmselfsup/models/backbones/cae_pretrain_vit.py

mmselfsup/models/heads/cae_head.py

mmselfsup/models/heads/mae_head.py

mmselfsup/models/utils/transformer_blocks.py

YuanLiuuuuuu · 2022-04-27T07:00:22Z

configs/selfsup/_base_/datasets/imagenet_cae.py

configs/selfsup/_base_/models/cae.py

docs/en/model_zoo.md

docs/zh_cn/model_zoo.md

codecov · 2022-04-27T14:49:04Z

Codecov Report

Merging #284 (a038f3e) into dev_v0.9.0 (955632e) will increase coverage by 1.67%.
The diff coverage is 83.60%.

❗ Current head a038f3e differs from pull request most recent head 6a7aa99. Consider uploading reports for the commit 6a7aa99 to get more accurate results

@@              Coverage Diff               @@
##           dev_v0.9.0     #284      +/-   ##
==============================================
+ Coverage       70.30%   71.98%   +1.67%     
==============================================
  Files             114      120       +6     
  Lines            4099     4698     +599     
  Branches          658      757      +99     
==============================================
+ Hits             2882     3382     +500     
- Misses           1092     1162      +70     
- Partials          125      154      +29

Flag	Coverage Δ
unittests	`71.98% <83.60%> (+1.67%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmselfsup/models/backbones/mae_vit.py	`96.36% <ø> (ø)`
mmselfsup/core/hooks/cosineAnnealing_hook.py	`16.21% <5.88%> (-8.79%)`	⬇️
mmselfsup/models/heads/mae_head.py	`78.78% <33.33%> (-17.05%)`	⬇️
mmselfsup/datasets/pipelines/transforms.py	`85.20% <71.55%> (-14.80%)`	⬇️
mmselfsup/models/utils/dall_e.py	`83.33% <83.33%> (ø)`
mmselfsup/models/utils/transformer_blocks.py	`88.69% <88.69%> (ø)`
mmselfsup/models/backbones/mim_cls_vit.py	`95.91% <90.90%> (-1.46%)`	⬇️
mmselfsup/models/heads/cae_head.py	`92.10% <92.10%> (ø)`
mmselfsup/models/backbones/cae_vit.py	`94.73% <94.73%> (ø)`
mmselfsup/datasets/pipelines/__init__.py	`100.00% <100.00%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 955632e...6a7aa99. Read the comment docs.

configs/selfsup/cae/metafile.yaml

fangyixiao18 · 2022-04-28T01:45:44Z

configs/benchmarks/classification/imagenet/vit-base-p16_ft-8xb128-coslr-100e-rpe_in1k.py

@@ -0,0 +1,38 @@
+_base_ = 'vit-b-p16_ft-8xb128-coslr-100e_in1k.py'


change the 'base' file name?

mmselfsup/models/heads/cae_head.py

* [Feature]: MAE pre-training with fp16 (#271) * [Feature]: MAE pre-training with fp16 * [Fix]: Fix lint * [Fix]: Fix SimMIM config link, and add SimMIM to model_zoo (#272) * [Fix]: Fix link error * [Fix]: Add SimMIM to model zoo * [Fix]: Fix lint * [Fix] fix 'no init_cfg' error for pre-trained model backbones (#256) * [UT] add unit test for apis (#276) * [UT] add unit test for apis * ignore pytest log * [Feature] Add extra dataloader settings in configs. (#264) * [Feature] support to set validation samples per gpu independently * set default to be cfg.data.samples_per_gpu * modify the tools/test.py * using 'train_dataloader', 'val_dataloader', 'test_dataloader' for specific settings * test 'evaluation' branch * [Fix]: Change imgs_per_gpu to samples_per_gpu MAE (#278) * [Feature]: Add SimMIM 192 pt 224 ft (#280) * [Feature]: Add SimMIM 192 pt 224 ft * [Feature]: Add simmim 192 pt 224 ft to readme * [Fix] fix key error bug when registering custom hooks (#273) * [UT] remove pytorch1.5 test (#288) * [Benchmark] rename linear probing config file names (#281) * [Benchmark] rename linear probing config file names * update config links * Avoid GPU memory leak with prefetch dataloader (#277) * [Feature] barlowtwins (#207) * [Fix]: Fix mmcls upgrade bug (#235) * [Feature]: Add multi machine dist_train (#232) * [Feature]: Add multi machine dist_train * [Fix]: Change bash to sh * [Fix]: Fix missing sh suffix * [Refactor]: Change bash to sh * [Refactor] Add unit test (#234) * [Refactor] add unit test * update workflow * update * [Fix] fix lint * update test * refactor moco and densecl unit test * fix lint * add unit test * update unit test * remove modification * [Feature]: Add MAE metafile (#238) * [Feature]: Add MAE metafile * [Fix]: Fix lint * [Fix]: Change LARS to AdamW in the metafile of MAE * Add barlowtwins * Add unit test for barlowtwins * Adjust training params * add decorator to pass CI * adjust params * Add barlowtwins * Add unit test for barlowtwins * Adjust training params * add decorator to pass CI * adjust params * add barlowtwins configs * revise LatentCrossCorrelationHead * modify ut to save memory * add metafile * add barlowtwins results to model zoo * add barlow twins to homepage * fix batch size bug * add algorithm readme * add type hints * reorganize the model zoo * remove one config * recover the config * add missing docstring * revise barlowtwins * reorganize coco and voc benchmark * add barlowtwins to index.rst * revise docstring Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Yixiao Fang <36138628+fangyixiao18@users.noreply.github.com> Co-authored-by: fangyixiao18 <fangyx18@hotmail.com> * [Fix] fix --local-rank (#290) * [UT] reduce memory usage while runing unit test (#291) * [Feature]: CAE Supported (#284) * [Feature]: Add mc * [Feature]: Add dataset of CAE * [Feature]: Init version of CAE * [Feature]: Add mc * [Fix]: Change beta to (0.9, 0.999) * [Fix]: New feature * [Fix]: Decouple the qkv bias * [Feature]: Decouple qkv bias in MultiheadAttention * [Feature]: New mask generator * [Fix]: Fix TransformEncoderLayer bug * [Feature]: Add MAE CAE linear prob * [Fix]: Fix config * [Fix]: Delete redundant mc * [Fix]: Add init value in mim cls vit * [Fix]: Fix cae ft config * [Fix]: Delete repeated init_values * [Fix]: Change bs from 64 to 128 in CAE ft * [Fix]: Add mc in cae pt * [Fix]: Fix momemtum update bug * [Fix]: Add no weight_decay for gamma * [Feature]: Add mc for cae pt * [Fix]: Delete mc * [Fix]: Delete redundant files * [Fix]: Fix lint * [Feature]: Add docstring to algo, backbone, neck and head * [Fix]: Fix lint * [Fix]: network * [Feature]: Add docstrings for network blocks * [Feature]: Add docstring to ToTensor * [Feature]: Add docstring to transoform * [Fix]: Add type hint to BEiTMaskGenerator * [Fix]: Fix lint * [Fix]: Add copyright to dalle_e * [Fix]: Fix BlockwiseMaskGenerator * [Feature]: Add UT for CAE * [Fix]: Fix dalle state_dict path not existed bug * [Fix]: Delete file_client_args related code * [Fix]: Remove redundant code * [Refactor]: Add fp16 to the name of cae pre-train config * [Refactor]: Use FFN from mmcv * [Refactor]: Change network_blocks to trasformer_blocks * [Fix]: Fix mask generator name bug * [Fix]: cae pre-train config bug * [Fix]: Fix docstring grammar * [Fix]: Fix mc related code * [Fix]: Add object parent to transform * [Fix]: Delete unnecessary modification * [Fix]: Change blockwisemask generator to simmim mask generator * [Refactor]: Change cae mae pretrain vit to cae mae vit * [Refactor]: Change lamb to lambd * [Fix]: Remove blank line * [Fix]: Fix lint * [Fix]: Fix UT * [Fix]: Delete modification to swin * [Fix]: Fix lint * [Feature]: Add README and metafile * [Feature]: Update index.rst * [Fix]: Update model_zoo * [Fix]: Change MAE to CAE in algorithm * [Fix]: Change SimMIMMaskGenerator to CAEMaskGenerator * [Fix]: Fix model zoo * [Fix]: Change to dalle_encoder * [Feature]: Add download link for dalle * [Fix]: Fix lint * [Fix]: Fix UT * [Fix]: Update metafile * [Fix]: Change b to base * [Feature]: Add dalle download link in warning * [Fix] add arxiv link in readme Co-authored-by: Jiahao Xie <52497952+Jiahao000@users.noreply.github.com> * [Enhance] update SimCLR models and results (#295) * [Enhance] update simclr models and results * [Fix] revise comments to indicate settings * Update version (#296) * [Feature]: Update to 0.9.0 * [Feature]: Add version constrain for mmcls * [Fix]: Fix bug * [Fix]: Fix version bug * [Feature]: Update version in install.md * update changelog * update readme * [Fix] fix uppercase * [Fix] fix uppercase * [Fix] fix uppercase * update version dependency * add cae to readme Co-authored-by: fangyixiao18 <fangyx18@hotmail.com> Co-authored-by: Jiahao Xie <52497952+Jiahao000@users.noreply.github.com> Co-authored-by: Yixiao Fang <36138628+fangyixiao18@users.noreply.github.com> Co-authored-by: Ming Li <73068772+mitming@users.noreply.github.com> Co-authored-by: xcnick <xcnick0412@gmail.com> Co-authored-by: fangyixiao18 <fangyx18@hotmail.com> Co-authored-by: Jiahao Xie <52497952+Jiahao000@users.noreply.github.com>

YuanLiuuuuuu requested review from fangyixiao18 and Jiahao000 April 20, 2022 03:54

fangyixiao18 requested changes Apr 21, 2022

View reviewed changes

YuanLiuuuuuu requested a review from fangyixiao18 April 22, 2022 12:39

Jiahao000 requested changes Apr 24, 2022

View reviewed changes

fangyixiao18 requested changes Apr 25, 2022

View reviewed changes

YuanLiuuuuuu force-pushed the feature/cae_align branch from bbcc8cb to fb250db Compare April 25, 2022 13:03

YuanLiuuuuuu requested review from Jiahao000 and fangyixiao18 April 25, 2022 13:46

YuanLiuuuuuu force-pushed the feature/cae_align branch from 471ce80 to ef82f42 Compare April 27, 2022 07:56

Jiahao000 reviewed Apr 27, 2022

View reviewed changes

configs/selfsup/_base_/datasets/imagenet_cae.py Outdated Show resolved Hide resolved

Jiahao000 requested changes Apr 27, 2022

View reviewed changes

configs/selfsup/_base_/models/cae.py Outdated Show resolved Hide resolved

docs/en/model_zoo.md Outdated Show resolved Hide resolved

docs/zh_cn/model_zoo.md Outdated Show resolved Hide resolved

YuanLiuuuuuu force-pushed the feature/cae_align branch from a03e57e to d26bfef Compare April 27, 2022 13:44

YuanLiuuuuuu requested a review from Jiahao000 April 27, 2022 13:45

Jiahao000 requested changes Apr 27, 2022

View reviewed changes

configs/selfsup/cae/metafile.yaml Outdated Show resolved Hide resolved

fangyixiao18 requested changes Apr 28, 2022

View reviewed changes

YuanLiuuuuuu added 7 commits April 28, 2022 11:52

[Feature]: Add mc

95f730d

[Feature]: Add dataset of CAE

971e31e

[Feature]: Init version of CAE

9dd9dee

[Feature]: Add mc

afb6f54

[Fix]: Change beta to (0.9, 0.999)

c83336c

[Fix]: New feature

02076db

[Fix]: Decouple the qkv bias

dd42e04

YuanLiuuuuuu added 21 commits April 28, 2022 11:52

[Fix]: Change blockwisemask generator to simmim mask generator

5b34459

[Refactor]: Change cae mae pretrain vit to cae mae vit

78ad665

[Refactor]: Change lamb to lambd

25865e4

[Fix]: Remove blank line

2f1891f

[Fix]: Fix lint

711694d

[Fix]: Fix UT

414e38d

[Fix]: Delete modification to swin

e956fdd

[Fix]: Fix lint

4379543

[Feature]: Add README and metafile

65e6660

[Feature]: Update index.rst

5d0d8fb

[Fix]: Update model_zoo

c01cf7a

[Fix]: Change MAE to CAE in algorithm

6c47b30

[Fix]: Change SimMIMMaskGenerator to CAEMaskGenerator

c174fe5

[Fix]: Fix model zoo

d37366c

[Fix]: Change to dalle_encoder

b3de8f1

[Feature]: Add download link for dalle

bedcb40

[Fix]: Fix lint

b348868

[Fix]: Fix UT

e3f3a1a

[Fix]: Update metafile

a6a1c07

[Fix]: Change b to base

cbc0b8d

[Feature]: Add dalle download link in warning

a038f3e

YuanLiuuuuuu force-pushed the feature/cae_align branch from cd7f798 to a038f3e Compare April 28, 2022 03:55

YuanLiuuuuuu requested review from fangyixiao18 and Jiahao000 April 28, 2022 03:56

fangyixiao18 approved these changes Apr 28, 2022

View reviewed changes

[Fix] add arxiv link in readme

6a7aa99

Jiahao000 approved these changes Apr 28, 2022

View reviewed changes

Jiahao000 merged commit c532a11 into open-mmlab:dev_v0.9.0 Apr 28, 2022

YuanLiuuuuuu deleted the feature/cae_align branch April 28, 2022 09:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature]: CAE Supported #284

[Feature]: CAE Supported #284

YuanLiuuuuuu commented Apr 20, 2022

liming-ai commented Apr 21, 2022

YuanLiuuuuuu commented Apr 21, 2022

liming-ai commented Apr 21, 2022

YuanLiuuuuuu commented Apr 21, 2022

fangyixiao18 left a comment

fangyixiao18 Apr 21, 2022

YuanLiuuuuuu Apr 22, 2022

fangyixiao18 Apr 28, 2022

liming-ai commented Apr 21, 2022

YuanLiuuuuuu commented Apr 27, 2022

codecov bot commented Apr 27, 2022 •

edited

Loading

fangyixiao18 Apr 28, 2022

		@@ -0,0 +1,38 @@
		_base_ = 'vit-b-p16_ft-8xb128-coslr-100e_in1k.py'

[Feature]: CAE Supported #284

[Feature]: CAE Supported #284

Conversation

YuanLiuuuuuu commented Apr 20, 2022

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

liming-ai commented Apr 21, 2022

YuanLiuuuuuu commented Apr 21, 2022

liming-ai commented Apr 21, 2022

YuanLiuuuuuu commented Apr 21, 2022

fangyixiao18 left a comment

Choose a reason for hiding this comment

fangyixiao18 Apr 21, 2022

Choose a reason for hiding this comment

YuanLiuuuuuu Apr 22, 2022

Choose a reason for hiding this comment

fangyixiao18 Apr 28, 2022

Choose a reason for hiding this comment

liming-ai commented Apr 21, 2022

YuanLiuuuuuu commented Apr 27, 2022

codecov bot commented Apr 27, 2022 • edited Loading

Codecov Report

fangyixiao18 Apr 28, 2022

Choose a reason for hiding this comment

codecov bot commented Apr 27, 2022 •

edited

Loading