SoC4SS-FGVC

This repo is the official Pytorch implementation of our paper:

Roll with the Punches: Expansion and Shrinkage of Soft Label Selection for Semi-supervised Fine-Grained Learning
Authors: Yue Duan, Zhen Zhao, Lei Qi, Luping Zhou, Lei Wang and Yinghuan Shi

• 🔗 Quick links:

  • Code Download
  • [PDF/Abs-arXiv | PDF/Abs-Published | Video | Poster/Slides]

• 📰 Latest news:

  • Slides and video explaining our paper are now online!
  • Our paper is accepted by AAAI Conference on Artificial Intelligence (AAAI) 2024 🎉🎉. Thanks to users.

• 📑 Related works:

  • 🆕 [LATEST] Interested in the cross-modal retrieval with noisy correspondence? 👉 Check out our ACMMM'24 paper PC2 [PDF-arXiv | Code].
  • [SSL] Interested in robust SSL in MNAR setting with mismatched distributions? 👉 Check out our ICCV'23 paper PRG [PDF-arXiv | Code] or our ECCV'22 paper RDA [PDF-arXiv | Code].
  • [SSL] Interested in the conventional SSL or application of complementary label in SSL? 👉 Check out our TNNLS paper MutexMatch [PDF-arXiv | Code].

Requirements

• numpy==1.20.3
• Pillow==9.3.0
• scikit_learn==1.1.3
• torch==1.8.0
• torchvision==0.9.0

How to Train

Important Args

• --net [resnet18/resnet50/resnet101/wrn/wrnvar/preresnet/cnn13]: By default, ResNet-50 is used for experiments. We provide alternatives as follows: ResNet-18/101, Wide ResNet, Wide ResNet-Var (WRN-37-2), PreAct ResNet and CNN-13.
• --dataset [semi-fungi/semi-aves] and --data_dir: Your dataset name and path. We support two datasets: Semi-Fungi and Semi-Aves (See "A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification" for details).
• --unlabel [in/inout]: The unlabeled data used for training. The unlabeled data of Semi-Fungi and Semi-Aves contains two subsets. One has in-distribution data only, the other one has both in-distribution and out-of-distribution data.
• --alpha: $\alpha$ for confidence-aware k selection.
• --num_tracked_batch: $N_{B}$ for class transition tracking (CTT).
• --resume, --load_path, --load_path_soc and --pretrained:
  • If you want to resume training without soc.pkl, i.e., saved data of centroids (for CTT-based k-means clustering) and label_matrix (for CTT), please set --resume --load_path @path to your checkpoint@.
  • If you want to resume training with soc.pkl, please set --resume --load_path @path to your checkpoint@ --load_path_soc @path to your soc.pkl@.
  • If you want to use the expert model or MoCo model, please set --resume --load_path @path to expert model/MoCo model@ --pretrained.

Training with Single GPU

python train_soc.py --rank 0 --gpu [0/1/...] @@@other args@@@

Training with Multi-GPUs

• Using DataParallel

python train_soc.py --world-size 1 --rank 0 @@@other args@@@

• Using DistributedDataParallel with single node

python train_soc.py --world-size 1 --rank 0 --multiprocessing-distributed @@@other args@@@

Examples of Running

To better reproduce our experimental results, it is recommended to use multi-GPUs with DataParallel for training.

Using In-distribuion Unlabeleda Data

Training from scratch

python train_soc.py --world-size 1 --rank 0 --seed 1 --num_eval_iter 2000 --overwrite --save_name aves_in_sc --dataset semi_aves --num_classes 200 --unlabel in 

Training from scratch with MoCo

python train_soc.py --world-size 1 --rank 0 --seed 1 --num_eval_iter 1000 --overwrite --save_name aves_in_sc_moco --dataset semi_aves --num_classes 200 --unlabel in --resume --load_path @path to MoCo pre-trained model@ --pretrained --num_train_iter 200000

Training from expert or expert with MoCo

python train_soc.py --world-size 1 --rank 0 --seed 1 --num_eval_iter 500 --overwrite --save_name aves_in_pre --dataset semi_aves --num_classes 200 --unlabel in --resume --load_path @path to pre-trained model@ --pretrained --lr 0.001 --num_train_iter 50000

The expert models and MoCo models can be obtained here (provided by https://github.com/cvl-umass/ssl-evaluation).

Using Out-of-Distribuion Unlabeleda Data

Training from scratch

python train_soc.py --world-size 1 --rank 0 --seed 1 --num_eval_iter 2000 --overwrite --save_name aves_inout_sc --dataset semi_aves --num_classes 200 --unlabel inout 

---

Evaluation

Each time you start training, the evaluation results of the current model will be displayed. If you want to evaluate a model, use its checkpoints to resume training, i.e., use --resume --load_path @path to your checkpoint@.

Results (e.g. seed=1)

Dateset	Unlabeled Data	Pre-training	Top1-Acc (%)	Top5-Acc (%)	Checkpoint
Semi-Aves	in-distribution	-	32.3	55.5	here
		MoCo	39.5	62.5	here
		ImageNet	56.8	79.1	here
		ImageNet + MoCo	57.1	79.1	here
		iNat	71.0	88.4	here
		iNat + MoCo	70.2	88.3	here
	out-of-distribution	-	27.5	50.7	here
		MoCo	40.4	65.9	here
Semi-Fungi	in-distribution	-	38.50	61.35	here
		MoCo	46.9	71.4	here
		ImageNet	61.1	83.2	here
		ImageNet + MoCo	61.8	85.9	here
		iNat	62.3	85.0	here
		iNat + MoCo	62.2	84.4	here
	out-of-distribution	-	35.6	60.6	here
		MoCo	50.0	74.8	here

Citaion

Please cite our paper if you find SoC useful:

@inproceedings{duan2024roll,
  title={Roll with the Punches: Expansion and Shrinkage of Soft Label Selection for Semi-supervised Fine-Grained Learning},
  author={Duan, Yue and Zhao, Zhen and Qi, Lei and Zhou, Luping and Wang, Lei and Shi, Yinghuan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={10},
  pages={11829--11837},
  year={2024}
}

or

@article{duan2023roll,
  title={Roll With the Punches: Expansion and Shrinkage of Soft Label Selection for Semi-supervised Fine-Grained Learning},
  author={Duan, Yue and Zhao, Zhen and Qi, Lei and Zhou, Luping and Wang, Lei and Shi, Yinghuan},
  journal={arXiv preprint arXiv:2312.12237},
  year={2023}
}