Skip to content

Official implementation of VIAT (Viewpoint-Invariant Adversarial Training) and GMVFool

Notifications You must be signed in to change notification settings

Heathcliff-saku/VIAT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

VIAT: Viewpoint-Invariant Adversarial Training

This repository contains the official implementation of VIAT and GMVFool for the paper "Towards Viewpoint-Invariant Visual Recognition via Adversarial Training" (ICCV2023)

and "Improving viewpoint robustness for visual recognition via adversarial training" (Extended Version)

By Shouwei Ruan, Yinpeng Dong, Hang Su, Jianteng Peng, Ning Chen, Xingxing Wei

fig1

⚙️ 1.Prerequisites

Ensure that you have these environments:

  • Pytorch (1.11.0)
  • torchvision (0.13.1)
  • timm (0.5.4)
  • pytorch-lighting(1.6.5)
  1. For the complete running environment of VIAT, please refer to ./requirements.txt.
  2. (🔥Extremely important) Please ensure to follow the ngp_pl(Instant-NGP based on pytorch-lighting) to install the relevant environment for Instant-NGP, since VIAT and GMVFool depend on Instant-NGP for viewpoint rendering.

💾 2.Datasets

The released data consists of two parts:

  1. IM3D: A multi-view dataset used for training, containing 100 hemispherical sampled viewpoint renderings of 1000 virtual 3D objects.
  1. ImageNet-V+, A larger adversarial viewpoint benchmark, including 100K adversarial viewpoint samples captured by GMVFool on IM3D.

🪄 3.Training NeRF for IM3D Objects

Before beginning, we need to use Instant-NGP to construct NeRF representations for the 1000 objects in IM3D, which will take approximately 24 hours. However, if you only want to conduct attacks or run simple demos, you can opt to train NeRF for a subset of the objects. Due to limited upload space, I regret that I cannot directly share the NeRF weights.

If you have correctly downloaded the IM3D dataset, extract it into a folder within the project. Then, training nerf can be conducted using the following script:

cd ./run_train_nerf
sh run_train_NeRF_1.sh

Note that you should modified --root_dir in run_train_NeRF_1.sh to the path of IM3D datasets, and the checkpoint will be saved in .\run_train_NeRF\ckpts\nerf

⚔️ 4.GMVFool

Using the following script to run GMVFool:

cd ./viat
sh run_Attack_1.sh

Note that you should modified --ckpt_attack_path to the path of nerf checkpoint, and --treat_model is the target model to attack. The weights of target model can be found in downloadlink, these are the weights we fine-tuned on the 100 classes from ImageNet, corresponding to the categories in IM3D.

🛡️ 5.VIAT

For VIAT using GMVFool:

cd ./viat
sh run_train_AT_final_res_viatgf_R.sh

For VIAT using ViewFool(Instant-NGP version):

sh run_train_AT_final_res_viatvf_R.sh

💽 6.ImageNet-V+ Benchmark

fig2

⚒️ 7. ViewRS

... The code is currently being organized, thank you for your patience.

Citation

If you find our methods useful or use the IM3D and imagenet-v+ dataset, please consider citing:

@inproceedings{ruan2023towards,
  title={Towards Viewpoint-Invariant Visual Recognition via Adversarial Training},
  author={Ruan, Shouwei and Dong, Yinpeng and Su, Hang and Peng, Jianteng and Chen, Ning and Wei, Xingxing},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={4709--4719},
  year={2023}
}
@article{ruan2023improving,
  title={Improving viewpoint robustness for visual recognition via adversarial training},
  author={Ruan, Shouwei and Dong, Yinpeng and Su, Hang and Peng, Jianteng and Chen, Ning and Wei, Xingxing},
  journal={arXiv preprint arXiv:2307.11528},
  year={2023}
}

This project uses Unofficial implementation of Instant-NGP (ngp_pl), Thanks to @kwea123:

@misc{queianchen_nerf,
  author={Quei-An, Chen},
  title={Nerf_pl: a pytorch-lightning implementation of NeRF},
  url={https://github.com/kwea123/nerf_pl/},
  year={2020},
}

If you are interested in viewpoint robustness, welcome to check our previous work: ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpoints (NIPS2022) [project].

😊 Contact

If you have any questions or suggestions about the paper or code, look forward to your contact with us:

About

Official implementation of VIAT (Viewpoint-Invariant Adversarial Training) and GMVFool

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published