This repository provides the official implementation of Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models at ICCV 2023.
- [2023.07.18] π₯ Release the code and instructions. π€
- [2023.07.14] π₯ Our paper IDPT has been accepted by ICCV 2023! ππ Many thanks to all the collaborators and anonymous reviewers! π₯°
We first explore prompt tuning for pre-trained point cloud models and propose a novel Instance-aware Dynamic Prompt Tuning (IDPT) strategy to enhance the model robustness against distributional diversity (caused by various noises) in real-world point clouds. IDPT generally utilizes a lightweight prompt generation module to perceive the semantic prior features and generate instance-aware prompt tokens for the pre-trained point cloud model. Compared with the common (static) prompt tuning strategies like Visual Prompt Tuning (VPT), IDPT shows notable improvement in downstream adaptation. IDPT is also competitive with full fine-tuning while requiring only ~7% of the trainable parameters.
In the following, we will guide you how to use this repository step by step. π€
git clone git@github.com:zyh16143998882/ICCV23-IDPT.git
cd ICCV23-IDPT/
- gcc >= 4.9
- cuda >= 9.0
- python >= 3.7
- pytorch >= 1.7.0 < 1.11.0
- anaconda
- torchvision
conda create -y -n idpt python=3.7
conda activate idpt
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt
# Chamfer Distance & emd
cd ./extensions/chamfer_dist
python setup.py install --user
cd ./extensions/emd
python setup.py install --user
# PointNet++
pip install "git+https://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"
# GPU kNN
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
pip install torch-scatter
Before running the code, we need to make sure that everything needed is ready. First, the working directory is expected to be organized as below:
click to expand π
ICCV23-IDPT/
βββ cfgs/
βββ data/
β βββ ModelNet/ # ModelNet40
β β βββ modelnet40_normal_resampled/
β β βββ modelnet40_shape_names.txt
β β βββ modelnet40_train.txt
β β βββ modelnet40_test.txt
β β βββ modelnet40_train_8192pts_fps.dat
β β βββ modelnet40_test_8192pts_fps.dat
β βββ ModelNetFewshot/ # ModelNet Few-shot
β β βββ 5way10shot/
β β β βββ 0.pkl
β β β βββ ...
β β β βββ 9.pkl
β β βββ 5way20shot/
β β β βββ ...
β β β ...
β β βββ 10way10shot/
β β β βββ ...
β β β ...
β β βββ 10way20shot/
β β βββ ...
β β ...
β βββ ScanObjectNN/ # ScanObjectNN
β β βββ main_split/
β β β βββ training_objectdataset_augmentedrot_scale75.h5
β β β βββ test_objectdataset_augmentedrot_scale75.h5
β β β βββ training_objectdataset.h5
β β β βββ test_objectdataset.h5
β β βββ main_split_nobg/
β β βββ training_objectdataset.h5
β β βββ test_objectdataset.h5
β βββ ShapeNet55-34/ # ShapeNet55/34
β β βββ shapenet_pc/
β β β βββ 02691156-1a04e3eab45ca15dd86060f189eb133.npy
β β β βββ 02691156-1a6ad7a24bb89733f412783097373bdc.npy
β β β βββ ...
β β β ...
β β βββ ShapeNet-55/
β β βββ train.txt
β β βββ test.txt
β βββ shapenetcore_partanno_segmentation_benchmark_v0_normal/ # ShapeNetPart
β βββ 02691156/
β β βββ 1a04e3eab45ca15dd86060f189eb133.txt
β β βββ ...
β β ...
β βββ ...
β βββ train_test_split/
β βββ synsetoffset2category.txt
βββ datasets/
βββ ...
...
Here we have also collected the download links of required datasets for you:
- ShapeNet55/34 (for pre-training): [link].
- ScanObjectNN: [link].
- ModelNet40: [link 1] (pre-processed) or [link 2] (raw).
- ModelNet Few-shot: [link].
- ShapeNetPart: [link].
To pre-train Point-MAE on ShapeNet training set, you can run the following command:
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/pretrain.yaml --exp_name <output_file_name>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/pretrain.yaml --exp_name pretrain_pointmae
If you want to try other models or change pre-training configuration, e.g., mask ratios, just create a new configuration file and pass its path to --config
.
For a quick start, we also have provided the pre-trained checkpoint of Point-MAE [link].
We take VPT and IDPT as two showcases of prompt tuning for pre-trained point cloud models. Executable commands of different downstream tasks are listed below.
VPT-Deep (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_modelnet_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name modelnet_vpt
# further enable voting mechanism
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet.yaml --test --vote --exp_name modelnet_vpt_vote --ckpts ./experiments/finetune_modelnet/cfgs/modelnet_vpt/ckpt-best.pth
IDPT (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_modelnet_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name modelnet_idpt
# further enable voting mechanism
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet.yaml --test --vote --exp_name modelnet_idpt_vote --ckpts ./experiments/finetune_modelnet/cfgs/modelnet_idpt/ckpt-best.pth
VPT-Deep (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objbg_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name bg_vpt
IDPT (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name bg_idpt
VPT-Deep (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objonly_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name only_vpt
IDPT (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name only_idpt
VPT-Deep (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_hardest_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name hard_vpt
IDPT (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name hard_idpt
VPT-Deep (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/fewshot_vpt.yaml --finetune_model --ckpts <path/to/pre-trained/model> --exp_name <output_file_name> --way <5 or 10> --shot <10 or 20> --fold <0-9>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/fewshot_vpt.yaml --finetune_model --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --exp_name fewshot_vpt --way 5 --shot 10 --fold 0
IDPT (click to expand π)
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/fewshot_idpt.yaml --finetune_model --ckpts <path/to/pre-trained/model> --exp_name <output_file_name> --way <5 or 10> --shot <10 or 20> --fold <0-9>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/fewshot_idpt.yaml --finetune_model --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --exp_name fewshot_idpt --way 5 --shot 10 --fold 0
IDPT (click to expand π)
cd segmentation
# python main.py --model prompt_pt3 --optimizer_part only_new --ckpts <path/to/pre-trained/model> --root path/to/data --learning_rate 0.0002 --epoch 300
CUDA_VISIBLE_DEVICES=0 python main.py --model prompt_pt3 --optimizer_part only_new --ckpts ../checkpoint/pretrain/mae/ckpt-last.pth --root ../data/shapenetcore_partanno_segmentation_benchmark_v0_normal/ --log_dir seg_idpt --learning_rate 0.0002 --epoch 300
For reproducibility, logs and checkpoints of tuned models via IDPT can be found in the table below.
Task | Dataset | Trainable Parameters | log | Acc. | Checkpoints Download |
---|---|---|---|---|---|
Pre-training | ShapeNet | 1.7M | - | N.A. | Point-MAE |
Classification | ScanObjectNN | 1.7M | finetune_scan_objbg.log | 93.63% | OBJ-BG |
Classification | ScanObjectNN | 1.7M | finetune_scan_objonly.log | 93.12% | OBJ-ONLY |
Classification | ScanObjectNN | 1.7M | finetune_scan_hardest.log | 88.51% | PB-T50-RS |
Classification | ModelNet40 | 1.7M | finetune_modelnet.log | 93.3% | ModelNet-1k |
Classification | ModelNet40 (vote) | - | finetune_modelnet_vote.log | 94.4% | - |
Task | Dataset | log | 5w10s (%) | 5w20s (%) | 10w10s (%) | 10w20s (%) |
---|---|---|---|---|---|---|
Few-shot learning | ModelNet40 | fewshot_logs | 97.3 Β± 2.1 | 97.9 Β± 1.1 | 92.8 Β± 4.1 | 95.4 Β± 2.9 |
The evaluation commands with checkpoints should be in the following format:
CUDA_VISIBLE_DEVICES=<GPU> python main.py --test --config <yaml_file_name> --exp_name <output_file_name> --ckpts <path/to/ckpt>
For example, click to expand π
# object classification on ScanObjectNN (PB-T50-RS)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --ckpts ./checkpoint/hardest/ckpt-best.pth --test --exp_name hard_test
# object classification on ScanObjectNN (OBJ-BG)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --ckpts ./checkpoint/bg/ckpt-best.pth --test --exp_name bg_test
# object classification on ScanObjectNN (OBJ-ONLY)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --ckpts ./checkpoint/only/ckpt-best.pth --test --exp_name only_test
# object classification on ModelNet40 (w/o voting)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --ckpts ./checkpoint/modelnet40/ckpt-best.pth --test --exp_name model_test
# object classification on ModelNet40 (w/ voting)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --test --vote --exp_name modelnet_idpt_vote --ckpts ./checkpoint/modelnet40/ckpt-best.pth
# few-show learning on ModelNet40
python parse_test_res.py ./experiments/all/fewshot --multi-exp --few-shot
We used t-SNE to visualize the results obtained by IDPT on the ModelNet40 and ScanObjectNN test sets. The execution commands are as follows:
CUDA_VISIBLE_DEVICES=<GPU> python main.py --tsne --config <yaml_file_name> --exp_name <output_file_name> --ckpts <path/to/ckpt>
For example, click to expand π
# t-SNE on ScanObjectNN (PB-T50-RS)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_hardest_idpt_tsne.yaml --ckpts ./checkpoint/hardest/ckpt-best.pth --tsne --exp_name hard_test_tsne
# object classification on ScanObjectNN (OBJ-BG)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_objbg_idpt_tsne.yaml --ckpts ./checkpoint/bg/ckpt-best.pth --tsne --exp_name bg_test_tsne
# object classification on ScanObjectNN (OBJ-ONLY)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_objonly_idpt_tsne.yaml --ckpts ./checkpoint/only/ckpt-best.pth --tsne --exp_name only_test_tsne
# object classification on ModelNet40
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_modelnet_idpt_tsne.yaml --ckpts ./checkpoint/modelnet40/ckpt-best.pth --tsne --exp_name model_test_tsne
You can also modify the code yourself to implement a customized t-SNE visualization. π€
If you find this code useful or use the toolkit in your work, please consider citing:
@inproceedings{zha2023_IDPT,
title={Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models},
author={Zha, Yaohua and Wang, Jinpeng and Dai, Tao and Chen, Bin and Wang, Zhi and Xia, Shu-Tao},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2023}
}
Our codes are built upon Point-MAE, Point-BERT, ACT, DGCNN and VPT. Thanks for their efforts.
If you have any question, you can raise an issue or email Yaohua Zha (chayh21@mails.tsinghua.edu.cn) and Jinpeng Wang (wjp20@mails.tsinghua.edu.cn). We will reply you soon.