Skip to content

jamycheung/Trans4PASS

Repository files navigation

Transformers for Panoramic Semantic Segmentation (Trans4PASS & Trans4PASS+)

Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation, CVPR 2022, [PDF]. trans4pass

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation, arxiv preprint, [PDF]. trans4pass+

PWC

PWC

PWC

PWC

Update

  • [08/2022], a new panoramic semantic segmentation benchmark SynPASS is released.

  • [08/2022], Trans4PASS+ is released.

Environments

conda create -n trans4pass python=3.8
conda activate trans4pass
cd ~/path/to/trans4pass 
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install mmcv-full==1.3.9 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html
pip install -r requirements.txt
python setup.py develop --user
# Optional: install apex follow: https://github.com/NVIDIA/apex

SynPASS dataset

SynPASS

SynPASS dataset contains 9080 panoramic images (1024x2048) and 22 categories.

The scenes include cloudy, foggy, rainy, sunny, and day-/night-time conditions.

The SynPASS dataset is now availabel at GoogleDrive.

SynPASS statistic information:

Cloudy Foggy Rainy Sunny ALL
Split train/val/test train/val/test train/val/test train/val/test train/val/test
#Frames 1420/420/430 1420/430/420 1420/430/420 1440/410/420 5700/1690/1690
Split day/night day/night day/night day/night day/night
#Frames 1980/290 1710/560 2040/230 1970/300 7700/1380
Total 2270 2270 2270 2270 9080

Data Preparation

Prepare datasets:

datasets/
├── cityscapes
│   ├── gtFine
│   └── leftImg8bit
├── Stanford2D3D
│   ├── area_1
│   ├── area_2
│   ├── area_3
│   ├── area_4
│   ├── area_5a
│   ├── area_5b
│   └── area_6
├── Structured3D
│   ├── scene_00000
│   ├── ...
│   └── scene_00199
├── SynPASS
│   ├── img
│   │   ├── cloud
│   │   ├── fog
│   │   ├── rain
│   │   └── sun
│   └── semantic
│       ├── cloud
│       ├── fog
│       ├── rain
│       └── sun
├── DensePASS
│   ├── gtFine
│   └── leftImg8bit

Prepare pretrained weights, which can be found in the public repository of SegFormer.

pretrained/
├── mit_b1.pth
└── mit_b2.pth

Network Define

The code of Network pipeline is in segmentron/models/trans4pass.py.

The code of backbone is in segmentron/models/backbones/trans4pass.py.

The code of DMLP decoder is in segmentron/modules/dmlp.py.

The code of DMLPv2 decoder is in segmentron/modules/dmlp2.py.

Train

For example, to use 4 1080Ti GPUs to run the experiments:

# Trans4PASS
python -m torch.distributed.launch --nproc_per_node=4 tools/train_cs.py --config-file configs/cityscapes/trans4pass_tiny_512x512.yaml
python -m torch.distributed.launch --nproc_per_node=4 tools/train_s2d3d.py --config-file configs/stanford2d3d/trans4pass_tiny_1080x1080.yaml
# Trans4PASS+, please modify the version at segmentron/models/trans4pass.py
python -m torch.distributed.launch --nproc_per_node=4 tools/train_cs.py --config-file configs/cityscapes/trans4pass_plus_tiny_512x512.yaml
python -m torch.distributed.launch --nproc_per_node=4 tools/train_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml
python -m torch.distributed.launch --nproc_per_node=4 tools/train_s2d3d.py --config-file configs/stanford2d3d/trans4pass_plus_tiny_1080x1080.yaml

Test

Download the models from GoogleDrive and save in ./workdirs folder as:

workdirs folder:
./workdirs
├── cityscapes
│   ├── trans4pass_plus_small_512x512
│   ├── trans4pass_plus_tiny_512x512
│   ├── trans4pass_small_512x512
│   └── trans4pass_tiny_512x512
├── cityscapes13
│   ├── trans4pass_plus_small_512x512
│   ├── trans4pass_plus_tiny_512x512
│   ├── trans4pass_small_512x512
│   └── trans4pass_tiny_512x512
├── stanford2d3d
│   ├── trans4pass_plus_small_1080x1080
│   ├── trans4pass_plus_tiny_1080x1080
│   ├── trans4pass_small_1080x1080
│   └── trans4pass_tiny_1080x1080
├── stanford2d3d8
│   ├── trans4pass_plus_small_1080x1080
│   ├── trans4pass_plus_tiny_1080x1080
│   ├── trans4pass_small_1080x1080
│   └── trans4pass_tiny_1080x1080
├── stanford2d3d_pan
│   ├── trans4pass_plus_small_1080x1080
│   ├── trans4pass_small_1080x1080
│   └── trans4pass_tiny_1080x1080
├── structured3d8
│   ├── trans4pass_plus_small_512x512
│   ├── trans4pass_plus_tiny_512x512
│   ├── trans4pass_small_512x512
│   └── trans4pass_tiny_512x512
├── synpass
│   ├── trans4pass_plus_small_512x512
│   ├── trans4pass_plus_tiny_512x512
│   ├── trans4pass_small_512x512
│   └── trans4pass_tiny_512x512
└── synpass13
    ├── trans4pass_plus_small_512x512
    ├── trans4pass_plus_tiny_512x512
    ├── trans4pass_small_512x512
    └── trans4pass_tiny_512x512

Some test examples:

# Trans4PASS
python tools/eval.py --config-file configs/cityscapes/trans4pass_tiny_512x512.yaml
python tools/eval_s2d3d.py --config-file configs/stanford2d3d/trans4pass_tiny_1080x1080.yaml
# Trans4PASS+
python tools/eval.py --config-file configs/cityscapes/trans4pass_plus_tiny_512x512.yaml
python tools/eval_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml
python tools/eval_dp13.py --config-file configs/synpass13/trans4pass_plus_tiny_512x512.yaml
python tools/eval_s2d3d.py --config-file configs/stanford2d3d/trans4pass_plus_tiny_1080x1080.yaml

Trans4PASS models CS -> DP:

from Cityscapes to DensePASS

python tools/eval.py --config-file configs/cityscapes/trans4pass_plus_tiny_512x512.yaml
Network CS DP Download
Trans4PASS (T) 72.49 45.89 model
Trans4PASS (S) 72.84 51.38 model
Trans4PASS+ (T) 72.67 50.23 model
Trans4PASS+ (S) 75.24 51.41 model

Trans4PASS models SPin -> SPan:

from Stanford2D3D_pinhole to Stanford2D3D_panoramic

python tools/eval_s2d3d.py --config-file configs/stanford2d3d/trans4pass_plus_tiny_1080x1080.yaml
Network SPin SPan Download
Trans4PASS (T) 49.05 46.08 model
Trans4PASS (S) 50.20 48.34 model
Trans4PASS+ (T) 48.99 46.75 model
Trans4PASS+ (S) 53.46 50.35 model

Trans4PASS models on SPan:

supervised trained on Stanford2D3D_panoramic

# modify fold id at segmentron/data/dataloader/stanford2d3d_pan.py
# modify TEST_MODEL_PATH at configs/stanford2d3d_pan/trans4pass_plus_small_1080x1080.yaml
python tools/eval_s2d3d.py --config-file configs/stanford2d3d_pan/trans4pass_plus_tiny_1080x1080.yaml
Network Fold SPan Download
Trans4PASS+ (S) 1 54.05 model
Trans4PASS+ (S) 2 47.70 model
Trans4PASS+ (S) 3 60.25 model
Trans4PASS+ (S) avg 54.00

Note: for the Trans4PASS versions (not Trans4PASS+), please modify the respective DMLP version, check here.

Trans4PASS models on SP:

supervised trained on SynPASS

python tools/eval_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml
Network Cloudy Foggy Rainy Sunny Day Night ALL (val) ALL (test) Download
Trans4PASS (T) 46.90 41.97 41.61 45.52 44.48 34.73 43.68 38.53 model
Trans4PASS (S) 46.74 43.49 43.39 45.94 45.52 37.03 44.80 38.57 model
Trans4PASS+ (T) 48.33 43.41 43.11 46.99 46.52 35.27 45.21 38.85 model
Trans4PASS+ (S) 48.87 44.80 45.24 47.62 47.17 37.96 46.47 39.16 model

Trans4PASS models Pin2Pan vs. Syn2Real settings:

  • (1) Indoor Pin2Pan: SPin8 -> SPan8
  • (2) Indoor Syn2Real: S3D8 -> SPan8
  • (3) Outdoor Pin2Pan: CS13 -> DP13
  • (4) Outdoor Syn2Real: SP13 -> DP13
python tools/eval_dp13.py --config-file configs/synpass13/trans4pass_plus_tiny_512x512.yaml
python tools/eval_dp13.py --config-file configs/cityscaps13/trans4pass_plus_tiny_512x512.yaml
python tools/eval_s2d3d8.py --config-file configs/structured3d8/trans4pass_plus_tiny_512x512.yaml
python tools/eval_s2d3d8.py --config-file configs/stanford2d3d8/trans4pass_plus_tiny_512x512.yaml
(1) Outdoor Pin2Pan: CS13 DP13 mIoU Gaps model
Trans4PASS (Tiny) 71.63 49.21 -22.42 model
Trans4PASS (Small) 75.21 50.96 -24.25 model
Trans4PASS+ (Tiny) 72.92 49.16 -23.76 model
Trans4PASS+ (Small) 74.52 51.40 -23.12 model
(2) Outdoor Syn2Real: SP13 DP13 mIoU Gaps
Trans4PASS (Tiny) 61.08 39.68 -21.40 model
Trans4PASS (Small) 62.76 43.18 -19.58 model
Trans4PASS+ (Tiny) 60.37 39.62 -20.75 model
Trans4PASS+ (Small) 61.59 43.17 -18.42 model
(3) Indoor Pin2Pan: SPin8 SPan8 mIoU Gaps
Trans4PASS (Tiny) 64.25 58.93 -5.32 model
Trans4PASS (Small) 66.51 62.39 -4.12 model
Trans4PASS+ (Tiny) 65.09 59.55 -5.54 model
Trans4PASS+ (Small) 65.29 63.08 -2.21 model
(4) Indoor Syn2Real: S3D8 SPan8 mIoU Gaps
Trans4PASS (Tiny) 76.84 48.63 -28.21 model
Trans4PASS (Small) 77.29 51.70 -25.59 model
Trans4PASS+ (Tiny) 76.91 50.60 -26.31 model
Trans4PASS+ (Small) 76.88 51.93 -24.95 model

Domain Adaptation

Train

After the model is trained on the source domain, the model can be further trained by the adversarial method for warm-up in the target domain.

cd adaptations
python train_warm.py

Then, use the warm up model to generate the pseudo label of the target damain.

python gen_pseudo_label.py

The proposed MPA method can be jointly used for perform domain adaptation.

# (optional) python train_ssl.py
python train_mpa.py

Domain adaptation with Trans4PASS+, for example, adapting model trained from pinhole to panoramic, i.e., from Cityscapes13 to DensePASS13 (CS13 -> DP13):

python train_warm_out_p2p.py
python train_mpa_out_p2p.py

Test

Download the models from GoogleDrive and save in adaptations/snapshots folder as:

adaptations/snapshots/
├── CS2DensePASS_Trans4PASS_v1_MPA
│   └── BestCS2DensePASS_G.pth
├── CS2DensePASS_Trans4PASS_v2_MPA
│   └── BestCS2DensePASS_G.pth
├── CS132CS132DP13_Trans4PASS_plus_v2_MPA
│   └── BestCS132DP13_G.pth
├── CS2DP_Trans4PASS_plus_v1_MPA
│   └── BestCS2DensePASS_G.pth
└── CS2DP_Trans4PASS_plus_v2_MPA
    └── BestCS2DensePASS_G.pth

Change the RESTORE_FROM in evaluate.py file. Or change the scales in evaluate.py for multi-scale evaluation.

# Trans4PASS
cd adaptations
python evaluate.py
# Trans4PASS+, CS132DP13
python evaluate_out13.py

References

We appreciate the previous open-source works.

License

This repository is under the Apache-2.0 license. For commercial use, please contact with the authors.

Citations

If you are interested in this work, please cite the following works:

Trans4PASS+ [PDF]

@article{zhang2022behind,
  title={Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation},
  author={Zhang, Jiaming and Yang, Kailun and Shi, Hao and Rei{\ss}, Simon and Peng, Kunyu and Ma, Chaoxiang and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer},
  journal={arXiv preprint arXiv:2207.11860},
  year={2022}
}

Trans4PASS [PDF]

@inproceedings{zhang2022bending,
  title={Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation},
  author={Zhang, Jiaming and Yang, Kailun and Ma, Chaoxiang and Rei{\ss}, Simon and Peng, Kunyu and Stiefelhagen, Rainer},
  booktitle={2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={16917--16927},
  year={2022}
}

About

Repository of Trans4PASS (accepted to CVPR2022)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published