Skip to content

Implementation of "MECPformer: Multi-estimations Complementary Patch with CNN-Transformers for Weakly Supervised Semantic Segmentation"

Notifications You must be signed in to change notification settings

ChunmengLiu1/MECPformer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

37 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MECPformer

Official Implementation of the paper: MECPformer: Multi-estimations Complementary Patch with CNN-Transformers for Weakly Supervised Semantic Segmentation.

Accepted to Neural Computing and Applications.

outline

Abstract

The initial seed based on the convolutional neural network (CNN) for weakly supervised semantic segmentation always highlights the most discriminative regions but fails to identify the global target information. Methods based on transformers have been proposed successively benefiting from the advantage of capturing long-range feature representations. However, we observe a flaw regardless of the gifts based on the transformer. Given a class, the initial seeds generated based on the transformer may invade regions belonging to other classes. Inspired by the mentioned issues, we devise a simple yet effective method with Multi-estimations Complementary Patch (MECP) strategy and Adaptive Conflict Module (ACM), dubbed MECPformer. Given an image, we manipulate it with the MECP strategy at different epochs, and the network mines and deeply fuses the semantic information at different levels. In addition, ACM adaptively removes conflicting pixels and exploits the network self-training capability to mine potential target information. Without bells and whistles, our MECPformer has reached new state-of-the-art 72.0% mIoU on the PASCAL VOC 2012 and 42.4% on MS COCO 2014 dataset.

Prerequisite

1. install dependencies

Ubuntu 20.04, with Python 3.6 and the following python dependencies.

pip install -r requirements.txt

2. Download dataset

Download the PASCAL VOC 2012 development kit.

3. Download pretrained weights

Download Conformer-S pretrained weights.

4. Download saliency map

Download saliency map.

Usage

1. Run the run.sh script for training MECPformer in the initial seeds stage

bash run.sh

2. Train semantic segmentation network

To train DeepLab-v2, we refer to deeplab-pytorch.

Testing

Download our trained weights

Stage Backbone Google drive mIoU (%)
Initial seeds Conformer-S Weights 66.6
Final predict ResNet101 Weights 72.0

Acknowledgements

This code is borrowed from TransCAM, CPN, and deeplab-pytorch.

Citing MECPformer

If you use MECPformer in your research, please use the follwing entry:

@article{liu2023mecpformer,
  title={MECPformer: Multi-estimations Complementary Patch with CNN-Transformers for Weakly Supervised Semantic Segmentation},
  author={Liu, Chunmeng and Li, Guangyao and Shen, Yao and Wang, Ruiqi},
  journal={Neural Comput & Applic},
  year={2023}
}

or


Liu, C., Li, G., Shen, Y. et al. MECPformer: multi-estimations complementary patch with CNN-transformers for weakly supervised semantic segmentation. Neural Comput & Applic 35, 23249–23264 (2023). https://doi.org/10.1007/s00521-023-08816-2

About

Implementation of "MECPformer: Multi-estimations Complementary Patch with CNN-Transformers for Weakly Supervised Semantic Segmentation"

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published