Modulated Contrast for Versatile Image Synthesis

Paper | Project page | Video

Fangneng Zhan, Jiahui Zhang, Yingchen Yu, Rongliang Wu, Shijian Lu
Nanyang Technological University, Singapore

MoNCE outperforms Perceptual Loss and PatchNCE Loss.

Abstract: This paper presents MoNCE, a versatile metric that introduces image contrast to learn a calibrated metric for the perception of multifaceted inter-image distances. Unlike vanilla contrast which indiscriminately pushes negative samples from the anchor regardless of their similarity, we propose to re-weight the pushing force of negative samples adaptively according to their similarity to the anchor, which facilitates the contrastive learning from informative negative samples. Since multiple patch-level contrastive objectives are involved in image distance measurement, we introduce optimal transport in MoNCE to modulate the pushing force of negative samples collaboratively across multiple contrastive objectives.

Prerequisites

• Linux or macOS
• Python3, PyTorch
• CPU or NVIDIA GPU + CUDA CuDNN

Installation

Please follow the guidance in SPADE and CUT.

Inference Using Pretrained Model

Unpaired Image Translation (CUT):

The pretrained model on Cityscapes, Horse2Zebra, Winter2Summer can be downloaded from Google Drive. Put them into CUT_MoNCE/checkpoints and run the command

cd CUT_MoNCE
bash test_cityscapes.sh

Paired Image Translation (SPADE):

The pretrained model on ADE20K, CelebA-HQ (semantic), DeepFashion can be downloaded from Google Drive. Put them into SPADE_MoNCE/checkpoints and run the command

cd SPADE_MoNCE
bash test_ade20k.sh

Training

Unpaired Image Translation (CUT):

Run the command

cd CUT_MoNCE
bash train_cityscapes.sh

Paired Image Translation (SPADE):

Run the command

cd SPADE_MoNCE
bash train_ade20k.sh

Citation

If you use this code for your research, please cite our papers.

@inproceedings{zhan2022modulated,
  title={Modulated contrast for versatile image synthesis},
  author={Zhan, Fangneng and Zhang, Jiahui and Yu, Yingchen and Wu, Rongliang and Lu, Shijian},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={18280--18290},
  year={2022}
}

Acknowledgments

This code borrows heavily from CUT and SPADE.