SketchySceneColorization - SIGA2019

Paper | Supplementary Material | Project Page

This repository hosts the datasets and the code for the SketchyScene Colorization system (SIGGRAPH Asia 2019). Please refer to our paper for more information: Language-based Colorization of Scene Sketches.

System Overview

Our system supports two-mode interactive colorization for a given input scene sketch and text-based colorization instructions, using three models, namely, the instance matching model, foreground colorization model, and background colorization model.

Outline

• Requirements
• Preparations
  • Instance Matching
  • Foreground Instance Colorization
  • Background Colorization
• Colorizing With The Whole Pipeline
• Citation

Requirements

• Python 3
• Tensorflow (>= 1.3.0)
• scipy
• PIL
• skimage

Preparations

Please follow the instructions in the following three sections (Instance Matching, Foreground Instance Colorization, and Background Colorization) to download the dataset and pre-trained models and place them in the correct directories.

Instance Matching

For the details of MATCHING dataset and the code, please refer to the Instance_Matching directory.

Foreground Instance Colorization

For the details of FOREGROUND dataset and the code, please refer to the Foreground_Instance_Colorization directory.

Background Colorization

For the details of BACKGROUND dataset and the code, please refer to the Background_Colorization directory.

Colorizing With The Whole Pipeline

Our system allows users to colorize the sketches through language instructions. If the result is not satisfactory, users can also withdraw the last instruction.

🔥 We have provided some test examples in examples directory.

1. To colorize a sketch, run the command like:

   python3 sketchyscene_colorization_main.py --image_id 9996 \
                                             --instruction 'the bus is orange with gray windows'
   

   • Set image_id to the sketch you want.
   • Try other instructions by changing the instruction.

   You will see the results in outputs directory.

2. To withdraw the last instruction, run the command like:

   python3 sketchyscene_colorization_main.py --command 'withdraw' --image_id 9996
   

   See what happens in outputs directory :)

Citation

Please cite the corresponding paper if you found the datasets or code useful:

@article{zouSA2019sketchcolorization,
  title = {Language-based Colorization of Scene Sketches},
  author = {Zou, Changqing and Mo, Haoran and Gao, Chengying and Du, Ruofei and Fu, Hongbo},
  journal = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2019)},
  year = {2019},
  volume = 38,
  number = 6,
  pages = {233:1--233:16}
}

@inproceedings{zou2018sketchyscene,
  title={Sketchyscene: Richly-annotated scene sketches},
  author={Zou, Changqing and Yu, Qian and Du, Ruofei and Mo, Haoran and Song, Yi-Zhe and Xiang, Tao and Gao, Chengying and Chen, Baoquan and Zhang, Hao},
  booktitle={Proceedings of the european conference on computer vision (ECCV)},
  pages={421--436},
  year={2018}
}