FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition

[Paper] [Project Page]

Sicheng Mo^1*, Fangzhou Mu^2*, Kuan Heng Lin¹, Yanli Liu³, Bochen Guan³, Yin Li², Bolei Zhou^1
1 UCLA, ² University of Wisconsin-Madison, ³ Innopeak Technology, Inc
^* Equal contribution
Computer Vision and Pattern Recognition (CVPR), 2024

Getting Started

Environment Setup

• We proovide a conda env file for environment setup.

conda env create -f environment.yml
conda activate freecontrol

Sample Semantic Bases

• We provide two example file under the scripts folder as an example of how to compute target semantic bases.
• You can also download from google drive to use our pre-computed bases.
• After downloading the file, you can put it under the dataset folder and use the gradio demo.

Gradio demo

• We provide the user interface for testing out method. Ruuning the following commend to start the demo.

python gradio_app.py

Galley:

We are building a gallery generated with FreeControl. You are wellcomed to share your generated images with us.

Contact

Sicheng Mo (smo3@cs.ucla.edu)

Reference

@article{mo2023freecontrol,
  title={FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition},
  author={Mo, Sicheng and Mu, Fangzhou and Lin, Kuan Heng and Liu, Yanli and Guan, Bochen and Li, Yin and Zhou, Bolei},
  journal={arXiv preprint arXiv:2312.07536},
  year={2023}
}