TextDeformer [Project Page]

Installation

conda create -y -n TextDeformer python=3.9
conda activate TextDeformer
pip3 install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu113
conda install -y -c conda-forge igl
pip install -r requirements.txt

Usage

NOTE: This repository requires a GPU to run.

Run examples

main.py is the primary script to use. You may pass arguments using the --config flag, which takes the path to a .yml file. See example_config.yml for an example. Alternatively, you may pass command line arguments manually, which override the arguments provided by the config file. Below, we provide example usage:

# Use all arguments provided by the example config
python main.py --config example_config.yml

# Change the optimized mesh to hand.obj, change the base and target text prompts
python main.py --config example_config.yml --mesh meshes/hand.obj --text_prompt 'an octupus' --base_text_prompt 'a hand'

# Now, increase the batch size, learning rate, and the training resolution
python main.py --config example_config.yml --mesh meshes/hand.obj --text_prompt 'an octopus' \
--base_text_prompt 'a hand' --batch_size 50 --lr 0.005 --train_res 1024

Outputs

The outputs will be saved to the path specified in the run configuration, which is ./outputs by default. The output folder will contain several folders: images contains intermittently saved samples of the rendered images passed to CLIP, logs will contain tensorboard logs of the optimization process, mesh_best_clip, mesh_best_total, and mesh_final contain the optimized meshes at the best CLIP score, the best total loss, and the final epoch. The configuration file is also saved at config.yml and a video of the optimization process is saved at video_log.mp4.

Common bugs

Mesh Orientation

Due to how the camera angles are sampled, the input mesh may need to be re-oriented to ensure proper performance. You can check your mesh orientation by checking images/epoch_0.png.

The Eiffel tower mesh is oriented poorly and CLIP will see the mesh rendered from strange viewpoints

Here, we re-orient the mesh so that CLIP will see the side profile of the tower

Jacobian temp files

The NeuralJacobianFields code in this repository will create several temporary files in outputs/tmp. Note that if these temporary files already exist, this code will attempt to read the existing files instead of overwriting them. This may cause issues if you run multiple examples with the same output path, intending to overwrite the output folder.

Citation

@InProceedings{Gao_2023_SIGGRAPH,
    author    = {Gao, William and Aigerman, Noam and Groueix Thibault and Kim, Vladimir and Hanocka, Rana},
    title     = {TextDeformer: Geometry Manipulation using Text Guidance},
    booktitle = {ACM Transactions on Graphics (SIGGRAPH)},
    year      = {2023},
}