This repository contains the code for the paper "Art Creation With Multi-Conditional StyleGANs" accepted at IJCAI 2022. The source code is based on StyleGAN2-ADA by Karras et al. from NVIDIA.
In general, instructions are the same as for the original StyleGAN2-ADA code. For multi-conditional training, you will also need to supply a prepared_dataset.json
file with the --cond-path
flag to the python train.py
command. The prepared_dataset.json
should contain the multi-conditions and have the following format:
{
"condition_order": ["painter", "keywords", "emotions"], // condition names in the order they appear in the concatenated vector
"shapes": [352, 768, 9], // the size of the vector representation for each condition part (in the same order)
"labels": [
["/path/to/image/within/data/dir", [<label>, <label>, <label>],
... // for all images in dataset
],
}
Each <label>
is either:
- an integer (which will be converted to vector representation through one-hot-encoding)
- an array containing one or more strings (which will be converted to vector representation with a pretrained TinyBERT embedding). In case of multiple strings, a single representation will be randomly sampled each time the training sample is shown to the model.
- an array containing floats, which should be a probability distribution and are directly used as vector representation
How to get the EnhancedArtEmis dataset?
Unfortunately, we cannot host the dataset or emotion annotations due to copyright and licencing. We outline the process to reproduce the dataset and prepare it for training with StyleGAN2 here.
- Follow the instructions in the ArtEmis repository to download and preprocess the emotion annotations. We do not use their preprocessing for "deep" networks.
- Download the actual image files for ArtEmis. You can use our
download_artemis_images.py
script. This can take a while. - Create a
dataset.json
file compatible with thedataset_tool.py
that maps image names to emotion labels. We convert multiple emotion annotations per image directly into a probability distribution. The format fordataset.json
should look like this:
{
"labels": [
["vincent-van-gogh_the-starry-night-1889.jpg", [0.0, 0.0, 0.2, 0.4, 0.0, 0.0, 0.2, 0.0, 0.2]],
["claude-monet_water-lilies-1919.jpg", [0.0, 0.8, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]],
... //for every image in the dataset
["pablo-picasso_guernica-1937.jpg", [0.0, 0.0, 0.0, 0.0, 0.6, 0.0, 0.4, 0.0, 0.0]],
]
}
- Prepare the image files and
dataset.json
for training with thedataset_tool.py
(instructions in the original StyleGAN2-ADA repository). Thedataset.json
should be placed inside the folder containing the images downloaded in step 2. We used a command like this (we scale non-square images instead of cropping):
python dataset_tool.py --source=./artemis-download/ --dest=./processed-artemis.zip --width=512 --height=512
- To get additional annotations scraped from Wikiart use our
scrape_additional_annotations.py
script. Or you can use theenhanced_annotations.json
we provide (scraped as of July 2021). - Now, we only need to create the
prepared_dataset.json
to enable our multi-conditional training. We have prepared thecreate_label_json.py
script for this. - 🚀 Wow, you made it! 🚀 Time to create some 🎨 art 🎨.
You can start a multi-conditional training with a command like this:
python train.py --outdir=./results-out --data=</path/to/data.zip> --cond-path ./annotations/painter-style-keywords/prepared_dataset.json --gpus=1 --snap=50 --workers=4 --batch=64 --cond=1 -n my-multiconditional-stylegan --dataset-cache-dir </path/to/cache/if/wanted>
In the paper, we propose the conditional truncation trick for StyleGAN. If you use the truncation trick together with conditional generation or on diverse datasets, give our conditional truncation trick a try (it's a drop-in replacement). The effect is illustrated below (figure taken from the paper):
The implementation is quite easy. The relevant lines of code can be found here:
multi-conditional-stylegan/training/networks.py
Lines 245 to 264 in 5b9ec5f
The source code in this repository is distributed under the same Nvidia Source Code License as the original StyleGAN2-ADA repository.
@inproceedings{dobler2022multiconditional,
title = {Art Creation with Multi-Conditional StyleGANs},
author = {Dobler, Konstantin and Hübscher, Florian and Westphal, Jan and Sierra-Múnera, Alejandro and de Melo, Gerard and Krestel, Ralf},
booktitle = {Proceedings of the Thirty-First International Joint Conference on
Artificial Intelligence, {IJCAI-22}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Lud De Raedt},
pages = {4936--4942},
year = {2022},
month = {7},
note = {AI and Arts}
doi = {10.24963/ijcai.2022/684},
url = {https://doi.org/10.24963/ijcai.2022/684},
}
If you use the code in this repository, please also cite StyleGAN2-ADA:
@inproceedings{Karras2020ada,
title = {Training Generative Adversarial Networks with Limited Data},
author = {Tero Karras and Miika Aittala and Janne Hellsten and Samuli Laine and Jaakko Lehtinen and Timo Aila},
booktitle = {Proc. NeurIPS},
year = {2020}
}