Skip to content

Latest commit

 

History

History
141 lines (101 loc) · 4.86 KB

README.md

File metadata and controls

141 lines (101 loc) · 4.86 KB
Raw

Combining Generative and Geometry Priors for Wide-Angle Portrait Correction

Lan Yao, Chaofeng Chen, Xiaoming Li, Zifei Yan, Wangmeng Zuo

we introduce a framework that leverages generative and geometry priors to rectify wide-angle distortions in faces and backgrounds. Experiments demonstrate that our approach outperforms previous methods by a large margin, excelling not only in quantitative measures such as line straightness and shape consistency metrics but also in terms of perceptual visual quality

Given a wide-angle image, LineCNet and FaceCNet deal with the correction in background and face region seperatly, then face fusion is done by a post-process.

TODO

  • Release the source code.
  • Release the checkpoint model.

Installation

All dependencies for defining the environment are provided in environment.yaml.

conda env create -f environment.yaml
conda actviate Dualpriors

Dataset Preparation

Public Dataset:

public dataset released by work Practical Wide-Angle Portraits Correction with Deep Structured Models

Generated distorted face by FFHQ and CelebA-HQ:

You can generate by code like below: first, generate the distorting flow:

python utils/gen_data.py --option flow --dir ./path_to_save_flow

then, generate the distorted face:

python utils/gen_data.py --option image --dir ./FFHQ

Evaluate

Because of our post process, ShapeAcc and Landmark Distance cannot directly generated by an output flow, which we calculate by using a face detection tool to get each faces keypoints and calculate according to ./evaluate.py.

Step 0: download the weights

checkpoints(Baidu Netdisk) should be put in ./pretrained_models/

Step 1: download the dataset

download the test dataset and put in ../test/.

Step 2: generate the output of test dataset

python evaluate.py \ 
--option generate \ 
--test-dir ../test/ \ 
--device cuda:0 \ 
--e4e-path ./pretrained_models/e4e_best_model.pth \ 
--linenet-path ./pretrained_models/linenet.pt \ 
--facenet-path ./pretrained_models/facenet.pt

The generated output is before post-process part, which means for each input xx.jpg, you now have a xx_out.jpg and xx_mask.jpg that you may need a image inpainting alogorithm to generate the final output. You can use lama for this part.

Step 3: Evaluate

Due to our method's approach, we don't generate a complete flow for the entire image during output creation. Consequently, calculating evaluation metrics requires additional processing to obtain the final combined flow, which significantly increases evaluation time compared to the output generation step.

To run the evaluation:

python evaluate.py \ 
--option evaluate \ 
--test-dir ../test/ \ 
--device cuda:0 \ 
--e4e-path ./pretrained_models/e4e_best_model.pth \ 
--linenet-path ./pretrained_models/linenet.pt \ 
--facenet-path ./pretrained_models/facenet.pt

Alternative evaluation method:

  1. Generate the output images (after post-process) .
  2. Use a face landmark detection method to obtain 81 corresponding face landmarks for each output.
  3. Save these landmarks as JSON files.
  4. Use these JSON files to directly calculate the metrics.

This method bypasses the need for combined flow calculation and may yield more accurate results.

Note: To ensure compatibility with the test dataset, we recommend using the Megvii Face++ Landmark Detection API. When using this API, remember to exclude the 'left_eye_pupil' and 'right_eye_pupil' landmarks to match the 81 landmarks in the test dataset.

Training

Data for finetune e4e encoder:

  • generated distorted face data using FFHQ and CelebA-HQ

Training Data for FaceCNet:

  • generated distorted face data using FFHQ and CelebA-HQ
  • cropped face from public datset
python train_facecnet.py
--dataset_type facecnet_data \
--exp_dir experiment/ \
--start_from_latent_avg \
--val_interval 40000 \
--save_interval 20000 \
--max_steps 200000 \
--image_interval 1000 \
--stylegan_size 512 \
--stylegan_weights pretrained_models/stylegan2.pt \
--workers 4 \
--batch_size 4 \
--test_batch_size 8 \
--test_workers 4 \
--checkpoint_path pretrained_models/e4e_best_model.pt \

Training Data for lineCNet:

python train_linenet.py \
--dataset_type linenet_data \
--exp_dir experiment/ \
--val_interval 20000 \
--batch_size 4 \
--max_steps 200000

Acknowledgement

This project is built based on encoder4editing.