AutoAvatar: Autoregressive Neural Fields for Dynamic Avatar Modeling

Ziqian Bai · Timur Bagautdinov · Javier Romero . Michael Zollhöfer · Ping Tan · Shunsuke Saito

ECCV 2022

AutoAvatar is an autoregressive approach for modeling dynamically deforming human bodies directly from raw scans without the need of precise surface registration.

Data Preparation of DFaust

• Create "DFaust" folder under "<workspace_folder>".

cd <workspace_folder>
mkdir DFaust

• Download SMPL+H parameters of DFaust from AMASS dataset to "<workspace_folder>/DFaust". Unzip to get the "DFaust_67" folder.

• Download Dfaust scan data from link. Here, we take subject 50002 as an example in the following steps. Unzip data to "<workspace_folder>/DFaust/scans/50002".

• Download SMPL model from link. Download SMPL meta data from link. Move SMPL related files "basicmodel_m_lbs_10_207_0_v1.0.0.pkl", "basicModel_f_lbs_10_207_0_v1.0.0.pkl", "uv_info.npz", and "smpl_resample_idxs.npz" into "<workspace_folder>/SMPL".

• Set up AMASS for DFaust data preprocessing. More specifically, download SMPL+H (smplh.tar.xz) and unzip to "<workspace_folder>/SMPL/smplh". Download DMPLs (dmpls.tar.xz) and unzip to "<workspace_folder>/SMPL/dmpls".

• clone this repo to "<workspace_folder>".

cd <workspace_folder>
git clone https://github.com/facebookresearch/AutoAvatar.git

• Now we should have the following folder structure:

    \<workspace_folder\>
    ├── DFaust
    │   ├── DFaust_67
    │   │   └── 50002
    │   │       └── *.npz
    │   └── scans
    │       └── 50002
    │           └── \<sequences_folders\>
    ├── SMPL
    │   ├── smplh
    │   │   ├── female
    │   │   ├── male
    │   │   └── neutral
    │   ├── dmpls
    │   │   ├── female
    │   │   ├── male
    │   │   └── neutral
    |   └── \<other_SMPL_related_files\>
    └── AutoAvatar

Environment Setup

• Install Anaconda or Miniconda. Then run the setup script.

cd AutoAvatar
conda create -n AutoAvatar python=3.8
conda activate AutoAvatar
bash setup.sh

• Create "external" folder and install human_body_prior for DFaust data preprocess.

mkdir external
cd external
git clone https://github.com/nghorbani/human_body_prior.git
cd human_body_prior
python setup.py develop

Data Preprocess

• Run "DFaust_generate.py" to preprocess data. Note that this may take a long time due to the mesh simplification (the open3d API mesh_o3d.simplify_quadric_decimation() in simplify_scans())! Mesh simplification is to speed up data loading during training.

cd AutoAvatar
export PYTHONPATH=<workspace_folder>/AutoAvatar
python data/DFaust_generate.py --ws_dir <workspace_folder>

Train

• Run "implicit_train_dfaust.py" to train the model.

cd AutoAvatar
export PYTHONPATH=<workspace_folder>/AutoAvatar
python exps/PosedDecKNN_dPoses_dHs/implicit_train_dfaust.py --ws_dir <workspace_folder> --configs_path configs/PosedDecKNN_dPoses_dHs/AutoRegr.yaml --configs_path_rollout configs/PosedDecKNN_dPoses_dHs/AutoRegr_Rollout2.yaml

Test

• Run "implicit_eval_dfaust.py" to test the model.

cd AutoAvatar
export PYTHONPATH=<workspace_folder>/AutoAvatar
python exps/PosedDecKNN_dPoses_dHs/implicit_eval_dfaust.py --ws_dir <workspace_folder> --ckpt_dir <checkpoint_folder>

Pretrained Model

• Download pretrained model for DFaust subject 50002 from link.

Publication

If you find our code or paper useful, please consider citing:

@inproceedings{bai2022autoavatar,
  title={AutoAvatar: Autoregressive Neural Fields for Dynamic Avatar Modeling},
  author={Bai, Ziqian and Bagautdinov, Timur and Romero, Javier and Zollh{\"o}fer, Michael and Tan, Ping and Saito, Shunsuke},
  booktitle={European conference on computer vision},
  year={2022},
}

License

CC-BY-NC 4.0. See the LICENSE file.