BOA - Bilevel Online Adaptation

🌟🌟🌟 New: Stronger DynaBOA is available at https://github.com/syguan96/DynaBOA.

Code repository for the paper:
Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction
Shanyan Guan*, Jingwei Xu*, Yunbo Wang^†, Bingbing Ni^†, Xiaokang Yang
CVPR 2021
[Paper] [project page] [Supp]

Citation

If you find this code useful for your research or the use data generated by our method, please consider citing this paper:

@inproceedings{syguan2021boa,
  Title          = {Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction},
  Author         = {Shanyan, Guan and Jingwei, Xu and Yunbo, Wang and Bingbing, Ni and Xiaokang, Yang},
  Booktitle      = {CVPR},
  Year           = {2021}
}

Requirements

1. Please run pip install -r requirements.txt to install all dependencies.
2. Downloading data related to SMPL:

• Download the SMPL model and then remove the dependency on Chumpy follwing this instruction. Then, put the processed models to data/smpl/.
• Download 3rd party files which is provided by SPIN. Then extact the file and put them to data/spin_data.

Get Started

Download the base model pre-trained on Human 3.6M. Run the following commond to excute Bilevel online optimization.

python boa.py --name boa-1

Preparing Dataset

Before running the BOA, we should process the datasets first.

• 3DPW
  Note that this is the guideline to get data according to the #PS protocol (i.e. processing 3DPW following SPIN). To obtain data according to the #PH protocol, please run the scripts in HMMR, and save the results.

  1. Download the 3DPW dataset. Then edit PW3D_ROOT in the config.py.
  2. Run the processing script:

     python process_data.py --dataset 3dpw
     

• MPI-INF-3DHP

  1. Download the MPI-INF-3DHP dataset. Then edit MPI_INF_3DHP_ROOT in the config.py
  2. Extracting frames by running

     cd utils/data_preprocess
     python extract_3dhp_frames.py
     

  3. Run the processing script:

     python process_data.py --dataset 3dhp
     

• Human 3.6M
  Mesh annotation is a necessary file. However I cannot provide it in public. If you get access to it, following the next intruction to process Human 3.6M. Note that installing spacepy is required. Please refer to this website to install it.

  1. Download Human 3.6M. Downloader is suggested.
  2. Unpack files:

     cd utils/data_preprocess
     python unpack_h36m.py
     

     And then edit H36M_ROOT in the config.py.
  3. Check if the mesh annotations need to be rectified:

     cd utils/data_preprocess
     python check_mosh.py
     

     If the joints is not aligned to the image, please rectify them by

     python rectify_pose.py
     

  4. Processing Human 3.6M:

     python process_data.py --dataset h36m
     

References

Here are some great resources we benefit:

• SPIN
• learn2learn
• VIBE
• SMPL-X