README.md

SKEL

This repository is part of the code release for the Siggraph Asia 2023 paper "From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans".

[paper] [project page]

This repo contains the pytorch SKEL loader and the code to align it to SMPL sequences.

SKEL is a parametric body shape and skeleton model. Its shape parameter θ lets you change the body shape and its pose parameter qlets you pose the skeleton and skin in an anatomically plausible way. Given shape and pose parameters, SKEL returns joint locations, a body mesh, and a skeleton mesh. SKEL is differentiable and can be fit to various data like motion capture or SMPL sequences.

For more details, please check our Siggraph 2023 paper: From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans.

Table of Contents

• Quickstart
• Installation
  • Set up the environment
  • Downloading SKEL
  • Aitviewer
  • SMPL and Mesh package
• Demos
  • Pose parameters
  • Shape space
  • Rigging
  • SKEL sequence
  • Aligning SKEL to SMPL mesh
  • Aligning SKEL to SMPL sequences
• Acknoledgments
• Citation
• License
• Contact

Quickstart

For a minimal use, you will have to follow the install instructions in Set up the environment and Downloading SKEL.

To test the SKEL model, you can then run:

python quickstart.py 

This runs the forward pass of SKEL and saves the output as separated body and skeleton meshes.

You can then try to change the SKEL parameters knowing the following. SKEL is controled by two parameters:

• betas controls the shape of the body (tall, small, skinny, etc...). It is a vector of size 10. The values should remain between -2 and 2. The first value of the vector roughly controls the height, the second value the weight.

• pose controls the body pose. It is a vector of size 46, most values are angles expressed in radian, and can vary between -3.14 and 3.14. You can see which of the 46 parameters controls which body articulation here .

SKEL(beta, pose) yields:

• A skeleton 3D mesh
• A body mesh
• Anatomical joint locations

The rest of this readme will let you setup our SKEL vizualizer and run some demo code. We hope you enjoy our work!

Installation

Set up the environment

Clone this repository

git clone https://github.com/MarilynKeller/SKEL
cd SKEL

Create a virtual environment and install the SKEL package

pip install -U pip   
python3.8 -m venv skel_venv
source skel_venv/bin/activate
pip install git+https://github.com/mattloper/chumpy 
pip install -e .

Downloading SKEL

Create an account on https://skel.is.tue.mpg.de/ (Necessary for the download to work).

Then download the SKEL model from the download page with the "Download Models" button. Extract the downloaded folder and edit the file SKEL/skel/config.py to specify the folder containing the downloaded SKEL model folder: skel_folder = '/path/to/skel_models_v1.0

Below, we explain how to set up the visualizer for SKEL (only supported on Linux and Mac), and run some example code.

Aitviewer

If you want to run the Demos, you will also need our aitviewer fork for visualization:

cd ..
git clone https://github.com/MarilynKeller/aitviewer-skel.git
cd aitviewer-skel 
pip install -e .

Edit then the file aitviewer/aitviewer/aitvconfig.yaml to point to the SKEL folder:

skel_models: "/path/to/skel_models_v1.0"

SMPL and MPI Mesh package

! Note that the MPI mesh package is only supported on Linux systems. If you are not on Linux, you will have to comment out the code depending on the package psbody.mesh and code your own visualization.

If you want to run an alignment to SMPL, you need to download the SMPL model. First, create an account on https://smpl.is.tue.mpg.de/. Then download this file: SMPL_python_v.1.1.0.zip from the download page. And run:

cd ../SKEL
python scripts/setup_smpl.py /path/to/SMPL_python_v.1.1.0.zip  

For visualizing the fitting process you need the MPI mesh package, you can install it with the following line:

pip install git+https://github.com/MPI-IS/mesh.git  

Note that the mesh package is only supported on Linux, but it is only necessary for the visualization of the fitting processes.

Demos

Pose parameters

Visualize the effects of the pose parameters of SKEL:

python examples/skel_poses.py --gender male

Shape space

Vizualize the shape space:

python examples/skel_betas.py --gender female 

Rigging

Visualize the skinning weights of the skin and bones to the SKEL kinematic tree:

python examples/skel_rigging.py --gender female  

Visualize the kinematic tree and joint locations of SKEL:

python examples/skel_kintree.py --gender female

You can see a visual of the joint ids here and their names and the list of degrees of freedom here

SKEL sequence

Visualize a SKEL sequence. You can find a sample SKEL motion in skel_models_v1.0/sample_motion/ and the corresponding SMPL motion.

python examples/skel_sequence.py /path/to/skel_models_v1.x/sample_motion/01_01_poses_skel.pkl -z 

To visualize the SMPL sequence alongside :

python examples/skel_sequence.py /path/to/skel_models_v1.0/sample_motion/01_01_poses_skel.pkl -z --smpl_seq /path/to/skel_models_v1.0/sample_motion/01_01_poses.npz

Aligning SKEL to a SMPL mesh

SKEL can be aligned to SMPL meshes:

python examples/align_to_SMPL_frame.py --smpl_data 'examples/samples/img_fit/emily-sea-coiWR0gT8Cw-unsplash_0.npz' 

Aligning SKEL to SMPL sequences

SKEL can be aligned to SMPL sequences, here is an example:

python examples/align_to_SMPL_seq.py examples/samples/amass_seq/CMU_01_01.npz -D 

You can visualize the result with:

python examples/skel_sequence.py output/CMU_01_01/CMU_01_01_skel.pkl -z 

You can download more SMPL sequences of this kind from the AMASS Download page, and selecting the SMPL+H G sequences.

Acknowledgments

This research was done partly at The Movement Lab in Stanford and Perceiving Systems at the Max Planck Institute for Intelligent Systems.

We thank Neelay Shah for generating all the necessary SMPL fits, A. A. Osman for his expertise on parametric body models, Shashank Tripathi for his help with MOYO, Giorgio Becherini for his expertise on AMASS and mocap fit evaluation, Peter Kultis, Yao Feng and Yuliang Xu for feedbacks on the paper. We also thank the TML lab and the NMBL lab at Stanford for the fruitful discussions, especially Jennifer Maier for her expertise on the shoulder.

Citation

If you use this software, please cite the following work and software:

@inproceedings{keller2023skel,
  title = {From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans},
  author = {Keller, Marilyn and Werling, Keenon and Shin, Soyong and Delp, Scott and 
            Pujades, Sergi and Liu, C. Karen and Black, Michael J.},
  booktitle = {ACM ToG, Proc.~SIGGRAPH Asia},
  volume = {42},
  number = {6},
  month = dec,
  year = {2023},
}

License

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE.txt file.

Contact

For any questions about SKEL loading, please contact skel@tuebingen.mpg.de.

For commercial licensing, please contact ps-licensing@tue.mpg.de