S²HAND: Model-based 3D Hand Reconstruction via Self-Supervised Learning

S²HAND presents a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. Specifically, we obtain geometric cues from the input image through easily accessible 2D detected keypoints. To learn an accurate hand reconstruction model from these noisy geometric cues, we utilize the consistency between 2D and 3D representations and propose a set of novel losses to rationalize outputs of the neural network. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations. For more details, please see our paper, video, and project page.

Code

Environment

Training is implemented with PyTorch. This code was developed under Python 3.6 and Pytorch 1.1.

Please compile the extension modules by running:

pip install tqdm tensorboardX transforms3d chumpy scikit-image

git clone https://github.com/TerenceCYJ/neural_renderer.git
cd neural_renderer
python setup.py install
rm -r neural_renderer

Note that we modified the neural_renderer/lighting.py compared to daniilidis-group/neural_renderer.

Data

For example, for 3D hand reconstruction task on the FreiHAND dataset:

• Download the FreiHAND dataset from the website.
• Modify the input and output directory accordingly in examples/config/FreiHAND/*.json.

For HO3D dataset:

• Download the HO3D dataset from the website.
• Modify the input and output directory accordingly in examples/config/HO3D/*.json.

Offline 2D Detection

• Offline 2D keypoint detection use a off-the-shelf detector like pytorch-openpose.

  • We also provide detected 2D keypoints for FreiHAND training set. You may downlad and change the self.open_2dj_lists in the examples/data/dataset.py accordingly.

  • Or Download the hand_pose_model.pth provided by pytorch-openpose, and put the file to examples/openpose_detector/src. Then use the following script and modify the input and output directory accordingly.

    python example/openpose_detector/hand_dectect.py

Training and Evaluation

HO3D

Evaluation: download the pretrained model [texturehand_ho3d.t7], and modify the "pretrain_model" in examples/config/HO3D/evaluation.json.

cd S2HAND
python3 ./examples/train.py --config_json examples/config/HO3D/evaluation.json

Training:

Stage-wise training:

python3 ./examples/train.py --config_json examples/config/HO3D/SSL-shape.json
python3 ./examples/train.py --config_json examples/config/HO3D/SSL-kp.json
python3 ./examples/train.py --config_json examples/config/HO3D/SSL-finetune.json

Or end-to-end training:

python3 ./examples/train.py --config_json examples/config/HO3D/SSL-e2e.json

Note: remember to check and inplace the dirs and files in the *.json files.

FreiHAND

Evaluation: download the pretrained model [texturehand_freihand.t7], and modify the "pretrain_model" in examples/config/FreiHAND/evaluation.json.

cd S2HAND
python3 ./examples/train.py --config_json examples/config/FreiHAND/evaluation.json

Training: refer to HO3D traing scripts.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{chen2021s2hand,
    title={Model-based 3D Hand Reconstruction via Self-Supervised Learning}, 
    author={Chen, Yujin and Tu, Zhigang and Kang, Di and Bao, Linchao and Zhang, Ying and Zhe, Xuefei and Chen, Ruizhi and Yuan, Junsong},
    booktitle={Conference on Computer Vision and Pattern Recognition},
    year={2021}
}