NSF: Neural Surface Field for Human Modeling from Monocular Depth

Project Page | Video | Paper

In ICCV 2023, Paris

Yuxuan Xue^1, , Bharat Lal Bhatnagar^2,, Riccardo Marin¹, Nikolaos Sarafianos², Yuanlu Xu², Gerard Pons-Moll¹, Tony Tung²

¹Real Virtual Human Group @ University of Tübingen & Tübingen AI Center & Max Planck Institute for Informatics
²Meta Reality Lab Research

News 🚩

• [2023/08/30] NSF paper is available on ArXiv.
• [2023/08/29] Code for NSF is available.
• [2023/07/14] NSF is accepted to ICCV 2023, Paris.

Key Insight 🙌

• Define neural fields on top of continuous surface
• Learned NSF generalizes to arbitrary resolution or topology of the surface
• Why benefitial For clothed human modelling:
  • eliminates need for Marching Cube / Poisson Reconstruction per frame => efficient
  • can reconstruct / animate human mesh resolution / topology => flexible
  • keeps mesh coherency across different frames => modelling

Instruction 📘

A. Preparation

1. Dependencies

Please refer to Dependencies for:

• install conda environment and required packages
• obtain SMPL model
• obtain prediffused SMPL skinning weights field

2. Data

Please refer to Data for:

• render depth frames from scan
• preprocess data

3. Path

Please specify ROOT_DIR and DATA_DIR here

B. Running

Learn Fusion Shape via SDF

Please refer to Fusion Shape for:

• learn implicit fusion shape
• fit SMPL-D to fusion shape
• project off-surface points onto fusion shape

Learn Neural Surface Field based on Fusion Shape

Please refer to NSF for:

• learn NSF to model clothed avatar
• infer clothed avatar at arbitrary resolution
• animate clothed avatar with desired pose sequences

C. Pre-trained Model

Please refer to Pretrained Models for:

• available pretrained models for subjects in BuFF/CAPE dataset

Citation ✍️

@inproceedings{xue2023nsf,
  title     = {{NSF: Neural Surface Field for Human Modeling from Monocular Depth}},
  author    = {Xue, Yuxuan and Bhatnagar, Bharat Lal and Marin, Riccardo and Sarafianos, Nikolaos and Xu, Yuanlu and Pons-Moll, Gerard and Tung, Tony.},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month     = {October},
  year      = {2023},
}