SIGGRAPH 2024 Journal Track
Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis

Zeyi Zhang*, Tenglong Ao*, Yuyao Zhang*, Qingzhe Gao, Chuan Lin, Baoquan Chen, Libin Liu†

---

- Video (YouTube) - Paper (arXiv) - Project Page (github) -

This is a reimplemention of Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis.

This codebase provides:

• SeG dataset
• pretrained models
• training & inference codes

If you use the dataset or codes, please cite our Paper

@article{
  Zhang2024SemanticGesture,
  author = {Zhang, Zeyi and Ao, Tenglong and Zhang, Yuyao and Gao, Qingzhe and Lin, Chuan and Chen, Baoquan and Liu, Libin},
  title = {Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis},
  journal = {ACM Trans. Graph.},
  issue_date = {July 2024},
  numpages = {17},
  doi = {10.1145/3658134},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {co-speech gesture synthesis, multi-modality, retrieval augmentation}
}

SeG Dataset

Collection Period of the semantic gesture inventory/list and the motion capture data: December 2023 - May 2024.

Copyright Source of semantic gestures: the "Copyright Source" column in SeG_list.xlsx indicates the source of each semantic gesture, with the copyright belonging to the respective references.

The MOCCA Semantic Gesture Dataset (SeG) comprises 208 types of semantic gestures commonly utilized globally. Each record in the dataset includes a sequence of captured motions and associated meta-information.

As shown below, the meta-information includes semantics-aware index, label, description, contextual meaning, and example sentence, facilitating a thorough organization and analysis of human gestures. The related information is stored in ./SeG_dataset/SeG_list.xlsx.

As for motion, we have open-sourced the motion capture data from a male performer, comprising 544 motion files in the .bvh format. Each gesture is represented, on average, in 2.6 distinct ways. These motion files are named using the format "label" + "represented index" and stored in ./SeG_dataset/bvh. You can also get these files in the .zip format from Google Drive.

Code

coming soon ...