GENEA Numerical Evaluations

Scripts for numerical evaluations for the GENEA Gesture Generation Challenge 2022: https://genea-workshop.github.io/2022/challenge/

This directory provides the scripts for quantitative evaluation of our gesture generation framework. We currently support the following measures:

• Average Jerk and Acceleration (AJ)
• Histogram of Moving Distance (HMD) for velocity and acceleration
• Hellinger distance between histograms
• Canonical Correlation Analysis (CCA) coefficient
• Fréchet Gesture Distance (FGD)

Obtain the data

Download the 3D coordinates of the GENEA Challenge 2022 systems at https://zenodo.org/record/6973297 . Create a data folder and put challenge system motions there as in data/UBA.

Run

calk_jerk_or_acceleration.py, calc_histogram.py, hellinger_distance.py and calc_cca.py support different quantitative measures, described below.

Average jerk and acceleration

Average Jerk (AJ) represent the characteristics of gesture motion.

To calculate AJ, you can use calk_jerk_or_acceleration.py.

# Compute AJ
python calk_jerk_or_acceleration.py -m jerk -c your_prediction_dir

Note: calk_jerk_or_acceleration.py computes AJ for both original and predicted gestures. The AJ of the original gestures will be stored in result/original by default. The AJ of the predicted gestures will be stored in result/your_prediction_dir.

The same script can be used to calculate average acceleration (AA):

# Compute AA
python calk_jerk_or_acceleration.py -m acceleration -c your_prediction_dir

Histogram of Moving Distance

Histogram of Moving Distance (HMD) shows the velocity/acceleration distribution of gesture motion.

To calculate HMD, you can use calc_histogram.py. You can select the measure to compute by --measure or -m option (default: velocity).
In addition, this script supports histogram visualization. To enable visualization, use --visualize or -v option.

# Compute velocity histogram
python calc_histogram.py -c your_prediction_dir -m velocity -w 0.05  # You can change the bin width of the histogram

# Compute acceleration histogram
python calc_histogram.py -c your_prediction_dir -m acceleration -w 0.05

Note: calc_histogram.py computes HMD for both original and predicted gestures. The HMD of the original gestures will be stored in result/original by default.

Hellinger distance

Hellinger distance indicates how close two histograms are to each other.

To calculate Hellinger distance, you can use hellinger_distance.py script.

Canonical Correlation Analysis

Canonical Correlation Analysis (CCA) is a way of inferring information from cross-covariance matrices. If we have two vectors X = (X1, ..., Xn) and Y = (Y1, ..., Ym) of random variables, and there are correlations among the variables, then canonical-correlation analysis will find linear combinations of X and Y which have maximum correlation with each other.

To calculate CCA coefficient, you can use calc_cca.py script.

Fréchet Gesture Distance

Please see README in the FGD folder.