Isometric translations are evaluated using standard translation quality measures (such as BLEU, BERTScore), and length compliance metrics. Below you will find the steps to run an evaluation script for Isometric SLT,
Before running the evaluation script, make sure to have files containing the input, translation output, and/or reference samples.
Install BERTScore and Sacrebleu
./setup-env.sh
For a source language en, a target language L=fr, source samples test.en, mt output test_mt.fr,
and reference samples test.fr, run the following command to compute the metrics for isometric SLT:
./compute_isometric_slt_stat.sh $L test.en test_mt.$L test.$L
#cat log.stat.$L
The above command should provide you with the full log of translation quality metrics (BERTScore, Sacrebleu), and length compliance scores (length ratio, length range within +-10%). For the Isometric SLT shared task, you should use BERTScore and Length range.