Official resources for "A Benchmark for Evaluating Machine Translation Metrics on Dialects Without Standard Orthography"
This repository contains the code and additional resources for the paper "A Benchmark for Evaluating Machine Translation Metrics on Dialects Without Standard Orthography" presented at WMT 2023.
In the evaluation
directory, you can find:
- the Swiss German translations of NTREX-128
- the systems translations and the human ratings
- the challenge set
- the code to evaluate metrics on our test sets.
In the training
directory, you can find the code for training the neural metrics evaluated in the paper.
We provide a fork of the mtm-metrics-eval
repository. Run the following commands to clone it before you follow instructions in the evaluation
directory.
git submodule init
git submodule update
If you use our code, data or models, please cite our paper:
@inproceedings{aepli-etal-2023-benchmark,
title = "A Benchmark for Evaluating Machine Translation Metrics on Dialects without Standard Orthography",
author = {Aepli, No{\"e}mi and
Amrhein, Chantal and
Schottmann, Florian and
Sennrich, Rico},
editor = "Koehn, Philipp and
Haddon, Barry and
Kocmi, Tom and
Monz, Christof",
booktitle = "Proceedings of the Eighth Conference on Machine Translation",
month = dec,
year = "2023",
address = "Singapore",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.wmt-1.99",
pages = "1045--1065"
}