müller-sennrich-2021-understanding

This directory uses the mbr package to reproduce an experiment from the paper Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation (Müller & Sennrich, ACL-IJCNLP 2021).

Setup

• Task: Machine translation
• Translation directions: dan–epo, aze–eng, bel–rus, deu–fra
• MBR metric: ChrF2 (Popović, 2015)
• Number of samples: 5–100
• Sampling approach: ancestral sampling
• Samples and references are the same
• Test set: Tatoeba (Tiedemann, 2020)
• Evaluation metric: ChrF2
• Baseline: beam search with beam size 5

Differences to the paper

• The paper used custom models trained without label smoothing, this reproduction uses open-source models from Opus-MT (Tiedemann & Thottingal, 2020).
• The paper reports averages over 2 runs, this reproduction uses a single run.

Results

Paper	Reproduction