variable number of gold annotations

[danyaljj]

The example in the introduction assumes that there is a fixed # of gold annotations for each generated text:

import sacrebleu
refs = [['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
        ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.']]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']
bleu = sacrebleu.corpus_bleu(sys, refs)
print(bleu.score)

Is there a call that accepts variable number of gold text for each generated text?

[martinpopel]

Yes, it is expected that each reference (gold) translation is available for all sentences. If this is not the case, and e.g. a third reference is missing for some sentences, you can fill the missing blanks with another reference (duplicating a reference should have no effect on BLEU):

refs = [['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
        ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
        ['This is the 3rd reference, which is available only for the first sentence.', 'No one was surprised.', 'The man had bitten the dog.'],]