Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to filter available testsets by language pair #211

Merged
merged 9 commits into from
Oct 4, 2022

Conversation

ZJaume
Copy link
Contributor

@ZJaume ZJaume commented Sep 14, 2022

Implement #210

Using

sacrebleu --list -l en-fr

test sets available in English-French are listed.

export SKIP_CHRF="true"; export SKIP_TER="true"; export SKIP_MECAB="true"; export SKIP_MECAB_KO="true"; ./test.sh

tests are passing.

Some examples:

sacrebleu --list -l is-en
The available test sets for is-en are:
wmt21                         : Official evaluation data for WMT21.
wmt21/dev                     : Development data for WMT21,if multiple references are available, the first one is used.
sacrebleu --list -l tr-en
The available test sets for tr-en are:
wmt16                         : Official evaluation data.
wmt16/dev                     : Development sets released for new languages in 2016.
wmt17                         : Official evaluation data.
wmt18                         : Official evaluation data.
wmt18/test-ts                 : Official evaluation sources with extra test sets interleaved.
sacrebleu --list -l fa-en
The available test sets for fa-en are:

Swapping the language pair gives the same output for consistence with download, that is able to find the set regardless of direction I guess.

Using
```
sacrebleu --list -l en-fr
```
test sets available in English-French are listed.
@ZJaume
Copy link
Contributor Author

ZJaume commented Sep 14, 2022

Please let me know if something with codestyle or style of implementation is wrong and I will change it. Hope it is useful!

@ZJaume
Copy link
Contributor Author

ZJaume commented Sep 30, 2022

I thought test.sh was running pytest. Now it should be fixed, python -m pytest -k available works for me.

@mjpost
Copy link
Owner

mjpost commented Oct 3, 2022

Thanks! Can you give me write permission to your branch, so I add a few small changes (e.g., I updated the CHANGELOG).

Copy link
Owner

@mjpost mjpost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the PR!

@mjpost
Copy link
Owner

mjpost commented Oct 3, 2022

@ZJaume Let me know if you can give me write access. If so, I'll push up a few changes and merge. If not, I'll have to create a local branch and have someone else approve it.

@ZJaume
Copy link
Contributor Author

ZJaume commented Oct 4, 2022

There you go! I gave you collaborator access to my fork.

@mjpost mjpost merged commit 1d04b6f into mjpost:master Oct 4, 2022
@mjpost
Copy link
Owner

mjpost commented Oct 4, 2022

Thanks very much! I'll release this with 2.2.2 soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants