Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add language_bcp47 tag #4753

Merged
merged 7 commits into from
Jul 27, 2022
Merged

Add language_bcp47 tag #4753

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e65d63b
fix ms_terms
lhoestq Jul 27, 2022
e0c09a8
fix ted_talks_iwslt
lhoestq Jul 27, 2022
01aae4a
move tags to language_bcp47
lhoestq Jul 27, 2022
0f30913
fix bad codes
lhoestq Jul 27, 2022
47e1263
add language_details and language_bcp47 in DatasetMetadata
lhoestq Jul 27, 2022
ecaa2b5
more fixes
lhoestq Jul 27, 2022
c7fd4c7
fix wiki_dpr
lhoestq Jul 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions datasets/ai2_arc/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ annotations_creators:
language_creators:
- found
language:
- en
language_bcp47:
- en-US
license:
- cc-by-sa-4.0
Expand Down
2 changes: 2 additions & 0 deletions datasets/arcd/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ annotations_creators:
language_creators:
- crowdsourced
language:
- ar
language_bcp47:
- ar-SA
license:
- mit
Expand Down
2 changes: 1 addition & 1 deletion datasets/bbaw_egyptian/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ language_creators:
- found
language:
- de
- en
- egy
- en
license:
- cc-by-4.0
multilinguality:
Expand Down
4 changes: 2 additions & 2 deletions datasets/blbooks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ annotations_creators:
language_creators:
- machine-generated
language:
- en
- fr
- de
- en
- es
- fr
- it
- nl
license:
Expand Down
2 changes: 1 addition & 1 deletion datasets/blbooksgenre/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ language_creators:
- crowdsourced
- expert-generated
language:
- en
- de
- en
- fr
- nl
license:
Expand Down
1 change: 0 additions & 1 deletion datasets/casino/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ task_ids:
- dialogue-modeling
pretty_name: Campsite Negotiation Dialogues
paperswithcode_id: casino

---


Expand Down
18 changes: 10 additions & 8 deletions datasets/cc100/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ language:
- be
- bg
- bn
- bn-Latn
- br
- bs
- ca
Expand All @@ -39,7 +38,6 @@ language:
- ha
- he
- hi
- hi-Latn
- hr
- ht
- hu
Expand Down Expand Up @@ -71,7 +69,6 @@ language:
- mr
- ms
- my
- my-x-zawgyi
- ne
- nl
- 'no'
Expand All @@ -87,9 +84,9 @@ language:
- ro
- ru
- sa
- si
- sc
- sd
- si
- sk
- sl
- so
Expand All @@ -100,26 +97,31 @@ language:
- sv
- sw
- ta
- ta-Latn
- te
- te-Latn
- th
- tl
- tn
- tr
- ug
- uk
- ur
- ur-Latn
- uz
- vi
- wo
- xh
- yi
- yo
- zh
- zu
language_bcp47:
- bn-Latn
- hi-Latn
- my-x-zawgyi
- ta-Latn
- te-Latn
- ur-Latn
- zh-Hans
- zh-Hant
- zu
license:
- unknown
multilinguality:
Expand Down
12 changes: 5 additions & 7 deletions datasets/ccaligned_multilingual/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ language:
- br
- bs
- ca
- ceb
- ckb
- cs
- ceb
- cy
- de
- dv
Expand Down Expand Up @@ -48,6 +48,7 @@ language:
- iu
- ja
- ka
- kac
- kg
- kk
- km
Expand All @@ -71,7 +72,6 @@ language:
- ms
- mt
- my
- my
- ne
- nl
- 'no'
Expand All @@ -83,15 +83,14 @@ language:
- pl
- ps
- pt
- shn
- kac
- rm
- ro
- ru
- rw
- sc
- sd
- se
- shn
- si
- sk
- sl
Expand All @@ -116,19 +115,18 @@ language:
- tr
- ts
- tt
- zgh
- ug
- uk
- ur
- uz
- ve
- vi
- wo
- war
- wo
- xh
- yi
- yo
- zh
- zgh
- zh
- zu
- zza
Expand Down
2 changes: 1 addition & 1 deletion datasets/code_x_glue_tc_text_to_code/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ annotations_creators:
language_creators:
- found
language:
- en
- code
- en
license:
- c-uda
multilinguality:
Expand Down
4 changes: 2 additions & 2 deletions datasets/code_x_glue_tt_text_to_text/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ language_creators:
- found
language:
- da
- nb
- en
- lv
- nb
- zh
- en
license:
- c-uda
multilinguality:
Expand Down
11 changes: 8 additions & 3 deletions datasets/common_language/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ language:
- eu
- fa
- fr
- fy-NL
- fy
- ia
- id
- it
Expand All @@ -36,17 +36,22 @@ language:
- nl
- pl
- pt
- rm-sursilv
- rm
- ro
- ru
- rw
- sah
- sl
- sv-SE
- sv
- ta
- tr
- tt
- uk
- zh
language_bcp47:
- fy-NL
- rm-sursilv
- sv-SE
- zh-CN
- zh-HK
- zh-TW
Expand Down
19 changes: 13 additions & 6 deletions datasets/common_voice/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ language:
- fa
- fi
- fr
- fy-NL
- ga-IE
- fy
- ga
- hi
- hsb
- hu
Expand All @@ -44,24 +44,31 @@ language:
- mt
- nl
- or
- pa-IN
- pa
- pl
- pt
- rm-sursilv
- rm-vallader
- rm
- ro
- ru
- rw
- sah
- sl
- sv-SE
- sv
- ta
- th
- tr
- tt
- uk
- vi
- vot
- zh
language_bcp47:
- fy-NL
- ga-IE
- pa-IN
- rm-sursilv
- rm-vallader
- sv-SE
- zh-CN
- zh-HK
- zh-TW
Expand Down
2 changes: 2 additions & 0 deletions datasets/conv_questions/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ annotations_creators:
language_creators:
- crowdsourced
language:
- en
language_bcp47:
- en-US
license:
- cc-by-4.0
Expand Down
31 changes: 17 additions & 14 deletions datasets/covost2/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,30 @@ language_creators:
- crowdsourced
- expert-generated
language:
- fr
- ar
- ca
- cy
- de
- es
- ca
- it
- ru
- zh-CN
- pt
- fa
- et
- fa
- fr
- id
- it
- ja
- lv
- mn
- nl
- tr
- ar
- sv-SE
- lv
- pt
- ru
- sl
- sv
- ta
- ja
- id
- cy
- tr
- zh
language_bcp47:
- sv-SE
- zh-CN
license:
- cc-by-nc-4.0
multilinguality:
Expand Down
2 changes: 2 additions & 0 deletions datasets/hendrycks_test/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ annotations_creators:
language_creators:
- expert-generated
language:
- en
language_bcp47:
- en-US
license:
- mit
Expand Down
4 changes: 2 additions & 2 deletions datasets/ilist/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ multilinguality:
task_ids:
- text-classification-other-language-identification
language:
- hi
- awa
- bho
- mag
- bra
- hi
- mag
annotations_creators:
- unknown
source_datasets:
Expand Down
3 changes: 3 additions & 0 deletions datasets/kan_hope/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ annotations_creators:
language_creators:
- crowdsourced
language:
- en
- kn
language_bcp47:
- en-IN
- kn-IN
license:
Expand Down
Loading