Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLDR-18155 Order languageData's scripts by number of users #4237

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

CLDR-18155 Order languageData's scripts by number of users #4237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f14ddb5
CLDR-18155 Order languageData's scripts by number of users
conradarcturus Dec 3, 2024
f03fd8c
CLDR-18155 Recommended edits
conradarcturus Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions common/supplemental/likelySubtags.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,7 +451,6 @@ not be patched by hand, as any changes made in that fashion may be lost.
<likelySubtag from="mai" to="mai_Deva_IN"/> <!--Maithili‧?‧? ➡ Maithili‧Devanagari‧India-->
<likelySubtag from="mak" to="mak_Latn_ID"/> <!--Makasar‧?‧? ➡ Makasar‧Latin‧Indonesia-->
<likelySubtag from="man" to="man_Latn_GM"/> <!--Mandingo‧?‧? ➡ Mandingo‧Latin‧Gambia-->
<likelySubtag from="man_GN" to="man_Nkoo_GN"/> <!--Mandingo‧?‧Guinea ➡ Mandingo‧N’Ko‧Guinea-->
<likelySubtag from="man_Nkoo" to="man_Nkoo_GN"/> <!--Mandingo‧N’Ko‧? ➡ Mandingo‧N’Ko‧Guinea-->
<likelySubtag from="mas" to="mas_Latn_KE"/> <!--Masai‧?‧? ➡ Masai‧Latin‧Kenya-->
<likelySubtag from="maz" to="maz_Latn_MX"/> <!--Central Mazahua‧?‧? ➡ Central Mazahua‧Latin‧Mexico-->
Expand Down Expand Up @@ -1254,7 +1253,8 @@ not be patched by hand, as any changes made in that fashion may be lost.
<likelySubtag from="und_Narb" to="xna_Narb_SA"/> <!--?‧Old North Arabian‧? ➡ Ancient North Arabian‧Old North Arabian‧Saudi Arabia-->
<likelySubtag from="und_Nbat" to="arc_Nbat_JO"/> <!--?‧Nabataean‧? ➡ Aramaic‧Nabataean‧Jordan-->
<likelySubtag from="und_Newa" to="new_Newa_NP"/> <!--?‧Newa‧? ➡ Newari‧Newa‧Nepal-->
<likelySubtag from="und_Nkoo" to="man_Nkoo_GN"/> <!--?‧N’Ko‧? ➡ Mandingo‧N’Ko‧Guinea-->
<likelySubtag from="und_Nkoo" to="nqo_Nkoo_GN"/> <!--?‧N’Ko‧? ➡ N’Ko‧N’Ko‧Guinea-->
<likelySubtag from="und_Nkoo_GM" to="man_Nkoo_GM"/> <!--?‧N’Ko‧Gambia ➡ Mandingo‧N’Ko‧Gambia-->
<likelySubtag from="und_Nkoo_ML" to="bm_Nkoo_ML"/> <!--?‧N’Ko‧Mali ➡ Bambara‧N’Ko‧Mali-->
<likelySubtag from="und_Nshu" to="zhx_Nshu_CN"/> <!--?‧Nüshu‧? ➡ Chinese (family)‧Nüshu‧China-->
<likelySubtag from="und_Ogam" to="sga_Ogam_IE"/> <!--?‧Ogham‧? ➡ Old Irish‧Ogham‧Ireland-->
Expand Down
37 changes: 20 additions & 17 deletions common/supplemental/supplementalData.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1346,7 +1346,7 @@ XXX Code for transations where no currency is involved
<language type="awa" scripts="Deva"/>
<language type="awa" territories="IN" alt="secondary"/>
<language type="ay" scripts="Latn" territories="BO"/>
<language type="az" scripts="Arab Cyrl Latn" territories="AZ"/>
<language type="az" scripts="Arab Latn Cyrl" territories="AZ"/>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likely subtags for az is az_Latn_AZ but I will note there are more Azerbaijani speakers in Iran (az_Arab_IR). It's just that they haven't transitioned to the internet like Azerbaijan has.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While more speakers, it appears that there is more written in Latin (AZ) that Arabic (IR). So probably better Latn Arab Cyrl.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This language points out a good flaw in population estimates alone -- and it depends on the usage.

Do people want likely subtags based on the largest corpora, for active internet users, for official government mandates, for any potential user (regardless of online status), for spoken content, for local content v global content (eg. AE's largest internet usage is en_AE not ar_AE).

An interesting idea (for LikelySubtags) could be to add alt="official" and potentially other alt tags when it's different. But yes, let's just stew on this thought and return to it if/when we have a good evidence it would be useful downstream.

Copy link
Contributor

@roozbehp roozbehp Dec 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strong agreement with what Mark said: despite spoken Azerbaijani being quite common in Iran with more speakers than in Azerbaijan, literacy in the writing system is pretty low in Iran, due to it not being formally taught in schools. There are books and magazines published in az-Arab in Iran, but most Iranian Azerbaijani speakers have difficulty reading them, even if they are literate in Persian (which is is theoretically a close writing system). So if anyone runs into written Azerbaijani in the wild these days, there's a much higher chance it's in Latin than in Arabic.

I also believe there's much more spoken "content" (read "media") produced in the language in Azerbaijan than in Iran (there's limited Azerbaijani radio and TV stations in Iran, and I believe less podcasts and songs produced in Iran). But either way, I don't think we want to tag such content as az-Arab or az-Latn. That's just az (or azb/azj if you insist on using ISO 639-3), and not az-Latn or az-Arab.

<language type="az" territories="IQ IR RU" alt="secondary"/>
<language type="ba" scripts="Cyrl"/>
<language type="ba" territories="RU" alt="secondary"/>
Expand Down Expand Up @@ -1424,7 +1424,7 @@ XXX Code for transations where no currency is involved
<language type="brh" scripts="Latn" territories="PK" alt="secondary"/>
<language type="brx" scripts="Deva"/>
<language type="brx" territories="IN" alt="secondary"/>
<language type="bs" scripts="Cyrl Latn" territories="BA"/>
<language type="bs" scripts="Latn Cyrl" territories="BA"/>
<language type="bsc" scripts="Latn"/>
<language type="bsc" territories="SN" alt="secondary"/>
<language type="bss" scripts="Latn"/>
Expand All @@ -1448,7 +1448,7 @@ XXX Code for transations where no currency is involved
<language type="car" scripts="Latn"/>
<language type="cay" scripts="Latn"/>
<language type="cch" scripts="Latn"/>
<language type="ccp" scripts="Beng Cakm"/>
<language type="ccp" scripts="Cakm Beng"/>
<language type="ccr" scripts="Latn"/>
<language type="ce" scripts="Cyrl"/>
<language type="ce" territories="RU" alt="secondary"/>
Expand Down Expand Up @@ -1477,7 +1477,7 @@ XXX Code for transations where no currency is involved
<language type="ckt" scripts="Cyrl"/>
<language type="clc" scripts="Latn"/>
<language type="co" scripts="Latn"/>
<language type="cop" scripts="Arab Copt Grek" territories="EG" alt="secondary"/>
<language type="cop" scripts="Copt Arab Grek" territories="EG" alt="secondary"/>
<language type="cps" scripts="Latn"/>
<language type="cr" scripts="Cans Latn"/>
<language type="crg" scripts="Latn"/>
Expand Down Expand Up @@ -1653,7 +1653,7 @@ XXX Code for transations where no currency is involved
<language type="gvr" scripts="Deva"/>
<language type="gwi" scripts="Latn"/>
<language type="gwi" territories="CA" alt="secondary"/>
<language type="ha" scripts="Arab Latn"/>
<language type="ha" scripts="Latn Arab"/>
<language type="ha" territories="NE NG" alt="secondary"/>
<language type="hai" scripts="Latn"/>
<language type="hak" scripts="Hans Hant" territories="TW"/>
Expand Down Expand Up @@ -1784,7 +1784,7 @@ XXX Code for transations where no currency is involved
<language type="kjg" scripts="Laoo"/>
<language type="kjg" scripts="Latn" alt="secondary"/>
<language type="kjh" scripts="Cyrl"/>
<language type="kk" scripts="Arab Cyrl" territories="KZ"/>
<language type="kk" scripts="Cyrl Arab" territories="KZ"/>
<language type="kk" territories="CN" alt="secondary"/>
<language type="kkj" scripts="Latn"/>
<language type="kl" scripts="Latn" territories="GL"/>
Expand Down Expand Up @@ -1824,7 +1824,7 @@ XXX Code for transations where no currency is involved
<language type="ksb" territories="TZ" alt="secondary"/>
<language type="ksf" scripts="Latn"/>
<language type="ksh" scripts="Latn"/>
<language type="ku" scripts="Arab Cyrl Latn"/>
<language type="ku" scripts="Latn Arab Cyrl"/>
<language type="ku" territories="SY TR" alt="secondary"/>
<language type="kum" scripts="Cyrl"/>
<language type="kum" territories="RU" alt="secondary"/>
Expand All @@ -1840,7 +1840,7 @@ XXX Code for transations where no currency is involved
<language type="kxp" scripts="Arab"/>
<language type="kxv" scripts="Latn"/>
<language type="kxv" scripts="Deva Orya Telu" alt="secondary"/>
<language type="ky" scripts="Arab Cyrl Latn" territories="KG"/>
<language type="ky" scripts="Cyrl Arab Latn" territories="KG"/>
<language type="kyu" scripts="Kali"/>
<language type="la" scripts="Latn"/>
<language type="la" territories="VA" alt="secondary"/>
Expand Down Expand Up @@ -1973,7 +1973,7 @@ XXX Code for transations where no currency is involved
<language type="mrj" scripts="Cyrl"/>
<language type="mro" scripts="Latn"/>
<language type="mro" scripts="Mroo" alt="secondary"/>
<language type="ms" scripts="Arab Latn" territories="BN MY SG"/>
<language type="ms" scripts="Latn Arab" territories="BN MY SG"/>
<language type="ms" territories="CC ID" alt="secondary"/>
<language type="mt" scripts="Latn" territories="MT"/>
<language type="mtr" scripts="Deva"/>
Expand Down Expand Up @@ -2200,7 +2200,7 @@ XXX Code for transations where no currency is involved
<language type="sg" scripts="Latn" territories="CF"/>
<language type="sga" scripts="Latn Ogam" alt="secondary"/>
<language type="sgs" scripts="Latn"/>
<language type="shi" scripts="Arab Latn Tfng"/>
<language type="shi" scripts="Tfng Latn Arab"/>
<language type="shi" territories="MA" alt="secondary"/>
<language type="shn" scripts="Mymr"/>
<language type="shn" territories="MM" alt="secondary"/>
Expand Down Expand Up @@ -2289,7 +2289,7 @@ XXX Code for transations where no currency is involved
<language type="teo" territories="UG" alt="secondary"/>
<language type="ter" scripts="Latn"/>
<language type="tet" scripts="Latn" territories="TL"/>
<language type="tg" scripts="Arab Cyrl Latn" territories="TJ"/>
<language type="tg" scripts="Cyrl Arab Latn" territories="TJ"/>
<language type="th" scripts="Thai" territories="TH"/>
<language type="thl" scripts="Deva"/>
<language type="thq" scripts="Deva"/>
Expand All @@ -2300,7 +2300,7 @@ XXX Code for transations where no currency is involved
<language type="tig" territories="ER" alt="secondary"/>
<language type="tiv" scripts="Latn"/>
<language type="tiv" territories="NG" alt="secondary"/>
<language type="tk" scripts="Arab Cyrl Latn" territories="TM"/>
<language type="tk" scripts="Latn Arab Cyrl" territories="TM"/>
<language type="tk" territories="AF IR" alt="secondary"/>
<language type="tkl" scripts="Latn" territories="TK"/>
<language type="tkr" scripts="Latn"/>
Expand Down Expand Up @@ -2365,9 +2365,9 @@ XXX Code for transations where no currency is involved
<language type="unx" scripts="Beng Deva"/>
<language type="ur" scripts="Arab" territories="PK"/>
<language type="ur" territories="GB IN" alt="secondary"/>
<language type="uz" scripts="Arab Cyrl Latn" territories="UZ"/>
<language type="uz" scripts="Latn Cyrl Arab" territories="UZ"/>
<language type="uz" territories="AF" alt="secondary"/>
<language type="vai" scripts="Latn Vaii"/>
<language type="vai" scripts="Vaii Latn"/>
<language type="ve" scripts="Latn"/>
<language type="ve" territories="ZA" alt="secondary"/>
<language type="vec" scripts="Latn"/>
Expand Down Expand Up @@ -2435,7 +2435,7 @@ XXX Code for transations where no currency is involved
<language type="yrk" scripts="Cyrl"/>
<language type="yrl" scripts="Latn"/>
<language type="yua" scripts="Latn"/>
<language type="yue" scripts="Hans Hant" territories="MO"/>
<language type="yue" scripts="Hant Hans" territories="MO"/>
conradarcturus marked this conversation as resolved.
Show resolved Hide resolved
<language type="yue" territories="CN HK" alt="secondary"/>
<language type="za" scripts="Latn"/>
<language type="za" scripts="Hans" territories="CN" alt="secondary"/>
Expand Down Expand Up @@ -3140,13 +3140,15 @@ XXX Code for transations where no currency is involved
<territory type="GM" gdp="7905000000" literacyPercent="51.1" population="2523330"> <!--Gambia-->
<languagePopulation type="en" populationPercent="40" officialStatus="official"/> <!--English-->
<languagePopulation type="man" populationPercent="29"/> <!--Mandingo-->
<languagePopulation type="man_Nkoo" writingPercent="5" populationPercent="29" references="R1021"/> <!--Mandingo (N’Ko)-->
<languagePopulation type="ff" populationPercent="0"/> <!--Fula-->
<languagePopulation type="ff_Adlm" populationPercent="0"/> <!--Fula (Adlam)-->
</territory>
<territory type="GN" gdp="56660000000" literacyPercent="41" population="13986200"> <!--Guinea-->
<languagePopulation type="fr" populationPercent="27" officialStatus="official" references="R1352"/> <!--French-->
<languagePopulation type="ff" populationPercent="26"/> <!--Fula-->
<languagePopulation type="man_Nkoo" populationPercent="23"/> <!--Mandingo (N’Ko)-->
<languagePopulation type="man" populationPercent="23"/> <!--Mandingo-->
<languagePopulation type="man_Nkoo" writingPercent="5" populationPercent="23" references="R1021"/> <!--Mandingo (N’Ko)-->
<languagePopulation type="sus" populationPercent="11"/> <!--Susu-->
<languagePopulation type="nqo" populationPercent="5" references="R1286"/> <!--N’Ko-->
<languagePopulation type="kpe" populationPercent="3.8"/> <!--Kpelle-->
Expand Down Expand Up @@ -3591,7 +3593,7 @@ XXX Code for transations where no currency is involved
<languagePopulation type="en" populationPercent="14" references="R1050"/> <!--English-->
<languagePopulation type="tzm" literacyPercent="25" populationPercent="9.8" officialStatus="official"/> <!--Central Atlas Tamazight-->
<languagePopulation type="shi" populationPercent="8.7"/> <!--Tachelhit-->
<languagePopulation type="shi_Latn" populationPercent="8.7" references="R1181"/> <!--Tachelhit (Latin)-->
<languagePopulation type="shi_Latn" writingPercent="5" populationPercent="8.7" references="R1181"/> <!--Tachelhit (Latin)-->
<languagePopulation type="rif" writingPercent="5" populationPercent="4.9"/> <!--Riffian-->
<languagePopulation type="rif_Tfng" writingPercent="5" populationPercent="4.9"/> <!--Riffian (Tifinagh)-->
<languagePopulation type="es" populationPercent="0.065"/> <!--Spanish-->
Expand Down Expand Up @@ -5552,6 +5554,7 @@ XXX Code for transations where no currency is involved
<reference type="R1018" uri="https://www.cia.gov/cia/publications/factbook/geos/gj.html">English official; the figure is derived from literacy * lang pop</reference>
<reference type="R1019" uri="https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810021601">Canada 2021 Census language &quot;Knowledge of Language&quot;; official status from Wikipedia Languages_of_Canada</reference>
<reference type="R1020" uri="http://en.wikipedia.org/wiki/Ascension_Island">[missing]</reference>
<reference type="R1021">Actually literacy in Nko writing unknown but historically they used the Latin script</reference>
<reference type="R1022" uri="https://www.cia.gov/cia/publications/factbook/geos/gi.html">English official, the figure is derived from literacy * lang pop</reference>
<reference type="R1024" uri="http://www.ethnologue.com/show_language.asp?code=ltz">Some 99% of users are literate in French or German. For languages not customarily written, the writing population is artificially set to 5% in the absence of better information.</reference>
<reference type="R1025" uri="http://www.ethnologue.com/language/ewo">2nd lang literacy 15-25%</reference>
Expand Down
6 changes: 3 additions & 3 deletions common/testData/localeIdentifiers/likelySubtags.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1575,9 +1575,9 @@ und-NP ; ne-Deva-NP ; ne ;
und-NR ; en-Latn-NR ; en-NR ;
und-NU ; en-Latn-NU ; en-NU ;
und-NZ ; en-Latn-NZ ; en-NZ ;
und-Nkoo ; man-Nkoo-GN ; man-Nkoo ; man-GN
und-Nkoo-AQ ; man-Nkoo-AQ ; ;
und-Nkoo-GN ; man-Nkoo-GN ; man-Nkoo ; man-GN
und-Nkoo ; nqo-Nkoo-GN ; nqo ;
und-Nkoo-AQ ; nqo-Nkoo-AQ ; nqo-AQ ;
und-Nkoo-GN ; nqo-Nkoo-GN ; nqo ;
und-OM ; ar-Arab-OM ; ar-OM ;
und-Olck ; sat-Olck-IN ; sat ;
und-Olck-AQ ; sat-Olck-AQ ; sat-AQ ;
Expand Down
Loading
Loading