ICU-22325 Integrate CLDR release-44-m1 to ICU #2530

DraganBesevic · 2023-07-20T23:34:01Z

Checklist

Required: Issue filed: https://unicode-org.atlassian.net/browse/ICU-22325
Required: The PR title must be prefixed with a JIRA Issue number.
Required: The PR description must include the link to the Jira Issue, for example by completing the URL in the first checklist item
Required: Each commit message must be prefixed with a JIRA Issue number.
Issue accepted (done by Technical Committee after discussion)
Tests included, if applicable
API docs and/or User Guide docs changed or added, if applicable

Integrates CLDR release-44-m1 plus the fix for ko_CN in unicode-org/cldr#3107 and the fix for personName test data in unicode-org/cldr#3122.
Some tests results are temporarily changed because CLDR reverted some data items to provisional at the beginning of the submission cycle (these will be fixed with the alpha0 integration).

Fixed pre-existing bug in RBBITestMonkey and logged a known issue for problem with testing limits on Hebrew calendar ICU-22441

Also needed to add personName test data from CLDR per ant copy-cldr-testdata in tools/cldr. Had to add a copyright header for those files in CLDR (currently a known issue with most of them saying they are for unit tests, will fix in CLDR). This also entailed changing the tools/cldr/build.xml code to add a copyright header for the generated catalog.txt file.

pedberg-icu · 2023-07-20T23:37:24Z

/azp run CI-Exhaustive

azure-pipelines · 2023-07-20T23:37:31Z

Azure Pipelines successfully started running 1 pipeline(s).

richgillam

Too big to review, and looks to be all mechanically generated anyway. Rubber-stamping.

markusicu · 2023-07-21T17:42:47Z

Too big to review, and looks to be all mechanically generated anyway. Rubber-stamping.

Suggestion for future integration pull requests:

When creating the pull request, split it into multiple commits so that we can review them separately. For example, split all of the .res files (or all of the binary files including .jar files) into a separate commit.
In the end, you will of course need to squash, but then we can just rubber-stamp confidently.

markusicu · 2023-07-21T17:56:42Z

What is “unit of pressure: gasoline-equivalent”?
I know of mpge but that's a energy consumption unit, not a pressure unit.

markusicu

rslgtm2 after spot check
but that new unit seems fishy

pedberg-icu · 2023-07-21T22:47:50Z

What is “unit of pressure: gasoline-equivalent”?
I know of mpge but that's a energy consumption unit, not a pressure unit.

@macchiati is the best person to answer that. It may be a pressure dimensionally even though it is not used for pressure measurements.

macchiati · 2023-07-22T04:59:08Z

I think you found a problem, Markus! It is ~33.705 kilowatt-hours, so the base unit should be watt-seconds. Since watt is "*kilogram-square-meter-per-cubic-second*" it should be kilogram-square-meter-per-*square*-second which is joules, not pascals.

…

On Fri, Jul 21, 2023, 15:48 Peter Edberg ***@***.***> wrote: What is “unit of pressure: gasoline-equivalent”? I know of mpge <https://en.wikipedia.org/wiki/Miles_per_gallon_gasoline_equivalent> but that's a energy consumption unit, not a pressure unit. @macchiati <https://github.com/macchiati> is the best person to answer that. It may be a pressure dimensionally even though it is not used for pressure measurements. — Reply to this email directly, view it on GitHub <#2530 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMA5JE5FWPREAWX3U3TXRMBKBANCNFSM6AAAAAA2SDF6U4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

jira-pull-request-webhook · 2023-07-26T20:09:22Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

DraganBesevic · 2023-07-26T20:15:11Z

/azp run CI-Exhaustive

azure-pipelines · 2023-07-26T20:15:17Z

Commenter does not have sufficient privileges for PR 2530 in repo unicode-org/icu

DraganBesevic · 2023-07-26T20:17:13Z

/azp run CI-Exhaustive

azure-pipelines · 2023-07-26T20:17:20Z

Azure Pipelines successfully started running 1 pipeline(s).

pedberg-icu · 2023-07-26T20:49:22Z

I think you found a problem, Markus! It is ~33.705 kilowatt-hours, so the base unit should be watt-seconds. Since watt is "kilogram-square-meter-per-cubic-second" it should be kilogram-square-meter-per-square-second which is joules, not pascals.

@macchiati Can you file a CLDR ticket to fix gasoline-equivalent on the CLDR side? That should be fixed with a future CLDR-ICU integration for the current integration we will use the existing category.

macchiati · 2023-07-26T21:03:51Z

Will do.

macchiati · 2023-07-26T23:32:09Z

Hi, https://unicode-org.atlassian.net/browse/CLDR-15296?focusedCommentId=170939 is not yet closed, so will make a PR for that right away.

…

On Wed, Jul 26, 2023 at 1:49 PM Peter Edberg ***@***.***> wrote: I think you found a problem, Markus! It is ~33.705 kilowatt-hours, so the base unit should be watt-seconds. Since watt is " *kilogram-square-meter-per-cubic-second*" it should be kilogram-square-meter-per-*square*-second which is joules, not pascals. @macchiati <https://github.com/macchiati> Can you file a CLDR ticket to fix gasoline-equivalent on the CLDR side? That should be fixed with a future CLDR-ICU integration for the current integration we will use the existing category. — Reply to this email directly, view it on GitHub <#2530 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMHG4KTN4OHS3UFJ2IDXSF7FZANCNFSM6AAAAAA2SDF6U4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

macchiati · 2023-07-27T01:23:06Z

After investigating and writing some tests, I realized that the quantity was correct (because it simply represents the most common usage of the base unit). I also documented why the unit seems so odd. description="Constructed so that 1 gallon-gasoline-equivalent = 33.705 kWh as per https://www3.epa.gov/otaq/gvg/learn-more-technology.htm" unicode-org/cldr#3121

…

On Wed, Jul 26, 2023 at 4:31 PM Mark Davis Ⓤ ***@***.***> wrote: Hi, https://unicode-org.atlassian.net/browse/CLDR-15296?focusedCommentId=170939 is not yet closed, so will make a PR for that right away. On Wed, Jul 26, 2023 at 1:49 PM Peter Edberg ***@***.***> wrote: > I think you found a problem, Markus! It is ~33.705 kilowatt-hours, so the > base unit should be watt-seconds. Since watt is " > *kilogram-square-meter-per-cubic-second*" it should be > kilogram-square-meter-per-*square*-second which is joules, not pascals. > > @macchiati <https://github.com/macchiati> Can you file a CLDR ticket to > fix gasoline-equivalent on the CLDR side? That should be fixed with a > future CLDR-ICU integration for the current integration we will use the > existing category. > > — > Reply to this email directly, view it on GitHub > <#2530 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACJLEMHG4KTN4OHS3UFJ2IDXSF7FZANCNFSM6AAAAAA2SDF6U4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

jira-pull-request-webhook · 2023-07-27T15:50:42Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2023-07-27T17:22:35Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

pedberg-icu · 2023-07-27T17:27:40Z

/azp run CI-Exhaustive

azure-pipelines · 2023-07-27T17:27:47Z

Azure Pipelines successfully started running 1 pipeline(s).

jira-pull-request-webhook · 2023-07-27T22:22:47Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

pedberg-icu · 2023-07-27T22:26:35Z

/azp run CI-Exhaustive

azure-pipelines · 2023-07-27T22:26:43Z

Azure Pipelines successfully started running 1 pipeline(s).

macchiati

I spot-checked the data files. Markus and Rich would be better suited to check the rest.

richgillam

Either I forgot to actually rubber-stamp this the first time around, or my stamp got removed by activity in here in the meantime. Rubber-stamping again.

…BITestMonkey

jira-pull-request-webhook · 2023-07-28T18:35:58Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

pedberg-icu · 2023-07-28T18:44:56Z

/azp run CI-Exhaustive

azure-pipelines · 2023-07-28T18:45:04Z

Azure Pipelines successfully started running 1 pipeline(s).

pedberg-icu · 2023-07-28T18:46:17Z

Either I forgot to actually rubber-stamp this the first time around, or my stamp got removed by activity in here in the meantime. Rubber-stamping again.

@richgillam Well I needed to update this again to add a copyright for the personName testdata files (currently most of them say they are for unit tests, known issue that I will fix in CLDR). But this will need another look, sorry.

richgillam

Still looks good. Huge thanks (and deep apologies) for the PersonName changes!

richgillam · 2023-07-28T19:44:12Z

icu4j/main/tests/core/src/com/ibm/icu/dev/test/format/ExhaustivePersonNameFormatterTest.java

+                String[] FILENAMES_TO_SKIP = {"gaa.txt", "dsb.txt", "syr.txt", "hsb.txt", "lij.txt",
+                      "yue_Hans.txt", "fa.txt", "ja.txt", "ka.txt", "zh_Hant_HK.txt", "zh_Hant.txt",
+                      "bn.txt", "zh.txt", "nl.txt", "to.txt", "uk.txt", "my.txt", "yue.txt",
+                      "bg.txt", "tk.txt", "ps.txt", "ko.txt", "kk.txt", "ms.txt"};


Thanks; I'll try to fix this after I get the algorithms back in sync and Mark regenerates the test data files (or I figure out how to do it).

richgillam · 2023-07-28T19:45:29Z

icu4j/main/tests/core/src/com/ibm/icu/dev/test/format/PersonNameFormatterTest.java

+//                { "nl_NL", "LONG",   "MONOGRAM",   "FORMAL",   "DEFAULT", "", "WvP" },   Temporary change the test because data was set to provisional
+//                { "nl_NL", "LONG",   "MONOGRAM",   "INFORMAL", "DEFAULT", "", "WvP" },
+                { "nl_NL", "LONG",   "MONOGRAM",   "FORMAL",   "DEFAULT", "", "WV" },
+                { "nl_NL", "LONG",   "MONOGRAM",   "INFORMAL", "DEFAULT", "", "WV" },


Sounds like I have some cleanup and sync to do here too...

NO, this is because we reset a lot of personName data to provisional before the beginning of data collection for CLDR 44; provisional data dopes not get picked up for ICU. Once we have the CLDR 44 Survey Tool data in xml and redo the integration, these problems should go away.

richgillam · 2023-07-28T19:45:41Z

icu4j/main/tests/core/src/com/ibm/icu/dev/test/format/PersonNameFormatterTest.java

@@ -556,7 +558,8 @@ public void TestLocaleDerivation() {
                {"en", "MEDIUM", "REFERRING", "FORMAL",   "DEFAULT", "", "陳港生Test"},
            }),
            new NameAndTestCases("given=港生,surname=陳,given2=Test,locale=zh_Hant", new String[][]{
-                {"en", "MEDIUM", "REFERRING", "FORMAL",   "DEFAULT", "", "陳港生T."},
+//                {"en", "MEDIUM", "REFERRING", "FORMAL",   "DEFAULT", "", "陳港生T."},  Temporary change the test because data was set to provisional
+                {"en", "MEDIUM", "REFERRING", "FORMAL",   "DEFAULT", "", "陳港生Test"},


And here...

pedberg-icu · 2023-07-28T23:39:53Z

Still looks good. Huge thanks (and deep apologies) for the PersonName changes!

@richgillam No problem. But can you re-approve? All of the updates to this PR cancelled the previous approvals.

richgillam

Approving again.

richgillam · 2023-07-28T23:42:06Z

Still looks good. Huge thanks (and deep apologies) for the PersonName changes!

@richgillam No problem. But can you re-approve? All of the updates to this PR cancelled the previous approvals.

Done. Thank you!

Was my response to Frank in https://unicode-org.atlassian.net/browse/ICU-22304 accurate? If not, would you mind commenting over there? If so, I'll close it again after this PR lands.

pedberg-icu · 2023-07-28T23:43:52Z

@richgillam @markusicu @macchiati We have had some updates to this PR to get exhaustive tests to pass. Can one of you re-approve? Thanks!

richgillam · 2023-07-28T23:52:56Z

@richgillam @markusicu @macchiati We have had some updates to this PR to get exhaustive tests to pass. Can one of you re-approve? Thanks!

I did re-approve. I think you're okay (at least it looks like it on my side...).

pedberg-icu · 2023-07-28T23:56:00Z

Still looks good. Huge thanks (and deep apologies) for the PersonName changes!

@richgillam No problem. But can you re-approve? All of the updates to this PR cancelled the previous approvals.

Done. Thank you!

Was my response to Frank in https://unicode-org.atlassian.net/browse/ICU-22304 accurate? If not, would you mind commenting over there? If so, I'll close it again after this PR lands.

Yes, that is accurate...

pedberg-icu requested review from macchiati, markusicu and richgillam July 20, 2023 23:35

pedberg-icu assigned macchiati Jul 20, 2023

richgillam previously approved these changes Jul 21, 2023

View reviewed changes

pedberg-icu changed the title ~~ICU-22325 Integrate CLDR 44.1 to ICU~~ ICU-22325 Integrate CLDR release-44-m1 to ICU Jul 21, 2023

markusicu previously approved these changes Jul 21, 2023

View reviewed changes

DraganBesevic dismissed stale reviews from markusicu and richgillam via 3c86935 July 26, 2023 20:04

DraganBesevic force-pushed the ICU-22325-brs74rc-integrate-cldr-44-m1 branch from 3c86935 to ab86739 Compare July 26, 2023 20:09

DraganBesevic force-pushed the ICU-22325-brs74rc-integrate-cldr-44-m1 branch from dfb4033 to 123e0de Compare July 27, 2023 15:50

pedberg-icu force-pushed the ICU-22325-brs74rc-integrate-cldr-44-m1 branch from 123e0de to 340ce7c Compare July 27, 2023 17:22

pedberg-icu force-pushed the ICU-22325-brs74rc-integrate-cldr-44-m1 branch from 55bdec8 to d23d741 Compare July 27, 2023 22:22

macchiati reviewed Jul 28, 2023

View reviewed changes

richgillam previously approved these changes Jul 28, 2023

View reviewed changes

pedberg-icu dismissed richgillam’s stale review via 7c4af1f July 28, 2023 18:33

ICU-22325 Integrate CLDR 44.1 to ICU, add personName testdata, fix RB…

9500fa4

…BITestMonkey

pedberg-icu force-pushed the ICU-22325-brs74rc-integrate-cldr-44-m1 branch from 7c4af1f to 9500fa4 Compare July 28, 2023 18:35

pedberg-icu requested review from macchiati, richgillam and markusicu July 28, 2023 18:46

pedberg-icu mentioned this pull request Jul 28, 2023

CLDR-16936 For personNameTest files: Add copyright headers, _readme, _header unicode-org/cldr#3122

Merged

1 task

richgillam reviewed Jul 28, 2023

View reviewed changes

richgillam approved these changes Jul 28, 2023

View reviewed changes

pedberg-icu merged commit 1f07d2b into unicode-org:main Jul 28, 2023
104 checks passed

pedberg-icu deleted the ICU-22325-brs74rc-integrate-cldr-44-m1 branch July 28, 2023 23:53

markusicu mentioned this pull request Sep 11, 2023

ICU-22325 Update ICU4C API Change Reports - frontload #2589

Merged

7 tasks

ICU-22325 Integrate CLDR release-44-m1 to ICU #2530

ICU-22325 Integrate CLDR release-44-m1 to ICU #2530

Conversation

DraganBesevic commented Jul 20, 2023 • edited by jira bot Loading

Checklist

pedberg-icu commented Jul 20, 2023

azure-pipelines bot commented Jul 20, 2023

richgillam left a comment

Choose a reason for hiding this comment

markusicu commented Jul 21, 2023

markusicu commented Jul 21, 2023

markusicu left a comment

Choose a reason for hiding this comment

pedberg-icu commented Jul 21, 2023

macchiati commented Jul 22, 2023 via email

jira-pull-request-webhook bot commented Jul 26, 2023

DraganBesevic commented Jul 26, 2023

azure-pipelines bot commented Jul 26, 2023

DraganBesevic commented Jul 26, 2023

azure-pipelines bot commented Jul 26, 2023

pedberg-icu commented Jul 26, 2023

macchiati commented Jul 26, 2023

macchiati commented Jul 26, 2023 via email

macchiati commented Jul 27, 2023 via email

jira-pull-request-webhook bot commented Jul 27, 2023

jira-pull-request-webhook bot commented Jul 27, 2023

pedberg-icu commented Jul 27, 2023

azure-pipelines bot commented Jul 27, 2023

jira-pull-request-webhook bot commented Jul 27, 2023

pedberg-icu commented Jul 27, 2023

azure-pipelines bot commented Jul 27, 2023

macchiati left a comment

Choose a reason for hiding this comment

richgillam left a comment

Choose a reason for hiding this comment

jira-pull-request-webhook bot commented Jul 28, 2023

pedberg-icu commented Jul 28, 2023

azure-pipelines bot commented Jul 28, 2023

pedberg-icu commented Jul 28, 2023

richgillam left a comment

Choose a reason for hiding this comment

richgillam Jul 28, 2023

Choose a reason for hiding this comment

richgillam Jul 28, 2023

Choose a reason for hiding this comment

pedberg-icu Jul 28, 2023

Choose a reason for hiding this comment

richgillam Jul 28, 2023

Choose a reason for hiding this comment

richgillam Jul 28, 2023

Choose a reason for hiding this comment

pedberg-icu commented Jul 28, 2023

richgillam left a comment

Choose a reason for hiding this comment

richgillam commented Jul 28, 2023

pedberg-icu commented Jul 28, 2023

richgillam commented Jul 28, 2023

pedberg-icu commented Jul 28, 2023

DraganBesevic commented Jul 20, 2023 •

edited by jira bot

Loading