ICU-21592 Update cj normal/loose linebreak per CSS

[pedberg-icu]

Checklist

• Required: Issue filed: https://unicode-org.atlassian.net/browse/ICU-21592
• Required: The PR title must be prefixed with a JIRA Issue number.
• Required: The PR description must include the link to the Jira Issue, for example by completing the URL in the first checklist item
• Required: Each commit message must be prefixed with a JIRA Issue number.
• Issue accepted (done by Technical Committee after discussion)
• Tests included, if applicable
• API docs and/or User Guide docs changed or added, if applicable

This updates the CJ normal & loose linebreak tailorings per a recent change agreed on for CSS:

• For normal, CSS no longer allows breaks before \u2010 HYPHEN or \u2013 EN DASH, i.e. it makes the behavior for these back to what standard linebreak does. These are normally in linebreak class BA but the cj normal tailoring has split them off into a separate class BAX to allow breaks; this PR retracts that.
• For loose, CSS allows breaks before \u2010 HYPHEN or \u2013 EN DASH only after characters of class ID; for other characters it no longer does. These characters are still split into a separate class BAX, but the first part of rule LB 21 that formerly allowed a break before BAX now needs to be split into two rules, one for non-ID and one for ID. (BAX also needed to ba added to $CanFollowIS in rule LB 14a).

The separate rules for RBBIMonkeyTest in icu4c/source/test/testdata/break_rules/ and icu4j/main/tests/core/src/com/ibm/icu/dev/test/rbbi/break_rules/ needed to be updated accordingly. However they also needed one additional update for line_loose_cj.txt: They needed the following addition corresponding to something that has been in the standard rules for a while, but whose absence in the RBBIMonkeyTest did not previously cause a problem:

LB21.3:      CM+ [BA HY NS];

While the CI tests are running I will in parallel run an extended-duration version of RBBIMonkeyTest locally to check for any hard-to-find problems, and note the result below.

[jira-pull-request-webhook]

Notice: the branch changed across the force-push!

• icu4c/source/test/testdata/break_rules/line_loose_cj.txt is different
• icu4c/source/test/testdata/rbbitst.txt is different
• icu4j/main/shared/data/icudata.jar is different
• icu4j/main/shared/data/icutzdata.jar is different
• icu4j/main/shared/data/testdata.jar is different
• icu4j/main/tests/core/src/com/ibm/icu/dev/test/rbbi/break_rules/line_loose_cj.txt is different
• icu4j/main/tests/core/src/com/ibm/icu/dev/test/rbbi/rbbitst.txt is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

[jira-pull-request-webhook]

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

[jira-pull-request-webhook]

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

[pedberg-icu]

/azp run CI-Exhaustive

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[pedberg-icu]

I ran the ICU4C testMonkey for 400000 iterations (compared to 100 for a normal quick test), which took about 45 min; no problems found.

[FrankYFTang]

@allensu05 please also take a look.

[aheninger]

The changes to the main rules look good.

I think the LB21 changes in the line_loose_cj.txt monkey test rules can be simplified, but they're probably good enough for the moment, since you're under time pressure.

I'll play with them, and make a followup PR if I come up with something better.

[aheninger]

I don't think the long negated set is necessary; the hard breaks should never reach this point, having been handled by earlier rules. Monkey test rules are handled sequentially, unlike the main production rules, which are run in parallel.

[pedberg-icu]

Thanks Andy for looking at this and for approving!