Add a collator implementation under experimental/ #1706

hsivonen · 2022-03-17T14:14:56Z

I'd like to land the current collator work in progress under experimental/, so that it would participate in provider refactorings without having to catch up on a branch.

What works

&str (guaranteed-valid UTF-8), &[u8] (potentially-invalid UTF-8), and &[u16]` (potentially-invalid UTF-16) comparison.
Passes CollationTest_CLDR_NON_IGNORABLE.txt and CollationTest_CLDR_SHIFTED.txt
- Tests with unpaired surrogates are skipped, because this implementation treats UTF errors as U+FFFD instead of trying to treat errors as having collation semantics.
Loading collation tailorings and default settings (excluding reordering) by locale without alias resolution.
Passes a bunch of language-specific tests ported from ICU4C.
Discovering the available locales and variants by listing the TOML files in a given input directory.
Backward second level (Canadian French)
Identical (fifth) level.
Script reordering.
Loading a collation by variant.
Lithuanian diacritics.

What doesn't work

Byte-wise skipping over an identical prefix. (This is an optimization that's out of scope for now.)
Jamo in search collations. (When designing this, I mistakenly assumed that the jamo CE32s would be self-contained. They aren't in search tailorings. Instead, they refer to expansion data. Since the Latin mini expansion bit space is unused, it could probably be repurposed for archaic jamo mini expansions.)
Provider capabilities
- Resolving locales that are aliases. (This includes lack of support for even resolving en as und.)
- Falling back to less specific locale (e.g. from fi-FI to fi, from fi-u-co-bogus to fi, from sv-u-co-bogus to equivalent of sv-u-co-reformed, from zh-u-co-bogus to equilavent of zh-u-co-pinyin.)

Post-landing help would be particularly helpful on

Consolidating the near-identical providers.
Deciding whether script reordering should be overridable via the API or only implied by the locale. (Per meeting: No.)
Deciding how variants are requested, whether "search" usage should be a separate flag, and, if so, what to do with searchlj. (Per meeting: Via -u-co- in the Locale.)
Designing the alias handling mechanism. It needs to be able to expand region/script to variant. Most notably, zh-HK, zh-MO, zh-TW, zh-Hant, zh-Hant-HK, zh-Hant-MO, zh-Hant-TW to zh with variant stroke.

(I've edited this text to reflect the code updates.)

provider/testdata/Cargo.toml

jira-pull-request-webhook · 2022-03-31T11:26:42Z

Notice: the branch changed across the force-push!

Cargo.lock is different
Cargo.toml is different
components/properties/src/lib.rs is different
experimental/char16trie/src/char16trie.rs is different
experimental/collator/Cargo.toml is different
experimental/collator/src/lib.rs is different
experimental/normalizer/Cargo.toml is different
experimental/normalizer/src/lib.rs is different
provider/datagen/Cargo.toml is now changed in the branch
provider/datagen/src/bin/datagen.rs is now changed in the branch
provider/datagen/src/lib.rs is now changed in the branch
provider/datagen/src/uprops/canonical_decompositions.rs is now changed in the branch
provider/datagen/src/uprops/codepointtrie.rs is now changed in the branch
provider/datagen/src/uprops/collation_data.rs is now changed in the branch
provider/datagen/src/uprops/collation_diacritics.rs is now changed in the branch
provider/datagen/src/uprops/collation_jamo.rs is now changed in the branch
provider/datagen/src/uprops/collation_metadata.rs is now changed in the branch
provider/datagen/src/uprops/collation_serde.rs is now changed in the branch
provider/datagen/src/uprops/decompositions_serde.rs is now changed in the branch
provider/datagen/src/uprops/enum_codepointtrie.rs is now changed in the branch
provider/datagen/src/uprops/mod.rs is now changed in the branch
provider/datagen/src/uprops/uprops_helpers.rs is now changed in the branch
provider/testdata/Cargo.toml is different
provider/testdata/data/testdata.postcard is different
provider/uprops/Cargo.toml is no longer changed in the branch
provider/uprops/src/canonical_decompositions.rs is no longer changed in the branch
provider/uprops/src/codepointtrie.rs is no longer changed in the branch
provider/uprops/src/collation_data.rs is no longer changed in the branch
provider/uprops/src/collation_diacritics.rs is no longer changed in the branch
provider/uprops/src/collation_jamo.rs is no longer changed in the branch
provider/uprops/src/collation_metadata.rs is no longer changed in the branch
provider/uprops/src/collation_serde.rs is no longer changed in the branch
provider/uprops/src/decompositions_serde.rs is no longer changed in the branch
provider/uprops/src/enum_codepointtrie.rs is no longer changed in the branch
provider/uprops/src/lib.rs is no longer changed in the branch
provider/uprops/src/uprops_helpers.rs is no longer changed in the branch
provider/uprops/src/uprops_serde.rs is no longer changed in the branch
tools/datagen/Cargo.toml is no longer changed in the branch
tools/datagen/src/bin/datagen.rs is no longer changed in the branch
tools/datagen/src/lib.rs is no longer changed in the branch

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

robertbastian

Left some comments on datagen/provider stuff

Also, I cannot comment this in the file because it's too big:
nit: move experimental/collator/src/CollationTest_*.txt to a testdata directory

provider/datagen/Cargo.toml

provider/datagen/src/bin/datagen.rs

experimental/collator/src/provider.rs

experimental/datatest/Cargo.toml

experimental/normalizer/Cargo.toml

provider/testdata/src/paths.rs

jira-pull-request-webhook · 2022-04-21T08:55:15Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2022-04-22T08:43:56Z

Notice: the branch changed across the force-push!

Cargo.lock is different
components/properties/src/lib.rs is different
experimental/char16trie/src/char16trie.rs is different
experimental/collator/Cargo.toml is different
experimental/collator/README.md is different
experimental/collator/src/lib.rs is different
experimental/collator/src/provider.rs is different
experimental/datatest/Cargo.lock is no longer changed in the branch
experimental/datatest/Cargo.toml is no longer changed in the branch
experimental/datatest/src/main.rs is no longer changed in the branch
experimental/normalizer/Cargo.toml is different
experimental/normalizer/src/lib.rs is different
experimental/normalizer/src/provider.rs is different
provider/datagen/Cargo.toml is different
provider/datagen/src/bin/datagen.rs is different
provider/datagen/src/collator/mod.rs is different
provider/datagen/src/collator/transform.rs is different
provider/datagen/src/lib.rs is different
provider/datagen/src/uprops/canonical_decompositions.rs is different
provider/datagen/src/uprops/enum_codepointtrie.rs is different
provider/datagen/src/uprops/mod.rs is different
provider/testdata/Cargo.toml is different
provider/testdata/data/json/collator/data@1/big5han/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/data@1/gb2312han/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/data@1/search/sv.json is no longer changed in the branch
provider/testdata/data/json/collator/data@1/standard/sv.json is no longer changed in the branch
provider/testdata/data/json/collator/data@1/stroke/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/data@1/sv-u-co-search.json is now changed in the branch
provider/testdata/data/json/collator/data@1/sv-u-co-standard.json is now changed in the branch
provider/testdata/data/json/collator/data@1/unihan/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/data@1/zh-u-co-big5han.json is now changed in the branch
provider/testdata/data/json/collator/data@1/zh-u-co-gb2312.json is now changed in the branch
provider/testdata/data/json/collator/data@1/zh-u-co-stroke.json is now changed in the branch
provider/testdata/data/json/collator/data@1/zh-u-co-unihan.json is now changed in the branch
provider/testdata/data/json/collator/data@1/zh-u-co-zhuyin.json is now changed in the branch
provider/testdata/data/json/collator/data@1/zhuyin/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/dia@1/traditional/vi.json is no longer changed in the branch
provider/testdata/data/json/collator/dia@1/vi-u-co-trad.json is now changed in the branch
provider/testdata/data/json/collator/meta@1/big5han/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/meta@1/gb2312han/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/meta@1/search/sv.json is no longer changed in the branch
provider/testdata/data/json/collator/meta@1/standard/sv.json is no longer changed in the branch
provider/testdata/data/json/collator/meta@1/stroke/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/meta@1/sv-u-co-search.json is now changed in the branch
provider/testdata/data/json/collator/meta@1/sv-u-co-standard.json is now changed in the branch
provider/testdata/data/json/collator/meta@1/unihan/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/meta@1/zh-u-co-big5han.json is now changed in the branch
provider/testdata/data/json/collator/meta@1/zh-u-co-gb2312.json is now changed in the branch
provider/testdata/data/json/collator/meta@1/zh-u-co-stroke.json is now changed in the branch
provider/testdata/data/json/collator/meta@1/zh-u-co-unihan.json is now changed in the branch
provider/testdata/data/json/collator/meta@1/zh-u-co-zhuyin.json is now changed in the branch
provider/testdata/data/json/collator/meta@1/zhuyin/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/reord@1/big5han/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/reord@1/gb2312han/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/reord@1/stroke/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/reord@1/unihan/zh.json is no longer changed in the branch
provider/testdata/data/json/collator/reord@1/zh-u-co-big5han.json is now changed in the branch
provider/testdata/data/json/collator/reord@1/zh-u-co-gb2312.json is now changed in the branch
provider/testdata/data/json/collator/reord@1/zh-u-co-stroke.json is now changed in the branch
provider/testdata/data/json/collator/reord@1/zh-u-co-unihan.json is now changed in the branch
provider/testdata/data/json/collator/reord@1/zh-u-co-zhuyin.json is now changed in the branch
provider/testdata/data/json/collator/reord@1/zhuyin/zh.json is no longer changed in the branch
provider/testdata/data/json/normalizer/nfd@1.json is no longer changed in the branch
provider/testdata/data/json/normalizer/nfd@1/und.json is now changed in the branch
provider/testdata/data/testdata.postcard is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2022-04-22T08:53:38Z

Notice: the branch changed across the force-push!

provider/datagen/src/collator/transform.rs is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

experimental/collator/src/lib.rs

experimental/normalizer/src/lib.rs

experimental/collator/src/lib.rs

provider/datagen/src/uprops/mod.rs

experimental/collator/src/lib.rs

echeran

I'm trying to stick to Conventional Comments, but I forgot that all of these non-optional comments need to be accompanied with the proper Github review flags being set.

jira-pull-request-webhook · 2022-05-11T08:22:32Z

Notice: the branch changed across the force-push!

experimental/collator/src/elements.rs is different
provider/datagen/src/collator/transform.rs is different
provider/datagen/src/uprops/codepointtrie.rs is no longer changed in the branch
provider/datagen/src/uprops/decompositions_serde.rs is different
provider/datagen/src/uprops/mod.rs is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

robertbastian

I had written some comments but forgot to send them, they might be outdated.

experimental/collator/fuzz/Cargo.toml

experimental/collator/src/provider.rs

experimental/normalizer/Cargo.toml

experimental/normalizer/fuzz/Cargo.toml

experimental/normalizer/src/provider.rs

provider/datagen/src/collator/transform.rs

provider/datagen/src/uprops/canonical_decompositions.rs

provider/testdata/src/paths.rs

jira-pull-request-webhook · 2022-05-16T10:39:19Z

Notice: the branch changed across the force-push!

Cargo.lock is different
Cargo.toml is different
components/properties/src/lib.rs is different
experimental/collator/Cargo.toml is different
experimental/collator/src/comparison.rs is different
experimental/normalizer/Cargo.toml is different
experimental/normalizer/src/lib.rs is different
provider/datagen/Cargo.toml is different
provider/datagen/src/bin/datagen.rs is no longer changed in the branch
provider/datagen/src/collator/mod.rs is no longer changed in the branch
provider/datagen/src/collator/transform.rs is no longer changed in the branch
provider/datagen/src/lib.rs is no longer changed in the branch
provider/datagen/src/registry.rs is now changed in the branch
provider/datagen/src/source.rs is now changed in the branch
provider/datagen/src/transform/collator/mod.rs is now changed in the branch
provider/datagen/src/transform/collator/transform.rs is now changed in the branch
provider/datagen/src/transform/mod.rs is now changed in the branch
provider/datagen/src/transform/uprops/canonical_decompositions.rs is now changed in the branch
provider/datagen/src/transform/uprops/decompositions_serde.rs is now changed in the branch
provider/datagen/src/transform/uprops/mod.rs is now changed in the branch
provider/datagen/src/uprops/canonical_decompositions.rs is no longer changed in the branch
provider/datagen/src/uprops/decompositions_serde.rs is no longer changed in the branch
provider/datagen/src/uprops/enum_codepointtrie.rs is no longer changed in the branch
provider/datagen/src/uprops/mod.rs is no longer changed in the branch
provider/testdata/Cargo.toml is different
provider/testdata/data/testdata.postcard is different
provider/testdata/src/bin/datagen.rs is now changed in the branch
provider/testdata/src/paths.rs is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2022-05-16T11:08:09Z

Notice: the branch changed across the force-push!

Cargo.lock is different
Cargo.toml is different
provider/datagen/Cargo.toml is different
provider/datagen/src/source.rs is different
provider/testdata/data/testdata.postcard is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

…onal block

Co-authored-by: Robert Bastian <robertbastian@users.noreply.github.com>

robertbastian

LGTM for everything outside experimental; I haven't looked at the actual algorithms in-depth.

experimental/collator/src/elements.rs

experimental/collator/src/lib.rs

robertbastian

Please wait for Elango's review before submitting

echeran

Looks good. A couple of minor things, but feel free to merge this in now, and create a followup PR later.

experimental/collator/src/options.rs

experimental/collator/src/lib.rs

experimental/collator/Cargo.toml

experimental/collator/fuzz/fuzz_targets/compare_utf16.rs

experimental/collator/src/comparison.rs

experimental/collator/src/lib.rs

experimental/collator/src/options.rs

experimental/collator/src/provider.rs

experimental/normalizer/src/lib.rs

sffc · 2022-05-20T03:39:01Z

@hsivonen Feel free to hit the merge button, and start opening smaller PRs to resolve remaining open comments. Please make sure all the open comments are associated with an open issue.

Great work!

hsivonen · 2022-05-20T06:04:18Z

Thanks. Merged. I will make sure issues are filed for the comments.

hsivonen requested review from sffc and a team as code owners March 17, 2022 14:14

hsivonen requested a review from echeran March 28, 2022 11:15

This was referenced Mar 28, 2022

Create a Normalizer component #972

Closed

Experiment with ICU4X boa-dev/boa#1180

Closed

sffc reviewed Mar 28, 2022

View reviewed changes

provider/testdata/Cargo.toml Outdated Show resolved Hide resolved

hsivonen force-pushed the collation branch from c90040c to 4f19aeb Compare March 31, 2022 11:26

hsivonen requested review from robertbastian and Manishearth as code owners March 31, 2022 11:26

hsivonen removed request for Manishearth and robertbastian April 1, 2022 08:08

robertbastian requested changes Apr 19, 2022

View reviewed changes

hsivonen force-pushed the collation branch from f13203b to 0a145e5 Compare April 21, 2022 08:55

hsivonen force-pushed the collation branch from 0a145e5 to 124db55 Compare April 22, 2022 08:43

hsivonen force-pushed the collation branch from 124db55 to b608122 Compare April 22, 2022 08:53

echeran reviewed Apr 28, 2022

View reviewed changes

provider/datagen/src/uprops/mod.rs Outdated Show resolved Hide resolved

experimental/collator/src/lib.rs Outdated Show resolved Hide resolved

echeran requested changes Apr 28, 2022

View reviewed changes

hsivonen force-pushed the collation branch from 3008914 to fead5e5 Compare May 11, 2022 08:22

robertbastian reviewed May 11, 2022

View reviewed changes

hsivonen force-pushed the collation branch from ccdec3d to af7a47f Compare May 16, 2022 10:39

hsivonen force-pushed the collation branch from fc4b2fa to 208a36d Compare May 16, 2022 11:08

hsivonen force-pushed the collation branch from e807320 to 2872468 Compare May 16, 2022 12:07

hsivonen and others added 5 commits May 18, 2022 08:56

Wrap even the debug_assert! access of iter_exhausted in a conditi…

ddbddfb

…onal block

Address clippy lints

bd77175

Hoist push_str out of match

fee64d9

Co-authored-by: Robert Bastian <robertbastian@users.noreply.github.com>

Comment out collator tests for locales that are not in the testdata set

a7dbdd6

Tweak links in doc comments

8327d2a

robertbastian requested changes May 18, 2022

View reviewed changes

experimental/collator/src/elements.rs Outdated Show resolved Hide resolved

experimental/collator/src/lib.rs Outdated Show resolved Hide resolved

hsivonen added 6 commits May 18, 2022 11:32

Make assertion conditional without a block

c68b812

Ignore tests that need more data instead of commenting them out

73e886d

Add README files

5061f13

Try to pass documentation CI checks

845da73

LICENSE files for fuzz crates

065edfc

Make CI happy about URLs in doc comments

8931e5f

hsivonen requested a review from robertbastian May 18, 2022 12:37

robertbastian previously approved these changes May 18, 2022

View reviewed changes

Merge branch 'main' into collation

2d4b5d6

hsivonen dismissed robertbastian’s stale review via 2d4b5d6 May 18, 2022 17:53

echeran approved these changes May 19, 2022

View reviewed changes

experimental/collator/src/options.rs Show resolved Hide resolved

experimental/collator/src/lib.rs Show resolved Hide resolved

sffc approved these changes May 20, 2022

View reviewed changes

hsivonen merged commit e40ba55 into unicode-org:main May 20, 2022

hsivonen deleted the collation branch June 2, 2022 08:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a collator implementation under experimental/ #1706

Add a collator implementation under experimental/ #1706

hsivonen commented Mar 17, 2022 •

edited

Loading

jira-pull-request-webhook bot commented Mar 31, 2022

robertbastian left a comment

jira-pull-request-webhook bot commented Apr 21, 2022

jira-pull-request-webhook bot commented Apr 22, 2022

jira-pull-request-webhook bot commented Apr 22, 2022

echeran left a comment

jira-pull-request-webhook bot commented May 11, 2022

robertbastian left a comment

jira-pull-request-webhook bot commented May 16, 2022

jira-pull-request-webhook bot commented May 16, 2022

robertbastian left a comment

robertbastian left a comment

echeran left a comment

sffc commented May 20, 2022

hsivonen commented May 20, 2022

Add a collator implementation under experimental/ #1706

Add a collator implementation under experimental/ #1706

Conversation

hsivonen commented Mar 17, 2022 • edited Loading

What works

What doesn't work

Post-landing help would be particularly helpful on

jira-pull-request-webhook bot commented Mar 31, 2022

robertbastian left a comment

Choose a reason for hiding this comment

jira-pull-request-webhook bot commented Apr 21, 2022

jira-pull-request-webhook bot commented Apr 22, 2022

jira-pull-request-webhook bot commented Apr 22, 2022

echeran left a comment

Choose a reason for hiding this comment

jira-pull-request-webhook bot commented May 11, 2022

robertbastian left a comment

Choose a reason for hiding this comment

jira-pull-request-webhook bot commented May 16, 2022

jira-pull-request-webhook bot commented May 16, 2022

robertbastian left a comment

Choose a reason for hiding this comment

robertbastian left a comment

Choose a reason for hiding this comment

echeran left a comment

Choose a reason for hiding this comment

sffc commented May 20, 2022

hsivonen commented May 20, 2022

hsivonen commented Mar 17, 2022 •

edited

Loading