Connect properties provider to the icu4x_datagen exporter tool

[echeran]

Closes #936

[jira-pull-request-webhook]

Notice: the branch changed across the force-push!

• components/properties/src/props.rs is now changed in the branch
• provider/uprops/src/enum_codepointtrie.rs is different
• tools/datagen/src/bin/datagen.rs is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

[aethanyc]

I find it a bit strange that the tool generates data for CodePointMaps keys GENERAL_CATEGORY_V1 and SCRIPT_V1 under uniset/ folder such as uniset/sc@1.json and uniset/gc@1.json.

[echeran]

I find it a bit strange that the tool generates data for CodePointMaps keys GENERAL_CATEGORY_V1 and SCRIPT_V1 under uniset/ folder such as uniset/sc@1.json and uniset/gc@1.json.

Agreed. Weigh in on the discussion in #1196

[sffc]

Looks great! Just some small things

[sffc]

Question: why do we need these serde impls now when we didn't need them before?

[echeran]

Deserialize is needed for unit tests that need to load data as a UnicodePropertyMap data struct (which in turns carries a CodePointMap, which takes these enums as parameters). In the reverse direction, serialization happens when generating testdata.

This is the error from cargo test when I comment out these serde impls:

error[E0277]: the trait bound `GeneralSubcategory: serde::de::Deserialize<'_>` is not satisfied
  --> src/maps.rs:56:5
   |
9  |     maps::get_general_category(&provider)
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `serde::de::Deserialize<'_>` is not implemented for `GeneralSubcategory`
   |
   = note: required because of the requirements on the impl of `for<'de> serde::de::Deserialize<'_>` for `UnicodePropertyMapV1<'de, GeneralSubcategory>`
   = note: 1 redundant requirements hidden
   = note: required because of the requirements on the impl of `for<'de> serde::de::Deserialize<'de>` for `yoke::trait_hack::YokeTraitHack<UnicodePropertyMapV1<'de, GeneralSubcategory>>`
   = note: required because of the requirements on the impl of `icu_provider::data_provider::DataProvider<'_, UnicodePropertyMapV1Marker<GeneralSubcategory>>` for `icu_provider_fs::fs_data_provider::FsDataProvider`
note: required by a bound in `get_general_category`
  --> /usr/local/google/home/elango/oss/icu4x/components/properties/src/maps.rs:67:8
   |
67 |     D: DataProvider<'data, UnicodePropertyMapV1Marker<GeneralSubcategory>> + ?Sized,
   |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `get_general_category`

Similarly, this is the error from cargo make testdata:

error[E0277]: the trait bound `GeneralSubcategory: Serialize` is not satisfied
   --> provider/uprops/src/enum_codepointtrie.rs:116:1
    |
116 | / icu_provider::impl_dyn_provider!(EnumeratedPropertyCodePointTrieProvider, {
117 | |     key::GENERAL_CATEGORY_V1 => UnicodePropertyMapV1Marker<GeneralSubcategory>,
118 | |     key::SCRIPT_V1 => UnicodePropertyMapV1Marker<Script>,
119 | | }, SERDE_SE, 'data);
    | |____________________^ the trait `Serialize` is not implemented for `GeneralSubcategory`
    |
    = note: required because of the requirements on the impl of `for<'a> Serialize` for `UnicodePropertyMapV1<'a, GeneralSubcategory>`
    = note: 1 redundant requirements hidden
    = note: required because of the requirements on the impl of `for<'a> Serialize` for `&'a UnicodePropertyMapV1<'a, GeneralSubcategory>`
    = note: required because of the requirements on the impl of `UpcastDataPayload<'_, UnicodePropertyMapV1Marker<GeneralSubcategory>>` for `SerdeSeDataStructMarker`
note: required by `upcast`
   --> /usr/local/google/home/elango/oss/icu4x/provider/core/src/dynutil.rs:40:5
    |
40  | /     fn upcast(
41  | |         other: crate::prelude::DataPayload<'data, M>,
42  | |     ) -> crate::prelude::DataPayload<'data, Self>;
    | |__________________________________________________^
    = note: this error originates in the macro `$crate::impl_dyn_provider` (in Nightly builds, run with -Z macro-backtrace for more info)

[sffc]

Okay, so I think we do actually need the serde impls here, because human_readable serialization of the ZeroVec needs those impls on T.