Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LocaleFallbackProvider to tutorial examples #3334

Merged
merged 4 commits into from
Aug 29, 2023
Merged

Add LocaleFallbackProvider to tutorial examples #3334

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c1d012c
Add LocaleFallbackProvider to tutorial examples
sffc Apr 15, 2023
87058d8
Imports
sffc Apr 15, 2023
bc8002f
Introduce why we need LocaleFallbackProvider
sffc Aug 26, 2023
6590244
Fix tutorials test?
sffc Aug 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/tutorials/data_management.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ You should generate it automatically at build time if:

If you check in the generated data, it is recommended that you configure a job in continuous integration that verifies that the data in your repository reflects the latest CLDR/Unicode releases; otherwise, your app may drift out of date.

## Locale Fallbacking

Data generated with `--format blob` or `--format fs` supports only exact matches for locales, not locales requiring _fallback_. For example, if `en-US` is requested but only `en` data is available, then the data request will fail. Because of this, it is often desirable to configure a `LocaleFallbackProvider`, as illustrated in the remainder of the examples on this page.

# 3. Using the generated data

Once we have generated some data, it needs to be loaded as a data provider. The blob format we chose can be loaded by `BlobDataProvider` from the `icu_provider_blob` crate.
Expand All @@ -65,6 +69,7 @@ We can then use the provider in our code:
use icu::locid::{locale, Locale};
use icu::calendar::DateTime;
use icu::datetime::{DateTimeFormatter, options::length};
use icu_provider_adapters::fallback::LocaleFallbackProvider;
use icu_provider_blob::BlobDataProvider;

const LOCALE: Locale = locale!("ja");
Expand All @@ -75,6 +80,11 @@ fn main() {
BlobDataProvider::try_new_from_blob(blob.into_boxed_slice())
.expect("Failed to initialize Data Provider.");

// Wrapping the raw BlobDataProvider in a LocaleFallbackProvider enables
// locales to fall back to other locales, like "en-US" to "en".
let buffer_provider = LocaleFallbackProvider::try_new_with_buffer_provider(buffer_provider)
.expect("Provider should contain fallback rules");

let options = length::Bag::from_date_time_style(length::Date::Long, length::Time::Medium);

let dtf = DateTimeFormatter::try_new_with_buffer_provider(&buffer_provider, &LOCALE.into(), options.into())
Expand Down Expand Up @@ -109,6 +119,7 @@ We can instead use `TypedDateTimeFormatter<Gregorian>`, which only supports form
use icu::locid::{locale, Locale};
use icu::calendar::{DateTime, Gregorian};
use icu::datetime::{TypedDateTimeFormatter, options::length};
use icu_provider_adapters::fallback::LocaleFallbackProvider;
use icu_provider_blob::BlobDataProvider;

const LOCALE: Locale = locale!("ja");
Expand All @@ -119,6 +130,11 @@ fn main() {
BlobDataProvider::try_new_from_blob(blob.into_boxed_slice())
.expect("Failed to initialize Data Provider.");

// Wrapping the raw BlobDataProvider in a LocaleFallbackProvider enables
// locales to fall back to other locales, like "en-US" to "en".
let buffer_provider = LocaleFallbackProvider::try_new_with_buffer_provider(buffer_provider)
.expect("Provider should contain fallback rules");

let options = length::Bag::from_date_time_style(length::Date::Long, length::Time::Medium);

let dtf = TypedDateTimeFormatter::<Gregorian>::try_new_with_buffer_provider(&buffer_provider, &LOCALE.into(), options.into())
Expand Down Expand Up @@ -168,6 +184,7 @@ extern crate alloc; // required as my-data-mod is written for #[no_std]
use icu::locid::{locale, Locale};
use icu::calendar::DateTime;
use icu::datetime::{TypedDateTimeFormatter, options::length};
use icu_provider_adapters::fallback::LocaleFallbackProvider;

const LOCALE: Locale = locale!("ja");

Expand All @@ -178,6 +195,11 @@ impl_data_provider!(UnstableProvider);
fn main() {
let unstable_provider = UnstableProvider;

// Wrapping the raw UnstableProvider in a LocaleFallbackProvider enables
// locales to fall back to other locales, like "en-US" to "en".
let unstable_provider = LocaleFallbackProvider::try_new_unstable(unstable_provider)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying this in my code but I get an error like this:

et provider = LocaleFallbackProvider::try_new_unstable(UnstableProvider)?;
    |                    ---------------------------------------- ^^^^^^^^^^^^^^^^ the trait `icu_provider::DataProvider<LocaleFallbackLikelySubtagsV1Marker>` is not implemented for `UnstableProvider`
    |                    |
    |                    required by a bound introduced by this call
    |
    = help: the following other types implement trait `icu_provider::DataProvider<M>`:
              <UnstableProvider as icu_provider::DataProvider<CanonicalDecompositionDataV1Marker>>
              <UnstableProvider as icu_provider::DataProvider<CanonicalDecompositionTablesV1Marker>>
              <UnstableProvider as icu_provider::DataProvider<CollationDataV1Marker>>
              <UnstableProvider as icu_provider::DataProvider<CollationDiacriticsV1Marker>>
              <UnstableProvider as icu_provider::DataProvider<CollationJamoV1Marker>>
              <UnstableProvider as icu_provider::DataProvider<CollationMetadataV1Marker>>
              <UnstableProvider as icu_provider::DataProvider<CollationReorderingV1Marker>>
              <UnstableProvider as icu_provider::DataProvider<CollationSpecialPrimariesV1Marker>>
note: required by a bound in `LocaleFallbackProvider::<P>::try_new_unstable`
   --> /home/autarch/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/icu_provider_adapters-1.2.0/src/fallback/adapter.rs:54:8
    |
54  |     P: DataProvider<LocaleFallbackLikelySubtagsV1Marker>
    |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `LocaleFallbackProvider::<P>::try_new_unstable`

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this is because I had to add more keys to the data generated by icu4x-datagen. The workflow around this is pretty frustrating:

  1. Generate all the keys.
  2. Build my program, which takes forever since I generated all the keys.
  3. Generate using --keys-for-bin.
  4. Add a new locale-related bit of code, go to step Transpilation approach #1.

It'd be nice if there was more guidance on which keys are needed for each part of the icu crate's API.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, yes, acknowledged that this process is a bit frustrating. It was designed to give more control over the data lifecycle than ICU4C, and it goes very much to the extreme in that direction. For the next 1.3 release, we have #2945 to streamline this for users who don't need all the control.

That said, there are ways you can speed up your development loop. For example, in step 1 and 2, if you generate data with --locales none, it builds a lot faster, and still works with --keys-for-bin.

It'd be nice if there was more guidance on which keys are needed for each part of the icu crate's API.

You can see the list of keys in the signature of the try_new_unstable functions. For example: LineSegmenter::try_new_auto_unstable has the following bounds:

DataProvider<LineBreakDataV1Marker> + DataProvider<LstmForWordLineAutoV1Marker> + DataProvider<GraphemeClusterBreakDataV1Marker>

If you click into those three data markers, it gives you the key in the docs string.

.expect("Provider should contain fallback rules");

let options = length::Bag::from_date_time_style(length::Date::Long, length::Time::Medium);

let dtf = TypedDateTimeFormatter::try_new_unstable(&unstable_provider, &LOCALE.into(), options.into())
Expand Down Expand Up @@ -224,6 +246,7 @@ $ cargo add icu_provider_fs
use icu::locid::{locale, Locale};
use icu::calendar::DateTime;
use icu::datetime::{TypedDateTimeFormatter, options::length};
use icu_provider_adapters::fallback::LocaleFallbackProvider;
use icu_provider_fs::FsDataProvider;

const LOCALE: Locale = locale!("ja");
Expand All @@ -232,6 +255,11 @@ fn main() {
let buffer_provider = FsDataProvider::try_new("my-data-dir")
.expect("Failed to initialize Data Provider");

// Wrapping the raw BlobDataProvider in a LocaleFallbackProvider enables
// locales to fall back to other locales, like "en-US" to "en".
let buffer_provider = LocaleFallbackProvider::try_new_with_buffer_provider(buffer_provider)
.expect("Provider should contain fallback rules");

let options = length::Bag::from_date_time_style(length::Date::Long, length::Time::Medium);

let dtf = TypedDateTimeFormatter::try_new_with_buffer_provider(&buffer_provider, &LOCALE.into(), options.into())
Expand Down
2 changes: 1 addition & 1 deletion docs/tutorials/testing/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ displaydoc = "0.2.3"
icu = "1.2"
icu_datagen = { version = "1.2", default-features = false}
icu_provider = { version = "1.2", features = ["deserialize_json"] }
icu_provider_adapters = "1.2"
icu_provider_adapters = { version = "1.2", features = ["serde"] }
icu_provider_blob = "1.2"
icu_provider_fs = "1.2"
icu_testdata = "1.2"
Expand Down
Loading