Improve strategy for internationalization of MFEs

[arbrandes]

Motivation

There are a series of pain points currently identified by the community when it comes to i18n and l10n of MFEs:

• Need for forking: MFE languages are hard coded (see here and here)
• Complicated process to update translations: we need to do it manually for every repository
• Complicated process to update translations: MFEs use json files; edx-platform uses .po files
• Build size: The build includes all languages: the image builder should be able to select which ones they want
• l10n: Currency does not change with localization; no source of truth for currency
• MFEs don't support all the languages that edx-platform does

Open questions

• How many Open edX instances actually offer more than one selectable language?

Action items

• docs: add OEP-0058 - Translations Management open-edx-proposals#367
• Expand list of supported Transifex languages to push/pull in Makefile wg-frontend#119
• Discovery: What is the process to get a new language supported on an MFE? axim-engineering#479
• Documentation: How to add a language to the supported list in the current way docs.openedx.org#154
• Support standard set of languages studio-frontend#357
• Review RacoonGang's i18n improvement ideas axim-engineering#508
• feat: add js translations job openedx-translations#18
• Extend language support: add basic list languages that Open edX supports (currently 130 languages)
• Build: Minimize build by restricting the number of languages that are included
• Per-user l10n: Offer on-the-fly language selector in the MFEs
• Per-user l10n: Restrict the number of languages (dark lang for MFEs)
• Per-user l10n: Unify user-facing translation system: account settings vs selector vs session cookies

Misc

This issue is also tracked in the TWG Trello Board

[github-actions]

Thanks for your submission, @openedx/open-edx-project-managers will review shortly.

[ARMBouhali]

@arbrandes If l10n settings are loaded through an API call, we would then solve any l10n pain points above from the root.

Changes to frontend-platform

• frontend-platform's i18n functionality can be rewritten to load the necessary i18n configuration from an API endpoint, such as:
• available languages + language status (supported, beta, etc.)
• localization settings (currency, number & date formats)
• a function to be used by MFE apps and components at page load to retrieve the translation file (<language_code>.json) if needed.

Changes to MFE apps and components:

• use the new frontend-platform i18n functions to load all l10n-related settings and resources.
• All i18n JSON files can then be removed.
• No more i18n chore commits.

Changes to Transifex workflows:

• Original strings will still be loaded from their current source.
• Translations will all be pushed periodically to a single GitHub repository (for example openedx/frontend-i18n) which will be tagged for releases as any other Open edX repo.

Quick overview of the API endpoint(s)

• Implementation: may be deployed as a Django IDA, but I'd prefer an edx-platform plugin.
• Dynamic configurations such as DarkLang, released languages, and default language can be exposed as an API endpoint.
• i18n JSON files can be loaded manually using a Django management command, or automatically upon configuration change (post_save signal). They can be saved as static files and served to all MFE apps and codes.

Concerns

We need to provide a clear method to support l10n overrides and custom locales.

• L10n file overrides can be done simply by changing the source of truth (e.g using a fork of frontend-i18n), but it's a coarse-grained approach which I don't like.
• For custom (sub)locales, a discussion is necessary.

P.S: IMO the proposition above goes the same direction as this WIP for MFE runtime configuration

[arbrandes]

Adding a pre-existing BD-49 discovery doc @jmakowski1123 pointed out:

https://docs.google.com/document/d/1-q0yeq9MUpbylO5uIl_LFJJBPR6sNgmX1zXbeyoFRvw/edit

It makes me think that this is a good potential candidate for tCRIL-funded development.

[arbrandes]

@ARMBouhali, those are all great ideas. We should work with @Carlos-Muniz on the backend, in particular regarding:

• Translations will all be pushed periodically to a single GitHub

This is something he's already working on, via a new OEP (OEP-58). Care to take a look?

As for MFE runtime configuration: we'd theoretically want to take advantage of that endpoint for the translation APIs.

[arbrandes]

Evgen at RacoonGang has questions and a proposal as well:

https://discuss.openedx.org/t/language-settings-unconsistency/8466

[idegtiarov]

Adding a pre-existing BD-49 discovery doc @jmakowski1123 pointed out:

https://docs.google.com/document/d/1-q0yeq9MUpbylO5uIl_LFJJBPR6sNgmX1zXbeyoFRvw/edit

It makes me think that this is a good potential candidate for tCRIL-funded development.

@arbrandes The doc mentioned in your comment is not public. Could you please open it to become public or share the doc's summary and those great ideas in this thread? ))

[arbrandes]

@idegtiarov, done! Not sure why it was private in the first place.

[idegtiarov]

@arbrandes thanks!

[arbrandes]

@brian-smith-tcril, the Translation Working Group needs help streamlining the MFE i18n/l10n process. I figure this is a good next step for you, and it will be very useful to the community. Would you be willing to take this epic on, starting this sprint?

What does that entail, you might ask? As you'll see from the issue description, the specifics are still unclear, so initially it's all about discovery. The following is a good place to start:

• Discovery: What is the process to get a new language supported on an MFE? axim-engineering#479

Getting in touch with @Carlos-Muniz would be a good starting point as far as community coordination goes, and I'm of course available as well. We should probably schedule a kickstart session early in the week.

[brian-smith-tcril]

@arbrandes If l10n settings are loaded through an API call, we would then solve any l10n pain points above from the root.

Changes to frontend-platform

• frontend-platform's i18n functionality can be rewritten to load the necessary i18n configuration from an API endpoint, such as:
• available languages + language status (supported, beta, etc.)
• localization settings (currency, number & date formats)
• a function to be used by MFE apps and components at page load to retrieve the translation file (<language_code>.json) if needed.

I'm not sure what exactly you're envisioning for this API. I feel like it makes sense to have most of this set at build time, so if the API is set up to provide this data at build time that could make sense. If you could elaborate on how you see this API being used, that could go a long way towards helping me get a clearer picture of what you're describing here.

Changes to MFE apps and components:

• use the new frontend-platform i18n functions to load all l10n-related settings and resources.
• All i18n JSON files can then be removed.
• No more i18n chore commits.

I completely agree we should get the i18n json files out of the MFE repos. I could see this working in a couple different ways. I was thinking we could utilize https://github.com/openedx/openedx-atlas to pull in translation .json files right before running the webpack build, but i'm very open to other ideas!

Changes to Transifex workflows:

• Original strings will still be loaded from their current source.
• Translations will all be pushed periodically to a single GitHub repository (for example openedx/frontend-i18n) which will be tagged for releases as any other Open edX repo.

This is already happening! Aside from the tagging for release part, there is a github action set up on https://github.com/openedx/openedx-translations that:

• pulls from the MFE repos
• runs make extract_translations
• commits the generated transifex_input.json files to https://github.com/openedx/openedx-translations

and then the transifex github app pulls from the files in that repo (using https://github.com/openedx/openedx-translations/blob/main/transifex.yml for config)

this is still a work in progress, and is currently connected to a sandbox environment on transifex

Quick overview of the API endpoint(s)

• Implementation: may be deployed as a Django IDA, but I'd prefer an edx-platform plugin.
• Dynamic configurations such as DarkLang, released languages, and default language can be exposed as an API endpoint.
• i18n JSON files can be loaded manually using a Django management command, or automatically upon configuration change (post_save signal). They can be saved as static files and served to all MFE apps and codes.

I'm curious as to how much refactoring this would require on each MFE. Are languages added/updated enough that we should decouple it from running a build?

Concerns

We need to provide a clear method to support l10n overrides and custom locales.

• L10n file overrides can be done simply by changing the source of truth (e.g using a fork of frontend-i18n), but it's a coarse-grained approach which I don't like.
• For custom (sub)locales, a discussion is necessary.

P.S: IMO the proposition above goes the same direction as this WIP for MFE runtime configuration

I'm curious about the override use case. My first thought is that a fork wouldn't be the worst way to handle it, but I see how having a "patch style" way to handle overrides could be useful.

Overall these are some great ideas! I'd love to hear any more you have as we work towards improving the translation architecture!

[brian-smith-tcril]

not sure where to put this but i figure a comment on the epic works

concerns were raised about i18n linting (ensuring pluralization etc. is handled correctly in source translation strings) as part of the CI process in this comment on OEP-58

it's not clear what the current strategy for this is, and if there is any consistency across repos. this is definitely a place where some discovery should be done

[arbrandes]

I'm leaving this in the FWG backlog for us to revisit once the current work on i18n is complete.