Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(v2): core v2 i18n support + Docusaurus site Crowdin integration #3325

Merged
merged 130 commits into from
Nov 26, 2020

Conversation

slorber
Copy link
Collaborator

@slorber slorber commented Aug 21, 2020

Motivation

Initial i18n POC

See RFC #3317


Current WIP state:

Markdown translation

Markdown translations work for docs (including versioning), blog, pages.

Localized markdown files should be put in ./i18n/<locale>/<pluginName><-pluginId>/**/*.md

If the markdown exist at the correct localized location, it will be used in priority over the original document.

Note: some questions asked to Crowdin: https://gist.github.com/slorber/30643299196c7efa77084eec10c1c609

Key/values translations

Key/value translations are helpful to translate the plugin's data (content plugins) and UIs (theme plugins).

Key/value translations use Chrome i18n JSON format:

  • it allows adding context/description to messages, useful for translators
  • it's widely supported by many SaaS

SaaS documentation:

You can "init" the localized JSON files by using the cli command docusaurus write-translations --locales all.

Then you need to modify the JSON files in ./i18n/<locale>/<pluginName><-pluginId>/**/*.json

Code translations

Code translations is a subset of key/value translations that are found in UI / JSX, using the Docusaurus i18n runtime.

You can use translations in pages and components, with a new client API:

import Translate, { translate } from "@docusaurus/Translate";

const Examples = {
  // component API: useful for regular JSX translations
  content: (
    <Translate
      id="homepage.quotes.christopher-chedeau"
      description="Quote of Christopher Chedeau on the home page"
    >
      Default quote message of Christopher Chedeau
    </Translate>
  ),

  // imperative API: useful for placeholders, alt or regular string props...
  title: translate({
    id: "homepage.quotes.christopher-chedeau.title",
    message: "Lead Prettier Developer",
    description: "Title of quote of Christopher Chedeau on the home page"
  })
};

ìdanddescription` are optional args.

Code translations are extracted with the docusaurus write-translations cli at ./i18n/<locale>/code.json

Note: it's not possible to use dynamic translation values, because translations are handled through Babel, the id/message/description must be statically evaluable.

Site config

The initial config is small, but we'll add more customization options later (locale route paths, fs paths, ltr, charset encoding...)

export default {
  i18n: {
    defaultLocale: "en",
    locales: ["en", "fr"]
  }
};

CLI

Note: each locale leads to a single independent SPA site. It's not one SPA site containing all locales, that's why you can't start a Docusaurus site with all locales at once (but we may add this later).

  • docusaurus start: start your site on the default locale

  • docusaurus start --locale fr: start your site on the french locale

  • docusaurus build: will build your site in all available locales

  • docusaurus build --locale fr: will only build the french site

  • docusaurus write-translations: will extract translations from code and plugins

Use --help on cli for some extra doc.

Plugin lifecycles for localization

As Docusaurus v2 is modular, each plugin is responsible:

  • to declare its own translations (ie, write JSON files to FS at the correct location with cli command docusaurus write-translations)
  • to read those JS files from FS
  • use the FS translations to translate the loaded content (plugin data), or themeConfig (
type Plugin = {
  // return a descriptor of all files that will be written by cli docusaurus write-translations
  getTranslationFiles?(): Promise<TranslationFiles>;

  // will be called after "plugin.loadContent()" with the localized translation files of this plugin
  // you should return the localized content
  // the localized content will be passed to "plugin.contentLoaded()"
  // Usecase: translate docs titles, sidebars, version names...
  translateContent?({
    content,
    translationFiles
  }: {
    content: T; // the content loaded by this plugin instance
    translationFiles: TranslationFiles;
  }): T;

  // Translate the themeConfig
  // Usecase: translate themes like the classic theme (navbar, footer etc...)
  translateThemeConfig?({
    themeConfig,
    translationFiles
  }: {
    themeConfig: ThemeConfig;
    translationFiles: TranslationFiles;
  }): ThemeConfig;
};

Test deployments:

Markdown translation examples (Markdown data from Crowdin):

One domain for all locales:

One domain per locale:

@slorber slorber requested a review from yangshun as a code owner August 21, 2020 18:40
@slorber slorber marked this pull request as draft August 21, 2020 18:40
@slorber slorber added the pr: new feature This PR adds a new API or behavior. label Aug 21, 2020
@facebook-github-bot facebook-github-bot added the CLA Signed Signed Facebook CLA label Aug 21, 2020
@docusaurus-bot
Copy link
Contributor

docusaurus-bot commented Aug 21, 2020

Deploy preview for docusaurus-2 ready!

Built with commit f7ba5f3

https://deploy-preview-3325--docusaurus-2.netlify.app

@slorber slorber mentioned this pull request Aug 23, 2020
@slorber slorber merged commit 3166fab into master Nov 26, 2020
@slorber slorber changed the title feat(v2): core v2 i18n support + Crowdin integration feat(v2): core v2 i18n support + Docusaurus site Crowdin integration Nov 26, 2020
@slorber
Copy link
Collaborator Author

slorber commented Nov 26, 2020

Hi everyone.

This PR is now merged, but it's not the end of the story.
We'll test this code on Docusaurus + Jest, before advertising and documenting the i18n support.

In the meantime, you can still use the doc of this PR to figure out how it works, but keep in mind that until the official announcement, it's likely there will be a few breaking changes.

I'll probably look soon to onboard some of you privately before the official i18n release.
If you are interested, please let me know some details to see if you are a good candidate:

  • Is your site public, and easy to contribute to? (Github repo + convenient hosting with deploy previews)
  • Where do you want to store the translations? ( Git / Git submodule / Crowdin / Other ?)
  • Your Discord username?

@yangshun yangshun deleted the slorber/i18n branch November 26, 2020 11:56
@krillboi
Copy link

krillboi commented Nov 26, 2020

Hi @slorber, awesome news!

We'd love to be onboarded as we are ready to translate our whole site (https://support.touchgfx.com/) to Chinese and just need to get started with the integration between Docusaurus and Crowdin.

  • Our site is public but not open for contribution.
  • We'd like to use Crowdin as translation tool.
  • KrillBoi#9646

Looking forward to it!

@jcayzac
Copy link

jcayzac commented Dec 4, 2020

@slorber from the changes it's clear how to provide translations for documents, but what about the website's footer / navbar / title / tagline ?

@jcayzac
Copy link

jcayzac commented Dec 4, 2020

With this in my site config:

    i18n: {
      defaultLocale: 'en',
      locales: ['en', 'ja',],
    },

When I run write-translations -l ja, I get the following error:

Error: Can't write-translation for locale that is not in the locale configuration file.
Unknown locale=[ja].
Available locales=[en]
    at writeTranslations (/Users/julien.cayzac/rakuten/rpay/docs/node_modules/@docusaurus/core/lib/commands/writeTranslations.js:41:15)

(with 2.0.0-alpha.bd62be93d)

@jcayzac
Copy link

jcayzac commented Dec 4, 2020

Nevermind, I'm stupid, I had the i18 object in themeConfig……… 🤦🏼

@clairefro
Copy link

clairefro commented Dec 7, 2020

Hi @slorber this looks amazing!

Could I please hop on the onboarding train for using i18n with Docusaurus 2?

I am helping JAM stack framework RedwoodJS (https://redwoodjs.com/) start a new site for localizing it's documentation. Docusaurus 2 + Crowdin looks like a stellar combo for us.

  • Is your site public, and easy to contribute to? (Github repo + convenient hosting with deploy previews)
    Yes - our new site "Learn RedwoodJS" would be in the same spirit as redwoodjs.com here. Our repo would be public where anyone can add EN content, and translations would be crowdsourced via Crowdin.

  • Where do you want to store the translations? ( Git / Git submodule / Crowdin / Other ?)
    Github

  • Your Discord username?
    Claire Froelich#3472 <-- hope that works

@slorber
Copy link
Collaborator Author

slorber commented Dec 10, 2020

Hi there

I posted an update of i18n progress here:
#3317 (comment)

There are still a few things to change in Docusaurus core to be able to make Jest production-ready, but this is progressing: https://jest-v2.netlify.app/

@LoicPoullain
Copy link
Contributor

@slorber the translation dropdown seems broken on jest. When I select another language, I'm redirected to the home page with still English as language. Tested on Chrome and Firefox.

PS: Docusaurus is really cool and you're doing a great job. Keep it up! 👍

@slorber
Copy link
Collaborator Author

slorber commented Dec 14, 2020

@LoicPoullain thanks

The locale dropdown is hardcoded for now, will implement this very soon and make Jest production ready :)

@clairefro
Copy link

clairefro commented Dec 14, 2020

@slorber This may be intentional behavior but just dropping this here as a suggestion

Localized docs based on source docs that point to static assets (like ./static/img/logo.svg) break if the static assets are are not also defined in the localized directory.

Example error for localized version of shipped sample doc1 :

./i18n/fr/docusaurus-plugin-content-docs/current/doc1.md
Error: Image i18n/fr/docusaurus-plugin-content-docs/static/img/logo.svg used in i18n/fr/docusaurus-plugin-content-docs/current/doc1.md not found.
    at async Promise.all (index 1)

While this is a chance to optionally localize images that contain diagrams/text, it would be great if it were not required to copy assets to all locales.

Instead the i18n asset paths could resolve to the source assets by default, just like the i18n markdown files default to the source markdowns if no locale counterpart is found.

@slorber
Copy link
Collaborator Author

slorber commented Dec 15, 2020

hi @clairefro

If an asset is colocated, this means the asset will be duplicated for both versioning and localization, allowing you to use a different asset per version/locale (some images might contain localized text).

The localized doc folder should rather contain a full copy of the upstream doc folder, so that all relative imports keep working, or you should avoid using relative imports/ asset paths.

Note: if you don't need to version/localize an asset, you can put your svg in static and import it with @site/static/img/logo.svg

In the future, I'd like to allow "partial" localized doc sets more easily, and fallback to the original upstream doc if localized doc/asset does not exist, but it requires that I figure out how to do this with Webpack module resolution systems. This will not be part of the initial i18n release but will come after.

This will fallback as you expect:

  • upstream doc exist but localized doc does not exist
  • localized doc 1 link to ./doc2.md but doc2 localized doc does not exist

This won't fallback (as of today):

  • doc1 is localized and use a relative image path that does not exist: [](./colocatedImage.png)
  • doc1 is localized and import a mdx partial that is not localized: import MDXPartial from "./partials/_mdxPartial.mdx"

@krillboi
Copy link

krillboi commented Jan 5, 2021

Hi @slorber,

I am starting out with trying to use the i18n feature and looked at the Docusaurus v2 project on Crowdin. I noticed that compared to Docusaurus v1, the source files are a lot more messy to look at and understand for translators due to import statements and markdown/html cluttering up the view.

Take a look at: https://crowdin.com/translate/docusaurus-v2/128/en-fr?filter=basic&value=0

Compared to: https://crowdin.com/translate/docusaurus/2524/en-fr?filter=basic&value=0

Links in the v1 document are evaluated as markdown while v2 is just raw text (for example a lot of divs). In our docs, we use a lot of different React components, so this seems like it might be difficult for translators without a technical background to read and translate.

Is this something that can be fixed?

@slorber
Copy link
Collaborator Author

slorber commented Jan 5, 2021

Hi @krillboi

The i18n doc will soon be available.

I noticed that compared to Docusaurus v1, the source files are a lot more messy to look at and understand for translators due to import statements and markdown/html cluttering up the view.

You mean due to the fs organization?

Or just the embedded React components?

How is it more annoying than embedded HTML?

Links in the v1 document are evaluated as markdown while v2 is just raw text (for example a lot of divs).

That seems related to using the .mdx extension instead of .md, and how it is interpreted by Crowdin.
This v2 doc seems to work better for you no?
https://crowdin.com/translate/docusaurus-v2/126/en-fr?filter=basic&value=0

I see advantages of using the MDX extension, as IDE's can understand the embedded React code, Prettier can format it etc...
However, there's no first-class support for MDX in Crowdin.

You can "force" any extension to be interpreted as markdown in crowdin.yaml:
https://support.crowdin.com/configuration-file/#uploading-files-to-specified-path-with-specified-type

Their customer support can also enable custom processors too (not possible through the UI afaik).

Also, if Crowdin ask translators to translate strings that should rather not be translated (like mdx import statements), it is possible for you to use the "hide strings" feature so that only the crowdin admins can translate these strings


Is this something that can be fixed?

I'm in touch with Crowdin to see how to fix these problems, but keep in mind that Crowdin is an external SaaS software on which I have absolutely 0 control myself, and it's also not the one and only solution that you can use to translate a docusaurus project.

The Docusaurus v2 i18n is not coupled to any SaaS or Crowdin, and it just requires you to put files at the correct path.

We will try to document Crowdin usage the best we can, but in the end, if you want to use Crowdin to translate your project, you should understand how it works and read its doc, and also get in touch with their support to figure things out if needed.

In my opinion, Crowdin is far from perfect, but works not too badly either. There might be better translation SaaS alternatives that I'm not aware of, unfortunately don't have time to explore those, but hopefully some community contributors will be able to provide feedbacks and write documentation about other possible solutions?

Treat i18n in the same way as deployment doc: we historically document well the GitHub pages deployment, but many other options are possible. And in the end, you have to read the doc and understand the tradeoffs of the underlying solution.

@krillboi
Copy link

krillboi commented Jan 5, 2021

@slorber ah I understand, specifying the uploaded type as .md is a good first step I think and then a custom processor .mdx later maybe would make sense.

No worries, was just curious if anything was on the horizon 👍

@slorber
Copy link
Collaborator Author

slorber commented Jan 5, 2021

I'm constantly giving feedback to Crowdin support so that they can improve their product.

image

Here's just a few of my initial questions, but had many more recent ones 😉
https://gist.github.com/slorber/30643299196c7efa77084eec10c1c609

And they may implement some of my suggestions (they already did) but it may take time and I don't know when they will be released

@slorber
Copy link
Collaborator Author

slorber commented Jan 8, 2021

Hi there.

You'll find a i18n documentation preview on #4014 at https://deploy-preview-4014--docusaurus-2.netlify.app/classic/docs/next/i18n/introduction

Please give me feedbacks on the documentation PR if some things are not clear enough

slorber added a commit that referenced this pull request Jul 22, 2021
…3325)

* docs i18n initial poc

* docs i18n initial poc

* docs i18n initial poc

* docs i18n initial poc

* crowdin-v2 attempt

* fix source

* use crowdin env variable

* try to install crowdin on netlify

* try to install crowdin on netlify

* try to use crowdin jar directly

* try to curl the crowdin jar

* add java version cmd

* try to run crowdin jar in netlify

* fix translatedDocsDirPath

* fix loadContext issue due to site baseUrl not being modified in generted config file

* real validateLocalesFile

* add locale option to deploy command

* better LocalizationFile type

* create util getPluginI18nPath

* better core localization context loading code

* More explicit VersionMetadata type for localized docs folders

* Ability to translate blog posts with Crowdin!

* blog: refactor markdown loader + report broken links + try to get linkify working better

* upgrade crowdin config to upload all docs folder files except source code related files

* try to support translated pages

* make markdown pages translation work

* add write-translations cli command template

* fix site not  reloaded with correct options

* refactor a bit the read/write of @generated/i18n.json file

* Add <Translate> + translate() API + use it on the docusaurus homepage

* watch locale translation dir

* early POC of adding babel parsing for translation extraction

* fs.stat => pathExists

* add install:fast script

* TSC: noUnusedLocals false as it's already checked  by eslint

* POC of extracting translations from source code

* minor typo

* fix extracted key to code

* initial docs extracted translations

* stable plugin translations POC

* add crowdin commands

* quickfix for i18n deployment

* POC  of themeConfig translation

* add ability to have localized site without path prefix

* sidebar typo

* refactor translation system to output multiple translation files

* translate properly  the docs plugin

* improve theme classic translation

* rework translation extractor to handle new Chrome I18n JSON format (include id/description)

* writeTranslations: allow to pass locales cli arg

* fix ThemeConfig TS issues

* fix localizePath errors

* temporary add write-translations to netlify deploy preview

* complete example of french translated folder

* update fr folder

* remove all translations from repo

* minor translation  refactors

* fix all docs-related tests

* fix blog feed tests

* fix last blog tests

* refactor i18n context a bit, extract codeTranslations in an extra generated file

* improve @generated/i18n type

* fix some i18n todos

* minor refactor

* fix logo typing issue after merge

* move i18n.json to siteConfig instead

* try to fix windows CI build

* fix config test

* attempt to fix windows non-posix path

* increase v1 minify css jest timeout due to flaky test

* proper support for localizePath on windows

* remove non-functional install:fast

* docs, fix docsDirPathLocalized

* fix Docs i18n / md linkify issues

* ensure theme-classic swizzling will use "nextjs" sources (transpiled less aggressively, to make them human readable)

* fix some snapshots

* improve themeConfig translation code

* refactor a bit getPluginI18nPath

* readTranslationFileContent => ensure files are valid, fail fast

* fix versions tests

* add extractSourceCodeAstTranslations comments/resource links

* ignore eslint: packages/docusaurus-theme-classic/lib-next/

* fix windows CI with cross-env

* crowdin ignore .DS_Store

* improve writeTranslations + add exhaustive tests for translations.ts

* remove typo

* Wire currentLocale to algolia search

* improve i18n locale error

* Add tests for translationsExtractor.ts

* better code translation extraction regarding statically evaluable code

* fix typo

* fix typo

* improve theme-classic transpilation

* refactor  +  add i18n tests

* typo

* test new utils

* add missing snapshots

* fix snapshot

* blog onBrokenMarkdownLink

* add sidebars tests

* theme-classic index should now use ES modules

* tests for theme-classic translations

* useless comment

* add more translation tests

* simplify/cleanup writeTranslations

* try to fix Netlify fr deployment

* blog: test translated md is used during feed generation

* blog: better i18n tests regarding editUrl + md translation application

* more i18n tests for docs plugin

* more i18n tests for docs plugin

* Add tests for pages i18n

* polish docusaurus build i18n logs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed Signed Facebook CLA pr: new feature This PR adds a new API or behavior.
Projects
None yet
Development

Successfully merging this pull request may close these issues.