Help with multilingual markdown

[hrgdavor]

I would like to ask for advice.

I am in process of making some sort tutorial based on makdown documents for jscad.
I am wondering if you could give some ideas how to handle translations syncing.

If I have a markdown page with some small tutorial, it is easy to translate it once ...
but what would be a good practice to handle when original markdown doc changes and translations needs to be updated to match ?

[ryul1206]

Thank you for opening an interesting discussion.
When I first started this package, I also had an issue with syncing translations.

For translation synchronization, it is necessary to track changes, and we can broadly classify tracking methods into three categories.

1. Fully manual

Currently, I have designed this MMG-CLI to be minimal, so you need to synchronize it fully manually.
But even if you should find sections that require sync, you may not miss them if they are adjacent.
So, I put all translations into one *.base.md file. (This is the current style of MMG-CLI.)

However, it is only useful when you have few languages to translate and translate something alone.

2. Fully automated

Fully automated without *.base.md may require a new text editor to compare different languages' versions of all text sections.
For example, we can imagine an editor that parses the following files and manages those section id and ver automatically.

<!-- [section id: "8a1201" | ver: 00001] -->
# README.en.md
some intro
<!-- [section id: "2fe31a" | ver: 00211] -->
some description

<!-- [section id: "8a1201" | ver: 00002] -->
# README.fr.md
some intro
<!-- [section id: "2fe31a" | ver: 00211] -->
some description

I don't know this kind of editor exists.

3. Semi-automated

Suppose I update the MMG-CLI to track sections automatically. In that case, we can call this method semi-auto since we have to set section names manually.

Let assume a *.base.md like below:

# README.base.md
<!-- [en | section id: "intro"] -->
Introduntion in English
<!-- [fr | section id: "intro"] -->
Introduntion in French

Then we can get the hash values of each section and create a file similar to package-lock.json.
Let's call this section-lock.yaml.

- intro
    - en: 32ef11
    - fr: 91021a

This method is cumbersome to have to write down the section IDs one by one. It would be nice if there were a way to generate the section ids automatically, but I don't have a good idea right now.

First of all, I will remember that the semi-auto sync feature for the next new version.
Due to my busy schedule until September, I may start later.

[hrgdavor]

well, maybe generating section id's would not be that hard. If using UUID, there would be no worries about uniqueness. UUID also embeds timestamp by design

Also along with id the hash value could be generated

Files anatomy:

• base language file contains own hash
• translation file contains hash of the translation that it is in sync with
• translation file contains copy of the en section it is in sync with

Instead of comparing all languages, my idea is that in my case EN is primary, and others are checked against it

• does the language have same sections
• does section hash match section hash in EN

.. I am thinking of a silly idea ... a bit brainstorming so bare with me :)

step 1 create

let's start with EN, and it will only contain original, also hash values are manual to be descriptive. note:

readme.md

<!--section-->
Introduction in English

then automatic part takes over and convert readme.md

<!-- [en | id:"UUID1" hash:"en-hash-1" -->
Introduction in English

then automatic part takes over and generates readme.fr.md (fr.hash is empty to mark it not translated yet)

<!-- [en | id:"UUID1" hash:"en-hash-1" -->
Introduction in English
<!-- [fr | id:"UUID1" hash:"" -->
Introduction in English

and readme.de.md (de.hash is empty to mark it not translated yet)

<!-- [en | id:"UUID1" hash:"en-hash-1" -->
Introduction in English
<!-- [de | id:"UUID1" hash:"" -->
Introduction in English

It may seem silly having copy of english in each translation, but it has a nice use-case later on

step 2 sync translations

user translates the text in readme.fr.md and updates the hash to mark translated

<!-- [en | id:"UUID1" hash:"en-hash-1" -->
Introduction in English
<!-- [fr | id:"UUID1" hash:"en-hash-1" -->
Introduction in French

and readme.de.md

<!-- [en | id:"UUID1" hash:"en-hash-1" -->
Introduction in English
<!-- [de | id:"UUID1" hash:"en-hash-1" -->
Introduction in German

step 3 update en

let's start with EN, and it will only contain original, also hash names are manual to be descriptive. note:

update readme.md

<!-- [en | id:"UUID1" hash:"en-hash-2" -->
Updated Introduction in English

readme.fr.md stays the same

<!-- [en | id:"UUID1" hash:"en-hash-1" -->
Introduction in English
<!-- [fr | id:"UUID1" hash:"en-hash-1" -->
Introduction in English

and readme.de.md stays the same

<!-- [en | id:"UUID1" hash:"en-hash-1" -->
Introduction in English
<!-- [de | id:"UUID1" hash:"en-hash-1" -->
Introduction in German

What we have now is situation where code could be inspected and this info displayed

• fr out of date, section:UUID1 - Updated Introduction in English
• de out of date, section:UUID1 - Updated Introduction in English

After this an editor could be made that takes new en|section:UUID1 in readme.md and old en|section:UUID1 in readme.fr.md and shows diff to the user while user is updating the french translation

When user is done with sync of the section, editor can then copy current en|section:UUID1 into readme.fr.md effectively marking it up-to-date.

final thoughts

I am actually happy with this idea, and it came to me while writing reply to you. If you are interested, I could make a prototype editor that would streamline this

• show diff
• allow marking synced by clicking a button without user doing copy/paste of current english version of the section

[ryul1206]

Having UUIDs and copies of primary is a really great idea. Thank you.
With your idea, *.base.md is no longer required, and users can easily manage their translations.

I am very interested in your idea. But I have a question.
How about storing the hashes for each language as well?
Rarely, a user can forget the dedicated editor and update a section in a non-primary language before the primary changed.
(Actually, I made a similar mistake when I use MMG-CLI.)

[hrgdavor]

Yes, storing hashes for each section in secondary languages as well would definitely help catch case you have mentioned.

There is no real reason not to store the hash, esp. with the case you mention.

I googled javascript diff, and https://github.com/kpdecker/jsdiff

I already started making a mockup of editor that uses jsdiff and codemiror(with markdown support)

I am ok with your syntax for marking section for this mockup editor, but will likely for myself go in direction of having the section marks in the same line as the section title, (for me it is simpler to parse).

starting file:

# section title 
bla bla

## sub-section title

script would add section markers with syntax maybe something like this

# section title {{ "id":"UUID1", "hash":"hash1" }}
bla bla

## sub-section title {{ "id":"UUID2", "hash":"hash2" }}

fr translation file:

# section title {{ "id":"UUID1", "hash":"hash1" }}
bla bla

# section title fr {{ "parent":"UUID1", "hash":"hash3" }}
bla bla

## sub-section title {{ "id":"UUID2", "hash":"hash2" }}

## sub-section title fr {{ "parent":"UUID2", "hash":"hash4" }}

with files separated in folders, base md files in root, name of the language is not important to mark in the markup inside the md files, hence id,parent as an idea for property names

Would you even consider a such big departure from your current syntax and parsers, and in that case make it a new major version (converters could be made for migration) .. or maybe we open separate repo for this fancy version I am fantasizing about :D

[ryul1206]

A separate repo looks better because the approach is completely different.
I am not good at javascript. But I hope to contribute to your awesome repository. 👍
If this new method gets an editor and CLI, I'll link your repo in the header of README of this repo to recommend your new style.
And here will be passively maintained.

[hrgdavor]

@ryul1206 thanks for enthusiastic support, hopefully you will participate in some extent

• to test CLI and editor features
• or propose more improvements
• maybe even use it at some point :)

I have to take a little break from side projects, and leaving self reminder a month from now, but I had to put the ides into writing to clear it up from my brain :)

the checklist for initial release grew pretty quickly
https://github.com/hrgdavor/multilingual-markdown/

here is just the checklist separated from the linked document

TODO for intital release

• define inital format
• LIB: hashing rules
• LIB: uuid lib / func
• LIB: choose hash - MD5(cross platform and cross language) not for security, it is just helping detect changes in text
• create node project (package.json)
• CLI: create script with UUID and md5 included cli tutorial used
• LIB: setup first test (test sample text and md5 of it)
• LIB: sections parse - read md into obj model
• LIB: sections stringify - write md from obj model
• Editor: basic edit base file (drag and drop 1 file) sections outline, click to edit title and text separately one section at the time
• LIB: initialize sections (add uuids and hashes)
• Editor: publish as github page (docs folder) test version
• CLI: initialize base file (add uuids and hashes) mulmd init docs/editor/parameters.md
• Editor: initialize base file (add uuids and hashes)
• CLI: generate file for translation from base file mulmd init file.md file.fr.md
• Editor: generate file for translation from base file
• LIB: translation info (details)
  • Case 1 - missing-translation
  • Case 2 - missing-original
  • Case 3 - initialized
  • Case 4 - initialized-dirty
  • Case 5 - partial
  • Case 6 - partial-dirty
  • Case 7 - translated-dirty
  • Case 8 - translated
• Editor: basic edit translation file (drag and drop 2 files, together or 1 by 1)
• Editor: edit translation file: with diff of changes in base file (drag and drop 2 files, together or 1 by 1)
• CLI: sync translation by injecting missing sections
• Editor: sync translation by injecting missing sections
• CLI-pre-commit-hook: script spacification so it can be used to block unfinished edits
• CLI: allow automatic pushing code-block changes across translations (if it was unchanged in the translation)
• Editor: allow automatic pushing code-block changes across translations (if it was unchanged in the translation)
• Editor: electron app
• CLI-info: specification of the script (options and actions and infos that it can give)
• CLI-info: xxx
• CLI-info: yyy