Refactor langs list

[kolakachi]

This starts the process of changing from 'language' to 'model' when we're talking about OCR engine options. It makes the following changes:

• Move the main config file from /public/langs.json to /public/models.json`.
• Restructure that file to have the top level keys be engine IDs (google, tesseract, and transkribus), and then 'model IDs' below those (many of these look like language codes, but not all of them).
• Each model entry in models.json is keyed by the model ID, and can contain entries of languages, title, htr, and line (the latter two being for Transcribus only). The languages entries is an array of ISO639 language codes, by which users will be able to browse the models (i.e. it's a rough estimation of which languages will find which models most useful; mul will be a reasonable value for some, it sounds like). title is needed if the model ID does not match a language that Intuition knows about; if it does match then that language's name will be used for the title.
• Change a few methods in EngineBase to use the new terminology (but not all; subsequent patches will change the rest).
• Update some tests to work with the new structure.

Bug: T358433
Bug: T330061

[sohomdatta1]

nit: Do you have a script that generates ./models.json it might be useful to include that script as well.

[kolakachi]

nit: Do you have a script that generates ./models.json it might be useful to include that script as well.

I don't have a script at the moment.

[samwilson]

I guess the old langs.json is kept here for backwards compatibility? Could it perhaps be build programatically from the new models.json? (I.e. a new route added, rather than serving the static file.)

Also, it'd be great to document the schema of models.json a bit more clearly: basically, it's top level is engines, and in each is a list of models and each has at least a title and a set of languages. But are those languages given as ISO639 codes, Wikimedia lang codes, or something else? I think they should be standardized. For example, uzb_cyrl isn't a language (Uzbek is written in three scripts, I think? Latin, Cyrillic, and Arabic, so the OCR tool should be clear whether it's listing things by language or by script; I imagine the former is more common and useful although results in perhaps confusing states like trying to run a Cyrillic Uzbek text through a Arabic Uzbek model).

I guess the crucial things here are to

• be able to group by language,
• have unique identifiers for models (although perhaps re-use of identifiers between engines isn't a problem?), and
• be able to keep the list up to date easily.

[kolakachi]

I guess the old langs.json is kept here for backwards compatibility? Could it perhaps be build programatically from the new models.json? (I.e. a new route added, rather than serving the static file.)

Also, it'd be great to document the schema of models.json a bit more clearly: basically, it's top level is engines, and in each is a list of models and each has at least a title and a set of languages. But are those languages given as ISO639 codes, Wikimedia lang codes, or something else? I think they should be standardized. For example, uzb_cyrl isn't a language (Uzbek is written in three scripts, I think? Latin, Cyrillic, and Arabic, so the OCR tool should be clear whether it's listing things by language or by script; I imagine the former is more common and useful although results in perhaps confusing states like trying to run a Cyrillic Uzbek text through a Arabic Uzbek model).

I guess the crucial things here are to

• be able to group by language,
• have unique identifiers for models (although perhaps re-use of identifiers between engines isn't a problem?), and
• be able to keep the list up to date easily.

• The langs.json was left because we haven't fully transitioned into the new structure.
• models.json wasn't built programmatically, currently langs.json and models.json are both stand alone not depending on each other
• The languages are given as Wikimedia lang codes (they were all gotten from the prev langs.json)
• Thank you for pointing out the uzb_cyrl the intention was to list languages and not scripts, do we have other scripts? The focus was to change the structure, all langs/scripts were gotten from langs.json.

[stweil]

I renamed this Tesseract model from frk to deu_latf. It will take some time until all Linux distributions notice and follow this change, but they will. Therefore "de-frk" might not be the best choice.

[stweil]

"langs" is not the right word nowadays. We talk about "models". In some cases these models correspond to languages, in other cases they correspond to script, and future models will be just AI models which can read text but neither correspond to languages nor to scripts because they can read any text.

[samwilson]

I think I've brought this up to date, and it's perhaps ready to go. I'll leave it a bit in case anyone wants to review.

[Parthiv-M]

Deleted a comment by mistake, but I see some places where the model titles have been replaced by ភាសាខ្មែរ (square boxes) instead of UTF characters. These should be changed to their respective legible model title names

Here is an (hopefully) exhaustive list

• google: bo, dz, km, kn, ko, my, te
• tesseract: bod, dzo, khm, kan, kor, mya, tel

[samwilson]

I see some places where the model titles have been replaced by ភាសាខ្មែរ (square boxes) instead of UTF characters. These should be changed to their respective legible model title names

Are you missing some fonts or something? It looks okay to me, including your reply:

[Parthiv-M]

Indeed, I can see it properly on my phone, not on my laptop. That's weird!

Rest of it lgtm.