Checking the source dictionary

[jaumeortola]

The first version of the source dictionary is here: https://github.com/languagetool-org/english-pos-dict/tree/main/src-dict

I will be adding some comments and ideas here. We can open new issues for some parts of the work.

• We can proceed to separate the entries by groups: the ones that don't need review, the ones that need some manual review, and so on. For example:
  • words in all spelling dicts and tagged -> no need to review
  • words in all spelling dicts but not tagged -> maybe they can be tagged easily
  • words in US and GB and tagged -> maybe they can be accepted by all variants?
  • ...
• Check that variant labels are coherent with en-US-GB.txt (use scripting).
• Some sets of entries look suspicious: untagged words in GB with some prefixes (mis-, out-, over-, re-, under-) seem nonsense words. The same with some affixes (see: survivorshipably... survivorshipry).
• Words with the tag us-large come from a Hunspell US dictionary that we didn't use until now. It is mentioned in Explore differences between en-US and en-US-large #2
• We are using a simplified format for regular verbs: recharge=verb=all. (We use a few rules to cover more cases of regular verbs. See here). It would be useful to have something similar for nouns: a simple and quick way to tag a noun. We would need to define the format, and ways to write exceptions.
• What sources we consider authoritative to determine if a word is GB or US? And AU, CA, ZA, NZ? Are there dictionaries for those variants?

[jaumeortola]

Separating the dict in src-clean.txt (accepted entries), src-pending.txt and src-discarded.txt. 551de86

Done:

• tagged words in all spelling dicts -> moved to src-clean
• tagged words in GB and US -> moved to src-clean, variant modified to "all"
• untagged words in GB with some prefixes (mis-, out-, over-, re-, under-, in- + uppercase) -> moved to src-discarded

[AzadehSafakish]

Next steps, perhaps in this order(?)

• simplified format (shorthands) for dictionary entries #6
• Check that variant labels are coherent with en-US-GB.txt #7
• Tag words that are currently untagged  #8
• Establish authoritative sources for each variant (or at least US and GB) #9