Add support for other languages

[ajdapretnar]

Text version

3.5.dev-

Orange version

0.2.5

Expected behavior

Orange supports many key languages.

Actual behavior

No support for Latin, (old) Greek, Polish... Poor support for French, German, Spanish, Portuguese. Think about Chinese, Hindu and Arabic as well.

Steps to reproduce the behavior

Additional info (worksheets, data, screenshots, ...)

[gengyabc]

I wish this addon can support Chinese, and you may use https://github.com/fxsjy/jieba or https://github.com/hankcs/HanLP as the backend

[ajdapretnar]

It does support Chinese (UDPipe lemmatization and sentiment analysis). Word embedding is upcoming.

[ajdapretnar]

Ah, do you mean for tokenization? We have absolutely no clue how to work with Chinese, so someone else would have to set this up (especially for tests). HanLP doesn't have English documentation, but Jieba seems nice!

[gengyabc]

English version of HanLP is here: https://github.com/hankcs/HanLP/tree/master.

I have forked this repo, translated some widgets to chinese and added some Chinese tokenization support. but I have no idea of what nlp is, so I don't know how to do next.

(I am a teacher who teaches machine learning in China, and my students know little English, so I translated it)

[ajdapretnar]

Ok, I've checked both.
HanLP is in my opinion a no-go, because it depends on Tensorflow, which is (from what I know) an big dependency. Also, Text already have NLTK, which covers most of HanLP features.

From what I gathered, the only real issue in Chinese text processing is word segmentation. So we would need a specialized Chinese segmenter. This could be added to the new Preprocess Text as a separate option.
We could go with Jieba, but if possible, I'd like to avoid another dependency. Looks like there's a segmenter available also in NLTK. Perhaps we could check that first, then fallback to Jieba if NLTK is insufficient.

I would close this issue, as it is too broad and open a separate specific one.

Continued in #536.