Isolate single language processing library

[msarahan]

• Gensim
• NLTK

Reimplement internally

[msarahan]

Current usages:

Gensim: LDA (was used also for stopwords, but we don't really need that)
NLTK: collocation tokenizer
TextBlob: entities tokenizer (depends in turn on NLTK)

[gpfreitas]

@msarahan , did you guys consider spacy? If so, any thoughts on it?

Spacy has a big problem though (at least for now): it's for English text only. :/

[msarahan]

No, we didn't. Sorry, other than being aware of it by name, I don't know anything about it.

[gpfreitas]

No worries. I am trying to find out more about this library. Very few people seem to know it well. Thanks, Mike.