Tokenizers don't split words into sub-words

[tabergma]

Proposed changes:
To avoid the problem of our entity extractors predicting entity labels for just a part of the words, we introduced a cleaning method after the prediction was done. However, we should avoid the incorrect prediction in the first place.
To achieve this we will not tokenize words into sub-words anymore. We take the mean feature vectors of the sub-words as the feature vector of the word.

fixes #5755
closes https://github.com/RasaHQ/research/issues/83

Status (please check what you already did):

• added some tests for the functionality
• updated the documentation
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)

[dakshvar22]

@tabergma Do we have any performance numbers with and without this fix?

[tabergma]

Yes, I tested it on carbon bot and the results were the same (77.2% vs 77.8% for entities - branch composite entities). Also, verified locally on the smaller example bots, that the prediction are not on sub-tokens anymore.

[tabergma]

Results on Sara (2 fold cross validation):
master - micro f1: 83.8
fix-tokenization - micro f1: 85.5

[dakshvar22]

Tested on carbon bot with BERT as well. Performance for entities improves by 2 points 🚀
Just one comment for an additional test.

[dakshvar22]

Looks good! 🌟