EntityExtractor in v1.10.0 yield wrong entity value for language without spaces

[tomgun132]

Rasa version:
v1.10.0
Python version:
3.7
Operating system:
Windows, Linux

I'm not sure if this should be a bug or feature request, but here I go.

Description of Problem:
For language that doesn't have space that separate words, any EntityExtractor class will return the whole sentence as the value. When I see what changes have been made into that class, I see that there are few new functions that make up into clean_up_entities function. The problem appears from that function as I observe the process from the SpacyEntityExtractor. Based on the comment on function _token_clusters: "two tokens belong to the same word if there is no other character between them", I assume that all the process of the clean_up_entities function will merge whole input if there is no space between words.

Overview of the Solution:
I haven't really looked into the sub-words problem that requires this clean_up_entities function, so I can't really offer any solution except that I will still use v1.9.x. However, if this function will still be kept as it is for the future release, I either need to remake all of my training data to have space and preprocess the input to Rasa chatbot with custom components, or still stay on hold with version 1.9.x

Examples
On previous version:

>>> model.parse('木村拓哉は誰ですか')
'entities': [{'entity': 'Person',
   'value': '木村拓哉',
   'start': 0,
   'confidence': None,
   'end': 4,
   'extractor': 'SpacyEntityExtractor'}]

on version 1.10.0

>>> model.parse('木村拓哉は誰ですか')
'entities': [{'entity': 'Person',
   'value': '木村拓哉は誰ですか',
   'start': 0,
   'confidence': None,
   'end': 9,
   'extractor': 'SpacyEntityExtractor'}]

[tabergma]

Thanks @tomgun132 for raising the issue. I'm already working on a fix. Should hopefully be in the next patch release.

EDIT: our fix will be in the next minor release

[tomgun132]

Hello @tabergma,
Thank you very much for the quick response and I really appreciate it that you're already working on it.