fix for 6340 bug

[praneethgb]

Proposed changes:

If two entities are separated by a single space with U- tags, they are currently detected as one entity.

Example:
my ingredients are eggs lemon juice and milk results in
{'entity': 'eggs lemon juice', 'entity':: 'milk', 'extractor': 'DIETClassifier'}

This PR fixes this. Entities separated by a single space and BILOU tags will be listed as individual entities.
Related post: #6340

Status (please check what you already did):

• added some tests for the functionality
• updated the documentation
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)

[CLAassistant]

All committers have signed the CLA.

[sara-tagger]

Thanks for submitting a pull request 🚀 @melindaloubser1 will take a look at it as soon as possible ✨

[tabergma]

Looks great! Thanks 🚀

[tabergma]

Looks great! Thanks for addressing my comments 💯

[tabergma]

@praneethgb You need to sign the Contributor License Agreement. Otherwise I am not able to merge this PR. Thanks.

[praneethgb]

@tabergma, Signed. Thanks.

[elenaviter]

When use roles, there's still a problem to have the entity tokens B-entity ... L-entity to be collected as a whole because of the condition:

(

rasa/rasa/nlu/extractors/extractor.py

Line 152 in cfe49f4

group_or_role_changed = (

)
group_or_role_changed = (last_group_tag != current_group_tag or last_role_tag != current_role_tag)
Considering the role and the entity type are both associated with the same token, for me, this works instead

group_or_role_changed = (
                    last_group_tag != current_group_tag or bilou_utils.tag_without_prefix(last_role_tag) != bilou_utils.tag_without_prefix(current_role_tag)
            )

And the rest of the logic to reconstruct the compound entity based on bilou labels of the entity tag ensures the integrity of the approach.

Also the "current_role_tag" under construction of the entity -> bilou_utils.tag_without_prefix(current_role_tag) :

entity = self._create_new_entity(
                    list(tags.keys()),
                    current_entity_tag,
                    current_group_tag,
                    bilou_utils.tag_without_prefix(current_role_tag),
                    token,
                    idx,
                    confidences,
                )

[wochinge]

@tabergma have you seen the message from above?

[tabergma]

It is fixed on master (#6577) and I just created a PR for a minor release.