Rasa 2.0.x fails to use stories where the previous events in the current conversation are different to the previous events in a story when making predictions

[Devocado]

Rasa version: 2.0.3

Python version: 3.8.2

Operating system Ubuntu 20.04

Issue:
It appears that Rasa 2.0.x fails to use stories where the previous events in the current conversation are different to the previous events in the story when making predictions where Rasa 1.10.x does not

have create a demo repository to reproduce the issue here: https://github.com/Devocado/reproduce-stories-bug-rasa2 which demonstrates two instances of this.

• In the first case, circling back to a certain point in a story (demonstrated in the first test) ends in Rasa 2.0.x falling back, while Rasa 1.10.x correctly follows the same path as before.
• In the second case, where a story does not overlap with other stories Rasa 2.0.x will not predict the correct actions when the intent which begins the story is triggered. Again, the second test fails in Rasa 2.0.x while passing in rasa 1.10.x

It also gives the steps to reproduce this behaviour, which for completeness I'll add it here too:

Steps to reproduce:

• Train the assistant in Rasa 1.10.14.
• Run the tests

All tests will pass.

• Train the assistant in Rasa 2.0.x
• Run the tests

The first two tests will fail

[sara-tagger]

Thanks for raising this issue, @tmbo will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[wochinge]

@Ghostvv Do you have an idea?

[Ghostvv]

The only difference to 1.10 that come to mind is that we changed the padding logic from padding from the beginning to padding from the end. The fact that TED cannot learn to predict create_shipment_form is not surprising. The surprising part is that 1.10 ted can learn that

[Ghostvv]

Hi, I have a theory why performance might drop. Could you please try mcfly-ted branch of rasa with your data?