Cycle breaking in 2.x not working as expected / different from 1.x

[indam23]

Rasa version: 2.3.0

Python version: 3.7.9

Issue:

Given the stories with a cyclical checkpoint below, in rasa 1.10.x, 14 trackers are found, while in Rasa 2.3.0, 16 trackers are found. The increase in trackers in 2.3 vs 1.10 remains with higher numbers of max_history

Based on debug plots it appears that cycle breaking is not working as expected.

Error (including full traceback): rasa==2.3.0

2021-02-17 19:43:31 DEBUG rasa.shared.core.generator - Data generation rounds finished. 2021-02-17 19:43:31 DEBUG rasa.shared.core.generator - Found 0 unused checkpoints 2021-02-17 19:43:31 DEBUG rasa.shared.core.generator - Found 16 training trackers. 2021-02-17 19:43:31 DEBUG rasa.shared.core.generator - No rules found. 2021-02-17 19:43:31 DEBUG rasa.core.agent - Agent trainer got kwargs: {} 2021-02-17 19:43:31 DEBUG rasa.core.featurizers.tracker_featurizers - Creating states and action examples from collected trackers (by MaxHistoryTrackerFeaturizer(SingleStateFeaturizer))... Processed trackers: 100%|███████████████████████████████████████| 16/16 00:00<00:00, 551.10it/s, # actions=40 2021-02-17 19:43:31 DEBUG rasa.core.featurizers.tracker_featurizers - Created 40 action examples.

rasa==1.10.3

2021-02-17 19:44:41 DEBUG rasa.core.training.generator - Data generation rounds finished. 2021-02-17 19:44:41 DEBUG rasa.core.training.generator - Found 0 unused checkpoints 2021-02-17 19:44:41 DEBUG rasa.core.training.generator - Found 14 training trackers. 2021-02-17 19:44:41 DEBUG rasa.core.agent - Agent trainer got kwargs: {} 2021-02-17 19:44:41 DEBUG rasa.core.featurizers - Creating states and action examples from collected trackers (by MaxHistoryTrackerFeaturizer(LabelTokenizerSingleStateFeaturizer))... Processed trackers: 100%|███████████████████████████████████████| 14/14 00:00<00:00, 437.82it/s, # actions=40 2021-02-17 19:44:41 DEBUG rasa.core.featurizers - Created 40 action examples.

Command or request that led to error:

rasa train core --debug --augmentation 0 --debug-plots

Stories:

## intro

*   intro

*   action_intro
     > first_checkpoint

1\. 1\. first
 > first_checkpoint

*   slot {"random": "true"}
*   action_continue
     > cycle_checkpoint

1\. 1\. cycle
 > cycle_checkpoint

*   inform {"entity1":"something"}

 OR inform

 {"entity2":"somethingelse"} 

*   action_check
*   slot {"pass": "true"}
*   action_cycle
*   action_cycle_2
     > resolution

1\. 1\. break
 > cycle_checkpoint

*   inform {"entity1":"something"}

 OR inform

 {"entity2":"somethingelse"} 

*   action_check
*   slot {"pass": null}
*   action_break_cycle
     > cycle_checkpoint

1\. 1\. resolve
 > resolution

*   resolve

*   action_resolve

Content of configuration file (config.yml) (if relevant):

policies:

*   name: TEDPolicy
     epochs: 1
     max_history: 4
     pipeline:
     language:

Content of domain file (domain.yml) (if relevant):

intents:

*   intro
*   inform
*   resolve

entities:

*   entity1
*   entity2

actions:

*   action_intro
*   action_continue
*   action_step
*   action_check
*   action_cycle
*   action_cycle_2
*   action_break_cycle
*   action_resolve

slots:
 pass:
 type: categorical 
 values:

*   "true"
*   "other"
     random:
     type: categorical
     values:
*   "true"
     

Debug plot for 1.10:

Debug plot for 2.3:

[Ghostvv]

Exalate commented:

Ghostvv commented:

I think it might be related to

rasa/rasa/shared/core/training_data/structures.py

Line 107 in e0163e7

self.id = "{}_{}".format(STEP_COUNT, uuid.uuid4().hex)

[Ghostvv]

Exalate commented:

Ghostvv commented:

most probably, both graphs are correct. Just global step changed, so ordering behaves differently.

Though the fact that the second graph still has a cycle, looks suspicious

[Ghostvv]

Exalate commented:

Ghostvv commented:

when I substituted self.id = uuid.uuid4().hex, it sometimes creates one graph, sometimes another

[stale]

Exalate commented:

stale[bot] commented:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sync-by-unito]

➤ Maxime Verger commented:

💡 Heads up! We're moving issues to Jira: https://rasa-open-source.atlassian.net/browse/OSS.

From now on, this Jira board is the place where you can browse (without an account) and create issues (you'll need a free Jira account for that). This GitHub issue has already been migrated to Jira and will be closed on January 9th, 2023. Do not forget to subscribe to the corresponding Jira issue!

➡️ More information in the forum: https://forum.rasa.com/t/migration-of-rasa-oss-issues-to-jira/56569.