add e2e docs

docs/docs/components.mdx

@@ -1429,7 +1429,7 @@ However, additional parameters exist that can be adapted.
 +=================================+==================+==============================================================+
 | hidden_layers_sizes             | text: []         | Hidden layer sizes for layers before the embedding layers    |
 |                                 | label: []        | for user messages and labels. The number of hidden layers is |
-|                                 |                  | equal to the length of the corresponding.                    |
+|                                 |                  | equal to the length of the corresponding list.                    |
 +---------------------------------+------------------+--------------------------------------------------------------+
 | share_hidden_layers             | False            | Whether to share the hidden layer weights between user       |
 |                                 |                  | messages and labels.                                         |

docs/docs/stories.mdx

@@ -18,23 +18,23 @@ Here's an example of a dialogue in the Rasa story format:
 stories:
 - story: collect restaurant booking info  # name of the story - just for debugging
   steps:
-  - intent: greet                         # user message with no entities
-  - action: utter_ask_howcanhelp
-  - intent: inform                        # user message with no entities
+  - intent: greet                          # user message with no entities
+  - action: utter_ask_howcanhelp           # action that the bot should execute
+  - intent: inform                         # user message with entities
     entities:
     - location: "rome"
     - price: "cheap"
-  - action: utter_on_it                  # action that the bot should execute
+  - bot: On it                             # actual text that bot can output
   - action: utter_ask_cuisine
-  - intent: inform
-    entities:
-    - cuisine: "spanish"
+  - user: I would like [spanish](cuisine). # actual text that user input
   - action: utter_ask_num_people
 ```
 
 ### User Messages
 
-While writing stories, you do not have to deal with the specific contents of
+While writing stories, you have two options.
+
+1) You do not have to deal with the specific contents of
 the messages that the users send. Instead, you can take advantage of the output
 from the NLU pipeline, which lets you use just the combination of an intent and
 entities to refer to all the possible messages the users can send to mean the
@@ -45,6 +45,19 @@ to predict the next action based on a *combination* of both the intent and
 entities (you can, however, change this behavior using the
 [use_entities](./domain.mdx#ignoring-entities-for-certain-intents) attribute).
 
+2) You do not have to deal with the specific
+intents of the messages that are extracted by the NLU pipeline. Instead, you can simply put
+the text of the user message.
+We call it end-to-end training because actual text is used for predictions.
+
+:::caution
+This feature is experimental.
+We introduce experimental features to get feedback from our community, so we encourage you to try it out!
+However, the functionality might be changed or removed in the future.
+If you have feedback (positive or negative) please share it with us on the [Rasa Forum](https://forum.rasa.com).
+
+:::
+
 ### Actions
 
 All actions executed by the bot, including [responses](./responses.mdx) are listed
@@ -54,6 +67,15 @@ You can use a response from your domain as an action by listing it as one
 in a story. Similarly, you can indicate that a story should call a custom action by including
 the name of the custom action from the `actions` list in your domain.
 
+You can also put bot utterance directly in the stories.
+
+:::caution
+This feature is experimental.
+We introduce experimental features to get feedback from our community, so we encourage you to try it out!
+However, the functionality might be changed or removed in the future.
+If you have feedback (positive or negative) please share it with us on the [Rasa Forum](https://forum.rasa.com).
+
+:::
 
 ### Events
 

docs/docs/training-data-format.mdx

@@ -380,26 +380,26 @@ stories:
 
 Each step can be one of the following:
 
-  - A [user message](#user-messages), represented by **intent** and **entities**.
-  - An [or statement](#or-statement), which includes two or more user messages under it
-  - A bot [action](#actions)
-  - A [form](#forms)
-  - A [slot was set](#slots) event
-  - A [checkpoint](#checkpoints), which connects the story to another story
+  - A [user message](#user-messages), represented by **intent**, **entities** or **user**.
+  - An [or statement](#or-statement), which includes two or more user messages under it.
+  - A bot [action](#actions), represented by **action** or **bot**.
+  - A [form](#forms).
+  - A [slot was set](#slots) event.
+  - A [checkpoint](#checkpoints), which connects the story to another story.
 
 
 #### User Messages
 
-All user messages are specified with the `intent:`
-key and an optional `entities:` key.
+User messages are specified with the `intent:`
+key and an optional `entities:` key or with `user:` key.
 
-While writing stories, you do not have to deal with the specific
+If you use `intent:` key while writing stories, you do not have to deal with the specific
 contents of the messages that the users send. Instead, you can take
 advantage of the output from the NLU pipeline, which uses
 a combination of an intent and entities to refer to all possible
 messages the users can send with the same meaning.
 
-User messages follow the format:
+These user messages follow the format:
 
 ```yaml-rasa
 stories:
@@ -428,6 +428,41 @@ policies learn to predict the next action based on a *combination* of
 both the intent and entities (you can, however, change this behavior
 using the [`use_entities`](#entities) attribute).
 
+If you use `user:` key while writing stories, you do not have to deal with the specific
+intents of the messages that are extracted by the NLU pipeline. Instead, you can simply put
+the text of the user message.
+We call it end-to-end training because actual text is used for predictions.
+
+:::caution
+This feature is experimental.
+We introduce experimental features to get feedback from our community, so we encourage you to try it out!
+However, the functionality might be changed or removed in the future.
+If you have feedback (positive or negative) please share it with us on the [Rasa Forum](https://forum.rasa.com).
+
+:::
+
+These end-to-end user messages follow the format:
+
+```yaml-rasa
+stories:
+- story: user message structure
+  steps:
+    - user: the actual text of the user message  # Required
+```
+
+In addition, you can add entity tags that will be extracted by end-to-end policy.
+The syntax for entity tags is the same as in
+[the entities in the NLU training data](./training-data-format.mdx#entities)
+For example, to represent the sentence
+`I want to check my credit balance`, where `credit` is an entity:
+
+```yaml-rasa {4}
+stories:
+- story: story with entities
+  steps:
+  - user: I want to check my [credit](account_type) balance
+  - action: action_credit_account_balance
+```
 
 #### Actions
 
@@ -458,6 +493,35 @@ stories:
   - action: action_store_feedback
 ```
 
+If you want to put bot utterance directly in the stories, use `bot:` key followed by
+the text that you want your bot to say
+
+:::caution
+This feature is experimental.
+We introduce experimental features to get feedback from our community, so we encourage you to try it out!
+However, the functionality might be changed or removed in the future.
+If you have feedback (positive or negative) please share it with us on the [Rasa Forum](https://forum.rasa.com).
+
+:::
+
+```yaml-rasa {5}
+stories:
+- story: story with an end-to-end response
+  steps:
+  - intent: greet
+  - bot: Hello
+```
+
+You can also have full end-to-end story:
+
+```yaml-rasa {4-5}
+stories:
+- story: full end-to-end story
+  steps:
+  - user: Hello
+  - bot: Hello
+```
+
 #### Forms
 
 

rasa/core/policies/ted_policy.py

@@ -165,9 +165,9 @@ class TEDPolicy(Policy):
         CONCAT_DIMENSION: {TEXT: 128, ACTION_TEXT: 128, f"{LABEL}_{ACTION_TEXT}": 128},
         ENCODING_DIMENSION: 50,
         # Number of units in sequence transformer
-        TRANSFORMER_SIZE: 128,
+        f"{SEQUENCE}_{TRANSFORMER_SIZE}": 128,
         # Number of sequence transformer layers
-        NUM_TRANSFORMER_LAYERS: 1,
+        f"{SEQUENCE}_{NUM_TRANSFORMER_LAYERS}": 1,
         # Number of units in dialogue transformer
         f"{DIALOGUE}_{TRANSFORMER_SIZE}": 128,
         # Number of dialogue transformer layers
@@ -233,7 +233,7 @@ class TEDPolicy(Policy):
         # Dropout rate for embedding layers of label, e.g. action, features.
         DROP_RATE_LABEL: 0.0,
         # Dropout rate for attention.
-        DROP_RATE_ATTENTION: 0,
+        DROP_RATE_ATTENTION: 0.0,
         # Sparsity of the weights in dense layers
         WEIGHT_SPARSITY: 0.8,
         # If 'True' apply dropout to sparse input tensors
@@ -312,7 +312,7 @@ def _load_params(self, **kwargs: Dict[Text, Any]) -> None:
         self.config = copy.deepcopy(self.defaults)
         self.config.update(kwargs)
 
-        self.config = train_utils.check_deprecated_options(self.config)
+        self.config = train_utils.check_core_deprecated_options(self.config)
 
         self.config = train_utils.update_similarity_type(self.config)
         self.config = train_utils.update_evaluation_parameters(self.config)
@@ -850,13 +850,13 @@ def _prepare_layers(self) -> None:
         for name in self.data_signature.keys():
             self._prepare_sparse_dense_layer_for(name, self.data_signature)
             if name in SEQUENCE_FEATURES_TO_ENCODE:
-                self._prepare_sequence_layers(name)
+                self._prepare_sequence_layers(name, prefix=SEQUENCE)
             self._prepare_encoding_layers(name)
 
         for name in self.label_signature.keys():
             self._prepare_sparse_dense_layer_for(name, self.label_signature)
             if name in SEQUENCE_FEATURES_TO_ENCODE:
-                self._prepare_sequence_layers(name)
+                self._prepare_sequence_layers(name, prefix=SEQUENCE)
             self._prepare_encoding_layers(name)
 
         self._prepare_transformer_layer(

rasa/utils/tensorflow/models.py

@@ -834,12 +834,22 @@ def _prepare_input_layers(self, name: Text) -> None:
                 prefix="concat_layer",
             )
 
-    def _prepare_sequence_layers(self, name: Text) -> None:
+    def _prepare_sequence_layers(
+        self, name: Text, prefix: Optional[Text] = None
+    ) -> None:
         self._prepare_input_layers(name)
+
+        if prefix:
+            size_key = f"{prefix}_{TRANSFORMER_SIZE}"
+            num_layers_key = f"{prefix}_{NUM_TRANSFORMER_LAYERS}"
+        else:
+            size_key = TRANSFORMER_SIZE
+            num_layers_key = NUM_TRANSFORMER_LAYERS
+
         self._prepare_transformer_layer(
             name,
-            self.config[NUM_TRANSFORMER_LAYERS],
-            self.config[TRANSFORMER_SIZE],
+            self.config[num_layers_key],
+            self.config[size_key],
             self.config[DROP_RATE],
             self.config[DROP_RATE_ATTENTION],
         )

rasa/utils/train_utils.py

@@ -20,7 +20,10 @@
     AUTO,
     INNER,
     COSINE,
+    TRANSFORMER_SIZE,
+    NUM_TRANSFORMER_LAYERS,
 )
+from rasa.core.constants import DIALOGUE
 
 if TYPE_CHECKING:
     from rasa.nlu.classifiers.diet_classifier import EntityTagSpec
@@ -192,6 +195,28 @@ def check_deprecated_options(config: Dict[Text, Any]) -> Dict[Text, Any]:
     return config
 
 
+def check_core_deprecated_options(config: Dict[Text, Any]) -> Dict[Text, Any]:
+    """
+    If old model configuration parameters are present in the provided config, replace
+    them with the new parameters and log a warning.
+    Args:
+        config: model configuration
+
+    Returns: updated model configuration
+    """
+
+    # note: call _replace_deprecated_option() here when there are options to deprecate
+
+    config = _replace_deprecated_option(
+        TRANSFORMER_SIZE, f"{DIALOGUE}_{TRANSFORMER_SIZE}", config
+    )
+    config = _replace_deprecated_option(
+        NUM_TRANSFORMER_LAYERS, f"{DIALOGUE}_{NUM_TRANSFORMER_LAYERS}", config
+    )
+
+    return config
+
+
 def entity_label_to_tags(
     model_predictions: Dict[Text, Any],
     entity_tag_specs: List["EntityTagSpec"],