Merge branch 'master' into e2e

Makefile

@@ -66,11 +66,17 @@ lint:
 	poetry run black --check rasa tests
 	make lint-docstrings
 
-BRANCH ?= master # Compare against `master` if no branch was provided
+# Compare against `master` if no branch was provided
+BRANCH ?= "master"
 lint-docstrings:
-     # Lint docstrings only against the the diff to avoid too many errors.
-     # Check only production code. Ignore other flake errors which are captured by `lint`
-	git diff $(BRANCH) -- rasa | poetry run flake8 --select D --diff
+	# Lint docstrings only against the the diff to avoid too many errors.
+	# Check only production code. Ignore other flake errors which are captured by `lint`
+	# Diff of committed changes (shows only changes introduced by your branch)
+	if [[ -n "$(BRANCH)" ]]; then \
+	    git diff $(BRANCH)...HEAD -- rasa | poetry run flake8 --select D --diff; \
+	fi
+	# Diff of uncommitted changes for running locally
+	git diff HEAD -- rasa | poetry run flake8 --select D --diff
 
 types:
 	# FIXME: working our way towards removing these

changelog/6498.improvement.md

@@ -0,0 +1,2 @@
+Adding configurable prefixes to Redis [Tracker](./tracker-stores.mdx) and [Lock Stores](./lock-stores.mdx) so that a single Redis instance (and logical DB) can support multiple conversation trackers and locks.
+By default, conversations will be prefixed with `tracker:...` and all locks prefixed with `lock:...`. Additionally, you can add an alphanumeric-only `prefix: value` in `endpoints.yml` such that keys in redis will take the form `value:tracker:...` and `value:lock:...` respectively.

data/test_endpoints/example_endpoints.yml

@@ -14,6 +14,7 @@ tracker_store:
   port: 6379
   db: 0
   password: password
+  key_prefix: conversation
   record_exp: 30000
 # example of mongoDB external tracker store config
 #tracker_store:

docs/docs/components.mdx

@@ -2714,11 +2714,6 @@ See [starspace paper](https://arxiv.org/abs/1709.03856) for details.
 You can create a custom component to perform a specific task which NLU doesn't currently offer (for example, sentiment analysis).
 Below is the specification of the `rasa.nlu.components.Component`] class with the methods you'll need to implement.
 
-:::tip follow the tutorial
-There is a detailed tutorial on building custom components on the [Rasa Blog](https://blog.rasa.com/enhancing-rasa-nlu-with-custom-components/).
-
-:::
-
 You can add a custom component to your pipeline by adding the module path.
 So if you have a module called `sentiment`
 containing a `SentimentAnalyzer` class:

docs/docs/event-brokers.mdx

@@ -123,19 +123,22 @@ If using the `SSL` protocol, the endpoints file should look like:
 
 ### Adding a Kafka Broker in Python
 
-The code below shows an example on how to instantiate a Kafka producer in you script.
+The code below shows an example on how to instantiate a Kafka producer in your script.
 
 ```python
 from rasa.core.brokers.kafka import KafkaEventBroker
 from rasa.core.tracker_store import InMemoryTrackerStore
+from rasa.shared.core.domain import Domain
 
-kafka_broker = KafkaEventBroker(host='localhost:9092',
+kafka_broker = KafkaEventBroker(url='localhost:9092',
                                 topic='rasa_events')
 
+
+domain = Domain.load("domain.yml")
 tracker_store = InMemoryTrackerStore(domain=domain, event_broker=kafka_broker)
 ```
 
-The host variable can be either a list of brokers addresses or a single one.
+The `url` argument can be either a list of broker addresses or a single one.
 If only one broker address is available, the client will connect to it and
 request the cluster Metadata.
 Therefore, the remaining brokers in the cluster can be discovered

docs/docs/lock-stores.mdx

@@ -57,6 +57,7 @@ address the same node when sending messages for a given conversation ID.
          port: <port of your redis instance, usually 6379>
          password: <password used for authentication>
          db: <number of your database within redis, e.g. 0>
+         key_prefix: <alphanumeric value to prepend to lock store keys>
      ```
 
   3. To start the Rasa Core server using your Redis backend, add the `--endpoints`
@@ -76,10 +77,13 @@ address the same node when sending messages for a given conversation ID.
 
   * `db` (default: `1`): The number of your redis database
 
+  * `key_prefix` (default: `None`): The prefix to prepend to lock store keys. Must
+     be alphanumeric
+
   * `password` (default: `None`): Password used for authentication
     (`None` equals no authentication)
 
   * `use_ssl` (default: `False`): Whether or not the communication is encrypted
 
   * `socket_timeout` (default: `10`): Time in seconds after which an
-     error is raised if Redis doesn't answer
+     error is raised if Redis doesn't answer

docs/docs/tracker-stores.mdx

@@ -179,14 +179,12 @@ To set up Rasa Open Source with Redis the following steps are required:
      type: redis
      url: <url of the redis instance, e.g. localhost>
      port: <port of your redis instance, usually 6379>
+     key_prefix: <alphanumeric value to prepend to tracker store keys>
      db: <number of your database within redis, e.g. 0>
      password: <password used for authentication>
      use_ssl: <whether or not the communication is encrypted, default `false`>
  ```
 
-3. To start the Rasa server using your configured Redis instance,
- add the `--endpoints` flag, e.g.:
-
  ```bash
  rasa run -m models --endpoints endpoints.yml
  ```
@@ -207,18 +205,20 @@ To set up Rasa Open Source with Redis the following steps are required:
       url: <url of the redis instance, e.g. localhost>
       port: <port of your redis instance, usually 6379>
       db: <number of your database within redis, e.g. 0>
+      key_prefix: <alphanumeric value to prepend to tracker store keys>
       password: <password used for authentication>
       use_ssl: <whether or not the communication is encrypted, default `false`>
  ```
 
-#### Configuration Parameters
-
 * `url` (default: `localhost`): The url of your redis instance
 
 * `port` (default: `6379`): The port which redis is running on
 
 * `db` (default: `0`): The number of your redis database
 
+* `key_prefix` (default: `None`): The prefix to prepend to tracker store keys. Must
+    be alphanumeric
+
 * `password` (default: `None`): Password used for authentication
 (`None` equals no authentication)
 

docs/docs/training-data-format.mdx

@@ -3,7 +3,7 @@ id: training-data-format
 sidebar_label: Training Data Format
 title: Training Data Format
 description: Description of the YAML format for training data
-abstract: This page provides an overview of the different types of training data
+abstract: This page describes the different types of training data
   that go into a Rasa assistant and how this training data is structured.
 ---
 
@@ -13,10 +13,9 @@ Rasa Open Source uses [YAML](https://yaml.org/spec/1.2/spec.html) as
 a unified and extendable way to manage all training data,
 including NLU data, stories and rules.
 
-With the YAML format, training data can be split over any number of YAML files,
-and every file can contain any kind of data.
-The training data parser will read the top level keys in each file to decide
-what kind of data is in a section at training time.
+You can split the training data over any number of YAML files,
+and each file can contain any combination of NLU data, stories, and rules.
+The training data parser determines the training data type using top level keys.
 
 The [domain](glossary.mdx#domain) uses
 the same YAML format as the training data and can also be split across
@@ -32,21 +31,20 @@ you can still find the documentation for <a href="https://legacy-docs-v1.rasa.co
 ### High-Level Structure
 
 Each file can contain one or more **keys** with corresponding training
-data. One file can contain multiple keys, as long as there is not more
-than one of a certain key in a single file. The available keys are:
+data. One file can contain multiple keys, but each key can only appear
+once in a single file. The available keys are:
 
 - `version`
 - `nlu`
 - `stories`
 - `rules`
 
-All YAML training data files should specify the `version` key to be parsed
-correctly. If you don't specify a version key in your training data file, Rasa
+You should specify the `version` key in all YAML training data files.
+If you don't specify a version key in your training data file, Rasa
 will assume you are using the latest training data format specification supported
 by the version of Rasa Open Source you have installed.
-Training data files with a version greater than is currently available in
-the version of Rasa Open Source you have installed will be
-skipped.
+Training data files with a Rasa Open Source version greater than the version you have
+installed on your machine will be skipped.
 Currently, the latest training data format specification for Rasa 2.x is 2.0.
 
 ### Example
@@ -84,7 +82,7 @@ rules:
 
 ```
 
-If you want to specify your test stories, you need to put them into a separate file:
+To specify your test stories, you need to put them into a separate file:
 ```yaml-rasa title="tests/test_stories.yml"
 stories:
 - story: greet and ask language
@@ -98,7 +96,8 @@ stories:
     intent: faq/language
   - action: utter_faq
 ```
-They use the same format as the story training data.
+[Test stories](#test-stories) use the same format as the story training data and should be placed
+in a separate file with the prefix `test_`.
 
 ## NLU Training Data
 
@@ -180,8 +179,10 @@ nlu:
       hey there!
 ```
 
-The `metadata` key can contain arbitrary key-value data that stays with an example and is accessible by the components in the NLU pipeline. In the example above, the sentiment of
-the example could be used by a custom component in the pipeline for sentiment analysis.
+The `metadata` key can contain arbitrary key-value data that is tied to an example and
+accessible by the components in the NLU pipeline.
+In the example above, the sentiment metadata could be used by a custom component in
+the pipeline for sentiment analysis.
 
 You can also specify this metadata at the intent level:
 
@@ -197,7 +198,7 @@ nlu:
       hey there!
 ```
 
-In this case, the content of the `metadata` key is passed on to every intent example.
+In this case, the content of the `metadata` key is passed to every intent example.
 
 
 If you want to specify [retrieval intents](glossary.mdx#retrieval-intent), then your NLU examples will look as follows:
@@ -218,8 +219,8 @@ nlu:
     - I like sunny days in Berlin.
 ```
 All retrieval intents have a suffix
-added to them which identifies a particular response key for your assistant, in the
-above example `ask_name` and `ask_weather` are the suffixes. The suffix is separated from
+added to them which identifies a particular response key for your assistant. In the
+above example, `ask_name` and `ask_weather` are the suffixes. The suffix is separated from
 the retrieval intent name by a `/` delimiter.
 
 :::note Special meaning of `/`
@@ -231,9 +232,9 @@ name of your intents.
 
 ### Entities
 
-[Entities](glossary.mdx#entity) are structured pieces of information that can be extracted from a user's message. 
+[Entities](glossary.mdx#entity) are structured pieces of information that can be extracted from a user's message.
 
-Entities are annotated in training examples with minimally the entity's name.
+Entities are annotated in training examples with the entity's name.
 In addition to the entity name, you can annotate an entity with [synonyms](nlu-training-data.mdx#synonyms), [roles, or groups](nlu-training-data.mdx#entities-roles-and-groups).
 
 In training examples, entity annotation would look like this:
@@ -255,15 +256,15 @@ The full possible syntax for annotating an entity is:
 ```
 
 The keywords `role`, `group`, and `value` are optional in this notation.
-The `value` keyword refers to synonyms. To understand what the labels `role` and `group` are
+The `value` field refers to synonyms. To understand what the labels `role` and `group` are
 for, see the section on [entity roles and groups](./nlu-training-data.mdx#entities-roles-and-groups).
 
 
 ### Synonyms
 
-Synonyms provide a way to normalize your training data by mapping an
+Synonyms normalize your training data by mapping an
 extracted entity to a value other than the literal text extracted.
-Synonyms can be defined in the format:
+You can define synonyms using the format:
 
 ```yaml-rasa
 nlu:
@@ -273,7 +274,7 @@ nlu:
     - credit account
 ```
 
-Synonyms can also be defined in-line in your training examples by
+You can also define synonyms in-line in your training examples by
 specifying the `value` of the entity:
 
 ```yaml-rasa
@@ -289,7 +290,7 @@ Read more about synonyms on the [NLU Training Data page](./nlu-training-data.mdx
 ### Regular Expressions
 
 You can use regular expressions to improve intent classification and
-entity extraction in combination with the [`RegexFeaturizer`](components.mdx#regexfeaturizer) and [`RegexEntityExtractor`](components.mdx#regexentityextractor) components in the pipeline.
+entity extraction using the [`RegexFeaturizer`](components.mdx#regexfeaturizer) and [`RegexEntityExtractor`](components.mdx#regexentityextractor) components.
 
 The format for defining a regular expression is as follows:
 
@@ -387,9 +388,9 @@ key and an optional `entities:` key.
 
 While writing stories, you do not have to deal with the specific
 contents of the messages that the users send. Instead, you can take
-advantage of the output from the NLU pipeline, which lets you use
-the combination of an intent and entities to refer to all the possible
-messages the users can send to mean the same thing.
+advantage of the output from the NLU pipeline, which uses
+a combination of an intent and entities to refer to all possible
+messages the users can send with the same meaning.
 
 User messages follow the format:
 
@@ -456,11 +457,11 @@ stories:
 A [form](glossary.mdx#form) is a specific kind of custom action that contains the logic to loop over
 a set of required slots and ask the user for this information. You
 [define a form](forms.mdx#defining-a-form) in the `forms` section in your domain.
-Once defined, the [happy path](glossary.mdx#happy--unhappy-paths)
-for a form should be specified as a [rule](forms.mdx), but interruptions of forms or
-other "unhappy paths" should be included in stories so that the model can
+Once defined, you should specify the [happy path](glossary.mdx#happy--unhappy-paths)
+for a form as a [rule](forms.mdx). You should include interruptions of forms or
+other "unhappy paths" in stories so that the model can
 generalize to unseen conversation sequences.
-As a step in a story, a form takes the following basic format:
+As a step in a story, a form takes the following format:
 
 
 ```yaml-rasa
@@ -643,15 +644,14 @@ rules:
 
 For more information about writing rules, see [Rules](rules.mdx#writing-a-rule).
 
-## Test Conversations
+## Test Stories
 
-Test conversations use stories to test your bot. The tests check if a message is
-classified correctly as well as the action predictions.
+Test stories check if a message is classified correctly as well as the action predictions.
 
 Test stories use the same format as [stories](#stories),
 except that user message steps can include a `user` to specify the actual
 text and entity annotations of the user message. Here's an example of a
-test conversation:
+test story:
 
 ```yaml-rasa
 stories:
@@ -671,7 +671,7 @@ stories:
   - action: utter_ask_price
 ```
 
-Running the tests can be done using the CLI:
+You can run the tests using the following command:
 ```bash
 rasa test
 ```