Merge original repo into fork (#1)

* backport fix * backport pika fix * pika fix backport * fix tests * test fixes * remove logger statement * Updated version of PikaMessageProcessor from the source branch * Small bugfix * changelog * prepared release of version 1.10.16 * Update CHANGELOG.rst Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> * add robots.txt and redirection headers * bump rasabaster version * Fix Pika multiprocessing for 1.10.x * Reformatting * Changelog entry * Updated tests * is_ready method is back * uninstall cached pydocstyle * also uninstall pydocstyle * uninstall in poetry env * Removing f-string * NoReturn -> None * prepared release of version 1.10.17 * Fix incorrect attribute for key prefix in RedisTrackerStore Fix RasaHQ#7306 * Add changelog entry * Remove the set-env command - 1.10.x (RasaHQ#7367) * Fix DynamoDB TrackerStore (RasaHQ#7340) * attempting to fix dynamo * fixed * fixed import * applied some more fixes * added deprecation warning for DynamoDB tracker store attribute * added changelog description for PR 7340 * fix formatting Co-authored-by: Tom Bocklisch <tom@rasa.com> * prepared release of version 1.10.18 (RasaHQ#7381) * Update docstrings * Update CHANGELOG.mdx * line length fix * Add print to confirm that the failed_test_stories.yml has been saved (RasaHQ#6793) * Add print to confirm that the failed_test_stories.yml has been saved (RasaHQ#3998) Co-authored-by: Henrike100 <henrikem100@gmail.com> * Change the way to show the file name * files formatted with black * add changelog * Update test core * Formatted with black * Update print Co-authored-by: mbslet <lmbs.geo@gmail.com> * Amend response schema to support missing top-level keys (RasaHQ#7232) * amend response schema to support missing top level keys * changelog entry * download mitie from github instead of s3 (RasaHQ#7434) Co-authored-by: melindaloubser1 <melinda.loubser@gmail.com> Co-authored-by: alwx <alwxndr@gmail.com> Co-authored-by: Roberto <43567378+rasabot@users.noreply.github.com> Co-authored-by: Melinda Loubser <32034278+melindaloubser1@users.noreply.github.com> Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> Co-authored-by: m-vdb <m.verger@rasa.com> Co-authored-by: Tobias Wochinger <t.wochinger@rasa.com> Co-authored-by: Duc-Minh Phan <alephvn@gmail.com> Co-authored-by: Federico Tedin <federicotdn@users.noreply.github.com> Co-authored-by: Tomasz Czekajlo <t.czekajlo@rasa.com> Co-authored-by: Tom Herold <tom.herold@scalableminds.com> Co-authored-by: Tom Bocklisch <tom@rasa.com> Co-authored-by: Henrique Martins <henrikem100@gmail.com> Co-authored-by: mbslet <lmbs.geo@gmail.com> Co-authored-by: pheel <philippe@mrbot.ai>
UTkzhang · Dec 2, 2020 · 961a8fd · 961a8fd
1 parent a099316
commit 961a8fd
Show file tree

Hide file tree

Showing 11 changed files with 131 additions and 25 deletions.
diff --git a/CHANGELOG.mdx b/CHANGELOG.mdx
@@ -895,6 +895,25 @@ https://github.com/RasaHQ/rasa/tree/master/changelog/ . -->
 ### Miscellaneous internal changes
 - [#5784](https://github.com/rasahq/rasa/issues/5784), [#5788](https://github.com/rasahq/rasa/issues/5788), [#6199](https://github.com/rasahq/rasa/issues/6199), [#6403](https://github.com/rasahq/rasa/issues/6403), [#6735](https://github.com/rasahq/rasa/issues/6735)
 
+## [1.10.18] - 2020-11-26
+
+### Bugfixes
+- [#7340](https://github.com/rasahq/rasa/issues/7340>): Fixed an issues with the DynamoDB TrackerStore creating a new table entry/object for each TrackerStore update. The column ``session_date`` has been deprecated and should be removed manually in existing DynamoDB tables.
+
+
+## [1.10.17] - 2020-11-12
+
+### Bugfixes
+- [#7219](https://github.com/rasahq/rasa/issues/7219): Prevent the message handling process in ``PikaEventBroker`` from being terminated.
+
+
+## [1.10.16] - 2020-10-15
+
+### Bugfixes
+- [#6703](https://github.com/rasahq/rasa/issues/6703): Update Pika event broker to be a separate process and make it use a
+  ``multiprocessing.Queue`` to send and process messages. This change should help
+  avoid situations when events stop being sent after a while.
+
 ## [1.10.15] - 2020-10-09
 
 ### Bugfixes

diff --git a/Makefile b/Makefile
@@ -26,6 +26,10 @@ help:
 	@echo "        Install system requirements for running tests on Windows."
 	@echo "    prepare-tests-files"
 	@echo "        Download all additional project files needed to run tests."
+	@echo "    prepare-spacy"
+	@echo "        Download all additional resources needed to use spacy as part of Rasa."
+	@echo "    prepare-mitie"
+	@echo "        Download all additional resources needed to use mitie as part of Rasa."
 	@echo "    test"
 	@echo "        Run pytest on tests/."
 	@echo "        Use the JOBS environment variable to configure number of workers (default: 1)."
@@ -103,13 +107,19 @@ types:
 	--disable-error-code no-redef \
 	--disable-error-code func-returns-value
 
-prepare-tests-files:
+prepare-spacy:
 	poetry install -E spacy
 	poetry run python -m spacy download en_core_web_md
 	poetry run python -m spacy download de_core_news_sm
 	poetry run python -m spacy link en_core_web_md en --force
 	poetry run python -m spacy link de_core_news_sm de --force
-	wget --progress=dot:giga -N -P data/ https://s3-eu-west-1.amazonaws.com/mitie/total_word_feature_extractor.dat
+
+prepare-mitie:
+	wget --progress=dot:giga -N -P data/ https://github.com/mit-nlp/MITIE/releases/download/v0.4/MITIE-models-v0.2.tar.bz2
+	tar -xvjf data/MITIE-models-v0.2.tar.bz2 --strip-components 2 -C data/ MITIE-models/english/total_word_feature_extractor.dat
+	rm data/MITIE*.bz2
+
+prepare-tests-files: prepare-spacy prepare-mitie
 
 prepare-wget-macos:
 	brew install wget || true

diff --git a/changelog/3998.improvement.md b/changelog/3998.improvement.md
@@ -0,0 +1 @@
+Added a message showing the location where the failed stories file was saved.
diff --git a/changelog/7232.improvement.md b/changelog/7232.improvement.md
@@ -0,0 +1 @@
+Add support for the top-level response keys `quick_replies`, `attachment` and `elements` refered to in `rasa.core.channels.OutputChannel.send_reponse`, as well as `metadata`.
diff --git a/changelog/7306.bugfix.md b/changelog/7306.bugfix.md
@@ -0,0 +1 @@
+Fix an erroneous attribute for Redis key prefix in `rasa.core.tracker_store.RedisTrackerStore`: 'RedisTrackerStore' object has no attribute 'prefix'.
diff --git a/docs/_redirects b/docs/_redirects
@@ -0,0 +1,4 @@
+# target all pages
+/*
+  # set noindex, no follow
+  X-Robots-Tag: noindex, no follow
diff --git a/docs/robots.txt b/docs/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /
diff --git a/rasa/cli/test.py b/rasa/cli/test.py
@@ -17,6 +17,7 @@
     DEFAULT_DATA_PATH,
     DEFAULT_RESULTS_PATH,
 )
+from rasa.core.test import FAILED_STORIES_FILE
 import rasa.shared.utils.validation as validation_utils
 import rasa.cli.utils
 
@@ -109,6 +110,10 @@ def run_core_test(args: argparse.Namespace) -> None:
     else:
         test_core_models(args.model, stories, output)
 
+    rasa.shared.utils.cli.print_info(
+        f"Failed stories written to '{os.path.join(output, FAILED_STORIES_FILE)}'"
+    )
+
 
 def run_nlu_test(args: argparse.Namespace) -> None:
     """Run NLU tests."""

diff --git a/rasa/core/tracker_store.py b/rasa/core/tracker_store.py
@@ -21,6 +21,8 @@
 )
 
 from boto3.dynamodb.conditions import Key
+from botocore.exceptions import ClientError
+
 import rasa.core.utils as core_utils
 import rasa.shared.utils.cli
 import rasa.shared.utils.common
@@ -346,18 +348,30 @@ def save(self, tracker, timeout=None):
             timeout = self.record_exp
 
         serialised_tracker = self.serialise_tracker(tracker)
-        self.red.set(self.prefix + tracker.sender_id, serialised_tracker, ex=timeout)
+        self.red.set(
+            self.key_prefix + tracker.sender_id, serialised_tracker, ex=timeout
+        )
 
     def retrieve(self, sender_id: Text) -> Optional[DialogueStateTracker]:
-        stored = self.red.get(self.prefix + sender_id)
+        """Retrieves tracker for the latest conversation session.
+
+        The Redis key is formed by appending a prefix to sender_id.
+
+        Args:
+            sender_id: Conversation ID to fetch the tracker for.
+
+        Returns:
+            Tracker containing events from the latest conversation sessions.
+        """
+        stored = self.red.get(self.key_prefix + sender_id)
         if stored is not None:
             return self.deserialise_tracker(sender_id, stored)
         else:
             return None
 
     def keys(self) -> Iterable[Text]:
-        """Returns keys of the Redis Tracker Store"""
-        return self.red.keys(self.prefix + "*")
+        """Returns keys of the Redis Tracker Store."""
+        return self.red.keys(self.key_prefix + "*")
 
 
 class DynamoTrackerStore(TrackerStore):
@@ -402,35 +416,69 @@ def get_or_create_table(
         except self.client.exceptions.ResourceNotFoundException:
             table = dynamo.create_table(
                 TableName=self.table_name,
-                KeySchema=[
-                    {"AttributeName": "sender_id", "KeyType": "HASH"},
-                    {"AttributeName": "session_date", "KeyType": "RANGE"},
-                ],
+                KeySchema=[{"AttributeName": "sender_id", "KeyType": "HASH"},],
                 AttributeDefinitions=[
                     {"AttributeName": "sender_id", "AttributeType": "S"},
-                    {"AttributeName": "session_date", "AttributeType": "N"},
                 ],
                 ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5},
             )
 
             # Wait until the table exists.
             table.meta.client.get_waiter("table_exists").wait(TableName=table_name)
-        return dynamo.Table(table_name)
+        else:
+            table = dynamo.Table(table_name)
+
+            column_names = [
+                attribute["AttributeName"] for attribute in table.attribute_definitions
+            ]
+            if "session_date" in column_names:
+                rasa.shared.utils.io.raise_warning(
+                    "Attribute 'session_date' is no longer required when using a "
+                    "DynamoDB TrackerStore. Please remove this attribute from "
+                    "any existing tables.",
+                    FutureWarning,
+                )
+
+        return table
 
     def save(self, tracker):
         """Saves the current conversation state"""
         if self.event_broker:
             self.stream_events(tracker)
-        self.db.put_item(Item=self.serialise_tracker(tracker))
+        serialized = self.serialise_tracker(tracker)
+
+        try:
+            self.db.put_item(Item=serialized)
+        except ClientError as e:
+            if "Missing the key session_date" in repr(e):
+                # the session_date attribute got removed as it was useless
+                # old databases will still contain an attribute for it though
+                # which we need to set (otherwise we are getting the error we
+                # just ran into) this section should be removed in 3.0
+                legacy_date = self._retrieve_latest_session_date(tracker.sender_id)
+
+                serialized["session_date"] = legacy_date
+                self.db.put_item(Item=serialized)
+            else:
+                raise
+
+    def _retrieve_latest_session_date(self, sender_id: Text) -> Optional[int]:
+        dialogues = self.db.query(
+            KeyConditionExpression=Key("sender_id").eq(sender_id),
+            Limit=1,
+            ScanIndexForward=False,
+        )["Items"]
+
+        if not dialogues:
+            return int(datetime.now(tz=timezone.utc).timestamp())
+
+        return dialogues[0].get("session_date")
 
     def serialise_tracker(self, tracker: "DialogueStateTracker") -> Dict:
         """Serializes the tracker, returns object with decimal types"""
         d = tracker.as_dialogue().as_dict()
         d.update(
-            {
-                "sender_id": tracker.sender_id,
-                "session_date": int(datetime.now(tz=timezone.utc).timestamp()),
-            }
+            {"sender_id": tracker.sender_id,}
         )
         # DynamoDB cannot store `float`s, so we'll convert them to `Decimal`s
         return core_utils.replace_floats_with_decimals(d)

diff --git a/rasa/shared/nlu/training_data/schemas/responses.yml b/rasa/shared/nlu/training_data/schemas/responses.yml
@@ -29,5 +29,25 @@ schema;responses:
                   type: "str"
                 payload:
                   type: "str"
+          quick_replies:
+            type: "seq"
+            sequence:
+            - type: "map"
+              allowempty: True
+              mapping:
+                title:
+                  type: "str"
+                payload:
+                  type: "str"
+          attachment:
+            type: "map"
+            allowempty: True
+          elements:
+            type: "seq"
+            sequence:
+            - type: "map"
+              allowempty: True
           channel:
             type: "str"
+          metadata:
+            type: "any"
diff --git a/tests/shared/core/test_trackers.py b/tests/shared/core/test_trackers.py
@@ -74,20 +74,15 @@
 
 
 class MockRedisTrackerStore(RedisTrackerStore):
-    # skipcq:PYL-W0231
-    # can't call call super init since it would init a redis connection
     def __init__(self, _domain: Domain) -> None:
+        super().__init__(_domain)
+
+        # Patch the Redis connection in RedisTrackerStore using fakeredis
         self.red = fakeredis.FakeStrictRedis()
-        self.record_exp = None
 
         # added in redis==3.3.0, but not yet in fakeredis
         self.red.connection_pool.connection_class.health_check_interval = 0
 
-        # Defined in RedisTrackerStore but needs to be added for the MockRedisTrackerStore
-        self.prefix = "tracker:"
-
-        TrackerStore.__init__(self, _domain)
-
 
 def stores_to_be_tested():
     temp = tempfile.mkdtemp()
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Added a message showing the location where the failed stories file was saved.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Add support for the top-level response keys `quick_replies`, `attachment` and `elements` refered to in `rasa.core.channels.OutputChannel.send_reponse`, as well as `metadata`.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Fix an erroneous attribute for Redis key prefix in `rasa.core.tracker_store.RedisTrackerStore`: 'RedisTrackerStore' object has no attribute 'prefix'.