[IMP] mail_tracking_mailgun: refactor to support modern webhooks

Before this patch, the module was designed after the [deprecated Mailgun webhooks][3]. However Mailgun had the [events API][2] which was quite different. Modern Mailgun has deprecated those webhooks and instead uses new ones that include the same payload as the events API, so you can reuse code.

However, this was incorrectly reusing the code inversely: trying to process the events API through the same code prepared for the deprecated webhooks.

Besides, both `failed` and `rejected` mailgun events were mapped to `error` state, but that was also wrong because [`mail_tracking` doesn't have an `error` state][1].

So the logic of the whole module is changed, adapting it to process the events API payload, both through controllers (prepared for the new webhooks) and manual updates that directly call the events API.

Also, `rejected` is now translated into `reject`, and `failed` is translated into `hard_bounce` or `soft_bounce` depending on the severity, as specified by [mailgun docs][2]. Also, `bounced` and `dropped` mailgun states are removed because they don't exist, and instead `failed` and `rejected` properly get their metadata.

Of course, to know the severity, now the method to obtain that info must change, it' can't be a simple dict anymore.

Added more parameters because for example modern Mailgun uses different keys for signing payload than for accessing the API. As there are so many parameters, configuration is now possible through `res.config.settings`. Go there to autoregister webhooks too.

Since the new webhooks are completely incompatible with the old supposedly-abstract webhooks controllers (that were never really that abstract), support for old webhooks is removed, and it will be removed in the future from `mail_tracking` directly. There is a migration script that attempts to unregister old webhooks and register new ones automatically.

[1]: https://github.com/OCA/social/blob/f73de421e28a43d018176f61725a3a59665f715d/mail_tracking/models/mail_tracking_event.py#L31-L42
[2]: https://documentation.mailgun.com/en/latest/api-events.html#event-types
[3]: https://documentation.mailgun.com/en/latest/api-webhooks-deprecated.html

mail_tracking_mailgun/README.rst

@@ -23,7 +23,7 @@ Mail tracking for Mailgun
     :target: https://runbot.odoo-community.org/runbot/205/14.0
     :alt: Try me on Runbot
 
-|badge1| |badge2| |badge3| |badge4| |badge5| 
+|badge1| |badge2| |badge3| |badge4| |badge5|
 
 This module integrates mail_tracking events with Mailgun webhooks.
 
@@ -38,33 +38,27 @@ function used here.
 .. contents::
    :local:
 
-Configuration
-=============
-
-You must configure Mailgun webhooks in order to receive mail events:
-
-1. Got a Mailgun account and validate your sending domain.
-2. Go to Webhook tab and configure the below URL for each event:
+Installation
+============
 
-.. code:: html
+If you're using a multi-database installation (with or without dbfilter option)
+where /web/databse/selector returns a list of more than one database, then
+you need to add ``mail_tracking_mailgun`` addon to wide load addons list
+(by default, only ``web`` addon), setting ``--load`` option.
 
-   https://<your_domain>/mail/tracking/all/<your_database>
+Example: ``--load=web,mail_tracking,mail_tracking_mailgun``
 
-Replace '<your_domain>' with your Odoo install domain name
-and '<your_database>' with your database name.
+Configuration
+=============
 
-In order to validate Mailgun webhooks you have to configure the following system
-parameters:
+To configure this module, you need to:
 
-- `mailgun.apikey`: You can find Mailgun api_key in your validated sending
-  domain.
-- `mailgun.api_url`: It should be fine as it is, but it could change in the
-  future.
-- `mailgun.domain`: In case your sending domain is different from the one
-  configured in `mail.catchall.domain`.
-- `mailgun.validation_key`: If you want to be able to check mail address
-  validity you must config this parameter with your account Public Validation
-  Key.
+#. Go to Mailgun, create an account and validate your sending domain.
+#. Go back to Odoo.
+#. Go to *Settings > General Settings > Discuss > Enable mail tracking with Mailgun*.
+#. Fill all the values. The only one required is the API key.
+#. Optionally click *Unregister Mailgun webhooks* and accept.
+#. Click *Register Mailgun webhooks*.
 
 You can also config partner email autocheck with this system parameter:
 
@@ -94,6 +88,11 @@ Known issues / Roadmap
 
 * There's no support for more than one Mailgun mail server.
 
+* Automate more webhook registration. It would be nice to not have to click the
+  "Unregister Mailgun webhooks" and "Register Mailgun webhooks" when setting up
+  Mailgun in Odoo. However, it doesn't come without its `conceptual complexities
+  <https://github.com/OCA/social/pull/787#discussion_r734275262>`__.
+
 Bug Tracker
 ===========
 
@@ -123,6 +122,7 @@ Contributors
   * David Vidal
   * Rafael Blasco
   * Ernesto Tejeda
+  * Jairo Llopis
   * Carlos Roca
 
 Other credits

mail_tracking_mailgun/__init__.py

@@ -1,3 +1,5 @@
 # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
 
+from . import controllers
 from . import models
+from . import wizards

mail_tracking_mailgun/__manifest__.py

@@ -6,13 +6,17 @@
 {
     "name": "Mail tracking for Mailgun",
     "summary": "Mail tracking and Mailgun webhooks integration",
-    "version": "14.0.1.0.0",
+    "version": "14.0.2.0.0",
     "category": "Social Network",
     "website": "https://github.com/OCA/social",
     "author": "Tecnativa, Odoo Community Association (OCA)",
     "license": "AGPL-3",
     "application": False,
     "installable": True,
     "depends": ["mail_tracking"],
-    "data": ["views/res_partner.xml", "views/mail_tracking_email.xml"],
+    "data": [
+        "views/res_partner.xml",
+        "views/mail_tracking_email.xml",
+        "wizards/res_config_settings_views.xml",
+    ],
 }

mail_tracking_mailgun/controllers/__init__.py

@@ -0,0 +1 @@
+from . import main

mail_tracking_mailgun/controllers/main.py

@@ -0,0 +1,75 @@
+# Copyright 2021 Tecnativa - Jairo Llopis
+# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl).
+
+import hashlib
+import hmac
+import logging
+from datetime import datetime, timedelta
+
+from werkzeug.exceptions import NotAcceptable
+
+from odoo import _
+from odoo.exceptions import ValidationError
+from odoo.http import request, route
+
+from ...mail_tracking.controllers import main
+from ...web.controllers.main import ensure_db
+
+_logger = logging.getLogger(__name__)
+
+
+class MailTrackingController(main.MailTrackingController):
+    def _mail_tracking_mailgun_webhook_verify(self, timestamp, token, signature):
+        """Avoid mailgun webhook attacks.
+
+        See https://documentation.mailgun.com/en/latest/user_manual.html#securing-webhooks
+        """  # noqa: E501
+        # Request cannot be old
+        processing_time = datetime.utcnow() - datetime.utcfromtimestamp(int(timestamp))
+        if not timedelta() < processing_time < timedelta(minutes=10):
+            raise ValidationError(_("Request is too old"))
+        # Avoid replay attacks
+        try:
+            processed_tokens = (
+                request.env.registry._mail_tracking_mailgun_processed_tokens
+            )
+        except AttributeError:
+            processed_tokens = (
+                request.env.registry._mail_tracking_mailgun_processed_tokens
+            ) = set()
+        if token in processed_tokens:
+            raise ValidationError(_("Request was already processed"))
+        processed_tokens.add(token)
+        params = request.env["mail.tracking.email"]._mailgun_values()
+        # Assert signature
+        if not params.webhook_signing_key:
+            _logger.warning(
+                "Skipping webhook payload verification. "
+                "Set `mailgun.webhook_signing_key` config parameter to enable"
+            )
+            return
+        hmac_digest = hmac.new(
+            key=params.webhook_signing_key.encode(),
+            msg=("{}{}".format(timestamp, token)).encode(),
+            digestmod=hashlib.sha256,
+        ).hexdigest()
+        if not hmac.compare_digest(str(signature), str(hmac_digest)):
+            raise ValidationError(_("Wrong signature"))
+
+    @route(["/mail/tracking/mailgun/all"], auth="none", type="json", csrf=False)
+    def mail_tracking_mailgun_webhook(self):
+        """Process webhooks from Mailgun."""
+        ensure_db()
+        # Verify and return 406 in case of failure, to avoid retries
+        # See https://documentation.mailgun.com/en/latest/user_manual.html#routes
+        try:
+            self._mail_tracking_mailgun_webhook_verify(
+                **request.jsonrequest["signature"]
+            )
+        except ValidationError as error:
+            raise NotAcceptable from error
+        # Process event
+        request.env["mail.tracking.email"].sudo()._mailgun_event_process(
+            request.jsonrequest["event-data"],
+            self._request_metadata(),
+        )

mail_tracking_mailgun/migrations/14.0.2.0.0/post-migration.py

@@ -0,0 +1,32 @@
+# Copyright 2021 Tecnativa - Jairo Llopis
+# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl).
+
+import logging
+
+from openupgradelib import openupgrade
+
+_logger = logging.getLogger(__name__)
+
+
+@openupgrade.migrate()
+def migrate(env, version):
+    """Update webhooks.
+
+    This version dropped support for legacy webhooks and added support for
+    webhook autoregistering. Do that process now.
+    """
+    settings = env["res.config.settings"].create({})
+    if not settings.mail_tracking_mailgun_enabled:
+        _logger.warning("Not updating webhooks because mailgun is not configured")
+        return
+    _logger.info("Updating mailgun webhooks")
+    try:
+        settings.mail_tracking_mailgun_unregister_webhooks()
+        settings.mail_tracking_mailgun_register_webhooks()
+    except Exception:
+        # Don't fail the update if you can't register webhooks; it can be a
+        # failing network condition or air-gapped upgrade, and that's OK, you
+        # can just update them later
+        _logger.warning(
+            "Failed to update mailgun webhooks; do that manually", exc_info=True
+        )