MagpieAdapter request/response hooks

CHANGES.rst

@@ -9,6 +9,15 @@ Changes
 `Unreleased <https://github.com/Ouranosinc/Magpie/tree/master>`_ (latest)
 ------------------------------------------------------------------------------------
 
+Features / Changes
+~~~~~~~~~~~~~~~~~~~~~
+* Add JSON schema validation of loaded `Service` configuration (``providers.cfg``).
+* Add optional ``hooks`` section under each `Service` definition of the ``providers.cfg`` or combined configuration
+  file that allows pre/post request/response processing operations using plugin Python scripts.
+* Store the validated `Service` configuration in ``magpie.services`` settings for later access to ``hooks`` definitions
+  by the ``MagpieAdapter``.
+* Rename the ``webhooks`` section stored in settings to ``magpie.webhooks`` to avoid possible name clashes.
+
 Bug Fixes
 ~~~~~~~~~~~~~~~~~~~~~
 * Fix typo in UI edit user page when listing order of resolution of permissions.

Dockerfile.adapter

@@ -3,7 +3,7 @@
 #   docker run will need to override ini file with mounted volume
 #   using config 'twitcher.adapter = magpie.adapter.MagpieAdapter'
 #
-FROM birdhouse/twitcher:v0.6.2
+FROM birdhouse/twitcher:v0.7.0
 LABEL Description="Configures MagpieAdapter on top of Twitcher application."
 LABEL Maintainer="Francis Charette-Migneault <francis.charette-migneault@crim.ca>"
 LABEL Vendor="CRIM"

README.rst

@@ -14,7 +14,9 @@ Behind the scene, it uses `Ziggurat-Foundations`_ and `Authomatic`_.
 .. start-badges
 
 .. list-table::
+    :header-rows: 0
     :stub-columns: 1
+    :widths: 10,90
 
     * - dependencies
       - | |py_ver| |dependencies|

config/providers.cfg

@@ -16,8 +16,38 @@
 #   c4i:        parameter passed down to Phoenix for service registration
 #   type:       service type to use for creation, must be one of the known Magpie service types
 #               (see: magpie.services.SERVICE_TYPE_DICT)
-#   sync_type:  service synchronization type, must be one of the known Magpie service sync-types (often equals to 'type')
-#               (see: magpie.cli.SYNC_SERVICES_TYPES)
+#   sync_type:  service synchronization type, must be one of the known Magpie service sync-types,
+#               often equals to 'type' (see: magpie.cli.SYNC_SERVICES_TYPES)
+#   hooks:      list of request processing hooks for the service
+#
+# Hooks: (requires Magpie>=3.25.0, Twitcher>=0.7.0)
+# ------
+#   When items are specified and that the *original* request filters match a configuration, the processing hooks are
+#   applied onto the proxied request or response using it. Each hook definition must be provided using the following
+#   structure under the relevant service.
+#
+#   hooks:
+#     - type: <HOOK_TYPE>         [required] (request|response)
+#       path: <PROXIED_PATH>      [required] service-specific request path / regex pattern (after proxy prefix path)
+#       query: <HTTP_QUERY>       [optional] request query string / regex pattern excluding '?' prefix (default: ".*")
+#       method: <HTTP_METHOD>     [optional] (HEAD|GET|POST|PUT|PATCH|DELETE|*) (default: "*")
+#       target: <FUNCTION_PATH>   [required] location of function to handle hook processing
+#                                            path should be absolute or relative to MAGPIE_ROOT
+#                                            (format: 'some/path/script.py:func')
+#     - <next hook>
+#     - <...>
+#
+#   Functions defined by request/response hook must respectively take as input the active request/response in the
+#   processing chain and return an equivalent request/response with desired modifications applied for following ones.
+#   Furthermore, they can specify an optional argument for the service definition that triggered the hook function.
+#   Permitted signatures of hooks are presented in:
+#   https://pavics-magpie.readthedocs.io/en/latest/configuration.html#service-hooks
+#
+#   Each hook that matches is applied iteratively in the listed order, allowing successive modifications of the
+#   request/responses as needed. When all request hooks are processed, the request is sent to the proxied service to
+#   obtain the response. This response then uses the same matching of the original request to apply response hook
+#   processing chain. The final response is returned if all steps succeeded all returned the expected request/response
+#   instances. If an error occurs or forbidden access happens during the request, following hooks are skipped entirely.
 #
 # Default behaviour:
 # ------------------
@@ -108,3 +138,28 @@ providers:
     c4i: false
     type: api
     sync_type: project-api
+
+  weaver:
+    url: http://${HOSTNAME}:4001
+    title: weaver-ogc-api-processes
+    public: true
+    c4i: false
+    type: api
+    sync_type: api
+    hooks:
+      - type: request
+        path: "/processes/[w+_-]/jobs"
+        method: POST
+        target: ../tests/hooks/request_hooks.py:add_x_wps_output_context
+      - type: request
+        path: "/jobs"
+        method: POST
+        target: ../tests/hooks/request_hooks.py:add_x_wps_output_context
+      - type: response
+        path: "/processes/[w+_-]/jobs/[a-z0-9-]+"
+        method: GET
+        target: ../tests/hooks/request_hooks.py:add_x_wps_output_link
+      - type: response
+        path: "/jobs/[a-z0-9-]+"
+        method: GET
+        target: ../tests/hooks/request_hooks.py:add_x_wps_output_link

docs/configuration.rst

@@ -192,6 +192,87 @@ field.
     Variable :envvar:`MAGPIE_WEBHOOKS_CONFIG_PATH` was added and will act in a similar fashion as their providers and
     permissions counterparts, to load definitions from multiple configuration files.
 
+.. _config_service_hooks:
+
+Service Hooks
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 3.25
+
+Under each :term:`Service` within `providers.cfg`_ or the :ref:`config_file`, it is possible to provide a section
+named``hooks`` that lists additional pre/post request/response processing operations to apply when matched against
+the given request filter conditions. These hooks are plugin-based Python scripts that can modify the proxied request
+and responses when `Magpie` and `Twitcher`_ work together using the :ref:`utilities_adapter<Magpie Adapter>`.
+Each hook must be configured using the following parameters.
+
+
+.. list-table::
+    :header-rows: 1
+    :stub-columns: 1
+    :widths: 10,10,10,70
+
+    * - Field
+      - Requirement
+      - Description
+    * - ``type``
+      - **required**
+      - Literal string ``{ request | response }`` of the desired instance where to invoke the hook.
+    * - ``path``
+      - **required**
+      - :term:`Service`-specific request path or regular expression pattern to be matched for invoking the hook.
+        Path starts after `Twitcher`_ proxy prefix path and :term:`Service` name (i.e.: path as if there was no proxy).
+    * - ``method``
+      - *optional*
+      - Literal string ``{ HEAD | GET | POST | PUT | PATCH | DELETE | * }`` (default: ``*`` representing any method).
+        HTTP method that must be matched for invoking the hook.
+    * - ``query``
+      - *optional*
+      - Request query string or regular expression pattern to be matched for invoking the hook (default: ``.*``).
+        Matches anything if not specified. To match explicitly no-query condition, provide an empty string (``""``).
+    * - ``target``
+      - **required**
+      - Location of the function that will handle hook processing when request matching conditions are met.
+        Path should be absolute or relative to :envvar:`MAGPIE_ROOT` and must be a valid Python file.
+        Path should include the function name using format: ``some/path/script.py:func``.
+
+More specifically, when a :term:`Service` or children :term:`Resource` is accessed, triggering a proxied request
+through `Twitcher`_, the authenticated and authorized request goes through ``hooks`` processing chain that can adjust
+certain request and response parameters (e.g.: add headers, filter the body, etc.), or even substitute the request
+definition entirely based on ``target`` implementations. Hooks are applied in the same order as they are defined in
+the configuration when they match the inbound request, propagating the request/response across each call.
+Plugin scripts can therefore apply some advanced logic to improve the synergy between the protected services.
+They can also be employed to apply some :term:`Service` specific operations such as filtering protected contents
+that `Magpie` and `Twitcher`_ cannot themselves process evidently.
+
+Permitted signatures of hook functions are as presented below.
+The first argument (``request`` or ``response`` accordingly) is always required. Its modified definition must be
+returned as well. The other parameters (``service``, ``hook``) are optional. They represent the specific configurations
+that triggered the ``target`` call. Optional arguments can be specified in any order or combination, but **MUST** use
+the exact argument names indicated below.
+
+.. code-block:: python
+
+    def request_hook(request: pyramid.request.Request) -> pyramid.request.Request: ...
+
+    def request_hook(request: pyramid.request.Request,
+                     service: magpie.typedefs.ServiceConfigItem,
+                     hook: magpie.typedefs.ServiceHookConfigItem) -> pyramid.request.Request: ...
+
+    def response_hook(response: pyramid.response.Response) -> pyramid.response.Response: ...
+
+    def response_hook(response: pyramid.response.Response,
+                      service: magpie.typedefs.ServiceConfigItem,
+                      hook: magpie.typedefs.ServiceHookConfigItem) -> pyramid.response.Response: ...
+
+.. seealso::
+    File `providers.cfg`_ presents contextual information and location of the ``hooks`` schema under
+    example provider definitions.
+
+    File |test-hooks|_ presents some examples of hook ``target`` functions with common operations to
+    update request and response parameters.
+
+.. |test-hooks| replace:: tests/hooks/request_hooks.py
+.. _test-hooks: https://github.com/Ouranosinc/Magpie/blob/master/tests/hooks/request_hooks.py
 
 .. _config_constants:
 
@@ -1409,7 +1490,9 @@ User Creation
 ~~~~~~~~~~~~~~~
 
 .. list-table::
+    :header-rows: 0
     :stub-columns: 1
+    :widths: 10,90
 
     * - Action
       - :attr:`WebhookAction.CREATE_USER`
@@ -1434,7 +1517,9 @@ User Deletion
 ~~~~~~~~~~~~~~~
 
 .. list-table::
+    :header-rows: 0
     :stub-columns: 1
+    :widths: 10,90
 
     * - Action
       - :attr:`WebhookAction.DELETE_USER`
@@ -1450,7 +1535,9 @@ User Status Update
 ~~~~~~~~~~~~~~~~~~~
 
 .. list-table::
+    :header-rows: 0
     :stub-columns: 1
+    :widths: 10,90
 
     * - Action
       - :attr:`WebhookAction.UPDATE_USER_STATUS`
@@ -1477,7 +1564,9 @@ Below :term:`Webhook` implementations can all be configured for any combination
 :term:`Permission` for a :term:`User` or :term:`Group`, and targeting either a :term:`Service` or a :term:`Resource`.
 
 .. list-table::
+    :header-rows: 0
     :stub-columns: 1
+    :widths: 10,90
 
     * - Action
       - :attr:`WebhookAction.CREATE_USER_PERMISSION`, :attr:`WebhookAction.DELETE_USER_PERMISSION`,

docs/references.rst

@@ -64,6 +64,6 @@
 .. _Magpie Security: https://github.com/Ouranosinc/Magpie/tree/master/magpie/security.py
 .. _permissions.cfg: https://github.com/Ouranosinc/Magpie/tree/master/config/permissions.cfg
 .. _postgres.env.example: https://github.com/Ouranosinc/Magpie/tree/master/env/postgres.env.example
-.. _providers.cfg: https://github.com/Ouranosinc/Magpie/tree/master/config/permissions.cfg
+.. _providers.cfg: https://github.com/Ouranosinc/Magpie/tree/master/config/providers.cfg
 .. _themes: https://github.com/Ouranosinc/Magpie/tree/master/magpie/ui/home/static/themes
 .. _tests: https://github.com/Ouranosinc/Magpie/tree/master/tests

docs/services.rst

@@ -98,6 +98,8 @@ follow parameters for parsing :term:`OWS` requests.
 .. list-table::
     :header-rows: 1
 
+    * - Attribute
+      - Description
     * - :attr:`ServiceOWS.params_expected` |br| (``List[str]``)
       - Represents specific parameter names that can be preprocessed during HTTP request parsing to ease following
         resolution of :term:`ACL` use cases.
@@ -110,6 +112,8 @@ Furthermore, some :term:`Services <Service>` specifically implement extended :te
 .. list-table::
     :header-rows: 1
 
+    * - Attribute
+      - Description
     * - :attr:`ServiceGeoserverBase.resource_scoped` |br| (``bool``)
       - Indicates if the :term:`Service` is allowed to employ scoped :class:`models.Workspace` naming, meaning that
         a :term:`Resource` of that type can be extracted either from the request path or the specific request parameter

magpie/adapter/__init__.py

@@ -1,3 +1,6 @@
+import copy
+import inspect
+import re
 import warnings
 from distutils.version import LooseVersion
 from typing import TYPE_CHECKING
@@ -32,7 +35,9 @@
     get_logger,
     get_magpie_url,
     get_settings,
+    import_target,
     is_json_body,
+    normalize_field_pattern,
     setup_cache_settings,
     setup_session_config
 )
@@ -48,10 +53,17 @@
 if LooseVersion(twitcher_version) >= LooseVersion("0.6.0"):
     from twitcher.owsregistry import OWSRegistry  # noqa  # pylint: disable=E0611  # Twitcher >= 0.6.x
 
-    if LooseVersion(twitcher_version) >= LooseVersion("0.7.0"):
+    if LooseVersion(twitcher_version) >= LooseVersion("0.8.0"):
         warnings.warn(
             "Magpie version is not guaranteed to work with newer versions of Twitcher. "
-            "This Magpie version offers compatibility with Twitcher 0.6.x. "
+            "This Magpie version offers compatibility with Twitcher 0.6.x and 0.7.x."
+            "Current package versions are (Twitcher: {}, Magpie: {})".format(twitcher_version, magpie_version),
+            ImportWarning
+        )
+    elif LooseVersion(twitcher_version) < LooseVersion("0.7.0"):
+        warnings.warn(
+            "Magpie version offers more capabilities than Twitcher 0.6.x is able to provide. "
+            "Consider updating to more recent Twitcher 0.7.x to make use of new functionalities. "
             "Current package versions are (Twitcher: {}, Magpie: {})".format(twitcher_version, magpie_version),
             ImportWarning
         )
@@ -79,9 +91,11 @@
     from pyramid.authentication import AuthTktCookieHelper
     from pyramid.config import Configurator
     from pyramid.request import Request
+    from pyramid.response import Response
 
-    from magpie.typedefs import JSON, AnyResponseType, AnySettingsContainer, Str
+    from magpie.typedefs import JSON, AnyResponseType, AnySettingsContainer, ServiceHookType, Str
 
+    from twitcher.models.service import ServiceConfig     # noqa  # pylint: disable=E0611  # Twitcher >= 0.6.3
     from twitcher.store import AccessTokenStoreInterface  # noqa  # pylint: disable=E0611  # Twitcher <= 0.5.x
 
 LOGGER = get_logger("TWITCHER|{}".format(__name__))
@@ -243,3 +257,83 @@ def configurator_factory(self, container):  # noqa: R0201
         config.add_view(verify_user, route_name="verify-user")
 
         return config
+
+    def _apply_hooks(self, instance, service_name, hook_type, method, path, query):
+        # type: (Union[Request, Response], Str, ServiceHookType, Str, Str, Str) -> Union[Request, Response]
+        """
+        Executes the hooks processing chain.
+        """
+        svc_config = self.settings.get("magpie.services", {}).get(service_name, {})
+        svc_hooks = svc_config.get("hooks", [])
+        # copy to avoid (un)intentional modifications to configurations
+        svc_config = copy.deepcopy(svc_config)
+        for hook_cfg in svc_hooks:
+            if hook_cfg["type"] != hook_type:
+                continue
+            if hook_cfg["method"] not in ["*", method]:
+                continue
+            hook_path = normalize_field_pattern(hook_cfg["path"])
+            if not re.match(hook_path, path):
+                continue
+            hook_query = normalize_field_pattern(hook_cfg["query"])
+            if not re.match(hook_query, query):
+                continue
+            hook_target = import_target(hook_cfg["target"])
+            hook_qs = "?" + query if query else ""
+            if not hook_target:
+                LOGGER.warning("Hook matched %s (%s %s%s) but specified target [%s] could not be loaded.",
+                               hook_type, method, path, hook_qs, hook_cfg["target"])
+                continue
+            LOGGER.debug("Hook matched %s (%s %s%s) [%s]", hook_type, method, path, hook_qs, hook_cfg["target"])
+            signature = inspect.Signature(hook_target)
+            kwargs = {}
+            if len(signature.parameters) > 1:
+                hook = copy.deepcopy(hook_cfg)
+                for key, val in [("service", svc_config), ("hook", hook)]:
+                    if key in signature.parameters:
+                        kwargs[key] = val
+            try:
+                instance = hook_target(instance, **kwargs)
+            except Exception as exc:
+                LOGGER.error("Hook failed %s (%s %s%s) [%s]",
+                             hook_type, method, path, hook_qs, hook_cfg["target"], exc_info=exc)
+                raise exc
+        return instance
+
+    def request_hook(self, request, service):
+        # type: (Request, ServiceConfig) -> Request
+        """
+        Apply modifications onto the request before sending it.
+
+        .. versionadded:: 3.25
+            Requires ``Twitcher >= 0.7.x``.
+
+        Request members employed after this hook is called include:
+        - :meth:`Request.headers`
+        - :meth:`Request.method`
+        - :meth:`Request.body`
+
+        This method can modified those members to adapt the request for specific service logic.
+        """
+        request = self._apply_hooks(
+            request, service["name"], "request",
+            request.method, request.path, request.query_string
+        )
+        return request
+
+    def response_hook(self, response, service):
+        # type: (Response, ServiceConfig) -> Response
+        """
+        Apply modifications onto the response from sent request.
+
+        .. versionadded:: 3.25
+            Requires ``Twitcher >= 0.7.x``.
+
+        The received response from the proxied service is normally returned directly.
+        This method can modify the response to adapt it for specific service logic.
+        """
+        response = self._apply_hooks(
+            response, service["name"], "response",
+            response.request.path, response.request.path, response.request.query_string
+        )
+        return response

magpie/api/requests.py

@@ -1,3 +1,4 @@
+import json
 from typing import TYPE_CHECKING
 
 import six