Implement notify-only support

.travis.yml

@@ -36,13 +36,13 @@ jobs:
       python: '3.9'
       env: TOXENV=py39
     - stage: test
-      python: '3.8'
+      python: '3.9'
       env: TOXENV=docs
     - stage: test
-      python: '3.8'
+      python: '3.9'
       env: TOXENV=docker
     - stage: deploy
-      python: '3.7'
+      python: '3.9'
       script: bash build_or_deploy.sh build
       after_success: echo after_success
       deploy:

CHANGES.rst

@@ -1,13 +1,15 @@
 Changelog
 =========
 
-Unreleased
-----------
+1.3.0 (2021-01-13)
+------------------
 
 * Fixes `#56 <https://github.com/manheim/manheim-c7n-tools/issues/56>`__ - Bump c7n version from 0.9.4 to `0.9.10 <https://github.com/cloud-custodian/cloud-custodian/releases/tag/0.9.10.0>`__ and c7n-mailer from 0.6.3 to 0.6.9.
 * Bump relax boto3 and botocore dependencies to work with c7n and new pip resolver.
-* Begin testing against Python 3.9
+* Add testing under Python 3.9; switch default Python version for tox/TravisCI to 3.9.
 * Bump base Docker image to latest ``python:3.9.1-alpine3.12``
+* Implement :ref:`policies.notify_only`.
+* Fix failing test.
 
 1.2.4 (2020-07-29)
 ------------------

docs/source/development.rst

@@ -12,11 +12,11 @@ Development
 Local Development and Testing
 =============================
 
-Clone this repo locally on a machine with Python 3.7. Then:
+Clone this repo locally on a machine with Python 3. Then:
 
 .. code-block:: shell
 
-    virtualenv --python=python3.7 .
+    virtualenv --python=python3 .
     source bin/activate
     pip install 'tox>=3.4.0'
     pip install -r requirements.txt

docs/source/manheim_c7n_tools.notifyonly.rst

@@ -0,0 +1,7 @@
+manheim\_c7n\_tools.notifyonly module
+=====================================
+
+.. automodule:: manheim_c7n_tools.notifyonly
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/manheim_c7n_tools.rst

@@ -21,6 +21,7 @@ Submodules
    manheim_c7n_tools.config
    manheim_c7n_tools.dryrun_diff
    manheim_c7n_tools.errorscan
+   manheim_c7n_tools.notifyonly
    manheim_c7n_tools.policygen
    manheim_c7n_tools.runner
    manheim_c7n_tools.s3_archiver

docs/source/policies.rst

@@ -252,6 +252,8 @@ The full list of top-level keys valid for a policy can be found by viewing the s
    :ref:`Actions <policies.actions>` section, below, for more information.
 -  **mode** - The ``mode`` key determines how the policy will be deployed and run. See the
    :ref:`Mode <policies.mode>` section, below, for more information.
+-  **notify_only** - This is a manheim-c7n-tools addition, which is used internally and removed from the policy before :ref:`policygen` generates the final YAML files for custodian. See :ref:`policies.notify_only` for further information.
+-  **disable** - This is a manheim-c7n-tools addition, which is used internally and removed from the policy before :ref:`policygen` generates the final YAML files for custodian. See :ref:`policies.disable` for further information.
 
 .. _`policies.filters`:
 
@@ -282,6 +284,8 @@ code for each (which is liked from that documentation).
 Actions
 -------
 
+.. note:: manheim-c7n-tools' :ref:`policies.notify_only` option on a policy can effect the actions specified. See that section for more information.
+
 Cloud-custodian has both generic/global actions (such as ``notify``) and resource-specific actions
 (such as ``stop`` and ``start``). Some actions are specified as only a string (i.e. ``stop`` or
 ``start``), whereas others need to be specified as a dictionary/hash/mapping including configuration options.
@@ -418,7 +422,7 @@ Other keys under the ``mode`` section include:
 Disabling a policy
 ------------------
 
-It is possible to disable a rule. Simply setting the ``disable`` key in a policy to ``true`` will stop that policy from being
+It is possible to disable a policy. Simply setting the ``disable`` key in a policy to ``true`` will stop that policy from being
 deployed.
 
 .. code:: yaml
@@ -454,6 +458,13 @@ are later enabled for corresponding policies, the actions might be taken
 immediately when enabled as a result of the "notify only" policies
 marking resources for action.
 
+As of version 1.3.0, manheim-c7n-tools supports a :ref:`policies.notify_only` flag to help simplify this transition. For older versions, or policies that existed prior to 1.3.0, see the following section on manual tag cleanup.
+
+.. _`policies.action_transition_manual_tag_cleanup`:
+
+Manual Tag Cleanup
+------------------
+
 As a result, when adding actions to policies that have been running in
 data collection mode, it's important to manually purge the relevant tags
 so the policies don't take any action based on tags applied during data
@@ -481,3 +492,24 @@ tags with something like (example for EC2 instances):
       echo "removing tag from: $i"
       aws ec2 delete-tags --resources $i --tags Key=$tagname
     done
+
+.. _`policies.notify_only`:
+
+Notify-Only Option for Policies
+===============================
+
+As described above in :ref:`policies.action_transition`, it's common to want to run new policies in a "notify only" mode that sends notifications (and collects data) but does not yet take actions, assess those notifications, and enable actually taking action at a later date.
+
+To support this, manheim-c7n-tools (specifically :ref:`policygen`) supports the addition of a boolean ``notify_only`` option at the top level of policy files, or in ``defaults.yml`` for account- / repository-wide notify-only. Setting this flag will cause :ref:`policygen` to pass the effected policies through :py:class:`~.NotifyOnlyPolicy` for pre-processing. This will cause the following changes to the final YAML policy:
+
+* The ``comment`` / ``comments`` / ``description`` fields will be prefixed with the string ``NOTIFY ONLY:``
+* If the policy has a ``tags`` list, a ``notify-only`` tag will be appended to it.
+* All tagging actions will have the string ``-notify-only`` appended to their tag names, to automate the above-described transition. Specifically:
+
+  * Any ``mark`` or ``tag`` actions in the actions list will have the string ``-notify-only`` appended to their ``tag`` or ``key`` values (if present) or appended to every item in their ``tags`` list (if present). If none of the above are present, the ``tag`` item will be set to custodian's ``DEFAULT_TAG`` value, with ``-notify-only`` appended.
+  * Any ``mark-for-op`` actions will have the string ``-notify-only`` appended to their ``tag`` value. If they do not already have a ``tag`` value, it will be set to custodian's ``DEFAULT_TAG`` value, with ``-notify-only`` appended.
+  * Any ``remove-tag`` / ``unmark`` / ``untag`` actions wukk have the string ``-notify-only`` appended to all items in their ``tags`` list.
+
+* All ``notify`` actions will have their ``violation_desc``, if present, prefixed with ``NOTIFY ONLY:``. Their ``action_desc``, if present, will be prefixed with ``in the future (currently notify-only)``.
+* Any ``filters`` items with ``tag:NAME`` keys, which match up with ``NAME`` tags used in ``mark-for-op`` actions, will be updated to ``tag:NAME-notify-only`` to retain their intended functionality.
+* All other action types, not listed above, will be **removed from the policy**. We enforce notify-only by only retaining specifically whitelisted actions in the policy.

docs/source/policygen.rst

@@ -113,6 +113,13 @@ Rules from higher-level rulesets can be disabled by creating a new rule with the
     name: rule-name
     disable: true
 
+For further information, see :ref:`policies.disable`.
+
+Notify-Only rules
+-----------------
+
+Policygen also has support for setting rules to a notify-only mode via a single flag. Please see :ref:`policies.notify_only` for further information.
+
 Mailer Templates
 ----------------
 

manheim_c7n_tools/notifyonly.py

@@ -0,0 +1,244 @@
+# Copyright 2017-2021 Manheim / Cox Automotive
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import List
+import logging
+from c7n.tags import DEFAULT_TAG
+
+logger = logging.getLogger(__name__)
+
+
+class NotifyOnlyPolicy:
+    """
+    This class converts a c7n policy to a "notify only" policy. See
+    :ref:`policies.notify_only` for details.
+
+    IMPORTANT: When making changes to this class, be SURE to update
+    :ref:`policies.notify_only` in the documentation.
+    """
+
+    def __init__(self, policy: dict):
+        """
+        Initialize a NotifyOnlyPolicy.
+
+        :param policy: the original policy
+        :type policy: dict
+        """
+        self._original: dict = policy
+        self._mark_for_op_tags: List[str] = []
+        self._fixed: dict = self._process(self._original)
+
+    def as_notify_only(self) -> dict:
+        """
+        Return the policy, converted to a notify-only version.
+
+        :return: converted policy
+        :rtype: dict
+        """
+        return self._fixed
+
+    def _process(self, policy: dict) -> dict:
+        """
+        Return the given policy, converted to notify-only.
+
+        :param policy: the original c7n policy
+        :type policy: dict
+        :return: policy, modified for notify-only mode
+        :rtype: dict
+        """
+        if 'notify_only' in policy:
+            del policy['notify_only']
+        for k in policy.keys():
+            if k in ['comment', 'comments', 'description']:
+                policy[k] = self._fix_comment(policy[k])
+            if k == 'tags':
+                policy[k] = self._fix_tags(policy[k])
+            if k == 'actions':
+                policy[k] = self._fix_actions(policy[k])
+        # this needs to happen AFTER all _fix_actions calls...
+        if 'filters' in policy:
+            policy['filters'] = self._fix_filters(policy['filters'])
+        return policy
+
+    def _fix_filters(self, filters: List) -> List:
+        """
+        Given a list of filters from a policy, update them for any tagging
+        changes.
+
+        :param filters: filters from policy, or a subset thereof
+        :type filters: list
+        :return: fixed filters
+        :rtype: list
+        """
+        tag_changes = {
+            f'tag:{x}': f'tag:{x}-notify-only' for x in self._mark_for_op_tags
+        }
+        for idx, item in enumerate(filters):
+            # each item should be a dict
+            if not isinstance(item, type({})):
+                continue
+            for k in list(item.keys()):
+                v = item[k]
+                if isinstance(v, type([])):
+                    item[k] = self._fix_filters(v)
+                if k in tag_changes:
+                    del item[k]
+                    item[tag_changes[k]] = v
+        return filters
+
+    def _fix_comment(self, comment: str) -> str:
+        """
+        Convert a policy comment/comments/description to a notify only version,
+        by prefixing it with the string "NOTIFY ONLY: ".
+
+        :param comment: the original policy comment
+        :type comment: str
+        :return: the modified comment
+        :rtype: str
+        """
+        return f'NOTIFY ONLY: {comment}'
+
+    def _fix_tags(self, tags: List[str]) -> List[str]:
+        """
+        Convert a policy tags list to a notify only version, by appending a
+        ``notify-only`` tag to the list.
+
+        :param tags: the original tags list
+        :type tags: list
+        :return: the modified list, with a notify-only tag appended
+        :rtype: list
+        """
+        return tags + ['notify-only']
+
+    def _fix_actions(self, original: List) -> List:
+        """
+        Given a list of actions from a policy, return a new list of notify-only
+        actions.
+
+        * ``notify`` actions will be included unmodified
+        * ``mark`` / ``tag`` actions will be passed through
+          :py:meth:`~._fix_tag_action` and the result included
+        * ``mark-for-op`` actions will be passed through
+          :py:meth:`~._fix_mark_for_op_action` and the result included
+        * ``remove-tag`` / ``unmark`` / ``untag`` actions will be passed through
+          :py:meth:`~._fix_untag_action` and the result included
+        * all other actions will be REMOVED
+
+        :param original: original policy actions list
+        :type original: list
+        :return: new list of actions
+        :rtype: list
+        """
+        result = []
+        for item in original:
+            if not isinstance(item, type({})):
+                logger.info('NotifyOnlyPolicy - removing action: %s', item)
+                continue
+            a_type = item.get('type', '')
+            if a_type == 'notify':
+                result.append(self._fix_notify_action(item))
+            if a_type == 'mark' or a_type == 'tag':
+                result.append(self._fix_tag_action(item))
+            elif a_type == 'mark-for-op':
+                result.append(self._fix_mark_for_op_action(item))
+            elif a_type in ['remove-tag', 'unmark', 'untag']:
+                result.append(self._fix_untag_action(item))
+            else:
+                logger.info(
+                    'NotifyOnlyPolicy - removing %s action: %s', a_type, item
+                )
+                continue
+        return result
+
+    def _fix_notify_action(self, item: dict) -> dict:
+        """
+        Fix a ``notify`` action for notify-only operation.
+
+        If the ``violation_desc`` key is present, its value will be prefixed
+        with ``NOTIFY ONLY: ``. If the ``action_desc`` key is present, its value
+        will be prefixed with the string
+        ``in the future (currently notify-only)``.
+
+        :param item: the original action
+        :type item: dict
+        :return: the modified action
+        :rtype: dict
+        """
+        if 'violation_desc' in item:
+            item['violation_desc'] = 'NOTIFY ONLY: ' + item['violation_desc']
+        if 'action_desc' in item:
+            item['action_desc'] = 'in the future (currently notify-only) ' + \
+                                  item['action_desc']
+        return item
+
+    def _fix_tag_action(self, item: dict) -> dict:
+        """
+        Fix a ``tag`` / ``mark`` action for notify-only operation.
+
+        The string ``-notify-only`` will be appended to the ``tag`` item,
+        ``key`` item, and/or every item in the ``tags`` list.
+
+        If none of these values are set, the ``tag`` item will be set to the
+        custodian ``DEFAULT_TAG``, suffixed with ``-notify-only``.
+
+        :param item: the original action
+        :type item: dict
+        :return: the modified action
+        :rtype: dict
+        """
+        if 'tag' in item:
+            item['tag'] = item['tag'] + '-notify-only'
+        if 'key' in item:
+            item['key'] = item['key'] + '-notify-only'
+        if 'tags' in item:
+            item['tags'] = {
+                f'{k}-notify-only': v for k, v in item['tags'].items()
+            }
+        if 'tag' not in item and 'key' not in item and 'tags' not in item:
+            item['tag'] = f'{DEFAULT_TAG}-notify-only'
+        return item
+
+    def _fix_mark_for_op_action(self, item: dict) -> dict:
+        """
+        Fix a ``mark-for-op`` action for notify-only operation.
+
+        The string ``notify-only`` will be appended to the tag name used.
+
+        :param item: the original action
+        :type item: dict
+        :return: the modified action
+        :rtype: dict
+        """
+        if 'tag' not in item:
+            item['tag'] = DEFAULT_TAG
+        self._mark_for_op_tags.append(item['tag'])
+        item['tag'] += '-notify-only'
+        return item
+
+    def _fix_untag_action(self, item: dict) -> dict:
+        """
+        Fix a ``remove-tag`` / ``unmark`` / ``untag`` action for notify-only
+        operation.
+
+        All tag names in the ``tags`` list will have ``-notify-only`` appended.
+
+        :param item: the original action
+        :type item: dict
+        :return: the modified action
+        :rtype: dict
+        """
+        if 'tags' not in item:
+            item['tags'] = [f'{DEFAULT_TAG}']
+        item['tags'] = [f'{tag}-notify-only' for tag in item['tags']]
+        return item