feat(invoke): reading Metadata adding to Resources in a template (aws…

…#907)

designs/resource_metadata_overriding.md

@@ -0,0 +1,170 @@
+Understand Resource Level Metadata
+==================================
+
+This is a design to capture how SAM CLI can support templates that are generated 
+from different frameworks, e.g. AWS Cloud Development Kit.
+
+Initially, the support will only be for processing Resource Metadata within the template, which enables support for
+customers using AWS Cloud Development Kit (CDK).
+
+What is the problem?
+--------------------
+
+Customers have different ways to define their AWS Resources. As of writing (Jan. 2109),
+SAM CLI supports the use case of defining an application in CloudFormation/SAM (a super
+set of CloudFormation). These CloudFormation/SAM applications are written in `json` or `yaml`
+and deployed through AWS CloudFormation. Frameworks like CDK offer customers an alternative
+in how they define their applications. SAM CLI should support the ability to invoke functions 
+defined through these other frameworks to enable them to locally debug or manage their 
+applications.
+
+What will be changed?
+---------------------
+
+To start, we will add support for processing Resource Metadata that is embedded into the template:
+SAM CLI will add a processing step on the templates it reads. This will consist of reading
+the template and for each resource reading the Metadata and replacing values as specified.
+
+In the future, we can support creating these templates from the different frameworks in a command directly within 
+SAM CLI but is out of scope in the initial implementation of support.
+
+Success criteria for the change
+-------------------------------
+
+* Ability to invoke functions locally that contain Metadata on a Resource
+* Process a template with CDK Metadata on a resource.
+
+Out-of-Scope
+------------
+
+* A command that will generate the template from the framework.
+
+User Experience Walkthrough
+---------------------------
+
+CDK is a framework that appends this Metadata to Resources within a template and will use this as an example.
+
+### Customer using CDK
+
+A customer will use CDK to generate the template. This can be done by generating a template and saving it to a file:
+`cdk synth > template.yaml`. Then will then be able to `sam local [invoke|start-api|start-lambda]` any 
+function they have defined [1].   
+
+
+[1] Note: The cdk version must be greater than v0.21.0 as the metadata needed to parse is not appended on older versions. 
+
+
+Implementation
+==============
+
+CLI Changes
+-----------
+
+For the features currently in scope, there are no changes to the CLI interface.
+
+### Breaking Change
+
+No breaking changes
+
+Design
+------
+
+All the providers, which are used to get resources out of the template provided to the command, call 
+`SamBaseProvider.get_template(template_dict, parameter_overrides)` to get a normalized template. This function call is 
+responsible for taking a SAM template dictionary and returning a cleaned copy of the template where SAM plugins have 
+been run and parameter values have been substituted. Given the current scope of this call, expanding it to also normalize 
+metadata, seems reasonable. We will expand `SamBaseProvider.get_tempalte()` to call a `ResourceMetadataNormalizer` class
+that will be responsible for understanding the metadata and normalizing the template with respect to the metadata. 
+
+Template snippet that contains the metadata SAM CLI will parse and understand.
+
+```yaml
+
+Resources:
+    MyFunction:
+      Type: AWS::Lambda::Function
+      Properties:
+        Code: 
+          S3Bucket: mybucket
+          S3Key: myKey
+        ...
+      Metadata:
+        aws:asset:path: '/path/to/function/code'
+        aws:asset:property: 'Code'
+```
+
+The two keys we will recognize are `aws:asset:path` and `aws:asset:property`. `aws:asset:path`'s value will be the path
+to the code, files, etc that are on the machine, while `aws:asset:property` is the Property of the Resource that
+needs to be replaced. So in the example above, the `Code` Property will be replaced with `/path/to/function/code`.
+
+Below algorithm to do this Metadata Normalization on the template.
+
+```python
+class ResourceMetadataNormalizer(object):
+
+    @staticmethod
+    def normalize(template_dict):
+        for resource in template_dict.get('Resources'):
+            if 'Metadata' in resource:
+                asset_property = resource.get('Metadata').get('aws:asset:property')
+                asset_path = resource.get('Metadata').get('aws:asset:path')
+                ResourceMetadataNormalizer.replace_property(asset_property, asset_path)
+```
+
+`.samrc` Changes
+----------------
+
+N/A
+
+Security
+--------
+
+*Tip: How does this change impact security? Answer the following
+questions to help answer this question better:*
+
+**What new dependencies (libraries/cli) does this change require?**
+
+None
+
+**What other Docker container images are you using?**
+
+None
+
+**Are you creating a new HTTP endpoint? If so explain how it will be
+created & used**
+
+No
+
+**Are you connecting to a remote API? If so explain how is this
+connection secured**
+
+No
+
+**Are you reading/writing to a temporary folder? If so, what is this
+used for and when do you clean up?**
+
+No
+
+**How do you validate new .samrc configuration?**
+
+N/A
+
+Documentation Changes
+---------------------
+
+* Blog or Documentation that explains how you can define an application in CDK and use SAM CLI to test/invoke
+
+Open Issues
+-----------
+
+Task Breakdown
+--------------
+
+-   \[x\] Send a Pull Request with this design document
+-   \[ \] Build the command line interface
+-   \[ \] Build the underlying library
+-   \[ \] Unit tests
+-   \[ \] Functional Tests
+-   \[ \] Integration tests
+-   \[ \] Run all tests on Windows
+-   \[ \] Update documentation

samcli/commands/local/lib/sam_base_provider.py

@@ -4,10 +4,12 @@
 
 import logging
 
-from samcli.lib.samlib.wrapper import SamTranslatorWrapper
 from samtranslator.intrinsics.resolver import IntrinsicsResolver
 from samtranslator.intrinsics.actions import RefAction
 
+from samcli.lib.samlib.wrapper import SamTranslatorWrapper
+from samcli.lib.samlib.resource_metadata_normalizer import ResourceMetadataNormalizer
+
 
 LOG = logging.getLogger(__name__)
 
@@ -60,6 +62,7 @@ def get_template(template_dict, parameter_overrides=None):
             template_dict = SamTranslatorWrapper(template_dict).run_plugins()
 
         template_dict = SamBaseProvider._resolve_parameters(template_dict, parameter_overrides)
+        ResourceMetadataNormalizer.normalize(template_dict)
         return template_dict
 
     @staticmethod

samcli/lib/samlib/resource_metadata_normalizer.py

@@ -0,0 +1,63 @@
+"""
+Class that Normalizes a Template based on Resource Metadata
+"""
+
+import logging
+
+RESOURCES_KEY = "Resources"
+PROPERTIES_KEY = "Properties"
+METADATA_KEY = "Metadata"
+ASSET_PATH_METADATA_KEY = "aws:asset:path"
+ASSET_PROPERTY_METADATA_KEY = "aws:asset:property"
+
+LOG = logging.getLogger(__name__)
+
+
+class ResourceMetadataNormalizer(object):
+
+    @staticmethod
+    def normalize(template_dict):
+        """
+        Normalize all Resources in the template with the Metadata Key on the resource.
+
+        This method will mutate the template
+
+        Parameters
+        ----------
+        template_dict dict
+            Dictionary representing the template
+
+        """
+        resources = template_dict.get(RESOURCES_KEY, {})
+
+        for logical_id, resource in resources.items():
+            resource_metadata = resource.get(METADATA_KEY, {})
+            asset_path = resource_metadata.get(ASSET_PATH_METADATA_KEY)
+            asset_property = resource_metadata.get(ASSET_PROPERTY_METADATA_KEY)
+
+            ResourceMetadataNormalizer._replace_property(asset_property, asset_path, resource, logical_id)
+
+    @staticmethod
+    def _replace_property(property_key, property_value, resource, logical_id):
+        """
+        Replace a property with an asset on a given resource
+
+        This method will mutate the template
+
+        Parameters
+        ----------
+        property str
+            The property to replace on the resource
+        property_value str
+            The new value of the property
+        resource dict
+            Dictionary representing the Resource to change
+        logical_id str
+            LogicalId of the Resource
+
+        """
+        if property_key and property_value:
+            resource.get(PROPERTIES_KEY, {})[property_key] = property_value
+        elif property_key or property_value:
+            LOG.info("WARNING: Ignoring Metadata for Resource %s. Metadata contains only aws:asset:path or "
+                     "aws:assert:property but not both", logical_id)

tests/integration/local/invoke/test_integrations_cli.py

@@ -36,6 +36,17 @@ def test_invoke_returncode_is_zero(self):
 
         self.assertEquals(return_code, 0)
 
+    def test_function_with_metadata(self):
+        command_list = self.get_command_list("FunctionWithMetadata",
+                                             template_path=self.template_path,
+                                             no_event=True)
+
+        process = Popen(command_list, stdout=PIPE)
+        process.wait()
+        process_stdout = b"".join(process.stdout.readlines()).strip()
+
+        self.assertEquals(process_stdout.decode('utf-8'), '"Hello World in a different dir"')
+
     def test_invoke_returns_execpted_results(self):
         command_list = self.get_command_list("HelloWorldServerlessFunction",
                                              template_path=self.template_path,

tests/integration/testdata/invoke/different_code_location/main.py

@@ -0,0 +1,2 @@
+def echo_hello_world(event, context):
+    return "Hello World in a different dir"

tests/integration/testdata/invoke/template.yml

@@ -88,6 +88,15 @@ Resources:
       Timeout:
         Ref: DefaultTimeout
 
+  FunctionWithMetadata:
+    Type: AWS::Serverless::Function
+    Properties:
+      Runtime: python3.6
+      Handler: main.echo_hello_world
+    Metadata:
+      aws:asset:property: CodeUri
+      aws:asset:path: ./different_code_location
+
   EchoEnvWithParameters:
     Type: AWS::Serverless::Function
     Properties:

tests/unit/commands/local/lib/test_sam_base_provider.py

@@ -178,11 +178,18 @@ def test_must_skip_empty_template(self):
 
 class TestSamBaseProvider_get_template(TestCase):
 
+    @patch("samcli.commands.local.lib.sam_base_provider.ResourceMetadataNormalizer")
     @patch("samcli.commands.local.lib.sam_base_provider.SamTranslatorWrapper")
     @patch.object(SamBaseProvider, "_resolve_parameters")
-    def test_must_run_translator_plugins(self, resolve_params_mock, SamTranslatorWrapperMock):
+    def test_must_run_translator_plugins(self,
+                                         resolve_params_mock,
+                                         SamTranslatorWrapperMock,
+                                         resource_metadata_normalizer_patch):
         translator_instance = SamTranslatorWrapperMock.return_value = Mock()
 
+        parameter_resolved_template = {"Key": "Value", "Parameter": "Resolved"}
+        resolve_params_mock.return_value = parameter_resolved_template
+
         template = {"Key": "Value"}
         overrides = {'some': 'value'}
 
@@ -191,3 +198,4 @@ def test_must_run_translator_plugins(self, resolve_params_mock, SamTranslatorWra
         SamTranslatorWrapperMock.assert_called_once_with(template)
         translator_instance.run_plugins.assert_called_once()
         resolve_params_mock.assert_called_once()
+        resource_metadata_normalizer_patch.normalize.assert_called_once_with(parameter_resolved_template)