Release v1.34.0 (#1893)

* Support DLQ, RetryPolicy properties for EventBridgeRule,Schedule event sources (#1842) * Add DeadLetterConfig,RetryPolicy properties for EventBridgeRule,Schedule event sources * Minor fix,rename function argument * Update test class name * Combine dlq extraction/generation into the utility class * Remove unused import * fix: propagate condition to sqs queue policy for sqssubscription (#1798) * fix: propagate condition to sqs queue policy for sqssubscription * Update unit test for function_event_conditions * Update black commands in Makefile to check only .py files * Update test with one more SNS event source with sqsSubscription set * Revert "Update black commands in Makefile to check only .py files" This reverts commit 115ff09. * chore: Remove biased language from pylintrc (#1847) * fix: Support new CodeDeploy ManagedPolicy (#1858) * fix: Support new CodeDeploy MangedPolicy in regions without AWSCodeDeployRoleForLambda CodeDeploy is migrating from AWSCodeDeployRoleForLambda to AWSCodeDeployRoleForLambdaLimited. Some partitions do not support AWSCodeDeployRoleForLambda and therefore we need to use the newer one in those partitions. We cannot widely update to AWSCodeDeployRoleForLambdaLimited since this can cause customer's stacks to fail unexpectedly. * Forgot to commit unit tests * Handle PR feedback Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> * fix: Update Slack invite link (#1877) Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> * feature: Support for custom checkpointing (#1883) Co-authored-by: Vinayak <vinayaks@amazon.com> * Fix: Description in AWS::Serverless::HttpApi (#1884) * Fix: Description in AWS::Serverless::HttpApi * Update _set to _add * Update AWS::S3::Bucket properties (#1885) * Update AWS::S3::Bucket properties * Fix type checking validators for AWS::S3::Bucket * Update to use any_type() in favor of supporing ref * Fix: Replaced invalid AMQ managed policy by providing policy statements (#1891) * Fix for invalid MQ event source managed policy * Fix for invalid managed policy for MQ, included support for new MQ event source property, updated test cases * Black reformatting * Test case changes * Changed policy name * Modified test cases with new policy name * chore: bump version 1.34.0 (#1892) * Fix: SAM crashes method_definition for path is invalid (#1802) * Fix: SAM crashes method_definition for path is invalid * Fix: SAM crashes whenmappings is null * Removed print statement in test_translator * Fix: Swagger security not a dict Co-authored-by: Mufaddal Makati <mmmakati@amazon.com> Co-authored-by: ejafarli <54083696+ejafarli@users.noreply.github.com> Co-authored-by: _sam <3804518+aahung@users.noreply.github.com> Co-authored-by: Jacob Fuss <32497805+jfuss@users.noreply.github.com> Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> Co-authored-by: vinayaksood <vinayaksood.282@gmail.com> Co-authored-by: Vinayak <vinayaks@amazon.com> Co-authored-by: Qingchuan Ma <69653965+qingchm@users.noreply.github.com> Co-authored-by: Mufaddal Makati <mufaddal@rawbytes.com> Co-authored-by: Mufaddal Makati <mmmakati@amazon.com>
aws · Jan 15, 2021 · cf126c6 · cf126c6
1 parent d2f18e7
commit cf126c6
Show file tree

Hide file tree

Showing 109 changed files with 5,107 additions and 362 deletions.
diff --git a/.pylintrc b/.pylintrc
@@ -7,7 +7,7 @@
 # pygtk.require().
 #init-hook=
 
-# Add files or directories to the blacklist. They should be base names, not
+# Add files or directories to the ignore list. They should be base names, not
 # paths.
 ignore=compat.py
 

diff --git a/README.md b/README.md
@@ -87,7 +87,7 @@ Read the [SAM Documentation Contribution Guide](https://github.com/awsdocs/aws-s
 started. 
 
 ### Join the SAM Community on Slack
-[Join the SAM developers channel (#samdev)](https://join.slack.com/t/awsdevelopers/shared_invite/zt-h82odes6-qYN2Cxit7hBGIvC6oMjGpg) on Slack to collaborate with fellow community members and the AWS SAM team.
+[Join the SAM developers channel (#samdev)](https://join.slack.com/t/awsdevelopers/shared_invite/zt-idww18e8-Z1kXhI7GNuDewkweCF3YjA) on Slack to collaborate with fellow community members and the AWS SAM team.
 
 
 
diff --git a/docs/cloudformation_compatibility.rst b/docs/cloudformation_compatibility.rst
@@ -169,6 +169,8 @@ CloudWatchEvent (superseded by EventBridgeRule, see below)
 Pattern                  All
 Input                    All
 InputPath                All
+DeadLetterConfig         All
+RetryPolicy              All
 ======================== ================================== ========================
 
 EventBridgeRule
@@ -179,6 +181,8 @@ EventBridgeRule
 Pattern                  All
 Input                    All
 InputPath                All
+DeadLetterConfig         All
+RetryPolicy              All
 ======================== ================================== ========================
 
 IotRule

diff --git a/docs/internals/generated_resources.rst b/docs/internals/generated_resources.rst
@@ -455,6 +455,8 @@ Example:
           Type: Schedule
           Properties:
             Input: rate(5 minutes)
+            DeadLetterConfig:
+              Type: SQS
       ...
 
 Additional generated resources:
@@ -464,6 +466,8 @@ CloudFormation Resource Type       Logical ID
 ================================== ================================
 AWS::Lambda::Permission            MyFunction\ **MyTimer**\ Permission
 AWS::Events::Rule                  MyFunction\ **MyTimer** 
+AWS::SQS::Queue                    MyFunction\ **MyTimer**\ Queue
+AWS::SQS::QueuePolicy              MyFunction\ **MyTimer**\ QueuePolicy
 ================================== ================================
 
 CloudWatchEvent (superseded by EventBridgeRule, see below)
@@ -523,6 +527,11 @@ Example:
               detail:
                 state:
                   - terminated
+              DeadLetterConfig:
+                Type: SQS
+              RetryPolicy:
+                MaximumEventAgeInSeconds: 600
+                MaximumRetryAttempts:3
       ...
 
 Additional generated resources:
@@ -532,6 +541,8 @@ CloudFormation Resource Type       Logical ID
 ================================== ================================
 AWS::Lambda::Permission            MyFunction\ **OnTerminate**\ Permission
 AWS::Events::Rule                  MyFunction\ **OnTerminate**
+AWS::SQS::Queue                    MyFunction\ **OnTerminate**\ Queue
+AWS::SQS::QueuePolicy              MyFunction\ **OnTerminate**\ QueuePolicy
 ================================== ================================
 
 AWS::Serverless::Api

diff --git a/samtranslator/__init__.py b/samtranslator/__init__.py
@@ -1 +1 @@
-__version__ = "1.33.0"
+__version__ = "1.34.0"
diff --git a/samtranslator/intrinsics/resolver.py b/samtranslator/intrinsics/resolver.py
@@ -1,6 +1,7 @@
 # Help resolve intrinsic functions
 
 from samtranslator.intrinsics.actions import Action, SubAction, RefAction, GetAttAction
+from samtranslator.model.exceptions import InvalidTemplateException, InvalidDocumentException
 
 # All intrinsics are supported by default
 DEFAULT_SUPPORTED_INTRINSICS = {action.intrinsic_name: action() for action in [RefAction, SubAction, GetAttAction]}
@@ -17,7 +18,9 @@ def __init__(self, parameters, supported_intrinsics=DEFAULT_SUPPORTED_INTRINSICS
         """
 
         if parameters is None or not isinstance(parameters, dict):
-            raise TypeError("parameters must be a valid dictionary")
+            raise InvalidDocumentException(
+                [InvalidTemplateException("'Mappings' or 'Parameters' is either null or not a valid dictionary.")]
+            )
 
         if not isinstance(supported_intrinsics, dict) or not all(
             [isinstance(value, Action) for value in supported_intrinsics.values()]

diff --git a/samtranslator/model/api/http_api_generator.py b/samtranslator/model/api/http_api_generator.py
@@ -112,6 +112,8 @@ def _construct_http_api(self):
         if self.disable_execute_api_endpoint is not None:
             self._add_endpoint_configuration()
 
+        self._add_description()
+
         if self.definition_uri:
             http_api.BodyS3Location = self._construct_body_s3_dict()
         elif self.definition_body:
@@ -124,9 +126,6 @@ def _construct_http_api(self):
                 "add a 'HttpApi' event to an 'AWS::Serverless::Function'.",
             )
 
-        if self.description:
-            http_api.Description = self.description
-
         return http_api
 
     def _add_endpoint_configuration(self):
@@ -586,6 +585,27 @@ def _construct_stage(self):
 
         return stage
 
+    def _add_description(self):
+        """Add description to DefinitionBody if Description property is set in SAM"""
+        if not self.description:
+            return
+
+        if not self.definition_body:
+            raise InvalidResourceException(
+                self.logical_id,
+                "Description works only with inline OpenApi specified in the 'DefinitionBody' property.",
+            )
+        if self.definition_body.get("info", {}).get("description"):
+            raise InvalidResourceException(
+                self.logical_id,
+                "Unable to set Description because it is already defined within inline OpenAPI specified in the "
+                "'DefinitionBody' property.",
+            )
+
+        open_api_editor = OpenApiEditor(self.definition_body)
+        open_api_editor.add_description(self.description)
+        self.definition_body = open_api_editor.openapi
+
     def to_cloudformation(self):
         """Generates CloudFormation resources from a SAM HTTP API resource
 

diff --git a/samtranslator/model/eventbridge_utils.py b/samtranslator/model/eventbridge_utils.py
@@ -0,0 +1,54 @@
+from samtranslator.model.sqs import SQSQueue, SQSQueuePolicy, SQSQueuePolicies
+from samtranslator.model.exceptions import InvalidEventException
+
+
+class EventBridgeRuleUtils:
+    @staticmethod
+    def create_dead_letter_queue_with_policy(rule_logical_id, rule_arn, queue_logical_id=None):
+        resources = []
+
+        queue = SQSQueue(queue_logical_id or rule_logical_id + "Queue")
+        dlq_queue_arn = queue.get_runtime_attr("arn")
+        dlq_queue_url = queue.get_runtime_attr("queue_url")
+
+        # grant necessary permission to Eventbridge Rule resource for sending messages to dead-letter queue
+        policy = SQSQueuePolicy(rule_logical_id + "QueuePolicy")
+        policy.PolicyDocument = SQSQueuePolicies.eventbridge_dlq_send_message_resource_based_policy(
+            rule_arn, dlq_queue_arn
+        )
+        policy.Queues = [dlq_queue_url]
+
+        resources.append(queue)
+        resources.append(policy)
+
+        return resources
+
+    @staticmethod
+    def validate_dlq_config(source_logical_id, dead_letter_config):
+        supported_types = ["SQS"]
+        is_arn_defined = "Arn" in dead_letter_config
+        is_type_defined = "Type" in dead_letter_config
+        if is_arn_defined and is_type_defined:
+            raise InvalidEventException(
+                source_logical_id, "You can either define 'Arn' or 'Type' property of DeadLetterConfig"
+            )
+        if is_type_defined and dead_letter_config.get("Type") not in supported_types:
+            raise InvalidEventException(
+                source_logical_id,
+                "The only valid value for 'Type' property of DeadLetterConfig is 'SQS'",
+            )
+        if not is_arn_defined and not is_type_defined:
+            raise InvalidEventException(source_logical_id, "No 'Arn' or 'Type' property provided for DeadLetterConfig")
+
+    @staticmethod
+    def get_dlq_queue_arn_and_resources(cw_event_source, source_arn):
+        """returns dlq queue arn and dlq_resources, assuming cw_event_source.DeadLetterConfig has been validated"""
+        dlq_queue_arn = cw_event_source.DeadLetterConfig.get("Arn")
+        if dlq_queue_arn is not None:
+            return dlq_queue_arn, []
+        queue_logical_id = cw_event_source.DeadLetterConfig.get("QueueLogicalId")
+        dlq_resources = EventBridgeRuleUtils.create_dead_letter_queue_with_policy(
+            cw_event_source.logical_id, source_arn, queue_logical_id
+        )
+        dlq_queue_arn = dlq_resources[0].get_runtime_attr("arn")
+        return dlq_queue_arn, dlq_resources
diff --git a/samtranslator/model/eventsources/pull.py b/samtranslator/model/eventsources/pull.py
@@ -34,12 +34,17 @@ class PullEventSource(ResourceMacro):
         "Broker": PropertyType(False, is_str()),
         "Queues": PropertyType(False, is_type(list)),
         "SourceAccessConfigurations": PropertyType(False, is_type(list)),
+        "SecretsManagerKmsKeyId": PropertyType(False, is_str()),
         "TumblingWindowInSeconds": PropertyType(False, is_type(int)),
+        "FunctionResponseTypes": PropertyType(False, is_type(list)),
     }
 
     def get_policy_arn(self):
         raise NotImplementedError("Subclass must implement this method")
 
+    def get_policy_statements(self):
+        raise NotImplementedError("Subclass must implement this method")
+
     def to_cloudformation(self, **kwargs):
         """Returns the Lambda EventSourceMapping to which this pull event corresponds. Adds the appropriate managed
         policy to the function's execution role, if such a role is provided.
@@ -87,6 +92,7 @@ def to_cloudformation(self, **kwargs):
         lambda_eventsourcemapping.Queues = self.Queues
         lambda_eventsourcemapping.SourceAccessConfigurations = self.SourceAccessConfigurations
         lambda_eventsourcemapping.TumblingWindowInSeconds = self.TumblingWindowInSeconds
+        lambda_eventsourcemapping.FunctionResponseTypes = self.FunctionResponseTypes
 
         destination_config_policy = None
         if self.DestinationConfig:
@@ -131,8 +137,17 @@ def _link_policy(self, role, destination_config_policy=None):
         :param model.iam.IAMRole role: the execution role generated for the function
         """
         policy_arn = self.get_policy_arn()
-        if role is not None and policy_arn not in role.ManagedPolicyArns:
-            role.ManagedPolicyArns.append(policy_arn)
+        policy_statements = self.get_policy_statements()
+        if role is not None:
+            if policy_arn is not None and policy_arn not in role.ManagedPolicyArns:
+                role.ManagedPolicyArns.append(policy_arn)
+            if policy_statements is not None:
+                if role.Policies is None:
+                    role.Policies = []
+                for policy in policy_statements:
+                    if policy not in role.Policies:
+                        if not policy.get("PolicyDocument") in [d["PolicyDocument"] for d in role.Policies]:
+                            role.Policies.append(policy)
         # add SQS or SNS policy only if role is present in kwargs
         if role is not None and destination_config_policy is not None and destination_config_policy:
             if role.Policies is None:
@@ -152,6 +167,9 @@ class Kinesis(PullEventSource):
     def get_policy_arn(self):
         return ArnGenerator.generate_aws_managed_policy_arn("service-role/AWSLambdaKinesisExecutionRole")
 
+    def get_policy_statements(self):
+        return None
+
 
 class DynamoDB(PullEventSource):
     """DynamoDB Streams event source."""
@@ -161,6 +179,9 @@ class DynamoDB(PullEventSource):
     def get_policy_arn(self):
         return ArnGenerator.generate_aws_managed_policy_arn("service-role/AWSLambdaDynamoDBExecutionRole")
 
+    def get_policy_statements(self):
+        return None
+
 
 class SQS(PullEventSource):
     """SQS Queue event source."""
@@ -170,6 +191,9 @@ class SQS(PullEventSource):
     def get_policy_arn(self):
         return ArnGenerator.generate_aws_managed_policy_arn("service-role/AWSLambdaSQSQueueExecutionRole")
 
+    def get_policy_statements(self):
+        return None
+
 
 class MSK(PullEventSource):
     """MSK event source."""
@@ -179,11 +203,69 @@ class MSK(PullEventSource):
     def get_policy_arn(self):
         return ArnGenerator.generate_aws_managed_policy_arn("service-role/AWSLambdaMSKExecutionRole")
 
+    def get_policy_statements(self):
+        return None
+
 
 class MQ(PullEventSource):
     """MQ event source."""
 
     resource_type = "MQ"
 
     def get_policy_arn(self):
-        return ArnGenerator.generate_aws_managed_policy_arn("service-role/AWSLambdaAMQExecutionRole")
+        return None
+
+    def get_policy_statements(self):
+        if not self.SourceAccessConfigurations:
+            raise InvalidEventException(
+                self.relative_id,
+                "No SourceAccessConfigurations for ActiveMQ provided.",
+            )
+        if not type(self.SourceAccessConfigurations) is list:
+            raise InvalidEventException(
+                self.relative_id,
+                "Provided SourceAccessConfigurations cannot be parsed into a list.",
+            )
+        # MQ only supports SourceAccessConfigurations with list size of 1
+        if not (len(self.SourceAccessConfigurations) == 1):
+            raise InvalidEventException(
+                self.relative_id,
+                "SourceAccessConfigurations for ActiveMQ only supports single configuration entry.",
+            )
+        if not self.SourceAccessConfigurations[0].get("URI"):
+            raise InvalidEventException(
+                self.relative_id,
+                "No URI property specified in SourceAccessConfigurations for ActiveMQ.",
+            )
+        document = {
+            "PolicyName": "SamAutoGeneratedAMQPolicy",
+            "PolicyDocument": {
+                "Statement": [
+                    {
+                        "Action": [
+                            "secretsmanager:GetSecretValue",
+                        ],
+                        "Effect": "Allow",
+                        "Resource": self.SourceAccessConfigurations[0].get("URI"),
+                    },
+                    {
+                        "Action": [
+                            "mq:DescribeBroker",
+                        ],
+                        "Effect": "Allow",
+                        "Resource": self.Broker,
+                    },
+                ]
+            },
+        }
+        if self.SecretsManagerKmsKeyId:
+            kms_policy = {
+                "Action": "kms:Decrypt",
+                "Effect": "Allow",
+                "Resource": {
+                    "Fn::Sub": "arn:${AWS::Partition}:kms:${AWS::Region}:${AWS::AccountId}:key/"
+                    + self.SecretsManagerKmsKeyId
+                },
+            }
+            document["PolicyDocument"]["Statement"].append(kms_policy)
+        return [document]