feat: Support maximum concurrency of Lambda with SQS as an event source #402

davidegiunchi · 2023-01-21T16:46:59Z

Description

Aws recently introduced maximum concurrency of lambda functions when using sqs as an event source ( https://aws.amazon.com/it/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/ ), i've added the support of this feature to this module.
This is my first contribution to this repo, please check the code since i might be wrong.

Motivation and Context

Limiting lambda concurrency with SQS without setting reserved concurrency.

Breaking Changes

If you want to set maximum_concurrency_sqs_scaling you need the aws provider >= 4.51.0, because the support for scaling_config has been introduced in 4.51.0: https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md#4510-january-19-2023

How Has This Been Tested?

I have updated at least one of the examples/* to demonstrate and validate my change(s)
I have tested and validated these changes using one or more of the provided examples/* projects

I have executed pre-commit run -a on my pull request

bryantbiggs · 2023-01-22T13:51:48Z

main.tf

@@ -280,6 +280,14 @@ resource "aws_lambda_event_source_mapping" "this" {
    }
  }

+  dynamic "scaling_config" {
+    for_each = try(each.value.maximum_concurrency_sqs_scaling, null) != null ? [true] : []


Suggested change

for_each = try(each.value.maximum_concurrency_sqs_scaling, null) != null ? [true] : []

for_each = try([each.value.scaling_config], [])

resolved in 748c6c2

bryantbiggs · 2023-01-22T13:52:47Z

main.tf

+  dynamic "scaling_config" {
+    for_each = try(each.value.maximum_concurrency_sqs_scaling, null) != null ? [true] : []
+    content {
+      maximum_concurrency = maximum_concurrency_sqs_scaling.value


Suggested change

maximum_concurrency = maximum_concurrency_sqs_scaling.value

maximum_concurrency = try(scaling_config.value.maximum_concurrency, null)

resolved in 748c6c2

bryantbiggs · 2023-01-22T13:53:58Z

examples/event-source-mapping/main.tf

-      function_response_types = ["ReportBatchItemFailures"]
+      event_source_arn                = aws_sqs_queue.this.arn
+      function_response_types         = ["ReportBatchItemFailures"]
+      maximum_concurrency_sqs_scaling = "2"


this would become

scaling_config = { maximum_concurrency = 2 }

resolved in 748c6c2

github-actions · 2023-02-22T00:16:38Z

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

davidegiunchi · 2023-02-22T15:47:52Z

not stale

DStranger · 2023-03-20T09:43:28Z

@davidegiunchi this is a welcome change! Curious, what is the status of this PR?

davidegiunchi · 2023-03-20T10:48:25Z

@DStranger the PR ha already been checked and corrected by Bryant, we are waiting for the PR to be merged (i think by Anton)

jomast · 2023-03-30T12:44:00Z

are we simply waiting for Bryant's suggestions to be implemented?

jomast · 2023-03-30T12:52:12Z

is this feature already implemented? https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_event_source_mapping#maximum_concurrency

antonbabenko · 2023-03-31T11:02:44Z

@davidegiunchi Please fix this PR as Bryant suggested, and I will be able to merge it.

jonathangueedes · 2023-04-03T18:37:02Z

@davidegiunchi help us with this please.

bryantbiggs · 2023-04-03T19:04:08Z

let me see if I can push changes - if I can, I can probably get it updated and released - one sec

…g and remove warning on deprecated EC2 platform attribute

bryantbiggs · 2023-04-03T20:35:43Z

examples/alias/main.tf

@@ -2,7 +2,6 @@ provider "aws" {
  region = "eu-west-1"

  # Make it faster by skipping something
-  skip_get_ec2_platforms      = true


removed due to

bryantbiggs · 2023-04-03T20:36:32Z

examples/event-source-mapping/main.tf

@@ -20,12 +27,15 @@ module "lambda_function" {
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

-  source_path = "${path.module}/../fixtures/python3.8-app1"
+  source_path = "${path.module}/../fixtures/python3.8-app1/index.py"


this was throwing an error around a specific version of python being required. Changing to point directly to the individual file fixed that. I don't believe that affects the intent of this example

bryantbiggs · 2023-04-03T20:36:59Z

examples/event-source-mapping/main.tf

-data "aws_vpc" "default" {
-  default = true
-}
+module "vpc" {


I don't have any default VPCs, so added a bare bones one to make it work for the RabbitMQ requirements

bryantbiggs · 2023-04-03T20:37:15Z

examples/event-source-mapping/main.tf

 }

 resource "aws_mq_broker" "this" {
  broker_name        = random_pet.this.id
  engine_type        = "RabbitMQ"
-  engine_version     = "3.8.11"


this is almost at EOS so updated to latest recommended by AWS

bryantbiggs

thank you!

## [4.13.0](v4.12.1...v4.13.0) (2023-04-03) ### Features * Support maximum concurrency of Lambda with SQS as an event source ([#402](#402)) ([268975c](268975c))

antonbabenko · 2023-04-03T20:46:17Z

This PR is included in version 4.13.0 🎉

github-actions · 2023-05-04T02:09:21Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

davidegiunchi added 2 commits January 21, 2023 17:13

maximum_concurrency support for source mapping of sqs

91e4b8e

fix correct module syntax

13e655a

bryantbiggs changed the title ~~support maximum concurrency of lambda with sqs as event_source~~ feat: Support maximum concurrency of Lambda with SQS as an event source Jan 22, 2023

bryantbiggs reviewed Jan 22, 2023

View reviewed changes

github-actions bot added the stale label Feb 22, 2023

github-actions bot removed the stale label Feb 23, 2023

Merge branch 'master' into master

4247b3c

chore: Modify for_each syntax; update example to use VPC for testin…

748c6c2

…g and remove warning on deprecated EC2 platform attribute

bryantbiggs reviewed Apr 3, 2023

View reviewed changes

bryantbiggs approved these changes Apr 3, 2023

View reviewed changes

bryantbiggs merged commit 268975c into terraform-aws-modules:master Apr 3, 2023

antonbabenko pushed a commit that referenced this pull request Apr 3, 2023

chore(release): version 4.13.0 [skip ci]

95e6a57

## [4.13.0](v4.12.1...v4.13.0) (2023-04-03) ### Features * Support maximum concurrency of Lambda with SQS as an event source ([#402](#402)) ([268975c](268975c))

antonbabenko mentioned this pull request Apr 18, 2023

feat: Adding variable principal_org_id to resource aws_lambda_permission #448

Merged

github-actions bot locked as resolved and limited conversation to collaborators May 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support maximum concurrency of Lambda with SQS as an event source #402

feat: Support maximum concurrency of Lambda with SQS as an event source #402

davidegiunchi commented Jan 21, 2023

bryantbiggs Jan 22, 2023 •

edited

Loading

bryantbiggs Apr 3, 2023

bryantbiggs Jan 22, 2023 •

edited

Loading

bryantbiggs Apr 3, 2023

bryantbiggs Jan 22, 2023

bryantbiggs Apr 3, 2023

github-actions bot commented Feb 22, 2023

davidegiunchi commented Feb 22, 2023

DStranger commented Mar 20, 2023

davidegiunchi commented Mar 20, 2023

jomast commented Mar 30, 2023

jomast commented Mar 30, 2023

antonbabenko commented Mar 31, 2023

jonathangueedes commented Apr 3, 2023

bryantbiggs commented Apr 3, 2023

bryantbiggs Apr 3, 2023

bryantbiggs Apr 3, 2023

bryantbiggs Apr 3, 2023

bryantbiggs Apr 3, 2023

bryantbiggs left a comment

antonbabenko commented Apr 3, 2023

github-actions bot commented May 4, 2023

	for_each = try(each.value.maximum_concurrency_sqs_scaling, null) != null ? [true] : []
	for_each = try([each.value.scaling_config], [])

	maximum_concurrency = maximum_concurrency_sqs_scaling.value
	maximum_concurrency = try(scaling_config.value.maximum_concurrency, null)

feat: Support maximum concurrency of Lambda with SQS as an event source #402

feat: Support maximum concurrency of Lambda with SQS as an event source #402

Conversation

davidegiunchi commented Jan 21, 2023

Description

Motivation and Context

Breaking Changes

How Has This Been Tested?

bryantbiggs Jan 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryantbiggs Jan 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 22, 2023

davidegiunchi commented Feb 22, 2023

DStranger commented Mar 20, 2023

davidegiunchi commented Mar 20, 2023

jomast commented Mar 30, 2023

jomast commented Mar 30, 2023

antonbabenko commented Mar 31, 2023

jonathangueedes commented Apr 3, 2023

bryantbiggs commented Apr 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryantbiggs left a comment

Choose a reason for hiding this comment

antonbabenko commented Apr 3, 2023

github-actions bot commented May 4, 2023

bryantbiggs Jan 22, 2023 •

edited

Loading

bryantbiggs Jan 22, 2023 •

edited

Loading