Add resources to support request locking mechanism #71

sfc-gh-nlele · 2022-09-28T15:01:02Z

Adding resources required for geff/pull/59, such as the DynamoDB table and related policies.

sfc-gh-pkommini · 2022-10-31T16:25:22Z

examples/complete/variables.tf

+variable "dynamodb_table_name" {
+  type        = string
+  description = "Either the name of the user-created DynamoDB table, or a custom name for the default table"
+  default     = "geff-request-locking-table"


let's use the prefix and lock the name up and not take it as an input.

sfc-gh-pkommini · 2022-10-31T16:25:38Z

examples/complete/variables.tf

@@ -103,6 +103,24 @@ variable "arn_format" {
  default     = "aws"
 }

+variable "use_custom_dynamodb_table" {


Is the usage of dynamodb table optional?

sfc-gh-pkommini · 2022-10-31T16:26:37Z

examples/complete/variables.tf

+  default     = "geff-request-locking-table"
+}
+
+variable "custom_dynamodb_table_arn" {


Are we make the the dynamodb an optional resource where it is created outside and input into the module or created within the module?

sfc-gh-bkou · 2023-04-26T18:18:29Z

examples/complete/variables.tf

+variable "user_managed_dynamodb_table_name" {
+  type        = string
+  description = "Name of the user-managed custom DynamoDB table."
+  default     = ""
+}


I think it's better to use "null" for the default value instead of empty string.

sfc-gh-bkou · 2023-04-26T18:20:30Z

variables.tf

+variable "user_managed_dynamodb_table_name" {
+  type        = string
+  description = "Name of the user-managed DynamoDB table."
+  default     = ""
+}


Same thing, I think it's better to use "null" for the default value instead of empty string.

sfc-gh-nlele · 2023-04-27T16:21:00Z

dynamodb.tf

+  }
+
+  ttl {
+    attribute_name = "TimeToExist"


sfc-gh-afedorov · 2023-04-27T16:22:48Z

dynamodb.tf

+  }
+
+  tags = {
+    name = "${local.geff_prefix}_batch_locking_table"


@sfc-gh-bkou do you know of a tagging strategy that might be helpful here?

Yeah, I think we should have general tagging on the provider and then overwrite/add at the resource level. I started putting general tags in the provider for Anecdotes. But I think we should agree on the format and what information we want to capture. For a module like this, we can look into having an input map variable for users to pass in multiple tags.

provider "aws" {
region = var.aws_region
default_tags {
tags = {
environment = "prod"
owner = "compliance"
project = "anecdotes"
}
}
}

But the existing resource naming in this module is using hyphen instead of underscore. For this PR, keeping it consistent with the existing resource naming should be fine for now until we agree on the general tagging name and approach.

resource "aws_api_gateway_rest_api" "ef_to_lambda" {
name = "${local.geff_prefix}-api-gateway"

endpoint_configuration {
types = [
"REGIONAL",
]
}
}

examples/complete/variables.tf

sfc-gh-nlele · 2023-05-02T15:14:54Z

dynamodb.tf

@@ -0,0 +1,30 @@
+resource "aws_dynamodb_table" "geff_batch_locking_table" {
+  count        = var.create_dynamodb_table ? 1 : 0
+  name         = "${local.geff_prefix}_batch_locking_table"


if user_managed_dynamodb_table_name is provided along with create_table, name this table user_managed_dynamodb_table_name

sfc-gh-nlele · 2023-05-02T15:15:54Z

dynamodb.tf

+}
+
+data "aws_dynamodb_table" "user_managed_table" {
+  count = var.user_managed_dynamodb_table_name != null ? 1 : 0


if user_managed_dynamodb_table_name and create_dynamodb_table are both provided, don't create this resource

sfc-gh-afedorov

getting there, but still seeing a couple of places where we can make things more clear

sfc-gh-afedorov · 2023-05-03T20:52:05Z

dynamodb.tf

+}
+
+locals {
+  dynamodb_table_name = var.user_managed_dynamodb_table_name != null ? (var.create_dynamodb_table ? aws_dynamodb_table.geff_batch_locking_table[0].name : data.aws_dynamodb_table.user_managed_table[0].name) : (var.create_dynamodb_table ?  aws_dynamodb_table.geff_batch_locking_table[0].name : null)


I think you repeat var.create_dynamodb_table ? aws_dynamodb_table.geff_batch_locking_table[0].name in both cases of the inner tertiaries, and repeat identical logic varying only .name, so you can instead do something like

locals { dynamodb_table = ( var.create_dynamodb_table ? aws_dynamodb_table.geff_batch_locking_table[0]: var.user_managed_dynamodb_table_name ? data.aws_dynamodb_table.user_managed_table[0]: null ) }

and then use local.dynamodb_table.name / local.dynamodb_table.arn in the rest of the code. does that make sense?

Makes sense 👍

sfc-gh-afedorov · 2023-05-03T20:54:50Z

examples/complete/variables.tf

+  default     = false
+}
+
+variable "user_managed_dynamodb_table_name" {


I don't think it makes sense to call this "user managed" anymore when it might be created by the other variable being set -- could we call it batch_locking_table_name and set the default to "${local.geff_prefix}_batch_locking_table" ?

But if this has a default value, we wouldn't be able to do the above condition right, unless the user specifies something like below?

batch_locking_table_name = null create_dynamodb_table = false

locals { dynamodb_table = ( var.create_dynamodb_table ? aws_dynamodb_table.geff_batch_locking_table[0]: var. batch_locking_table_name ? data.aws_dynamodb_table.user_managed_table[0]: null ) }

yes, and that seems like the ideal default, since the perspective of most users is ignorance about the 30s limitation of API Gateway having an impact on potential workloads, and all the need to know is that this creates a dynamodb table needed for some kind of batch synchronization mechanisms and go on with their installation. the few users that want to give it a custom name or not have a backend db or make a custom should be allowed to customize it, but most users should get it without having to worry about what it is.

on that thought, we might want to call it something more general, to support other features (e.g. rate limiting) in the future, without having to rename the table, maybe _batch_sync?

sfc-gh-afedorov · 2023-05-03T20:56:05Z

examples/complete/variables.tf

+variable "create_dynamodb_table" {
+  type        = bool
+  description = "Boolean for if a DynamoDB table is to be created for batch locking."
+  default     = false


default to true

sfc-gh-afedorov · 2023-05-03T21:00:28Z

dynamodb.tf

+}
+
+data "aws_dynamodb_table" "user_managed_table" {
+  count = var.user_managed_dynamodb_table_name != null && !var.create_dynamodb_table ? 1 : 0


does the first part do anything now? afaict, the predicate could be !var.create_dynamodb_table?

I think we'd have to check for both of these, for a case where the user sets create_dynamodb_table to be false, and batch_locking_table_name to be null.

sfc-gh-afedorov · 2023-05-03T21:18:38Z

iam.tf

+# 4. Policy for the DynamoDB table to be used as a backend for batch locking
+# -----------------------------------------------------------------------------
+resource "aws_iam_policy" "dynamodb_table_policy" {
+  count = var.create_dynamodb_table || var.user_managed_dynamodb_table_name != null ? 1 : 0


can this be simplified to local.dynamodb_table? it seems that would make it a clear connection to the Resource value below

It can, but the target needs to be specified on each apply.

The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the count depends on.

So this seemed simpler than that.

not sure if I follow, how does not being able to predict how many instances will be created apply to using a local variable in the count?

that said, no strong feeling -- the invariant that local.dynamodb_table_arn exists if one of the others is non-null seems relatively clear from the code above.

The explanation I got for this was that tf needs the count value during planning to know exactly how many resources are to be created, and that it's unable to do that when a local is used in such a condition since it derives its value from attributes of other resources. It is then unable to determine the value of count until the resources it depends on, have been created in an apply.

sfc-gh-afedorov · 2023-05-03T21:18:42Z

iam.tf

+}
+
+resource "aws_iam_role_policy_attachment" "dynamodb_table_policy_attachment" {
+  count = var.create_dynamodb_table || var.user_managed_dynamodb_table_name != null ? 1 : 0


can this be simplified to local.dynamodb_table?

sfc-gh-afedorov · 2023-05-03T21:21:36Z

variables.tf

+  default     = false
+}
+
+variable "user_managed_dynamodb_table_name" {


no longer user-managed

sfc-gh-afedorov

it seems like it might work, but there's a bit more clarifying cleanup that will help me be sure, could you make those and Slack me the tf plan, or put it in the description if there isn't anything private there?

sfc-gh-afedorov

looks like there are a few changes pending here. let's make sure the default behavior is adjusted to be on, for the user that is not aware of the technical details of API Gateways.

sfc-gh-afedorov · 2023-05-04T22:05:07Z

examples/complete/variables.tf

+  default     = false
+}
+
+variable "user_managed_dynamodb_table_name" {


yes, and that seems like the ideal default, since the perspective of most users is ignorance about the 30s limitation of API Gateway having an impact on potential workloads, and all the need to know is that this creates a dynamodb table needed for some kind of batch synchronization mechanisms and go on with their installation. the few users that want to give it a custom name or not have a backend db or make a custom should be allowed to customize it, but most users should get it without having to worry about what it is.

on that thought, we might want to call it something more general, to support other features (e.g. rate limiting) in the future, without having to rename the table, maybe _batch_sync?

sfc-gh-afedorov · 2023-05-04T22:14:11Z

iam.tf

+# 4. Policy for the DynamoDB table to be used as a backend for batch locking
+# -----------------------------------------------------------------------------
+resource "aws_iam_policy" "dynamodb_table_policy" {
+  count = var.create_dynamodb_table || var.user_managed_dynamodb_table_name != null ? 1 : 0


not sure if I follow, how does not being able to predict how many instances will be created apply to using a local variable in the count?

that said, no strong feeling -- the invariant that local.dynamodb_table_arn exists if one of the others is non-null seems relatively clear from the code above.

sfc-gh-nlele · 2023-05-05T15:04:14Z

@sfc-gh-afedorov terraform plan when

create_dynamodb_table    = true
batch_locking_table_name = null

and there are no dynamodb resources present.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
 <= read (data resources)

Terraform will perform the following actions:

  # module.geff.data.aws_iam_policy_document.geff_lambda_policy_doc will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "geff_lambda_policy_doc" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "logs:CreateLogStream",
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:logs:us-west-2:123456789:log-group:/aws/lambda/rlm-dev-geff-lambda:*",
            ]
          + sid       = "WriteCloudWatchLogs"
        }
      + statement {
          + actions   = [
              + "cloudwatch:PutMetricData",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "WriteCloudWatchMetrics"

          + condition {
              + test     = "StringLike"
              + values   = [
                  + "*",
                ]
              + variable = "cloudwatch:namespace"
            }
        }
      + statement {
          + actions   = [
              + "ecr:BatchGetImage",
              + "ecr:DescribeImageScanFindings",
              + "ecr:DescribeImages",
              + "ecr:DescribeRepositories",
              + "ecr:ListImages",
              + "ecr:StartImageScan",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "EcrScanImages"
        }
      + statement {
          + actions   = [
              + "s3:GetObject",
              + "s3:GetObjectVersion",
              + "s3:PutObject",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::rlm-dev-dev-bucket/*",
            ]
          + sid       = "WriteToS3"
        }
      + statement {
          + actions   = [
              + "lambda:InvokeFunction",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:lambda:us-west-2:123456789:function:rlm-dev-geff-lambda",
            ]
          + sid       = "InvokeLambda"
        }
      + statement {
          + actions   = [
              + "secretsmanager:DescribeSecret",
              + "secretsmanager:GetResourcePolicy",
              + "secretsmanager:GetSecretValue",
              + "secretsmanager:ListSecretVersionIds",
              + "secretsmanager:ListSecrets",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "AccessSecrets"
        }
      + statement {
          + actions   = [
              + "secretsmanager:ListSecrets",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "ListSecrets"
        }
    }

  # module.geff.aws_dynamodb_table.geff_batch_locking_table[0] will be created
  + resource "aws_dynamodb_table" "geff_batch_locking_table" {
      + arn              = (known after apply)
      + billing_mode     = "PAY_PER_REQUEST"
      + hash_key         = "batch_id"
      + id               = (known after apply)
      + name             = "rlm-dev-geff_batch_locking_table"
      + read_capacity    = (known after apply)
      + stream_arn       = (known after apply)
      + stream_label     = (known after apply)
      + stream_view_type = (known after apply)
      + tags_all         = {
          + "environment" = "dev"
        }
      + write_capacity   = (known after apply)

      + attribute {
          + name = "batch_id"
          + type = "S"
        }

      + point_in_time_recovery {
          + enabled = (known after apply)
        }

      + server_side_encryption {
          + enabled     = (known after apply)
          + kms_key_arn = (known after apply)
        }

      + ttl {
          + attribute_name = "ttl"
          + enabled        = true
        }
    }

  # module.geff.aws_iam_policy.dynamodb_table_policy[0] will be created
  + resource "aws_iam_policy" "dynamodb_table_policy" {
      + arn       = (known after apply)
      + id        = (known after apply)
      + name      = "rlm-dev-geff-dynamodb-table-policy"
      + path      = "/"
      + policy    = (known after apply)
      + policy_id = (known after apply)
      + tags_all  = {
          + "environment" = "dev"
        }
    }

  # module.geff.aws_iam_role_policy.geff_lambda_policy will be updated in-place
  ~ resource "aws_iam_role_policy" "geff_lambda_policy" {
        id     = "rlm-dev-geff-lambda:rlm-dev-geff-lambda-policy"
        name   = "rlm-dev-geff-lambda-policy"
      ~ policy = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "logs:PutLogEvents",
                          - "logs:CreateLogStream",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:us-west-2:123456789:log-group:/aws/lambda/rlm-dev-geff-lambda:*"
                      - Sid      = "WriteCloudWatchLogs"
                    },
                  - {
                      - Action    = "cloudwatch:PutMetricData"
                      - Condition = {
                          - StringLike = {
                              - "cloudwatch:namespace" = "*"
                            }
                        }
                      - Effect    = "Allow"
                      - Resource  = "*"
                      - Sid       = "WriteCloudWatchMetrics"
                    },
                  - {
                      - Action   = [
                          - "ecr:StartImageScan",
                          - "ecr:ListImages",
                          - "ecr:DescribeRepositories",
                          - "ecr:DescribeImages",
                          - "ecr:DescribeImageScanFindings",
                          - "ecr:BatchGetImage",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "EcrScanImages"
                    },
                  - {
                      - Action   = [
                          - "s3:PutObject",
                          - "s3:GetObjectVersion",
                          - "s3:GetObject",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:s3:::rlm-dev-dev-bucket/*"
                      - Sid      = "WriteToS3"
                    },
                  - {
                      - Action   = "lambda:InvokeFunction"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:lambda:us-west-2:123456789:function:rlm-dev-geff-lambda"
                      - Sid      = "InvokeLambda"
                    },
                  - {
                      - Action   = [
                          - "secretsmanager:ListSecrets",
                          - "secretsmanager:ListSecretVersionIds",
                          - "secretsmanager:GetSecretValue",
                          - "secretsmanager:GetResourcePolicy",
                          - "secretsmanager:DescribeSecret",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "AccessSecrets"
                    },
                  - {
                      - Action   = "secretsmanager:ListSecrets"
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "ListSecrets"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # module.geff.aws_iam_role_policy_attachment.dynamodb_table_policy_attachment[0] will be created
  + resource "aws_iam_role_policy_attachment" "dynamodb_table_policy_attachment" {
      + id         = (known after apply)
      + policy_arn = (known after apply)
      + role       = "rlm-dev-geff-lambda"
    }

  # module.geff.aws_lambda_function.geff_lambda will be updated in-place
  ~ resource "aws_lambda_function" "geff_lambda" {
        id                             = "rlm-dev-geff-lambda"
        tags                           = {}
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              + "DYNAMODB_TABLE_NAME" = "rlm-dev-geff_batch_locking_table"
                # (3 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

Plan: 3 to add, 2 to change, 0 to destroy.

sfc-gh-nlele · 2023-05-05T15:09:11Z

terraform plan when

create_dynamodb_table    = true
batch_locking_table_name = "rlm-custom-dynamodb-table"

and there are no dynamodb resources present.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
 <= read (data resources)

Terraform will perform the following actions:

  # module.geff.data.aws_iam_policy_document.geff_lambda_policy_doc will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "geff_lambda_policy_doc" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "logs:CreateLogStream",
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:logs:us-west-2:123456789:log-group:/aws/lambda/rlm-dev-geff-lambda:*",
            ]
          + sid       = "WriteCloudWatchLogs"
        }
      + statement {
          + actions   = [
              + "cloudwatch:PutMetricData",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "WriteCloudWatchMetrics"

          + condition {
              + test     = "StringLike"
              + values   = [
                  + "*",
                ]
              + variable = "cloudwatch:namespace"
            }
        }
      + statement {
          + actions   = [
              + "ecr:BatchGetImage",
              + "ecr:DescribeImageScanFindings",
              + "ecr:DescribeImages",
              + "ecr:DescribeRepositories",
              + "ecr:ListImages",
              + "ecr:StartImageScan",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "EcrScanImages"
        }
      + statement {
          + actions   = [
              + "s3:GetObject",
              + "s3:GetObjectVersion",
              + "s3:PutObject",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::rlm-dev-dev-bucket/*",
            ]
          + sid       = "WriteToS3"
        }
      + statement {
          + actions   = [
              + "lambda:InvokeFunction",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:lambda:us-west-2:123456789:function:rlm-dev-geff-lambda",
            ]
          + sid       = "InvokeLambda"
        }
      + statement {
          + actions   = [
              + "secretsmanager:DescribeSecret",
              + "secretsmanager:GetResourcePolicy",
              + "secretsmanager:GetSecretValue",
              + "secretsmanager:ListSecretVersionIds",
              + "secretsmanager:ListSecrets",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "AccessSecrets"
        }
      + statement {
          + actions   = [
              + "secretsmanager:ListSecrets",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "ListSecrets"
        }
    }

  # module.geff.aws_dynamodb_table.geff_batch_locking_table[0] will be created
  + resource "aws_dynamodb_table" "geff_batch_locking_table" {
      + arn              = (known after apply)
      + billing_mode     = "PAY_PER_REQUEST"
      + hash_key         = "batch_id"
      + id               = (known after apply)
      + name             = "rlm-custom-dynamodb-table"
      + read_capacity    = (known after apply)
      + stream_arn       = (known after apply)
      + stream_label     = (known after apply)
      + stream_view_type = (known after apply)
      + tags_all         = {
          + "environment" = "dev"
        }
      + write_capacity   = (known after apply)

      + attribute {
          + name = "batch_id"
          + type = "S"
        }

      + point_in_time_recovery {
          + enabled = (known after apply)
        }

      + server_side_encryption {
          + enabled     = (known after apply)
          + kms_key_arn = (known after apply)
        }

      + ttl {
          + attribute_name = "ttl"
          + enabled        = true
        }
    }

  # module.geff.aws_iam_policy.dynamodb_table_policy[0] will be created
  + resource "aws_iam_policy" "dynamodb_table_policy" {
      + arn       = (known after apply)
      + id        = (known after apply)
      + name      = "rlm-dev-geff-dynamodb-table-policy"
      + path      = "/"
      + policy    = (known after apply)
      + policy_id = (known after apply)
      + tags_all  = {
          + "environment" = "dev"
        }
    }

  # module.geff.aws_iam_role_policy.geff_lambda_policy will be updated in-place
  ~ resource "aws_iam_role_policy" "geff_lambda_policy" {
        id     = "rlm-dev-geff-lambda:rlm-dev-geff-lambda-policy"
        name   = "rlm-dev-geff-lambda-policy"
      ~ policy = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "logs:PutLogEvents",
                          - "logs:CreateLogStream",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:us-west-2:123456789:log-group:/aws/lambda/rlm-dev-geff-lambda:*"
                      - Sid      = "WriteCloudWatchLogs"
                    },
                  - {
                      - Action    = "cloudwatch:PutMetricData"
                      - Condition = {
                          - StringLike = {
                              - "cloudwatch:namespace" = "*"
                            }
                        }
                      - Effect    = "Allow"
                      - Resource  = "*"
                      - Sid       = "WriteCloudWatchMetrics"
                    },
                  - {
                      - Action   = [
                          - "ecr:StartImageScan",
                          - "ecr:ListImages",
                          - "ecr:DescribeRepositories",
                          - "ecr:DescribeImages",
                          - "ecr:DescribeImageScanFindings",
                          - "ecr:BatchGetImage",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "EcrScanImages"
                    },
                  - {
                      - Action   = [
                          - "s3:PutObject",
                          - "s3:GetObjectVersion",
                          - "s3:GetObject",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:s3:::rlm-dev-dev-bucket/*"
                      - Sid      = "WriteToS3"
                    },
                  - {
                      - Action   = "lambda:InvokeFunction"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:lambda:us-west-2:123456789:function:rlm-dev-geff-lambda"
                      - Sid      = "InvokeLambda"
                    },
                  - {
                      - Action   = [
                          - "secretsmanager:ListSecrets",
                          - "secretsmanager:ListSecretVersionIds",
                          - "secretsmanager:GetSecretValue",
                          - "secretsmanager:GetResourcePolicy",
                          - "secretsmanager:DescribeSecret",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "AccessSecrets"
                    },
                  - {
                      - Action   = "secretsmanager:ListSecrets"
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "ListSecrets"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # module.geff.aws_iam_role_policy_attachment.dynamodb_table_policy_attachment[0] will be created
  + resource "aws_iam_role_policy_attachment" "dynamodb_table_policy_attachment" {
      + id         = (known after apply)
      + policy_arn = (known after apply)
      + role       = "rlm-dev-geff-lambda"
    }

  # module.geff.aws_lambda_function.geff_lambda will be updated in-place
  ~ resource "aws_lambda_function" "geff_lambda" {
        id                             = "rlm-dev-geff-lambda"
        tags                           = {}
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              + "DYNAMODB_TABLE_NAME" = "rlm-custom-dynamodb-table"
                # (3 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

Plan: 3 to add, 2 to change, 0 to destroy.

sfc-gh-nlele · 2023-05-05T15:13:57Z

terraform plan when

create_dynamodb_table    = false
batch_locking_table_name = "rlm-custom-dynamodb-table"

and "rlm-custom-dynamodb-table" is already present, and is used above.

(this also errors as expected, if create_dynamodb_table is true and the custom table already exists.)

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
 <= read (data resources)

Terraform will perform the following actions:

  # module.geff.data.aws_iam_policy_document.geff_lambda_policy_doc will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "geff_lambda_policy_doc" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "logs:CreateLogStream",
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:logs:us-west-2:123456789:log-group:/aws/lambda/rlm-dev-geff-lambda:*",
            ]
          + sid       = "WriteCloudWatchLogs"
        }
      + statement {
          + actions   = [
              + "cloudwatch:PutMetricData",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "WriteCloudWatchMetrics"

          + condition {
              + test     = "StringLike"
              + values   = [
                  + "*",
                ]
              + variable = "cloudwatch:namespace"
            }
        }
      + statement {
          + actions   = [
              + "ecr:BatchGetImage",
              + "ecr:DescribeImageScanFindings",
              + "ecr:DescribeImages",
              + "ecr:DescribeRepositories",
              + "ecr:ListImages",
              + "ecr:StartImageScan",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "EcrScanImages"
        }
      + statement {
          + actions   = [
              + "s3:GetObject",
              + "s3:GetObjectVersion",
              + "s3:PutObject",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::rlm-dev-dev-bucket/*",
            ]
          + sid       = "WriteToS3"
        }
      + statement {
          + actions   = [
              + "lambda:InvokeFunction",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:lambda:us-west-2:123456789:function:rlm-dev-geff-lambda",
            ]
          + sid       = "InvokeLambda"
        }
      + statement {
          + actions   = [
              + "secretsmanager:DescribeSecret",
              + "secretsmanager:GetResourcePolicy",
              + "secretsmanager:GetSecretValue",
              + "secretsmanager:ListSecretVersionIds",
              + "secretsmanager:ListSecrets",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "AccessSecrets"
        }
      + statement {
          + actions   = [
              + "secretsmanager:ListSecrets",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
          + sid       = "ListSecrets"
        }
    }

  # module.geff.aws_iam_policy.dynamodb_table_policy[0] will be created
  + resource "aws_iam_policy" "dynamodb_table_policy" {
      + arn       = (known after apply)
      + id        = (known after apply)
      + name      = "rlm-dev-geff-dynamodb-table-policy"
      + path      = "/"
      + policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "dynamodb:GetItem",
                          + "dynamodb:PutItem",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:aws:dynamodb:us-west-2:123456789:table/rlm-custom-dynamodb-table"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + policy_id = (known after apply)
      + tags_all  = {
          + "environment" = "dev"
        }
    }

  # module.geff.aws_iam_role_policy.geff_lambda_policy will be updated in-place
  ~ resource "aws_iam_role_policy" "geff_lambda_policy" {
        id     = "rlm-dev-geff-lambda:rlm-dev-geff-lambda-policy"
        name   = "rlm-dev-geff-lambda-policy"
      ~ policy = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "logs:PutLogEvents",
                          - "logs:CreateLogStream",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:us-west-2:123456789:log-group:/aws/lambda/rlm-dev-geff-lambda:*"
                      - Sid      = "WriteCloudWatchLogs"
                    },
                  - {
                      - Action    = "cloudwatch:PutMetricData"
                      - Condition = {
                          - StringLike = {
                              - "cloudwatch:namespace" = "*"
                            }
                        }
                      - Effect    = "Allow"
                      - Resource  = "*"
                      - Sid       = "WriteCloudWatchMetrics"
                    },
                  - {
                      - Action   = [
                          - "ecr:StartImageScan",
                          - "ecr:ListImages",
                          - "ecr:DescribeRepositories",
                          - "ecr:DescribeImages",
                          - "ecr:DescribeImageScanFindings",
                          - "ecr:BatchGetImage",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "EcrScanImages"
                    },
                  - {
                      - Action   = [
                          - "s3:PutObject",
                          - "s3:GetObjectVersion",
                          - "s3:GetObject",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:s3:::rlm-dev-dev-bucket/*"
                      - Sid      = "WriteToS3"
                    },
                  - {
                      - Action   = "lambda:InvokeFunction"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:lambda:us-west-2:123456789:function:rlm-dev-geff-lambda"
                      - Sid      = "InvokeLambda"
                    },
                  - {
                      - Action   = [
                          - "secretsmanager:ListSecrets",
                          - "secretsmanager:ListSecretVersionIds",
                          - "secretsmanager:GetSecretValue",
                          - "secretsmanager:GetResourcePolicy",
                          - "secretsmanager:DescribeSecret",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "AccessSecrets"
                    },
                  - {
                      - Action   = "secretsmanager:ListSecrets"
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "ListSecrets"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # module.geff.aws_iam_role_policy_attachment.dynamodb_table_policy_attachment[0] will be created
  + resource "aws_iam_role_policy_attachment" "dynamodb_table_policy_attachment" {
      + id         = (known after apply)
      + policy_arn = (known after apply)
      + role       = "rlm-dev-geff-lambda"
    }

  # module.geff.aws_lambda_function.geff_lambda will be updated in-place
  ~ resource "aws_lambda_function" "geff_lambda" {
        id                             = "rlm-dev-geff-lambda"
        tags                           = {}
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              + "DYNAMODB_TABLE_NAME" = "rlm-custom-dynamodb-table"
                # (3 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

Plan: 2 to add, 2 to change, 0 to destroy.

sfc-gh-nlele · 2023-05-05T15:16:11Z

terraform plan when

create_dynamodb_table    = false
batch_locking_table_name = null

and there are no dynamodb resources present.

No changes. Your infrastructure matches the configuration.

lambda.tf

iam.tf

sfc-gh-afedorov

lgtm, but please test before merging

sfc-gh-pkommini · 2023-05-17T16:41:33Z

examples/complete/versions.tf

@@ -9,7 +9,7 @@ terraform {

    snowflake = {
      source  = "Snowflake-Labs/snowflake"
-      version = "~> 0.57.0"
+      version = ">= 0.64.0"


"~> 0.64.0" This might be better? It only upgrades to latest minor version and allows us to specify explicitly when you want to upgrade major versions.

Makes sense 👍

versions.tf

sfc-gh-pkommini

minor comments.

sfc-gh-pkommini

lgtm

sfc-gh-nlele requested a review from sfc-gh-pkommini October 31, 2022 14:21

sfc-gh-pkommini reviewed Oct 31, 2022

View reviewed changes

sfc-gh-pkommini requested a review from sfc-gh-bkou October 31, 2022 16:36

sfc-gh-nlele requested a review from sfc-gh-pkommini November 2, 2022 19:19

sfc-gh-nlele marked this pull request as ready for review April 6, 2023 12:38

sfc-gh-nlele changed the base branch from main to add_logging_metrics April 26, 2023 18:36

sfc-gh-nlele changed the base branch from add_logging_metrics to main April 26, 2023 18:36

sfc-gh-bkou approved these changes Apr 26, 2023

View reviewed changes

sfc-gh-nlele commented Apr 27, 2023

View reviewed changes

dynamodb.tf Outdated

}

ttl {

attribute_name = "TimeToExist"

Copy link

Collaborator Author

sfc-gh-nlele Apr 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ttl

sfc-gh-afedorov reviewed Apr 27, 2023

View reviewed changes

examples/complete/variables.tf Show resolved Hide resolved

sfc-gh-nlele requested a review from sfc-gh-afedorov April 28, 2023 13:35

sfc-gh-nlele commented May 2, 2023

View reviewed changes

sfc-gh-afedorov requested changes May 3, 2023

View reviewed changes

sfc-gh-afedorov reviewed May 4, 2023

View reviewed changes

sfc-gh-nlele requested a review from sfc-gh-afedorov May 5, 2023 15:26

sfc-gh-nlele commented May 5, 2023

View reviewed changes

lambda.tf Outdated Show resolved Hide resolved

sfc-gh-nlele commented May 5, 2023

View reviewed changes

iam.tf Show resolved Hide resolved

sfc-gh-afedorov approved these changes May 8, 2023

View reviewed changes

sfc-gh-pkommini reviewed May 17, 2023

View reviewed changes

versions.tf Show resolved Hide resolved

sfc-gh-pkommini requested changes May 17, 2023

View reviewed changes

sfc-gh-nlele and others added 22 commits May 25, 2023 17:42

Update variables.tf

8b27849

Initial commit

eff18d2

Improve formatting

e47e955

Make DynamoDB table creation optional

223cdc1

Allow custom DynamoDB tables to be used

5518fc4

Request locking variable changes

329f8f7

Add variable for TTL

6af8859

Update variables.tf

08081e6

Add DynamoDB TTL environment variable to lambda

c4a5c42

Update variables.tf

0780235

Use null instead of an empty string

6d35a59

Update main.tf

4b51585

Update variables.tf

827d71c

Rename 'request' locking to 'batch' locking

2eb6b8e

Change TTL attribute name

2d389a8

Add a batch_locking_enabled variable

caed8b5

Allow users to opt out of batch locking

287e7dd

Update dynamodb naming conditions

d793962

Update ternary logic, variable names and descriptions

441925a

Upgrade sf provider version

a8b1715

Update versions.tf

d2181c1

Update versions.tf

9eb3af2

sfc-gh-nlele force-pushed the add_req_locking_mechanism_support branch from cdcdbbd to 9eb3af2 Compare May 25, 2023 12:21

sfc-gh-nlele and others added 3 commits May 25, 2023 18:05

Undo commit

a0a921e

Formatting

793d76c

Update versions.tf

357ecc7

sfc-gh-nlele requested a review from sfc-gh-pkommini June 2, 2023 13:22

sfc-gh-pkommini approved these changes Jun 5, 2023

View reviewed changes

sfc-gh-nlele merged commit c6b9113 into main Jun 5, 2023

sfc-gh-nlele deleted the add_req_locking_mechanism_support branch June 5, 2023 15:46

Add resources to support request locking mechanism #71

Add resources to support request locking mechanism #71

Conversation

sfc-gh-nlele commented Sep 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-bkou May 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-afedorov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-nlele May 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-nlele May 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-afedorov left a comment

Choose a reason for hiding this comment

sfc-gh-afedorov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-nlele commented May 5, 2023 • edited Loading

sfc-gh-nlele commented May 5, 2023

sfc-gh-nlele commented May 5, 2023 • edited Loading

sfc-gh-nlele commented May 5, 2023 • edited Loading

sfc-gh-afedorov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-pkommini left a comment

Choose a reason for hiding this comment

sfc-gh-pkommini left a comment

Choose a reason for hiding this comment

sfc-gh-nlele commented Sep 28, 2022 •

edited

Loading

sfc-gh-bkou May 2, 2023 •

edited

Loading

sfc-gh-nlele May 4, 2023 •

edited

Loading

sfc-gh-nlele May 5, 2023 •

edited

Loading

sfc-gh-nlele commented May 5, 2023 •

edited

Loading

sfc-gh-nlele commented May 5, 2023 •

edited

Loading

sfc-gh-nlele commented May 5, 2023 •

edited

Loading