Fix: Conditional Datadog configurations

[TylerHendrickson]

Description

This PR includes two fixes and one minor improvement:

Fixes

• Bug: terraform apply using local.tfvars (i.e. in localstack usage) fails because no API keys exist for the Datadog provider.
  • Solution: Introduce a new datadog_dashboards_enabled input variable (default true) that is set to false in local.tfvars.
• Bug: terraform apply fails when the Lambda modules' additional_environment_variables input variable has a DD_TAGS attribute that is either null or an empty string.
  • Solution:
    • Ensure the for loop iterator building the first map() argument for local.dd_tags defaults to an empty list.
    • Use compact() on the resulting list to remove empty string values.

Improvements

• Add architecture diagrams (initially hidden/expandable with <details></details> tags to the README's "Architecture" section.

Testing

Apply terraform to a localstack using tflocal as outlined in the README:

$ tflocal apply -var-file=local.tfvars

and ensure that it succeeds.

Automated and Unit Tests

• [ ] Added Unit tests

Manual tests for Reviewer

• Added steps to test feature/functionality manually

Checklist

• Provided ticket and description
• Provided testing information
• Provided adequate test coverage for all new code
• Added PR reviewers

[github-actions]

Terraform Summary

Step	Result
🖌 Terraform Format & Style	✅
⚙️ Terraform Initialization	✅
🤖 Terraform Validation	✅
📖 Terraform Plan	✅

Output

Validation Output

Success! The configuration is valid.

Plan Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # datadog_dashboard.service_dashboard has moved to datadog_dashboard.service_dashboard[0]
    resource "datadog_dashboard" "service_dashboard" {
        id                      = "r2k-2tv-mui"
        # (9 unchanged attributes hidden)

        # (5 unchanged blocks hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.DownloadGrantsGovDB.source_size"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.DownloadGrantsGovDB.source_size"
-       type            = "gauge" -> null
        # (6 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.created"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.created"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.skipped"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.skipped"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.updated"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.updated"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-DownloadGrantsGovDB"
      ~ last_modified                  = "2023-04-05T16:10:26.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadGrantsGovDB:7" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadGrantsGovDB:7/invocations" -> (known after apply)
      ~ s3_key                         = "builds/2c72a14b68289efdffb2355f93456f3068b5788e838a494b9e323e1931a1c84c.zip" -> "builds/71becb3eb2f6e5ed2e220b5005f2895b6ff81757d5ffffac80f92fc5edf97489.zip"
      ~ s3_object_version              = "eennUj4zO6yTgabRJbGCTOz5wZUoB9pf" -> (known after apply)
        tags                           = {}
      ~ version                        = "7" -> (known after apply)
        # (23 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                        = "git.commit.sha:6c79b373e21a1de02f8ea191b3f39a3eef3d8043,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb" -> "git.commit.sha:dd69a0b1eff441b7c4a7aa431deff491f0fc588d,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb"
              ~ "DD_VERSION"                     = "6c79b373e21a1de02f8ea191b3f39a3eef3d8043" -> "dd69a0b1eff441b7c4a7aa431deff491f0fc588d"
                # (12 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_permission.current_version_triggers["Schedule"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "Schedule" -> (known after apply)
      ~ qualifier           = "7" -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "e16c0cf79d2f418066192841676c0f36-2" -> (known after apply)
      ~ id                     = "builds/2c72a14b68289efdffb2355f93456f3068b5788e838a494b9e323e1931a1c84c.zip" -> (known after apply)
      ~ key                    = "builds/2c72a14b68289efdffb2355f93456f3068b5788e838a494b9e323e1931a1c84c.zip" -> "builds/71becb3eb2f6e5ed2e220b5005f2895b6ff81757d5ffffac80f92fc5edf97489.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/2c72a14b68289efdffb2355f93456f3068b5788e838a494b9e323e1931a1c84c.zip" -> "builds/71becb3eb2f6e5ed2e220b5005f2895b6ff81757d5ffffac80f92fc5edf97489.zip"
-       tags                   = {} -> null
      ~ version_id             = "eennUj4zO6yTgabRJbGCTOz5wZUoB9pf" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-DownloadGrantsGovDB
                            cd bin/DownloadGrantsGovDB
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/71becb3eb2f6e5ed2e220b5005f2895b6ff81757d5ffffac80f92fc5edf97489.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/71becb3eb2f6e5ed2e220b5005f2895b6ff81757d5ffffac80f92fc5edf97489.plan.json"
+       id                   = (known after apply)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "860084327348974946" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/2c72a14b68289efdffb2355f93456f3068b5788e838a494b9e323e1931a1c84c.zip" -> "builds/71becb3eb2f6e5ed2e220b5005f2895b6ff81757d5ffffac80f92fc5edf97489.zip"
          ~ "timestamp" = "1680710913729629000" -> "1680903870403578000"
        }
    }

  # module.DownloadGrantsGovDB.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "1710816930940905303" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/2c72a14b68289efdffb2355f93456f3068b5788e838a494b9e323e1931a1c84c.zip" -> "builds/71becb3eb2f6e5ed2e220b5005f2895b6ff81757d5ffffac80f92fc5edf97489.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-SplitGrantsGovXMLDB"
      ~ last_modified                  = "2023-04-05T16:10:25.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-SplitGrantsGovXMLDB:6" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-SplitGrantsGovXMLDB:6/invocations" -> (known after apply)
      ~ s3_key                         = "builds/1a9f9cc9dfbcd45049eaff84d5e10e62e84f536b8bf075145dd28cfdfc0a74ec.zip" -> "builds/a723a9032aebcdb0390eb3026f45abd8c063fd51e59db74f62b533a0e592a235.zip"
      ~ s3_object_version              = "SCJUvnavg3d4NXcpdr0q8ogRziIf0BLJ" -> (known after apply)
        tags                           = {}
      ~ version                        = "6" -> (known after apply)
        # (23 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                          = "git.commit.sha:6c79b373e21a1de02f8ea191b3f39a3eef3d8043,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb" -> "git.commit.sha:dd69a0b1eff441b7c4a7aa431deff491f0fc588d,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb"
              ~ "DD_VERSION"                       = "6c79b373e21a1de02f8ea191b3f39a3eef3d8043" -> "dd69a0b1eff441b7c4a7aa431deff491f0fc588d"
                # (15 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "6" -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "ae2a20f8f0287457f0788173a43a4191-2" -> (known after apply)
      ~ id                     = "builds/1a9f9cc9dfbcd45049eaff84d5e10e62e84f536b8bf075145dd28cfdfc0a74ec.zip" -> (known after apply)
      ~ key                    = "builds/1a9f9cc9dfbcd45049eaff84d5e10e62e84f536b8bf075145dd28cfdfc0a74ec.zip" -> "builds/a723a9032aebcdb0390eb3026f45abd8c063fd51e59db74f62b533a0e592a235.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/1a9f9cc9dfbcd45049eaff84d5e10e62e84f536b8bf075145dd28cfdfc0a74ec.zip" -> "builds/a723a9032aebcdb0390eb3026f45abd8c063fd51e59db74f62b533a0e592a235.zip"
-       tags                   = {} -> null
      ~ version_id             = "SCJUvnavg3d4NXcpdr0q8ogRziIf0BLJ" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-SplitGrantsGovXMLDB
                            cd bin/SplitGrantsGovXMLDB
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/a723a9032aebcdb0390eb3026f45abd8c063fd51e59db74f62b533a0e592a235.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/a723a9032aebcdb0390eb3026f45abd8c063fd51e59db74f62b533a0e592a235.plan.json"
+       id                   = (known after apply)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "1576526221860595683" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/1a9f9cc9dfbcd45049eaff84d5e10e62e84f536b8bf075145dd28cfdfc0a74ec.zip" -> "builds/a723a9032aebcdb0390eb3026f45abd8c063fd51e59db74f62b533a0e592a235.zip"
          ~ "timestamp" = "1680710913485059000" -> "1680903870499571000"
        }
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "160764151572183357" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/1a9f9cc9dfbcd45049eaff84d5e10e62e84f536b8bf075145dd28cfdfc0a74ec.zip" -> "builds/a723a9032aebcdb0390eb3026f45abd8c063fd51e59db74f62b533a0e592a235.zip"
            # (4 unchanged elements hidden)
        }
    }

Plan: 10 to add, 6 to change, 8 to destroy.

Pusher: @TylerHendrickson, Action: pull_request, Workflow: Continuous Integration