Fix incorrect parsing of bill when 4-digit CFDA present in data #194

jakekreider · 2023-07-23T17:37:33Z

Ticket #193

Fixes #193

Description

From what I'm finding, CFDA's are strings with format NN.NNN. If the CFDA has a trailing zero, like 93.310, the FFIS spreadsheet recognizes that field as a decimal and will store it as 93.31. However the display format of the spreadsheet will include the trailing zero, masking the underlying issue. The way this handler works, it causes it to not see it as a CFDA (which it interprets as always marking the beginning of a data row) and instead treats it as a bill row.

It seems like this is just a quirk of the spreadsheet we'll need to handle, rather than calling it a data issue.

This change modifies it to recognize CFDA's as NN.NNN OR NN.NN. Additionally, it adds a trailing zero as needed.

Testing

tf up everything (e.g. docker compose down; docker compose up -d; cd terraform; tflocal init; tflocal apply -var-file=local.tfvars -auto-approve)
Upload the spreadsheet from the Slack thread, e.g. awslocal s3 cp --sse AES256 download.xlsx s3://grants-ingest-grantssourcedata-000000000000-us-west-2/sources/2023/07/23/ffis/download.xlsx
Wait a sec for the lambda to process
Get the file awslocal s3 cp s3://grants-ingest-grantsprepareddata-000000000000-us-west-2/349/349292/ffis.org/v1.json 349292.json and check contents.

This reproduces the issue before this code is applied, and should demonstrate the fix in this branch.

Automated and Unit Tests

Added Unit tests

Manual tests for Reviewer

Added steps to test feature/functionality manually

Checklist

Provided ticket and description
Provided testing information
Provided adequate test coverage for all new code
Added PR reviewers

… be parsed incorrectly

github-actions · 2023-07-23T17:39:16Z

Terraform Summary

Step	Result
🖌 Terraform Format & Style	✅
⚙️ Terraform Initialization	✅
🤖 Terraform Validation	✅
📖 Terraform Plan	✅

Output

Validation Output

Success! The configuration is valid.

Plan Output

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.PublishGrantEvents.module.lambda_function.aws_lambda_function.this[0] has changed
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-PublishGrantEvents"
      ~ last_modified                  = "2023-07-14T23:12:02.734+0000" -> "2023-07-14T23:12:10.838+0000"
        # (24 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # datadog_dashboard.service_dashboard[0] will be updated in-place
  ~ resource "datadog_dashboard" "service_dashboard" {
        id                      = "r2k-2tv-mui"
        tags                    = []
        # (9 unchanged attributes hidden)

      ~ template_variable {
-           default          = "production" -> null
          ~ defaults         = [
+               "production",
            ]
            name             = "env"
            # (2 unchanged attributes hidden)
        }
      ~ template_variable {
-           default          = "grants-ingest" -> null
          ~ defaults         = [
+               "grants-ingest",
            ]
            name             = "service"
            # (2 unchanged attributes hidden)
        }
      ~ template_variable {
-           default          = "*" -> null
          ~ defaults         = [
+               "*",
            ]
            name             = "version"
            # (2 unchanged attributes hidden)
        }

        # (3 unchanged blocks hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.DownloadGrantsGovDB.source_size"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.DownloadGrantsGovDB.source_size"
-       type            = "gauge" -> null
        # (6 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.PublishGrantEvents.event.published"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.PublishGrantEvents.event.published"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.PublishGrantEvents.grant_data.invalid"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.PublishGrantEvents.grant_data.invalid"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.PublishGrantEvents.invocation_batch_size"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.PublishGrantEvents.invocation_batch_size"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.PublishGrantEvents.item_image.build"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.PublishGrantEvents.item_image.build"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.PublishGrantEvents.item_image.malformatted_field"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.PublishGrantEvents.item_image.malformatted_field"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.PublishGrantEvents.record.failed"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.PublishGrantEvents.record.failed"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.created"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.created"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.failed"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.failed"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.skipped"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.skipped"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.updated"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.updated"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-DownloadFFISSpreadsheet"
      ~ last_modified                  = "2023-07-14T23:12:00.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadFFISSpreadsheet:14" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadFFISSpreadsheet:14/invocations" -> (known after apply)
      ~ s3_key                         = "builds/7ba51be5b5f8fd01546a13e95a1f7208fca744b1ebc5cc98b6d1c4d6c299f9e7.zip" -> "builds/d68b31f2719c6b7677e00772e9b1b608a8e5002a4560ce7b3d81fce689e893a8.zip"
      ~ s3_object_version              = "5yo6yvHs5WZHVf3tgEzO8TY3SXV0atLK" -> (known after apply)
        tags                           = {}
      ~ version                        = "14" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadffisspreadsheet" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadffisspreadsheet"
              ~ "DD_VERSION"                   = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (12 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.aws_lambda_permission.current_version_triggers["SQSQueueNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "SQSQueueNotification" -> (known after apply)
      ~ qualifier           = "14" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (4 unchanged attributes hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "aa4444fce4a2b296cdb54b6eb7c1532f-2" -> (known after apply)
      ~ id                     = "builds/7ba51be5b5f8fd01546a13e95a1f7208fca744b1ebc5cc98b6d1c4d6c299f9e7.zip" -> (known after apply)
      ~ key                    = "builds/7ba51be5b5f8fd01546a13e95a1f7208fca744b1ebc5cc98b6d1c4d6c299f9e7.zip" -> "builds/d68b31f2719c6b7677e00772e9b1b608a8e5002a4560ce7b3d81fce689e893a8.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/7ba51be5b5f8fd01546a13e95a1f7208fca744b1ebc5cc98b6d1c4d6c299f9e7.zip" -> "builds/d68b31f2719c6b7677e00772e9b1b608a8e5002a4560ce7b3d81fce689e893a8.zip"
-       tags                   = {} -> null
      ~ version_id             = "5yo6yvHs5WZHVf3tgEzO8TY3SXV0atLK" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-DownloadFFISSpreadsheet
                            cd bin/DownloadFFISSpreadsheet
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/d68b31f2719c6b7677e00772e9b1b608a8e5002a4560ce7b3d81fce689e893a8.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/d68b31f2719c6b7677e00772e9b1b608a8e5002a4560ce7b3d81fce689e893a8.plan.json"
+       id                   = (known after apply)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "3153009506637227711" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/7ba51be5b5f8fd01546a13e95a1f7208fca744b1ebc5cc98b6d1c4d6c299f9e7.zip" -> "builds/d68b31f2719c6b7677e00772e9b1b608a8e5002a4560ce7b3d81fce689e893a8.zip"
          ~ "timestamp" = "1689376288859068000" -> "1690252587703167000"
        }
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "1812814773429147444" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/7ba51be5b5f8fd01546a13e95a1f7208fca744b1ebc5cc98b6d1c4d6c299f9e7.zip" -> "builds/d68b31f2719c6b7677e00772e9b1b608a8e5002a4560ce7b3d81fce689e893a8.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-DownloadGrantsGovDB"
      ~ last_modified                  = "2023-07-14T23:12:00.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadGrantsGovDB:39" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadGrantsGovDB:39/invocations" -> (known after apply)
      ~ s3_key                         = "builds/0845573c431b367bbb93f58728aec2265645f5c48eab73c4df5fbf3d34b21359.zip" -> "builds/1717b389d3aacf0cc22d9e4e9ab60a900bf478d731d05ab0360247ed967c4b96.zip"
      ~ s3_object_version              = "KN1zf.v.ebDiUzIreZZjpBpU94muk12S" -> (known after apply)
        tags                           = {}
      ~ version                        = "39" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                        = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb"
              ~ "DD_VERSION"                     = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (12 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_permission.current_version_triggers["Schedule"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "Schedule" -> (known after apply)
      ~ qualifier           = "39" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "38d7ac5a01eeca87c1986f518593951f-2" -> (known after apply)
      ~ id                     = "builds/0845573c431b367bbb93f58728aec2265645f5c48eab73c4df5fbf3d34b21359.zip" -> (known after apply)
      ~ key                    = "builds/0845573c431b367bbb93f58728aec2265645f5c48eab73c4df5fbf3d34b21359.zip" -> "builds/1717b389d3aacf0cc22d9e4e9ab60a900bf478d731d05ab0360247ed967c4b96.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/0845573c431b367bbb93f58728aec2265645f5c48eab73c4df5fbf3d34b21359.zip" -> "builds/1717b389d3aacf0cc22d9e4e9ab60a900bf478d731d05ab0360247ed967c4b96.zip"
-       tags                   = {} -> null
      ~ version_id             = "KN1zf.v.ebDiUzIreZZjpBpU94muk12S" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-DownloadGrantsGovDB
                            cd bin/DownloadGrantsGovDB
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/1717b389d3aacf0cc22d9e4e9ab60a900bf478d731d05ab0360247ed967c4b96.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/1717b389d3aacf0cc22d9e4e9ab60a900bf478d731d05ab0360247ed967c4b96.plan.json"
+       id                   = (known after apply)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "857216909204372472" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/0845573c431b367bbb93f58728aec2265645f5c48eab73c4df5fbf3d34b21359.zip" -> "builds/1717b389d3aacf0cc22d9e4e9ab60a900bf478d731d05ab0360247ed967c4b96.zip"
          ~ "timestamp" = "1689376271330292000" -> "1690252570311246000"
        }
    }

  # module.DownloadGrantsGovDB.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "3476260685692148069" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/0845573c431b367bbb93f58728aec2265645f5c48eab73c4df5fbf3d34b21359.zip" -> "builds/1717b389d3aacf0cc22d9e4e9ab60a900bf478d731d05ab0360247ed967c4b96.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.EnqueueFFISDownload.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-EnqueueFFISDownload"
      ~ last_modified                  = "2023-07-14T23:11:57.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-EnqueueFFISDownload:19" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-EnqueueFFISDownload:19/invocations" -> (known after apply)
      ~ s3_key                         = "builds/860934197ab09f8a531d520dfe16823a9fb2f52ec2dc3c055902e5ad0f51118f.zip" -> "builds/95e58e22913a51df6d45d6450bb92d617ba6487849ec85dc099805af523b0366.zip"
      ~ s3_object_version              = "NDI99sLYXL80Bbmr46r1auQiDNJ9rJe5" -> (known after apply)
        tags                           = {}
      ~ version                        = "19" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:enqueueffisdownload" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:enqueueffisdownload"
              ~ "DD_VERSION"                   = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (12 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.EnqueueFFISDownload.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "19" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.EnqueueFFISDownload.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "0892a5083b2f967afb322af9878862fd-2" -> (known after apply)
      ~ id                     = "builds/860934197ab09f8a531d520dfe16823a9fb2f52ec2dc3c055902e5ad0f51118f.zip" -> (known after apply)
      ~ key                    = "builds/860934197ab09f8a531d520dfe16823a9fb2f52ec2dc3c055902e5ad0f51118f.zip" -> "builds/95e58e22913a51df6d45d6450bb92d617ba6487849ec85dc099805af523b0366.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/860934197ab09f8a531d520dfe16823a9fb2f52ec2dc3c055902e5ad0f51118f.zip" -> "builds/95e58e22913a51df6d45d6450bb92d617ba6487849ec85dc099805af523b0366.zip"
-       tags                   = {} -> null
      ~ version_id             = "NDI99sLYXL80Bbmr46r1auQiDNJ9rJe5" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.EnqueueFFISDownload.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-EnqueueFFISDownload
                            cd bin/EnqueueFFISDownload
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/95e58e22913a51df6d45d6450bb92d617ba6487849ec85dc099805af523b0366.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/95e58e22913a51df6d45d6450bb92d617ba6487849ec85dc099805af523b0366.plan.json"
+       id                   = (known after apply)
    }

  # module.EnqueueFFISDownload.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "5959741361938685684" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/860934197ab09f8a531d520dfe16823a9fb2f52ec2dc3c055902e5ad0f51118f.zip" -> "builds/95e58e22913a51df6d45d6450bb92d617ba6487849ec85dc099805af523b0366.zip"
          ~ "timestamp" = "1689376289063202000" -> "1690252587887152000"
        }
    }

  # module.EnqueueFFISDownload.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "8195113782754754347" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/860934197ab09f8a531d520dfe16823a9fb2f52ec2dc3c055902e5ad0f51118f.zip" -> "builds/95e58e22913a51df6d45d6450bb92d617ba6487849ec85dc099805af523b0366.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.ExtractGrantsGovDBToXML.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-ExtractGrantsGovDBToXML"
      ~ last_modified                  = "2023-07-14T23:12:01.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-ExtractGrantsGovDBToXML:8" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-ExtractGrantsGovDBToXML:8/invocations" -> (known after apply)
      ~ s3_key                         = "builds/de8549154296529c7aa59702c97b3584a3c0225db9b2cc4c101d33ebf464b6a8.zip" -> "builds/b4d9a578bfe74779dd1413ba65503f5182f2bf916f1e3b9132eb15aab3cc5d0e.zip"
      ~ s3_object_version              = "iPdU_58BO.CNQoMrOevLhJ7NVEDR2ST0" -> (known after apply)
        tags                           = {}
      ~ version                        = "8" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:extractgrantsgovdbtoxml" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:extractgrantsgovdbtoxml"
              ~ "DD_VERSION"                   = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (11 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.ExtractGrantsGovDBToXML.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "8" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.ExtractGrantsGovDBToXML.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "079d4e3cdfce0c986bd43f898d48f04a-2" -> (known after apply)
      ~ id                     = "builds/de8549154296529c7aa59702c97b3584a3c0225db9b2cc4c101d33ebf464b6a8.zip" -> (known after apply)
      ~ key                    = "builds/de8549154296529c7aa59702c97b3584a3c0225db9b2cc4c101d33ebf464b6a8.zip" -> "builds/b4d9a578bfe74779dd1413ba65503f5182f2bf916f1e3b9132eb15aab3cc5d0e.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/de8549154296529c7aa59702c97b3584a3c0225db9b2cc4c101d33ebf464b6a8.zip" -> "builds/b4d9a578bfe74779dd1413ba65503f5182f2bf916f1e3b9132eb15aab3cc5d0e.zip"
-       tags                   = {} -> null
      ~ version_id             = "iPdU_58BO.CNQoMrOevLhJ7NVEDR2ST0" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.ExtractGrantsGovDBToXML.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-ExtractGrantsGovDBToXML
                            cd bin/ExtractGrantsGovDBToXML
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/b4d9a578bfe74779dd1413ba65503f5182f2bf916f1e3b9132eb15aab3cc5d0e.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/b4d9a578bfe74779dd1413ba65503f5182f2bf916f1e3b9132eb15aab3cc5d0e.plan.json"
+       id                   = (known after apply)
    }

  # module.ExtractGrantsGovDBToXML.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "1868108409703660241" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/de8549154296529c7aa59702c97b3584a3c0225db9b2cc4c101d33ebf464b6a8.zip" -> "builds/b4d9a578bfe74779dd1413ba65503f5182f2bf916f1e3b9132eb15aab3cc5d0e.zip"
          ~ "timestamp" = "1689376288822685000" -> "1690252587987618000"
        }
    }

  # module.ExtractGrantsGovDBToXML.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "686517103685270286" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/de8549154296529c7aa59702c97b3584a3c0225db9b2cc4c101d33ebf464b6a8.zip" -> "builds/b4d9a578bfe74779dd1413ba65503f5182f2bf916f1e3b9132eb15aab3cc5d0e.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.PersistFFISData.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-PersistFFISData"
      ~ last_modified                  = "2023-07-14T23:12:01.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-PersistFFISData:9" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-PersistFFISData:9/invocations" -> (known after apply)
      ~ s3_key                         = "builds/f6316914e01ccfdfc7b92a69be0b08fc0751d5b0b97785c10fae96f6bbf19386.zip" -> "builds/26a12cc83a33ed0c1869e7973723d5d80e6d267f6e7b9aeb7e7363271a137f35.zip"
      ~ s3_object_version              = "MEh4qJ5UkiH_fbo2vPO5Y7mgYqMGBPPH" -> (known after apply)
        tags                           = {}
      ~ version                        = "9" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                       = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistffisdata" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistffisdata"
              ~ "DD_VERSION"                    = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (12 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.PersistFFISData.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "9" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.PersistFFISData.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "55bd3894cea97a40eb851c6c277d0689-2" -> (known after apply)
      ~ id                     = "builds/f6316914e01ccfdfc7b92a69be0b08fc0751d5b0b97785c10fae96f6bbf19386.zip" -> (known after apply)
      ~ key                    = "builds/f6316914e01ccfdfc7b92a69be0b08fc0751d5b0b97785c10fae96f6bbf19386.zip" -> "builds/26a12cc83a33ed0c1869e7973723d5d80e6d267f6e7b9aeb7e7363271a137f35.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/f6316914e01ccfdfc7b92a69be0b08fc0751d5b0b97785c10fae96f6bbf19386.zip" -> "builds/26a12cc83a33ed0c1869e7973723d5d80e6d267f6e7b9aeb7e7363271a137f35.zip"
-       tags                   = {} -> null
      ~ version_id             = "MEh4qJ5UkiH_fbo2vPO5Y7mgYqMGBPPH" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.PersistFFISData.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-PersistFFISData
                            cd bin/PersistFFISData
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/26a12cc83a33ed0c1869e7973723d5d80e6d267f6e7b9aeb7e7363271a137f35.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/26a12cc83a33ed0c1869e7973723d5d80e6d267f6e7b9aeb7e7363271a137f35.plan.json"
+       id                   = (known after apply)
    }

  # module.PersistFFISData.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "3320840181704114215" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/f6316914e01ccfdfc7b92a69be0b08fc0751d5b0b97785c10fae96f6bbf19386.zip" -> "builds/26a12cc83a33ed0c1869e7973723d5d80e6d267f6e7b9aeb7e7363271a137f35.zip"
          ~ "timestamp" = "1689376288959688000" -> "1690252587663941000"
        }
    }

  # module.PersistFFISData.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "6212922726728293016" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/f6316914e01ccfdfc7b92a69be0b08fc0751d5b0b97785c10fae96f6bbf19386.zip" -> "builds/26a12cc83a33ed0c1869e7973723d5d80e6d267f6e7b9aeb7e7363271a137f35.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-PersistGrantsGovXMLDB"
      ~ last_modified                  = "2023-07-14T23:12:00.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-PersistGrantsGovXMLDB:11" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-PersistGrantsGovXMLDB:11/invocations" -> (known after apply)
      ~ s3_key                         = "builds/2665d3f6d148430cd92a40faf3bb4f1f3c2fdd4f1dac11b97abc741bdbebedfb.zip" -> "builds/c13364aef8df4de2419790181c4649eb8577f99ea08151af1a294bfabd3e2921.zip"
      ~ s3_object_version              = "jpLoLz_kGb30awX91nDOooaNC3uOxSAo" -> (known after apply)
        tags                           = {}
      ~ version                        = "11" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                       = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistgrantsgovxmldb" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistgrantsgovxmldb"
              ~ "DD_VERSION"                    = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (12 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "11" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "d4159272d564f6a174f7bc51ab9c5499-2" -> (known after apply)
      ~ id                     = "builds/2665d3f6d148430cd92a40faf3bb4f1f3c2fdd4f1dac11b97abc741bdbebedfb.zip" -> (known after apply)
      ~ key                    = "builds/2665d3f6d148430cd92a40faf3bb4f1f3c2fdd4f1dac11b97abc741bdbebedfb.zip" -> "builds/c13364aef8df4de2419790181c4649eb8577f99ea08151af1a294bfabd3e2921.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/2665d3f6d148430cd92a40faf3bb4f1f3c2fdd4f1dac11b97abc741bdbebedfb.zip" -> "builds/c13364aef8df4de2419790181c4649eb8577f99ea08151af1a294bfabd3e2921.zip"
-       tags                   = {} -> null
      ~ version_id             = "jpLoLz_kGb30awX91nDOooaNC3uOxSAo" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-PersistGrantsGovXMLDB
                            cd bin/PersistGrantsGovXMLDB
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/c13364aef8df4de2419790181c4649eb8577f99ea08151af1a294bfabd3e2921.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/c13364aef8df4de2419790181c4649eb8577f99ea08151af1a294bfabd3e2921.plan.json"
+       id                   = (known after apply)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "692881246513526578" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/2665d3f6d148430cd92a40faf3bb4f1f3c2fdd4f1dac11b97abc741bdbebedfb.zip" -> "builds/c13364aef8df4de2419790181c4649eb8577f99ea08151af1a294bfabd3e2921.zip"
          ~ "timestamp" = "1689376271240367000" -> "1690252570239264000"
        }
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "2862697758826125335" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/2665d3f6d148430cd92a40faf3bb4f1f3c2fdd4f1dac11b97abc741bdbebedfb.zip" -> "builds/c13364aef8df4de2419790181c4649eb8577f99ea08151af1a294bfabd3e2921.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.PublishGrantEvents.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-PublishGrantEvents"
      ~ last_modified                  = "2023-07-14T23:12:10.838+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-PublishGrantEvents:1" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-PublishGrantEvents:1/invocations" -> (known after apply)
      ~ s3_key                         = "builds/c92372b54b03754b60c5ad5b754f8960a9db0c133579569c91df1685a9c3e4e9.zip" -> "builds/60d745fdac1c7d19f4659f875e28b0cc5d69714aaf55d4321bfe6ea7a7d1db06.zip"
      ~ s3_object_version              = "xXV64zOGh9RWh8goFEwgUjC6.lfLrX4f" -> (known after apply)
        tags                           = {}
      ~ version                        = "1" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:publishgrantevents" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:publishgrantevents"
              ~ "DD_VERSION"                   = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (11 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.PublishGrantEvents.module.lambda_function.aws_lambda_permission.current_version_triggers["dynamodb"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "dynamodb" -> (known after apply)
      ~ qualifier           = "1" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.PublishGrantEvents.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "7441c1633128bed126e7fdcf5777ae0a-2" -> (known after apply)
      ~ id                     = "builds/c92372b54b03754b60c5ad5b754f8960a9db0c133579569c91df1685a9c3e4e9.zip" -> (known after apply)
      ~ key                    = "builds/c92372b54b03754b60c5ad5b754f8960a9db0c133579569c91df1685a9c3e4e9.zip" -> "builds/60d745fdac1c7d19f4659f875e28b0cc5d69714aaf55d4321bfe6ea7a7d1db06.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/c92372b54b03754b60c5ad5b754f8960a9db0c133579569c91df1685a9c3e4e9.zip" -> "builds/60d745fdac1c7d19f4659f875e28b0cc5d69714aaf55d4321bfe6ea7a7d1db06.zip"
-       tags                   = {} -> null
      ~ version_id             = "xXV64zOGh9RWh8goFEwgUjC6.lfLrX4f" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.PublishGrantEvents.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-PublishGrantEvents
                            cd bin/PublishGrantEvents
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/60d745fdac1c7d19f4659f875e28b0cc5d69714aaf55d4321bfe6ea7a7d1db06.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/60d745fdac1c7d19f4659f875e28b0cc5d69714aaf55d4321bfe6ea7a7d1db06.plan.json"
+       id                   = (known after apply)
    }

  # module.PublishGrantEvents.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "1272216423898390078" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/c92372b54b03754b60c5ad5b754f8960a9db0c133579569c91df1685a9c3e4e9.zip" -> "builds/60d745fdac1c7d19f4659f875e28b0cc5d69714aaf55d4321bfe6ea7a7d1db06.zip"
          ~ "timestamp" = "1689376272512594000" -> "1690252571705462000"
        }
    }

  # module.PublishGrantEvents.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "1822574248775021981" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/c92372b54b03754b60c5ad5b754f8960a9db0c133579569c91df1685a9c3e4e9.zip" -> "builds/60d745fdac1c7d19f4659f875e28b0cc5d69714aaf55d4321bfe6ea7a7d1db06.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-SplitFFISSpreadsheet"
      ~ last_modified                  = "2023-07-14T23:12:01.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-SplitFFISSpreadsheet:10" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-SplitFFISSpreadsheet:10/invocations" -> (known after apply)
      ~ s3_key                         = "builds/8b4e30a017a06aa45aa99835002b81437f445de719fe10d9a9e2510906e5a077.zip" -> "builds/0a93420209e9b0917185d260d08e059663a5a068383642c745aedd498d2c093d.zip"
      ~ s3_object_version              = "JsTipvlhKiheTiI_KWYkXplTv4reeKwV" -> (known after apply)
        tags                           = {}
      ~ version                        = "10" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                          = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitffisspreadsheet" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitffisspreadsheet"
              ~ "DD_VERSION"                       = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (15 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "10" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "aacfe85588eb111d7dcab0a29f1b8bbe-2" -> (known after apply)
      ~ id                     = "builds/8b4e30a017a06aa45aa99835002b81437f445de719fe10d9a9e2510906e5a077.zip" -> (known after apply)
      ~ key                    = "builds/8b4e30a017a06aa45aa99835002b81437f445de719fe10d9a9e2510906e5a077.zip" -> "builds/0a93420209e9b0917185d260d08e059663a5a068383642c745aedd498d2c093d.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/8b4e30a017a06aa45aa99835002b81437f445de719fe10d9a9e2510906e5a077.zip" -> "builds/0a93420209e9b0917185d260d08e059663a5a068383642c745aedd498d2c093d.zip"
-       tags                   = {} -> null
      ~ version_id             = "JsTipvlhKiheTiI_KWYkXplTv4reeKwV" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-SplitFFISSpreadsheet
                            cd bin/SplitFFISSpreadsheet
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/0a93420209e9b0917185d260d08e059663a5a068383642c745aedd498d2c093d.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/0a93420209e9b0917185d260d08e059663a5a068383642c745aedd498d2c093d.plan.json"
+       id                   = (known after apply)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "666344473461857255" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/8b4e30a017a06aa45aa99835002b81437f445de719fe10d9a9e2510906e5a077.zip" -> "builds/0a93420209e9b0917185d260d08e059663a5a068383642c745aedd498d2c093d.zip"
          ~ "timestamp" = "1689376289108100800" -> "1690252587740542000"
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "7505932353551341034" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/8b4e30a017a06aa45aa99835002b81437f445de719fe10d9a9e2510906e5a077.zip" -> "builds/0a93420209e9b0917185d260d08e059663a5a068383642c745aedd498d2c093d.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-SplitGrantsGovXMLDB"
      ~ last_modified                  = "2023-07-14T23:11:58.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-SplitGrantsGovXMLDB:38" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-SplitGrantsGovXMLDB:38/invocations" -> (known after apply)
      ~ s3_key                         = "builds/528eb4d8431b2abac834d0383a15bf3dd37ba9b1468613566f0f7ff222339ebb.zip" -> "builds/cf020fc2c44aabf531be0cad05198a4a90ee5c34eca10b74577e8f25c5aa44c7.zip"
      ~ s3_object_version              = "i_iEyC9rQ78EyIAWemxX_wXRUQHc2jUI" -> (known after apply)
        tags                           = {}
      ~ version                        = "38" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                          = "git.commit.sha:2a1cf0a92bc51ffbae2956daa778ed0a43388a75,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb" -> "git.commit.sha:c42d4a02d892f387736ee19f87082dc0e483cf37,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb"
              ~ "DD_VERSION"                       = "2a1cf0a92bc51ffbae2956daa778ed0a43388a75" -> "c42d4a02d892f387736ee19f87082dc0e483cf37"
                # (15 unchanged elements hidden)
            }
        }

+       timeouts {}

        # (2 unchanged blocks hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "38" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "1c78e5e7b3fa067e274cdec2bb871fe7-2" -> (known after apply)
      ~ id                     = "builds/528eb4d8431b2abac834d0383a15bf3dd37ba9b1468613566f0f7ff222339ebb.zip" -> (known after apply)
      ~ key                    = "builds/528eb4d8431b2abac834d0383a15bf3dd37ba9b1468613566f0f7ff222339ebb.zip" -> "builds/cf020fc2c44aabf531be0cad05198a4a90ee5c34eca10b74577e8f25c5aa44c7.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/528eb4d8431b2abac834d0383a15bf3dd37ba9b1468613566f0f7ff222339ebb.zip" -> "builds/cf020fc2c44aabf531be0cad05198a4a90ee5c34eca10b74577e8f25c5aa44c7.zip"
-       tags                   = {} -> null
      ~ version_id             = "i_iEyC9rQ78EyIAWemxX_wXRUQHc2jUI" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-SplitGrantsGovXMLDB
                            cd bin/SplitGrantsGovXMLDB
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/cf020fc2c44aabf531be0cad05198a4a90ee5c34eca10b74577e8f25c5aa44c7.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/cf020fc2c44aabf531be0cad05198a4a90ee5c34eca10b74577e8f25c5aa44c7.plan.json"
+       id                   = (known after apply)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "1218660781217450387" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/528eb4d8431b2abac834d0383a15bf3dd37ba9b1468613566f0f7ff222339ebb.zip" -> "builds/cf020fc2c44aabf531be0cad05198a4a90ee5c34eca10b74577e8f25c5aa44c7.zip"
          ~ "timestamp" = "1689376271401673000" -> "1690252570412299000"
        }
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "7250306091143588501" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/528eb4d8431b2abac834d0383a15bf3dd37ba9b1468613566f0f7ff222339ebb.zip" -> "builds/cf020fc2c44aabf531be0cad05198a4a90ee5c34eca10b74577e8f25c5aa44c7.zip"
            # (4 unchanged elements hidden)
        }
    }

Plan: 45 to add, 21 to change, 36 to destroy.

Pusher: @jakekreider, Action: pull_request, Workflow: Continuous Integration

TylerHendrickson

Great find!

I'm now wondering what might happen for a (presumably valid?) CFDA value like "02.980", where the first character is a zero and the spreadsheet could coerce the interpreted float value to a display string of "2.98". If you think that's a reasonable enough (edge) case for us to anticipate, I left a suggestion that ought to do the trick. However, I'm also happy to accept the current solution if you think the work involved in updating how we detect the cell value as a CFDA would be unnecessary/ultimately more fragile.

cmd/SplitFFISSpreadsheet/sheet.go

…rse-4digit-cfda

jakekreider · 2023-07-25T02:36:34Z

Great find!

I'm now wondering what might happen for a (presumably valid?) CFDA value like "02.980", where the first character is a zero and the spreadsheet could coerce the interpreted float value to a display string of "2.98". If you think that's a reasonable enough (edge) case for us to anticipate, I left a suggestion that ought to do the trick. However, I'm also happy to accept the current solution if you think the work involved in updating how we detect the cell value as a CFDA would be unnecessary/ultimately more fragile.

Good call. The example spreadsheet won't add a leading zero so it seems less likely to cause confusion that way, but it could still happen if that's a valid CFDA. I added your suggestions (which make for cleaner number formatting anyway) and updated the test spreadsheet & test case to check for a leading zero as well.

Fix issue where a CFDA with a trailing zero caused the spreadsheet to…

8eead72

… be parsed incorrectly

jakekreider requested review from pearkes, TylerHendrickson and a team July 23, 2023 17:37

Forgot to add fixture

130fe9a

TylerHendrickson reviewed Jul 24, 2023

View reviewed changes

cmd/SplitFFISSpreadsheet/sheet.go Outdated Show resolved Hide resolved

TylerHendrickson reviewed Jul 24, 2023

View reviewed changes

cmd/SplitFFISSpreadsheet/sheet.go Outdated Show resolved Hide resolved

TylerHendrickson and others added 3 commits July 24, 2023 19:33

Merge branch 'main' into jk-parse-4digit-cfda

8f0faef

PR feedback - account for leading 0 as well

567ca6b

Merge remote-tracking branch 'origin/jk-parse-4digit-cfda' into jk-pa…

aeb0031

…rse-4digit-cfda

jakekreider requested a review from TylerHendrickson July 25, 2023 02:36

jakekreider enabled auto-merge (squash) July 25, 2023 02:36

TylerHendrickson approved these changes Jul 25, 2023

View reviewed changes

jakekreider merged commit dc06cb6 into main Jul 25, 2023

jakekreider deleted the jk-parse-4digit-cfda branch July 25, 2023 03:58

TylerHendrickson added bug Something isn't working go Pull requests that update Go code labels Aug 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix incorrect parsing of bill when 4-digit CFDA present in data #194

Fix incorrect parsing of bill when 4-digit CFDA present in data #194

jakekreider commented Jul 23, 2023 •

edited

Loading

github-actions bot commented Jul 23, 2023 •

edited

Loading

TylerHendrickson left a comment

jakekreider commented Jul 25, 2023

Fix incorrect parsing of bill when 4-digit CFDA present in data #194

Fix incorrect parsing of bill when 4-digit CFDA present in data #194

Conversation

jakekreider commented Jul 23, 2023 • edited Loading

Ticket #193

Description

Testing

Automated and Unit Tests

Manual tests for Reviewer

Checklist

github-actions bot commented Jul 23, 2023 • edited Loading

Terraform Summary

Output

TylerHendrickson left a comment

Choose a reason for hiding this comment

jakekreider commented Jul 25, 2023

jakekreider commented Jul 23, 2023 •

edited

Loading

github-actions bot commented Jul 23, 2023 •

edited

Loading