Split FFIS spreadsheet #94

pearkes · 2023-05-15T23:09:10Z

Ticket #12

Description

This Lambda function watches for a download.xlsx file in the source data bucket, downloads it, parses the rows as opportunities, and uploads a JSON file for each opportunity into the prepared data bucket. It is based on the patterns in the SplitGrantsGovXMLDB function and should look largely the same.

Some notes:

I don't know the subject matter extremely well so please check my naming of the serialized data, eg Bill.
Currently this assumes reasonably sized Excel files – it does not stream them row-by-row, but reads all rows into memory before iterating over the rows. I think this is likely fine based on examples I've seen but please let me know if we expect very large excel files (many megabytes?).
If there is a failure parsing data in a specific cell it will just log it and skip it, and it will only save opportunities with a properly formatted Grant ID.
I added a sanitized Excel file fixture for automated testing. I've tested this on the larger sheet as provided in that linked ticket, but currently we don't have any negative examples to write test cases for, I assume as formatting evolves those can be added.

cc/ @jakekreider not sure how to get bill, per #12 (comment), we should discuss.

Testing

Automated and Unit Tests

Added Unit tests

Manual tests for Reviewer

$ awslocal s3 ls
2023-05-15 15:46:46 local-terraform
2023-05-15 15:48:17 grants-ingest-grantssourcedata-000000000000-us-west-2
2023-05-15 15:48:17 grants-ingest-lambdaartifacts-000000000000-us-west-2
2023-05-15 15:48:17 grants-ingest-emaildelivery-000000000000-us-west-2
2023-05-15 15:48:17 grants-ingest-grantsprepareddata-000000000000-us-west-2
2023-05-15 15:48:20 awslambda-us-west-2-tasks
$ awslocal s3 ls grants-ingest-grantsprepareddata-000000000000-us-west-2
$ awslocal s3 ls grants-ingest-grantssourcedata-000000000000-us-west-2
$ awslocal s3 cp --sse AES256 ./cmd/SplitFFISSpreadsheet/fixtures/example_spreadsheet.xlsx s3://grants-ingest-grantssourcedata-000000000000-us-west-2/sources/2023/05/15/ffis/download.xlsx
upload: cmd/SplitFFISSpreadsheet/fixtures/example_spreadsheet.xlsx to s3://grants-ingest-grantssourcedata-000000000000-us-west-2/sources/2023/05/15/ffis/download.xlsx
$ awslocal s3 ls grants-ingest-grantsprepareddata-000000000000-us-west-2/123/123456/ffis.org/
2023-05-15 15:50:23        467 v1.json
$ awslocal s3 cp s3://grants-ingest-grantsprepareddata-000000000000-us-west-2/123/123456/ffis.org/v1.json .
download: s3://grants-ingest-grantsprepareddata-000000000000-us-west-2/123/123456/ffis.org/v1.json to ./v1.json
$ cat v1.json | jq
{
  "opportunity_agency": "Office of Energy Efficiency and Renewable Energy",
  "bill": "Infrastructure Investment and Jobs Act",
  "cfda": "81.086",
  "due_date": "2023-05-11T00:00:00Z",
  "eligibility": {
    "higher_education": false,
    "local": false,
    "non_profits": true,
    "other": false,
    "state": false,
    "tribal": false
  },
  "estimated_funding": 5000000,
  "expected_awards": "N/A",
  "grant_id": 123456,
  "match": false,
  "opportunity_number": "ABC-0003065",
  "opportunity_title": "Example Opportunity 1"
}

Checklist

Provided ticket and description
Provided testing information
Provided adequate test coverage for all new code
Added PR reviewers

github-actions · 2023-05-15T23:11:26Z

Terraform Summary

Step	Result
🖌 Terraform Format & Style	✅
⚙️ Terraform Initialization	✅
🤖 Terraform Validation	✅
📖 Terraform Plan	✅

Output

Validation Output

Success! The configuration is valid.

Plan Output

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] has changed
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-PersistGrantsGovXMLDB"
      ~ last_modified                  = "2023-05-15T13:49:38.319+0000" -> "2023-05-15T13:49:48.868+0000"
        # (28 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
  ~ update in-place
-/+ destroy and then create replacement
 <= read (data resources)

Terraform will perform the following actions:

  # aws_s3_bucket_notification.grant_source_data will be updated in-place
  ~ resource "aws_s3_bucket_notification" "grant_source_data" {
        id          = "grantsingest-staging-grantssourcedata-357150818708-us-west-2"
        # (2 unchanged attributes hidden)

+       lambda_function {
+           events              = [
+               "s3:ObjectCreated:*",
            ]
+           filter_prefix       = "sources/"
+           filter_suffix       = "/ffis/download.xlsx"
+           lambda_function_arn = (known after apply)
        }

        # (2 unchanged blocks hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.DownloadGrantsGovDB.source_size"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.DownloadGrantsGovDB.source_size"
-       type            = "gauge" -> null
        # (6 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.created"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.created"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.failed"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.failed"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.skipped"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.skipped"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.opportunity.updated"] will be updated in-place
  ~ resource "datadog_metric_metadata" "custom" {
        id              = "grants_ingest.SplitGrantsGovXMLDB.opportunity.updated"
-       type            = "gauge" -> null
        # (5 unchanged attributes hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-DownloadFFISSpreadsheet"
      ~ last_modified                  = "2023-05-15T13:49:46.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadFFISSpreadsheet:4" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadFFISSpreadsheet:4/invocations" -> (known after apply)
      ~ s3_key                         = "builds/676ea1efc698d46e6b0a30fcad7a922a22c75421defb0df63f30965f2d826976.zip" -> "builds/143377d6ae26fb4a1d05941a2b0d2027358b0dc739eb552a4b34c0de179ba43d.zip"
      ~ s3_object_version              = "V_0pBScSwzUXaFCvo9grTmK8Ar2zZUYl" -> (known after apply)
        tags                           = {}
      ~ version                        = "4" -> (known after apply)
        # (23 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:4d4e5d832e5c3a37f17f7f80ab6307fd8310741a,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadffisspreadsheet" -> "git.commit.sha:7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadffisspreadsheet"
              ~ "DD_VERSION"                   = "4d4e5d832e5c3a37f17f7f80ab6307fd8310741a" -> "7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348"
                # (12 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.aws_lambda_permission.current_version_triggers["SQSQueueNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "SQSQueueNotification" -> (known after apply)
      ~ qualifier           = "4" -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (4 unchanged attributes hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "f49986d569fe7964c4a6a0955aa02fb4-2" -> (known after apply)
      ~ id                     = "builds/676ea1efc698d46e6b0a30fcad7a922a22c75421defb0df63f30965f2d826976.zip" -> (known after apply)
      ~ key                    = "builds/676ea1efc698d46e6b0a30fcad7a922a22c75421defb0df63f30965f2d826976.zip" -> "builds/143377d6ae26fb4a1d05941a2b0d2027358b0dc739eb552a4b34c0de179ba43d.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/676ea1efc698d46e6b0a30fcad7a922a22c75421defb0df63f30965f2d826976.zip" -> "builds/143377d6ae26fb4a1d05941a2b0d2027358b0dc739eb552a4b34c0de179ba43d.zip"
-       tags                   = {} -> null
      ~ version_id             = "V_0pBScSwzUXaFCvo9grTmK8Ar2zZUYl" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-DownloadFFISSpreadsheet
                            cd bin/DownloadFFISSpreadsheet
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/143377d6ae26fb4a1d05941a2b0d2027358b0dc739eb552a4b34c0de179ba43d.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/143377d6ae26fb4a1d05941a2b0d2027358b0dc739eb552a4b34c0de179ba43d.plan.json"
+       id                   = (known after apply)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "190015383386577133" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/676ea1efc698d46e6b0a30fcad7a922a22c75421defb0df63f30965f2d826976.zip" -> "builds/143377d6ae26fb4a1d05941a2b0d2027358b0dc739eb552a4b34c0de179ba43d.zip"
          ~ "timestamp" = "1684158561507696000" -> "1684341436420792200"
        }
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "7234313076040092576" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/676ea1efc698d46e6b0a30fcad7a922a22c75421defb0df63f30965f2d826976.zip" -> "builds/143377d6ae26fb4a1d05941a2b0d2027358b0dc739eb552a4b34c0de179ba43d.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-DownloadGrantsGovDB"
      ~ last_modified                  = "2023-05-15T13:49:43.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadGrantsGovDB:29" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-DownloadGrantsGovDB:29/invocations" -> (known after apply)
      ~ s3_key                         = "builds/c61d41fc5744bd2b0bfe4d14e960d765976e412a599d4def72e619b9810a43f1.zip" -> "builds/d8743b13ef35e50702f7693b23df2e6de5b50eebf317b09c352e674630464cb5.zip"
      ~ s3_object_version              = "EsRFrnhIMglx0mbcblAK5pBwErvpKBbf" -> (known after apply)
        tags                           = {}
      ~ version                        = "29" -> (known after apply)
        # (23 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                        = "git.commit.sha:4d4e5d832e5c3a37f17f7f80ab6307fd8310741a,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb" -> "git.commit.sha:7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb"
              ~ "DD_VERSION"                     = "4d4e5d832e5c3a37f17f7f80ab6307fd8310741a" -> "7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348"
                # (12 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_permission.current_version_triggers["Schedule"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "Schedule" -> (known after apply)
      ~ qualifier           = "29" -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "f2b25a7e57871116edb98b5c5b4f51be-2" -> (known after apply)
      ~ id                     = "builds/c61d41fc5744bd2b0bfe4d14e960d765976e412a599d4def72e619b9810a43f1.zip" -> (known after apply)
      ~ key                    = "builds/c61d41fc5744bd2b0bfe4d14e960d765976e412a599d4def72e619b9810a43f1.zip" -> "builds/d8743b13ef35e50702f7693b23df2e6de5b50eebf317b09c352e674630464cb5.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/c61d41fc5744bd2b0bfe4d14e960d765976e412a599d4def72e619b9810a43f1.zip" -> "builds/d8743b13ef35e50702f7693b23df2e6de5b50eebf317b09c352e674630464cb5.zip"
-       tags                   = {} -> null
      ~ version_id             = "EsRFrnhIMglx0mbcblAK5pBwErvpKBbf" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-DownloadGrantsGovDB
                            cd bin/DownloadGrantsGovDB
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/d8743b13ef35e50702f7693b23df2e6de5b50eebf317b09c352e674630464cb5.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/d8743b13ef35e50702f7693b23df2e6de5b50eebf317b09c352e674630464cb5.plan.json"
+       id                   = (known after apply)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "2597633985486809045" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/c61d41fc5744bd2b0bfe4d14e960d765976e412a599d4def72e619b9810a43f1.zip" -> "builds/d8743b13ef35e50702f7693b23df2e6de5b50eebf317b09c352e674630464cb5.zip"
          ~ "timestamp" = "1684158549353123000" -> "1684341421980542800"
        }
    }

  # module.DownloadGrantsGovDB.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "3269007772528198687" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/c61d41fc5744bd2b0bfe4d14e960d765976e412a599d4def72e619b9810a43f1.zip" -> "builds/d8743b13ef35e50702f7693b23df2e6de5b50eebf317b09c352e674630464cb5.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.EnqueueFFISDownload.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-EnqueueFFISDownload"
      ~ last_modified                  = "2023-05-15T13:49:45.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-EnqueueFFISDownload:9" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-EnqueueFFISDownload:9/invocations" -> (known after apply)
      ~ s3_key                         = "builds/1eea7bc13bc55d316c28fa34d793618c08b5c26cac94f552678713223a47b694.zip" -> "builds/53402f42f6b933170523afc0fcdf53e51648fe1f2013696f082a971753b2b936.zip"
      ~ s3_object_version              = "1QHcGdIEU2pRSQblzEdKVyXsR82MgJT2" -> (known after apply)
        tags                           = {}
      ~ version                        = "9" -> (known after apply)
        # (23 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:4d4e5d832e5c3a37f17f7f80ab6307fd8310741a,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:enqueueffisdownload" -> "git.commit.sha:7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:enqueueffisdownload"
              ~ "DD_VERSION"                   = "4d4e5d832e5c3a37f17f7f80ab6307fd8310741a" -> "7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348"
                # (12 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.EnqueueFFISDownload.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "9" -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.EnqueueFFISDownload.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "dbd4554baff1f34aecaa0448943b1c82-2" -> (known after apply)
      ~ id                     = "builds/1eea7bc13bc55d316c28fa34d793618c08b5c26cac94f552678713223a47b694.zip" -> (known after apply)
      ~ key                    = "builds/1eea7bc13bc55d316c28fa34d793618c08b5c26cac94f552678713223a47b694.zip" -> "builds/53402f42f6b933170523afc0fcdf53e51648fe1f2013696f082a971753b2b936.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/1eea7bc13bc55d316c28fa34d793618c08b5c26cac94f552678713223a47b694.zip" -> "builds/53402f42f6b933170523afc0fcdf53e51648fe1f2013696f082a971753b2b936.zip"
-       tags                   = {} -> null
      ~ version_id             = "1QHcGdIEU2pRSQblzEdKVyXsR82MgJT2" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.EnqueueFFISDownload.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-EnqueueFFISDownload
                            cd bin/EnqueueFFISDownload
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/53402f42f6b933170523afc0fcdf53e51648fe1f2013696f082a971753b2b936.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/53402f42f6b933170523afc0fcdf53e51648fe1f2013696f082a971753b2b936.plan.json"
+       id                   = (known after apply)
    }

  # module.EnqueueFFISDownload.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "7602410211319493041" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/1eea7bc13bc55d316c28fa34d793618c08b5c26cac94f552678713223a47b694.zip" -> "builds/53402f42f6b933170523afc0fcdf53e51648fe1f2013696f082a971753b2b936.zip"
          ~ "timestamp" = "1684158561571302000" -> "1684341436752705000"
        }
    }

  # module.EnqueueFFISDownload.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "370616875097487959" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/1eea7bc13bc55d316c28fa34d793618c08b5c26cac94f552678713223a47b694.zip" -> "builds/53402f42f6b933170523afc0fcdf53e51648fe1f2013696f082a971753b2b936.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-PersistGrantsGovXMLDB"
      ~ last_modified                  = "2023-05-15T13:49:48.868+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-PersistGrantsGovXMLDB:1" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-PersistGrantsGovXMLDB:1/invocations" -> (known after apply)
      ~ s3_key                         = "builds/50e3e82faf3ff628d6c85f824d64c6c9dcda1abe56604431ea532c2b6ec0de78.zip" -> "builds/cc72edf00e0b915444ee2e2b040679d3fd56c1003dd8f975f88e2a57e2ea1879.zip"
      ~ s3_object_version              = "Jrfh_RlJ4.ZK88uLew_w4dxo0Q8WT4Ec" -> (known after apply)
        tags                           = {}
      ~ version                        = "1" -> (known after apply)
        # (23 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                       = "git.commit.sha:4d4e5d832e5c3a37f17f7f80ab6307fd8310741a,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistgrantsgovxmldb" -> "git.commit.sha:7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistgrantsgovxmldb"
              ~ "DD_VERSION"                    = "4d4e5d832e5c3a37f17f7f80ab6307fd8310741a" -> "7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348"
                # (12 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "1" -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "3a1e7aa1bc60d1c34fed8e33cb1798c3-2" -> (known after apply)
      ~ id                     = "builds/50e3e82faf3ff628d6c85f824d64c6c9dcda1abe56604431ea532c2b6ec0de78.zip" -> (known after apply)
      ~ key                    = "builds/50e3e82faf3ff628d6c85f824d64c6c9dcda1abe56604431ea532c2b6ec0de78.zip" -> "builds/cc72edf00e0b915444ee2e2b040679d3fd56c1003dd8f975f88e2a57e2ea1879.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/50e3e82faf3ff628d6c85f824d64c6c9dcda1abe56604431ea532c2b6ec0de78.zip" -> "builds/cc72edf00e0b915444ee2e2b040679d3fd56c1003dd8f975f88e2a57e2ea1879.zip"
-       tags                   = {} -> null
      ~ version_id             = "Jrfh_RlJ4.ZK88uLew_w4dxo0Q8WT4Ec" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-PersistGrantsGovXMLDB
                            cd bin/PersistGrantsGovXMLDB
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/cc72edf00e0b915444ee2e2b040679d3fd56c1003dd8f975f88e2a57e2ea1879.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/cc72edf00e0b915444ee2e2b040679d3fd56c1003dd8f975f88e2a57e2ea1879.plan.json"
+       id                   = (known after apply)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "6336361892767259113" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/50e3e82faf3ff628d6c85f824d64c6c9dcda1abe56604431ea532c2b6ec0de78.zip" -> "builds/cc72edf00e0b915444ee2e2b040679d3fd56c1003dd8f975f88e2a57e2ea1879.zip"
          ~ "timestamp" = "1684158549483749000" -> "1684341422093771000"
        }
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "2997621777616623147" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/50e3e82faf3ff628d6c85f824d64c6c9dcda1abe56604431ea532c2b6ec0de78.zip" -> "builds/cc72edf00e0b915444ee2e2b040679d3fd56c1003dd8f975f88e2a57e2ea1879.zip"
            # (4 unchanged elements hidden)
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.data.aws_iam_policy_document.logs[0] will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "logs" {
+       id   = (known after apply)
+       json = (known after apply)

+       statement {
+           actions   = [
+               "logs:CreateLogGroup",
+               "logs:CreateLogStream",
+               "logs:PutLogEvents",
            ]
+           effect    = "Allow"
+           resources = (known after apply)
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_cloudwatch_log_group.lambda[0] will be created
+   resource "aws_cloudwatch_log_group" "lambda" {
+       arn               = (known after apply)
+       id                = (known after apply)
+       name              = "/aws/lambda/grants_ingest-staging-SplitFFISSpreadsheet"
+       name_prefix       = (known after apply)
+       retention_in_days = 30
+       skip_destroy      = false
+       tags_all          = {
+           "env"        = "staging"
+           "management" = "terraform"
+           "owner"      = "grants"
+           "repo"       = "grants-ingest"
+           "service"    = "grants-ingest"
+           "usage"      = "workload"
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_iam_policy.additional_json[0] will be created
+   resource "aws_iam_policy" "additional_json" {
+       arn       = (known after apply)
+       id        = (known after apply)
+       name      = "grants_ingest-staging-SplitFFISSpreadsheet"
+       path      = "/"
+       policy    = jsonencode(
            {
+               Statement = [
+                   {
+                       Action   = "secretsmanager:GetSecretValue"
+                       Effect   = "Allow"
+                       Resource = "arn:aws:secretsmanager:us-west-2:357150818708:secret:grants_ingest-staging-datadog_api_key-JGC7E3"
+                       Sid      = "GetDatadogAPIKeySecretValue"
                    },
+                   {
+                       Action   = [
+                           "s3:ListBucket",
+                           "s3:GetObject",
                        ]
+                       Effect   = "Allow"
+                       Resource = [
+                           "arn:aws:s3:::grantsingest-staging-grantsprepareddata-357150818708-us-west-2/*/*/ffis.org/v1.json",
+                           "arn:aws:s3:::grantsingest-staging-grantsprepareddata-357150818708-us-west-2",
                        ]
+                       Sid      = "AllowInspectS3PreparedData"
                    },
+                   {
+                       Action   = "s3:GetObject"
+                       Effect   = "Allow"
+                       Resource = "arn:aws:s3:::grantsingest-staging-grantssourcedata-357150818708-us-west-2/sources/*/*/*/ffis/download.xlsx"
+                       Sid      = "AllowS3DownloadSourceData"
                    },
+                   {
+                       Action   = "s3:PutObject"
+                       Effect   = "Allow"
+                       Resource = "arn:aws:s3:::grantsingest-staging-grantsprepareddata-357150818708-us-west-2/*/*/ffis.org/v1.json"
+                       Sid      = "AllowS3UploadPreparedData"
                    },
                ]
+               Version   = "2012-10-17"
            }
        )
+       policy_id = (known after apply)
+       tags_all  = {
+           "env"        = "staging"
+           "management" = "terraform"
+           "owner"      = "grants"
+           "repo"       = "grants-ingest"
+           "service"    = "grants-ingest"
+           "usage"      = "workload"
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_iam_policy.logs[0] will be created
+   resource "aws_iam_policy" "logs" {
+       arn       = (known after apply)
+       id        = (known after apply)
+       name      = "grants_ingest-staging-SplitFFISSpreadsheet-logs"
+       path      = "/"
+       policy    = (known after apply)
+       policy_id = (known after apply)
+       tags_all  = {
+           "env"        = "staging"
+           "management" = "terraform"
+           "owner"      = "grants"
+           "repo"       = "grants-ingest"
+           "service"    = "grants-ingest"
+           "usage"      = "workload"
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_iam_role.lambda[0] will be created
+   resource "aws_iam_role" "lambda" {
+       arn                   = (known after apply)
+       assume_role_policy    = jsonencode(
            {
+               Statement = [
+                   {
+                       Action    = "sts:AssumeRole"
+                       Effect    = "Allow"
+                       Principal = {
+                           Service = "lambda.amazonaws.com"
                        }
+                       Sid       = ""
                    },
                ]
+               Version   = "2012-10-17"
            }
        )
+       create_date           = (known after apply)
+       force_detach_policies = true
+       id                    = (known after apply)
+       managed_policy_arns   = (known after apply)
+       max_session_duration  = 3600
+       name                  = "grants_ingest-staging-SplitFFISSpreadsheet"
+       name_prefix           = (known after apply)
+       path                  = "/"
+       permissions_boundary  = "arn:aws:iam::357150818708:policy/service-management-boundary"
+       tags_all              = {
+           "env"        = "staging"
+           "management" = "terraform"
+           "owner"      = "grants"
+           "repo"       = "grants-ingest"
+           "service"    = "grants-ingest"
+           "usage"      = "workload"
        }
+       unique_id             = (known after apply)

+       inline_policy {
+           name   = (known after apply)
+           policy = (known after apply)
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_iam_role_policy_attachment.additional_json[0] will be created
+   resource "aws_iam_role_policy_attachment" "additional_json" {
+       id         = (known after apply)
+       policy_arn = (known after apply)
+       role       = "grants_ingest-staging-SplitFFISSpreadsheet"
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_iam_role_policy_attachment.logs[0] will be created
+   resource "aws_iam_role_policy_attachment" "logs" {
+       id         = (known after apply)
+       policy_arn = (known after apply)
+       role       = "grants_ingest-staging-SplitFFISSpreadsheet"
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_function.this[0] will be created
+   resource "aws_lambda_function" "this" {
+       architectures                  = [
+           "arm64",
        ]
+       arn                            = (known after apply)
+       description                    = "Creates per-grant JSON representation of an FFIS Spreadsheet"
+       function_name                  = "grants_ingest-staging-SplitFFISSpreadsheet"
+       handler                        = "bootstrap"
+       id                             = (known after apply)
+       invoke_arn                     = (known after apply)
+       last_modified                  = (known after apply)
+       layers                         = [
+           "arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Extension-ARM:41",
        ]
+       memory_size                    = 1024
+       package_type                   = "Zip"
+       publish                        = true
+       qualified_arn                  = (known after apply)
+       qualified_invoke_arn           = (known after apply)
+       reserved_concurrent_executions = -1
+       role                           = (known after apply)
+       runtime                        = "provided.al2"
+       s3_bucket                      = "grantsingest-staging-lambdaartifacts-357150818708-us-west-2"
+       s3_key                         = "builds/a48cbe6b0dd30f9c96c15d93905f54ab0578b098c075f6a494eeae0ce131f864.zip"
+       s3_object_version              = (known after apply)
+       signing_job_arn                = (known after apply)
+       signing_profile_version_arn    = (known after apply)
+       source_code_hash               = (known after apply)
+       source_code_size               = (known after apply)
+       tags_all                       = {
+           "env"        = "staging"
+           "management" = "terraform"
+           "owner"      = "grants"
+           "repo"       = "grants-ingest"
+           "service"    = "grants-ingest"
+           "usage"      = "workload"
        }
+       timeout                        = 300
+       version                        = (known after apply)

+       environment {
+           variables = {
+               "DD_API_KEY_SECRET_ARN"            = (sensitive value)
+               "DD_APM_ENABLED"                   = "true"
+               "DD_CAPTURE_LAMBDA_PAYLOAD"        = "true"
+               "DD_ENV"                           = "staging"
+               "DD_SERVERLESS_APPSEC_ENABLED"     = "true"
+               "DD_SERVICE"                       = "grants-ingest"
+               "DD_SITE"                          = "datadoghq.com"
+               "DD_TAGS"                          = "git.commit.sha:7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitffisspreadsheet"
+               "DD_TRACE_ENABLED"                 = "true"
+               "DD_TRACE_RATE_LIMIT"              = "1000"
+               "DD_VERSION"                       = "7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348"
+               "DOWNLOAD_CHUNK_LIMIT"             = "20"
+               "GRANTS_PREPARED_DATA_BUCKET_NAME" = "grantsingest-staging-grantsprepareddata-357150818708-us-west-2"
+               "LOG_LEVEL"                        = "INFO"
+               "MAX_CONCURRENT_UPLOADS"           = "10"
+               "S3_USE_PATH_STYLE"                = "true"
+               "TZ"                               = "UTC"
            }
        }

+       ephemeral_storage {
+           size = 512
        }

+       tracing_config {
+           mode = (known after apply)
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] will be created
+   resource "aws_lambda_permission" "current_version_triggers" {
+       action              = "lambda:InvokeFunction"
+       function_name       = "grants_ingest-staging-SplitFFISSpreadsheet"
+       id                  = (known after apply)
+       principal           = "s3.amazonaws.com"
+       qualifier           = (known after apply)
+       source_arn          = "arn:aws:s3:::grantsingest-staging-grantssourcedata-357150818708-us-west-2"
+       statement_id        = "S3BucketNotification"
+       statement_id_prefix = (known after apply)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_permission.unqualified_alias_triggers["S3BucketNotification"] will be created
+   resource "aws_lambda_permission" "unqualified_alias_triggers" {
+       action              = "lambda:InvokeFunction"
+       function_name       = "grants_ingest-staging-SplitFFISSpreadsheet"
+       id                  = (known after apply)
+       principal           = "s3.amazonaws.com"
+       source_arn          = "arn:aws:s3:::grantsingest-staging-grantssourcedata-357150818708-us-west-2"
+       statement_id        = "S3BucketNotification"
+       statement_id_prefix = (known after apply)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_s3_object.lambda_package[0] will be created
+   resource "aws_s3_object" "lambda_package" {
+       acl                    = "private"
+       bucket                 = "grantsingest-staging-lambdaartifacts-357150818708-us-west-2"
+       bucket_key_enabled     = (known after apply)
+       content_type           = (known after apply)
+       etag                   = (known after apply)
+       force_destroy          = false
+       id                     = (known after apply)
+       key                    = "builds/a48cbe6b0dd30f9c96c15d93905f54ab0578b098c075f6a494eeae0ce131f864.zip"
+       kms_key_id             = (known after apply)
+       server_side_encryption = "AES256"
+       source                 = "builds/a48cbe6b0dd30f9c96c15d93905f54ab0578b098c075f6a494eeae0ce131f864.zip"
+       storage_class          = "ONEZONE_IA"
+       tags_all               = {
+           "env"        = "staging"
+           "management" = "terraform"
+           "owner"      = "grants"
+           "repo"       = "grants-ingest"
+           "service"    = "grants-ingest"
+           "usage"      = "workload"
        }
+       version_id             = (known after apply)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-SplitFFISSpreadsheet
                            cd bin/SplitFFISSpreadsheet
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/a48cbe6b0dd30f9c96c15d93905f54ab0578b098c075f6a494eeae0ce131f864.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/a48cbe6b0dd30f9c96c15d93905f54ab0578b098c075f6a494eeae0ce131f864.plan.json"
+       id                   = (known after apply)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.null_resource.archive[0] will be created
+   resource "null_resource" "archive" {
+       id       = (known after apply)
+       triggers = {
+           "filename"  = "builds/a48cbe6b0dd30f9c96c15d93905f54ab0578b098c075f6a494eeae0ce131f864.zip"
+           "timestamp" = "1684341436919987000"
        }
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] will be created
+   resource "null_resource" "sam_metadata_aws_lambda_function" {
+       id       = (known after apply)
+       triggers = {
+           "built_output_path"    = "builds/a48cbe6b0dd30f9c96c15d93905f54ab0578b098c075f6a494eeae0ce131f864.zip"
+           "original_source_code" = jsonencode(
                [
+                   {
+                       commands = [
+                           "task build-SplitFFISSpreadsheet",
+                           "cd bin/SplitFFISSpreadsheet",
+                           ":zip",
                        ]
+                       path     = "./.."
                    },
                ]
            )
+           "resource_name"        = "aws_lambda_function.this[0]"
+           "resource_type"        = "ZIP_LAMBDA_FUNCTION"
+           "source_code_property" = "path"
        }
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-staging-SplitGrantsGovXMLDB"
      ~ last_modified                  = "2023-05-15T13:49:44.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-SplitGrantsGovXMLDB:28" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-staging-SplitGrantsGovXMLDB:28/invocations" -> (known after apply)
      ~ s3_key                         = "builds/84d5d573c7f3b80dd7840411292b681f5c70aa164f6e6f28ce5219769ab04953.zip" -> "builds/3ae3e64939e9a675139a90f014b6d21b5ef4521c39eb14996fbc630734cc982a.zip"
      ~ s3_object_version              = "7XFHmkMoDpmaz2DEig3Gi2V.haznJebi" -> (known after apply)
        tags                           = {}
      ~ version                        = "28" -> (known after apply)
        # (23 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                          = "git.commit.sha:4d4e5d832e5c3a37f17f7f80ab6307fd8310741a,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb" -> "git.commit.sha:7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb"
              ~ "DD_VERSION"                       = "4d4e5d832e5c3a37f17f7f80ab6307fd8310741a" -> "7341e0f9dbfc7f6a683e8a1658ed7610cc5e9348"
                # (15 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "28" -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_s3_object.lambda_package[0] must be replaced
-/+ resource "aws_s3_object" "lambda_package" {
      ~ bucket_key_enabled     = false -> (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "83dfe2e8d3f19c5bebbedd7f22b9c0cb-2" -> (known after apply)
      ~ id                     = "builds/84d5d573c7f3b80dd7840411292b681f5c70aa164f6e6f28ce5219769ab04953.zip" -> (known after apply)
      ~ key                    = "builds/84d5d573c7f3b80dd7840411292b681f5c70aa164f6e6f28ce5219769ab04953.zip" -> "builds/3ae3e64939e9a675139a90f014b6d21b5ef4521c39eb14996fbc630734cc982a.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ source                 = "builds/84d5d573c7f3b80dd7840411292b681f5c70aa164f6e6f28ce5219769ab04953.zip" -> "builds/3ae3e64939e9a675139a90f014b6d21b5ef4521c39eb14996fbc630734cc982a.zip"
-       tags                   = {} -> null
      ~ version_id             = "7XFHmkMoDpmaz2DEig3Gi2V.haznJebi" -> (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.local_file.archive_plan[0] will be created
+   resource "local_file" "archive_plan" {
+       content              = jsonencode(
            {
+               artifacts_dir = "builds"
+               build_plan    = [
+                   [
+                       "sh",
+                       "..",
+                       <<-EOT
                            task build-SplitGrantsGovXMLDB
                            cd bin/SplitGrantsGovXMLDB
                        EOT,
                    ],
+                   [
+                       "zip:embedded",
+                       "..",
+                       null,
                    ],
                ]
+               filename      = "builds/3ae3e64939e9a675139a90f014b6d21b5ef4521c39eb14996fbc630734cc982a.zip"
+               runtime       = "provided.al2"
            }
        )
+       content_base64sha256 = (known after apply)
+       content_base64sha512 = (known after apply)
+       content_md5          = (known after apply)
+       content_sha1         = (known after apply)
+       content_sha256       = (known after apply)
+       content_sha512       = (known after apply)
+       directory_permission = "0755"
+       file_permission      = "0644"
+       filename             = "builds/3ae3e64939e9a675139a90f014b6d21b5ef4521c39eb14996fbc630734cc982a.plan.json"
+       id                   = (known after apply)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "8595114378880332771" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/84d5d573c7f3b80dd7840411292b681f5c70aa164f6e6f28ce5219769ab04953.zip" -> "builds/3ae3e64939e9a675139a90f014b6d21b5ef4521c39eb14996fbc630734cc982a.zip"
          ~ "timestamp" = "1684158549662465000" -> "1684341422057240000"
        }
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.null_resource.sam_metadata_aws_lambda_function[0] must be replaced
-/+ resource "null_resource" "sam_metadata_aws_lambda_function" {
      ~ id       = "3130237622886103111" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "built_output_path"    = "builds/84d5d573c7f3b80dd7840411292b681f5c70aa164f6e6f28ce5219769ab04953.zip" -> "builds/3ae3e64939e9a675139a90f014b6d21b5ef4521c39eb14996fbc630734cc982a.zip"
            # (4 unchanged elements hidden)
        }
    }

Plan: 38 to add, 11 to change, 20 to destroy.

Pusher: @pearkes, Action: pull_request, Workflow: Continuous Integration

pearkes · 2023-05-16T01:46:43Z

Not really sure why CI is failing on the build step, given the Taskfile was populated. Would appreciate help on that.

jakekreider · 2023-05-16T02:40:55Z

@pearkes I think it's a casing issue here
(SplitFFISSpreadSheet vs SplitFFISSpreadsheet)

pearkes · 2023-05-16T03:39:49Z

@jakekreider great eye thanks!

TylerHendrickson

Looking good!

cmd/SplitFFISSpreadsheet/handler.go

cmd/SplitFFISSpreadsheet/s3.go

cmd/SplitFFISSpreadsheet/handler.go

cmd/SplitFFISSpreadsheet/s3.go

cmd/SplitFFISSpreadsheet/sheet.go

Co-authored-by: Tyler Hendrickson <hendrickson.tsh@gmail.com>

pearkes · 2023-05-17T16:18:17Z

@TylerHendrickson Thank you for the great feedback! Incorporated almost all of it. This is ready for another look.

cmd/SplitFFISSpreadsheet/sheet.go

TylerHendrickson

🚀

* Initial commit - skeleton TF, lambda code * FFIS DTO, tests for parsing * Pulling in DynamoDB table for grants data Sourced from #46, WIP. * Single object per file * Update schema per #94 * Fix vars * Start of table update logic, WIP * DynamoDB ops, wip * Fixed bad merge * Use grant_id, no need for opp number * Fixed field mappings and conversion * Casing to match * Clean-up * Clean-up * Update cmd/PersistFFISData/main.go Co-authored-by: Tyler Hendrickson <hendrickson.tsh@gmail.com> * Apply suggestions from code review Co-authored-by: Tyler Hendrickson <hendrickson.tsh@gmail.com> * PR feedback --------- Co-authored-by: Tyler Hendrickson <hendrickson.tsh@gmail.com>

pearkes requested a review from a team as a code owner May 15, 2023 23:09

pearkes self-assigned this May 15, 2023

pearkes force-pushed the split-ffis-spreadsheet branch from 9e6536b to f025779 Compare May 15, 2023 23:11

pearkes force-pushed the split-ffis-spreadsheet branch 2 times, most recently from f5b5a7f to cd44cf9 Compare May 15, 2023 23:17

pearkes changed the title ~~Split ffis spreadsheet~~ Split FFIS spreadsheet May 15, 2023

jakekreider mentioned this pull request May 15, 2023

Split rows from downloaded FFIS excel spreadsheets into per-grant JSON files #12

Closed

jakekreider added a commit that referenced this pull request May 16, 2023

Update schema per #94

d4cd448

TylerHendrickson requested changes May 17, 2023

View reviewed changes

pearkes and others added 10 commits May 17, 2023 08:48

initial commit

dc85c8f

working terraform for SplitFFISSpreadsheet

ed202af

SplitFFISSpreadsheet: use correct key format

11016dd

SplitFFISSpreadsheet: logging, cleanup

0c5d536

SplitFFISSpreadsheet: staticcheck changes

e47f5d5

rename opp category to bill

8e850a7

fix casing in taskfile

a57ce8d

SplitFFISSpreadsheet: improvements from Tyler

861027f

Co-authored-by: Tyler Hendrickson <hendrickson.tsh@gmail.com>

SplitFFISSpreadsheet: review feedback

4367f80

SplitFFISSpreadsheet/s3: remove unused

9d216b3

pearkes force-pushed the split-ffis-spreadsheet branch from 50a91a1 to 9d216b3 Compare May 17, 2023 15:49

pearkes added 2 commits May 17, 2023 08:51

SplitFFISSpreadsheet: add logging for total opp

25c84c9

SplitFFISSpreadsheet: fix log format

8eb7a54

TylerHendrickson reviewed May 17, 2023

View reviewed changes

cmd/SplitFFISSpreadsheet/sheet.go Outdated Show resolved Hide resolved

SplitFFISSpreadsheet: log on date parse failure

bb53dcb

TylerHendrickson approved these changes May 17, 2023

View reviewed changes

pearkes merged commit 4ae117b into main May 17, 2023

pearkes deleted the split-ffis-spreadsheet branch May 17, 2023 18:08

TylerHendrickson added enhancement New feature or request go Pull requests that update Go code terraform Pull requests that update Terraform code labels Aug 24, 2023

TylerHendrickson mentioned this pull request Mar 22, 2024

Fix: Support new URL format for hyperlinked cells in FFIS spreadsheets #730

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split FFIS spreadsheet #94

Split FFIS spreadsheet #94

pearkes commented May 15, 2023 •

edited

Loading

github-actions bot commented May 15, 2023 •

edited

Loading

pearkes commented May 16, 2023

jakekreider commented May 16, 2023

pearkes commented May 16, 2023

TylerHendrickson left a comment

pearkes commented May 17, 2023

TylerHendrickson left a comment

Split FFIS spreadsheet #94

Split FFIS spreadsheet #94

Conversation

pearkes commented May 15, 2023 • edited Loading

Ticket #12

Description

Testing

Automated and Unit Tests

Manual tests for Reviewer

Checklist

github-actions bot commented May 15, 2023 • edited Loading

Terraform Summary

Output

pearkes commented May 16, 2023

jakekreider commented May 16, 2023

pearkes commented May 16, 2023

TylerHendrickson left a comment

Choose a reason for hiding this comment

pearkes commented May 17, 2023

TylerHendrickson left a comment

Choose a reason for hiding this comment

pearkes commented May 15, 2023 •

edited

Loading

github-actions bot commented May 15, 2023 •

edited

Loading