-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix: Handle missing LastUpdatedDate
attribute in DynamoDB items
#922
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
bug
Something isn't working
go
Pull requests that update Go code
labels
Sep 27, 2024
Terraform Summary
OutputValidation Output
Plan OutputTerraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place
-/+ destroy and then create replacement
Terraform will perform the following actions:
# datadog_metric_metadata.custom["grants_ingest.SplitGrantsGovXMLDB.record.skipped"] will be created
+ resource "datadog_metric_metadata" "custom" {
+ description = "Count of unchanged grant records from Grants.gov data skipped during invocation."
+ id = (known after apply)
+ metric = "grants_ingest.SplitGrantsGovXMLDB.record.skipped"
+ short_name = "Skipped grant records"
+ type = "gauge"
+ unit = "record"
}
# module.DownloadFFISSpreadsheet.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-DownloadFFISSpreadsheet"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-DownloadFFISSpreadsheet:63" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-DownloadFFISSpreadsheet:63/invocations" -> (known after apply)
tags = {}
~ version = "63" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadffisspreadsheet" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadffisspreadsheet"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (11 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.DownloadFFISSpreadsheet.module.lambda_function.aws_lambda_permission.current_version_triggers["SQSQueueNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "SQSQueueNotification" -> (known after apply)
~ qualifier = "63" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (4 unchanged attributes hidden)
}
# module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-DownloadGrantsGovDB"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-DownloadGrantsGovDB:63" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-DownloadGrantsGovDB:63/invocations" -> (known after apply)
tags = {}
~ version = "63" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (13 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_permission.current_version_triggers["Schedule"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "Schedule" -> (known after apply)
~ qualifier = "63" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
# module.EnqueueFFISDownload.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-EnqueueFFISDownload"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-EnqueueFFISDownload:63" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-EnqueueFFISDownload:63/invocations" -> (known after apply)
tags = {}
~ version = "63" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:enqueueffisdownload" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:enqueueffisdownload"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (11 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.EnqueueFFISDownload.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "S3BucketNotification" -> (known after apply)
~ qualifier = "63" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
# module.ExtractGrantsGovDBToXML.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-ExtractGrantsGovDBToXML"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-ExtractGrantsGovDBToXML:63" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-ExtractGrantsGovDBToXML:63/invocations" -> (known after apply)
tags = {}
~ version = "63" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:extractgrantsgovdbtoxml" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:extractgrantsgovdbtoxml"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (11 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.ExtractGrantsGovDBToXML.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "S3BucketNotification" -> (known after apply)
~ qualifier = "63" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
# module.PersistFFISData.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-PersistFFISData"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistFFISData:63" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistFFISData:63/invocations" -> (known after apply)
tags = {}
~ version = "63" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistffisdata" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistffisdata"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (11 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.PersistFFISData.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "S3BucketNotification" -> (known after apply)
~ qualifier = "63" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
# module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-PersistGrantsGovXMLDB"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistGrantsGovXMLDB:63" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistGrantsGovXMLDB:63/invocations" -> (known after apply)
tags = {}
~ version = "63" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistgrantsgovxmldb" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistgrantsgovxmldb"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (11 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "S3BucketNotification" -> (known after apply)
~ qualifier = "63" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
# module.PublishGrantEvents.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-PublishGrantEvents"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PublishGrantEvents:64" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PublishGrantEvents:64/invocations" -> (known after apply)
tags = {}
~ version = "64" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:publishgrantevents" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:publishgrantevents"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (11 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.PublishGrantEvents.module.lambda_function.aws_lambda_permission.current_version_triggers["dynamodb"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "dynamodb" -> (known after apply)
~ qualifier = "64" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
# module.ReceiveFFISEmail.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-ReceiveFFISEmail"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-ReceiveFFISEmail:62" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-ReceiveFFISEmail:62/invocations" -> (known after apply)
tags = {}
~ version = "62" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:receiveffisemail" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:receiveffisemail"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (12 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.ReceiveFFISEmail.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "S3BucketNotification" -> (known after apply)
~ qualifier = "62" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
# module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-SplitFFISSpreadsheet"
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-SplitFFISSpreadsheet:63" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-SplitFFISSpreadsheet:63/invocations" -> (known after apply)
tags = {}
~ version = "63" -> (known after apply)
# (21 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitffisspreadsheet" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitffisspreadsheet"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (14 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "S3BucketNotification" -> (known after apply)
~ qualifier = "63" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
# module.SplitGrantsGovXMLDB.module.lambda_artifact.aws_s3_object.lambda_function must be replaced
-/+ resource "aws_s3_object" "lambda_function" {
+ acl = (known after apply)
~ arn = "arn:aws:s3:::grantsingest-lambdaartifacts-357150818708-us-west-2/80e96fb0b8aa2259bc48dca65a8a2625.zip" -> (known after apply)
~ bucket_key_enabled = false -> (known after apply)
+ checksum_crc32 = (known after apply)
+ checksum_crc32c = (known after apply)
+ checksum_sha1 = (known after apply)
+ checksum_sha256 = (known after apply)
~ content_type = "binary/octet-stream" -> (known after apply)
~ etag = "d00c7b857ec29fd75e4cc3a42a11c6f7-3" -> (known after apply)
~ id = "80e96fb0b8aa2259bc48dca65a8a2625.zip" -> (known after apply)
~ key = "80e96fb0b8aa2259bc48dca65a8a2625.zip" -> "49619436296c4287a68f6bd8fc28c125.zip" # forces replacement
+ kms_key_id = (known after apply)
- metadata = {} -> null
~ storage_class = "STANDARD" -> (known after apply)
- tags = {} -> null
~ version_id = "XlciAkJiQWsYFJZSwESr25oQQ6fSSxak" -> (known after apply)
# (5 unchanged attributes hidden)
}
# module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
~ resource "aws_lambda_function" "this" {
id = "grants_ingest-SplitGrantsGovXMLDB"
~ last_modified = "2024-09-27T20:47:41.000+0000" -> (known after apply)
~ qualified_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-SplitGrantsGovXMLDB:63" -> (known after apply)
~ qualified_invoke_arn = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-SplitGrantsGovXMLDB:63/invocations" -> (known after apply)
~ s3_key = "80e96fb0b8aa2259bc48dca65a8a2625.zip" -> "49619436296c4287a68f6bd8fc28c125.zip"
tags = {}
~ version = "63" -> (known after apply)
# (19 unchanged attributes hidden)
~ environment {
~ variables = {
~ "DD_TAGS" = "git.commit.sha:17d62fc0846932d8cef4e34a8d78c4cb727f6c02,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb" -> "git.commit.sha:a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb"
~ "DD_VERSION" = "17d62fc0846932d8cef4e34a8d78c4cb727f6c02" -> "a994497e74be48dcd0463f4c2e4a7d8f6dd6eda5"
# (17 unchanged elements hidden)
}
}
# (3 unchanged blocks hidden)
}
# module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
~ id = "S3BucketNotification" -> (known after apply)
~ qualifier = "63" # forces replacement -> (known after apply) # forces replacement
+ statement_id_prefix = (known after apply)
# (5 unchanged attributes hidden)
}
Plan: 12 to add, 10 to change, 11 to destroy. Pusher: @TylerHendrickson, Action: |
as1729
approved these changes
Sep 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket #878
Description
This PR addresses a condition where a DynamoDB item has no
LastUpdatedDate
attribute, which can happen when FFIS data for a grant was ingested before the Grants.gov data for the same grant. Currently, this manifests as a time-parsing error, since an empty string cannot be parsed according to the expectedMMDDYYYY
format, which preventsprocessRecord()
from moving forward with uploading the record to S3.Cases where
LastUpdatedDate
cannot be parsed should generally result in an error, since we want to be alerted if records regularly contain un-parseable values. However, given that FFIS data may be ingested for a grant before Grants.gov data (although it should be fairly uncommon), we need to be able to handle the possibility thatLastUpdatedDate
may not yet be present for a grant record stored in DynamoDB. To handle this scenario gracefully, we simply check for a blankLastUpdatedDate
value before attempting to parse it. If the value is blank, we represent the*time.Time
value asnil
, which is consistent with the behavior implemented when a DynamoDB record is altogether missing for a grant.Unit tests have been revised to comprehensively cover each of the following scenarios that can be encountered when evaluating
LastUpdatedDate
values from DynamoDB:LastUpdatedDate
that is in the past relative to the record on-handLastUpdatedDate
that is in the future relative to the record on-handLastUpdatedDate
that is equal to that of the record on-handLastUpdatedDate
that is malformed / un-parseableLastUpdatedDate
is blank or missingTesting
Should be sufficient to run unit tests.
Automated and Unit Tests
Manual tests for Reviewer
Checklist