-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enqueue FFIS email downloads #53
Enqueue FFIS email downloads #53
Conversation
- Terraform skeleton - Command skeleton - Simple parser for plaintext email - Compose file for Localstack
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a few scattered comments, but overall this is looking good.
To address the points you listed for further discussion:
- I'm not familiar with the CloudPosse modules, but I didn't see where SES could be configured for the SQS queue, so I used the stock TF resource
We've used certain CloudPosse modules only where it (presumably) simplifies things, but they're by no means required – your implementation of the aws_sqs_queue
resource looks good to me.
For the TF structure, I put the queue in the EnqueueFFISDownload module. Should it be top-level?
Since this resource is common to multiple Lambdas (as it serve as the event source for the Lambda to-be-implemented in #11), I think it would be better to define it at the top-level and pass in the necessary attribute(s) to this module via input variable.
Am I using the log formatter correctly?
A few of the usages' arguments appear to be in the wrong order with regards to the main message – the INFO-level "Starting EnqueueFFISDownload"
usage is an example of correct usage. Put plainly, any call to log.Info()
or log.Debug()
requires some kind of (narrative/explanatory) message, and any additional arguments are intended to be key/value pairs that get added to the logged JSON output.
For example, this call in getEmailFromS3Event()
:
log.Debug(logger, bucket, "Reading from bucket")
will emit JSON that looks like this (formatted as multiline for readability):
{
"msg": "name-of-the-bucket",
"Reading from bucket": "(MISSING)",
"level": "debug",
"caller": "handler.go:145",
"ts": "2023-04-25T18:51:20.549912"
}
By contrast, a correct call would look like this:
log.Debug(logger, "Reading from bucket", "bucket_name", bucket)
and would produce the following JSON:
{
"msg": "Reading from bucket",
"bucket_name": "name-of-the-bucket",
"level": "debug",
"caller": "handler.go:145",
"ts": "2023-04-25T18:51:20.549912"
}
Any additional pairs of arguments are handled similarly:
log.Debug(logger, "Reading from bucket", "bucket", bucket, "key", s3Event.Records[0].S3.Object.Key)
produces:
{
"msg": "Reading from bucket",
"bucket": "name-of-the-bucket",
"key": "path/to/the/file.eml",
"level": "debug",
"caller": "handler.go:145",
"ts": "2023-04-25T18:51:20.549912"
}
- Moved queue out of module - Fixed log param ordering - Renamed handleS3Event - Fixed param required - Moved global vars to env object - Cleaned up docker-compose for localstack - Moved key vars from init-aws startup script - Changed lambda params for timeout and memory - Removed unneeded DD vars from lambda definition - Referenced full queue URL in lambda instead of name - Use reader for cleaner/more efficient IO in email handling
Thanks again for the detailed feedback @TylerHendrickson - Not sure why the log param ordering wasn't clicking with me but your explanation made perfect sense. I think I incorporated everything you requested. A couple of callouts:
SES terraform error
|
@jakekreider Responding to your callouts:
Yeah, looks like workflows that are initiated by the For the case at-hand, I think the easiest path here might be to change (rebase?) this PR to a branch local to this repository, which you should have permission to do as a team member.
This is also currently happening on the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a few very minor nits and suggested tweaks, most of which are related to logging.
We don't have any firm test coverage requirements, but I think the missing branch I mentioned (regarding the multiple.eml
case/fixture) in the comments would be good to have.
}{ | ||
{"good.eml", "https://mcusercontent.com/123456/files/file-01.xlsx"}, | ||
{"missing.eml", ""}, | ||
{"multiple.eml", ""}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Despite this test cases's fixture looking correct as far as I can tell, I don't see any coverage on the if len(matches) > 1
branch of parseURLFromEmailBody()
. Would you be able to look into that?
Hint: if you have Taskfile installed, you can run task test
to generate an HTML coverage report.
Co-authored-by: Tyler Hendrickson <hendrickson.tsh@gmail.com>
|
Ticket #4
Closes #4
Description
As described in the ticket,
Handler (in
cmd/EnqueueFFISDownload
) responds to S3 events in the grants bucket source, looks for a download link in it, and enqueues it into SQS for later processing.Some things to discuss:
I'm not familiar with the CloudPosse modules, but I didn't see where SES could be configured for the SQS queue, so I used the stock TF resourceFor the TF structure, I put the queue in the EnqueueFFISDownload module. Should it be top-level?Am I using the log formatter correctly?I expect some revs on this, but it was getting lengthy and I wanted to come up for air/iterate. Appreciate feedback as always!
Testing
docker compose up -d
) works heretflocal apply -var-file=local.tfvars -auto-approve
awslocal s3 cp ../cmd/EnqueueFFISDownload/fixtures/good.eml s3://grants-ingest-grantssourcedata-000000000000-us-west-2/sources/2023/04/24/ffis/raw.eml
curl -H "Accept: application/json" http://localhost:4566/_aws/sqs/messages?QueueUrl=http://localhost:4566/000000000000/ffis_downloads | jq '.ReceiveMessageResponse.ReceiveMessageResult.Message.Body'
Automated and Unit Tests
Manual tests for Reviewer
Checklist