Download queued FFIS spreadsheets to S3

[TylerHendrickson]

DOD: When a message containing a URL for an FFIS spreadsheet is sent to the "FFIS Downlaods" SQS queue, the spreadsheet file that the URL points to is saved to the "Grants Source Data" S3 bucket.

Blockers:

• Enqueue download jobs for FFIS digest spreadsheets #4

Implementation details:

• Create a "Download FFIS Spreadsheet" Lambda function that:
  • Sources invocation events from the "FFIS Downloads" SQS queue
  • When invoked, uses the URL from the SQS message in the invocation event to download the file at that URL.
  • Uses the download stream in order to upload the file as a new object in the "Grants Source Data" S3 bucket, with an object key prefix of /sources/YYYY/mm/dd/ffis/download.xlsx.

[jakekreider]

@TylerHendrickson What do you expect the /YYYY/mm/dd portion of the object key prefix to come from- the time of invocation of the Lambda, or copied from the path of the source email read by EnqueueFFISDownloads?

[TylerHendrickson]

@TylerHendrickson What do you expect the /YYYY/mm/dd portion of the object key prefix to come from- the time of invocation of the Lambda, or copied from the path of the source email read by EnqueueFFISDownloads?

@jakekreider The date-based object key path components are meant to reference the revision date of the data stored at that particular path. So in this case, we should copy the path of the source email, which represents the date we received the digest. Thanks for checking!