#2205 - Comp and Pen Batches

[k-macmillan]

Description

Test turned full implementation. Rather than process all records from the table with Celery Beat we instead run AWS Glue. Glue has been used as an ETL. It pulls all records from dynamodb, changes them into the required format, and then sends them to SQS in "batches" (homebrew batches, not sqs batches). This completed in under a minute on prod with 65,580 records, so DelaySeconds was added to slow it down.

Removed Comp and Pen feature flag. Removed notification_type from the bypass route because it was redundant.

issue #2205

How Has This Been Tested?

Deployed the notification-api and then ran the Glue script. It ingested all the records as expected and processed them correctly. The celery broker picked up the work and sent it to Celery tasks for processing.

• Task that processed (check out the notification id associated with record 42)
• Notification being delivered for the one item that had a valid va profile id
• Failed delivery due no VA Profile id found

The prod-bip-consumer-dead-letter-queue was purged and all prod records were sent there so they can be seen, though they contain an extra field that I have since deleted.
65,580 / 25 = 2623.2 (which would round to one more send_message call, or 2624):

It managed to process and enqueue all of those in under a minute:

Unit tests cover 100% of the new code:

Checklist

• I have assigned myself to this PR
• PR has an appropriate title: #9999 - What the thing does
• PR has a detailed description, including links to specific documentation
• I have added the appropriate labels to the PR.
• I did not remove any parts of the template, such as checkboxes even if they are not used
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to any documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works. Testing guidelines
• I have ensured the latest main is merged into my branch and all checks are green prior to review
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• The ticket was moved into the DEV test column when I began testing this change