Monitoring - Split PROD & STAGING build-error-check Canary into 2 Separate Canaries

[omgitsbillryan]

The Problem

The build-error-check AWS Synthetic Canary currently checks both staging.va.gov & va.gov. Each env should have it's own canary so they can have different alert rules & thresholds.

Extra Context

This could easily be done by using canary ENV variables - each canary would share the same JS code but have different ENV values. However, ENV vars for canaries are currently not supported in terraform. We could manually set these ENV vars in the AWS Canary UI, but they would get wiped each time the terraform for that canary gets applied.

The other alternative would be to duplicate the javascript code on our side - one for each canary. I opted against this and chose instead to just keep 1 canary that checks both envs.

Work to be Done

• Have 2 separate canaries running, one for staging and one for prod
• Each canary has its own CW alarm & SNS topic connections

[jhouse-solvd]

@pjhill & @JoeTice -
This was just a suggestion based on where we left the canary when we shipped it to cover the base use cases that Dror was interested in.
Can you take a look at this and see if it would still be relevant to the work that y'all are doing around synthetics?

[jbritt1]

@pjhill & @JoeTice, since synthetic-style monitoring has mostly gone to Datadog, I would feel this ticket to no longer be relevant or a priority. Could we close in your opinion(s)?

[jbritt1]

Bumping for feedback. Also posted in channel: https://dsva.slack.com/archives/C01CHAY3ULR/p1657739962454579