Preprod API is broken

[carlgieringer]

TODO

• Fix logging errors (Fix toJsonWithReplacer #687)
• Update code to handle missing parameter store parameters (Don't request missing parameter store parameters #688)
• Figure out how prod has permission to access scraping parameters. I don't know how prod was working. But I updated the IAM policy AccessDatabaseConnectionInfoSSMParameters to grant access to all parameters.
• Add missing parameters for pre-prod
• Release to preprod upon merge into master (Deploy to pre-prod after updating master #689)

Original issues:

Browser sees all 502s.

In CloudWatch I see:

2024-07-27T05:29:50.848Z	403228b5-1db2-4320-9f29-353019b5dcad	ERROR	Invoke Error 	
{
    "errorType": "TypeError",
    "errorMessage": "Converting circular structure to JSON\n    --> starting at object with constructor 'IncomingMessage'\n    |     property 'req' -> object with constructor 'ClientRequest'\n    --- property 'res' closes the circle",
    "stack": [
        "TypeError: Converting circular structure to JSON",
        "    --> starting at object with constructor 'IncomingMessage'",
        "    |     property 'req' -> object with constructor 'ClientRequest'",
        "    --- property 'res' closes the circle",
        "    at JSON.stringify (<anonymous>)",
        "    at AwsLogger3.makeJsonLogArguments [as makeLogArguments] (/howdju-service-common/lib/logging/AwsLogger.js:94:30)",
        "    at AwsLogger3.<anonymous> (/howdju-service-common/lib/logging/AwsLogger.js:105:26)",
        "    at routeRequest (/var/src/router.ts:208:26)"
    ]
}

[carlgieringer]

I think there is an underlying error, but when we log it we get another error due to not handling circularity. (See here.)

[carlgieringer]

After fixing the logging error, I can see:

"message": "User: arn:aws:sts::007899441171:assumed-role/premiserLambdaRole/premiserApi is not authorized to perform: ssm:GetParameters on resource: arn:aws:ssm:us-east-1:007899441171:parameter/pre_prod/scraping-ant-api-key because no identity-based policy allows the ssm:GetParameters action",