feat: add datadog back-end logging

[halfwhole]

Problem

Datadog currently does not receive any of our server logs

Solution

Added datadog-winston to send our server logs to Datadog. It works on staging, the staging server logs are visible on the log explorer

What would be a good name for service? Having something like [gov/edu/health]-[staging/production] seems right, but I also have some questions:

• Should we add a prefix like go to the beginning (e.g. go-health-production)? The log explorer contains logs from all projects across OGP, so having a prefix can help to differentiate our project from other teams'
• Our naming scheme on AWS is quite inconsistent, e.g. for the gov variant we have go-production on EB and gosg-production on RDS. Our team might want to discuss and standardise this?
• Using assetVariant directly might not be the best idea, but this might depend on how we want to define the constants/env variables eventually (pending discussions in feat: datadog integration to client side using datadog/browser-rum lib #1896)

Deploy Notes

New dependencies:

• datadog-winston: for sending winston logs to datadog

New dev dependencies:

• @types/datadog-winston

[yong-jie]

naming scheme on AWS is quite inconsistent, e.g. for the gov variant we have go-production on EB and gosg-production on RDS

agreed. for a bit more context, Go's infra was originally gosg-*, same with elastic beanstalk as well. The reason why you're seeing it as go-* was because we needed to migrate our EB platform to Amazon Linux 2, and using a different prefix like go-* was the best differentiation strategy we could think of 😓

of course, we did not need to make any modifications to our RDS instance and therefore the name is still gosg-*

Should we add a prefix like go to the beginning (e.g. go-health-production

i vote yes for the same reason you mentioned!

[thanhdatle]

agreed. for a bit more context, Go's infra was originally gosg-*, same with elastic beanstalk as well. The reason why you're seeing it as go-* was because we needed to migrate our EB platform to Amazon Linux 2, and using a different prefix like go-* was the best differentiation strategy we could think of 😓

This what we was guessing ytd

Should we add a prefix like go to the beginning (e.g. go-health-production

i vote yes for the same reason you mentioned!

It's a yes from me too!

[halfwhole]

nice will go with the new service naming then! the env variable DD_SERVICE on gov/edu/health should be set to go-[gov/edu/health]-[staging/production], so should be able to use that by default instead of setting the service manually

have also updated the tracing utility to enable log injections, which helps datadog to connect our logs with our traces (https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/nodejs/)

[yong-jie]

nice work, code mostly lgtm. i dropped some suggested changes to improve code safety that we can discuss about!