L3 231/add monitoring

[msterle]

PR 1/2 for L3-231

This PR adds monitoring to the king cluster, production cluster addressed in a follow-up PR

• Added kube-prometheus-stack chart for monitoring.
• Added ServiceMonitors for:
  • sqnc-node (for alice, bob and charlie)
  • sqnc-identity-service postgresql (for alice, bob and charlie)
  • sqnc-matchmaker-api postgresql (for alice, bob and charlie)
  • flux components
  • self-monitoring of monitoring components
• Add grafana dashboards for:
  • alertmanager
  • flux cluster stats (unsure if working, pulled from flux documentation)
  • flux control plane (pulled from flux documentation)
  • Flux2 (from grafana.com)
  • nginx (from grafana.com)
  • polkadot (from grafana.com)
  • postgresql (from grafana.com)

Notes:

• Currently, the flux cluster stats dashboard is broken due to a bug in the kube-state-metrics chart. This bug has been resolved, but the fix has not been pulled into kube-prometheus-stack yet

[dblane-digicatapult]

This looks good, we should ensure that alice, bob, charlie and the nginx/infra sync depend on the monitoring sync kustomization being completed first as this introduces the CRDs that will subsequently be used in those kustomizations/helm charts.