k8s-infra-prow-build: Add monitoring

[koksay]

Adds monitoring components for the k8s-infra-prow-build cluster.
Mainly copied from eks-prow-build-cluster configuration

Please note that, the alertmanager-slack-token.yam file has invalid data for this cluster. It will be updated later when the external secrets are in place.

[ameukam]

Probably drop the entire alerting config for the moment. we want metrics visualization first.

[koksay]

@ameukam I removed alertmanager folder and disabled it's configuration on prometheus.yaml

[xmudrii]

/assign

[xmudrii]

Thank you for taking care of this, @koksay!
/lgtm
/approve
/hold
for other folks to take a look

[upodroid]

Can we look into using Thanos to aggregate and store metrics for a period of time?

Also, I would prefer if prometheus operator was deployed via helm as the diff in Git would be very tiny.

[xmudrii]

We plan to do both, however:

• This PR replicates the EKS monitoring setup pretty much 1:1 and given that we're sure that this setup works well, we don't want to change it at this point
• Integrating with Thanos will take some time and we need the monitoring stack ASAP-ish to prepare the workshop that we have at the Contributor Summit
• The way we apply resources to the GKE cluster is not really compatible with Helm (e.g. I don't know if we even have Helm in the image that's used by post-k8sio-deploy-prow-build-resources job), so using Helm would require additional effort that's probably not worth it as we'll have ArgoCD pipeline for the GKE cluster at some point soon-ish

That said, I have preference to proceed with this PR as it is and iterating (for both GKE and EKS clusters) in the near future.

[upodroid]

Let's add Thanos after Kubecon
/hold cancel

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: koksay, upodroid, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• infra/gcp/terraform/k8s-infra-prow-build/prow-build/resources/OWNERS [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[BenTheElder]

Pretty sure this change is what is causing scheduling issues for some PR/release blocking jobs:

We have to be careful about increasing per-node resource utilization in the build clusters (as opposed to say, a deployment, which we can scale around just fine).
We have jobs that are best run as the sole tenant and request ~100% schedulable CPU to ensure this. See more in the linked issue.

I think we should consider reverting and then revisit shortly after, given kubernetes/kubernetes has code freeze pending and we're having trouble testing and merging PRs. kubernetes/test-infra#32157 (comment)

For revisit we can discuss in kubernetes/test-infra#32157 after confirming that CI is running smoothly again ...

[koksay]

@BenTheElder What if we remove the requests but keep limits for the node-exporter ds?