Remediate intermittent deploy failures/timeout on prometheus

[zachariahmiller]

Describe what should be investigated or refactored

Occasionally in local dev and frequently in CI Prometheus is failing to deploy successfully and deployment times out without manual intervention.

Additional context

This may be related to pepr istio container job termination and the admission/patch jobs not getting killed but we dont always have good feedback as to if this is the case in ci.

[corang]

Just encountered this myself, the istio sidecar isn't terminating correctly on the kube-prometheus patch something or other, job container is done but sidecar won't die

[zachariahmiller]

Yeah that's the assumption. Pepr is supposed to terminate that job. It's an edge case, but not totally sure if it's the watch getting dropped or something in the actual job termination code.

[corang]

I do want to say after like 8 minutes the sidecar did eventually terminate, so its like pepr is getting hung up on something

[docandrew]

I'm running into this as well in the monitoring namespace when trying UDS Core on RKE2. I had to manually kubectl debug into the pod and kill istio-proxy with the magic /quitquitquit URL to get the UDS Core deployment to continue.

[mjnagel]

Should be resolved by #419 - leaving open until that is released and we have feedback though.

[mjnagel]

Tentatively closing - please reopen or create a new issue if you encounter this problem in latest core versions.