amtool not working in prometheus-engine/alertmanager:v0.25.1-gmp.6-gke.0

[suryaincbe]

We have enabled Google Managed Prometheus in our cluster and the control plane is in version 1.30.2-gke.1023004.

We used to view firing alerts from alertmanager pods in gmp-system namespace using the command,
kubectl exec -n gmp-system alertmanager-0 -c alertmanager -- amtool --alertmanager.url=http://alertmanager:9093 alert

But now the alertmanger pods are using image prometheus-engine/alertmanager:v0.25.1-gmp.6-gke.0 in the 1.30 cluster version and we are not able use the amtool command.

[TheSpiritXIII]

Looks like the amtool was removed in GoogleCloudPlatform/alertmanager#81

This looks like something the team should consider adding back.

As a workaround, you could port-forward the Alertmanager container and use the embedded web UI. However, do note that some of the settings, e.g. silences, don't persist when the pod restarts!

There's also special synthetic metrics emitted from the alertmanager, which you might be able to use to see.

We apologize for the inconvenience!

[suryaincbe]

Looks like the amtool was removed in GoogleCloudPlatform/alertmanager#81

This looks like something the team should consider adding back.

As a workaround, you could port-forward the Alertmanager container and use the embedded web UI. However, do note that some of the settings, e.g. silences, don't persist when the pod restarts!

There's also special synthetic metrics emitted from the alertmanager, which you might be able to use to see.

We apologize for the inconvenience!

Hello,

We have many number of clusters, so to silence alerts in each of these clusters it would be difficult to use the port forward option.
It would really helpful if we have the amtool.

[TheSpiritXIII]

Another option you can try is using the old image in conjunction with a self-deployed alertmanager. The old image is still available so you can deploy it yourself and point GMP to it.

In the meanwhile I created a PR to reintroduce amtool: GoogleCloudPlatform/alertmanager#91