Add "alertmanager_provider_alerts" metric

[Pharb]

Fixes #1439 by adding a separate alertmanager_provider_alerts metric.

[Pharb]

@simonpasquier As you were responding to #1439 previously, if you have time, could you take a quick look at this and maybe give some feedback for my changes? Thank you 👍

[juliusv]

If this still preserves the state="active" label for resolved alerts, I think that's a bit confusing. Maybe rename that to suppressed="true|false"?

Or alternatively only expose non-resolved (actually active) alerts, to match what people would see on /alerts.

[juliusv]

Ah, state can also be unprocessed (and maybe more, didn't check), so just true/false for suppressed wouldn't work.

[Pharb]

@juliusv Thanks for your feedback. I agree that the whole alert "status/states" are a bit confusing in Alertmanager.

The alertmanager_alerts{state="firing"} metric from this PR should match the number of currently "active" alerts from the UI as I would expect as a user of Alertmanager.

[Pharb]

I rebased against the current master due to a minor merge conflict of imports in cmd/alertmanager/main.go.

[Pharb]

The continuous-integration/travis-ci/pr check failed due to an unrelated issue that currently affects master (e.g. https://travis-ci.org/prometheus/alertmanager/builds/468643002).

[nielsole]

Hey, are you still working on this? :)

[Pharb]

@nielsole Yes, I just noticed a few days ago that there are now merge conflicts due to the refactoring in #1741. I will try to get those fixed in the next few days.

Sadly I still cannot say if the PR will then be accepted, but at least I cannot run many Alertmanager instances in production without this fix included... (currently I have my fork deployed in multiple production clusters).

[Pharb]

Due to the recent refactoring in #1741 I changed the PR to add a new alertmanager_provider_alerts metric, because that seems to be the simplest solution at the moment.
Maybe that also avoids the confusion between having firing and active values on the same metric.

@juliusv Would be great to get this accepted, as this works pretty well for my use case and solves #1439 by providing an alternative to the alertmanager_alerts metric. Of course I'm also open for feedback on the implementation or other suggestions on how to solve this problem.

[nielsole]

Any updates on this? This week I would have time to support in any way :)

[Pharb]

Is there anything preventing this from being merged? Would be nice to get these reliable metrics built-in, this works quite well for me in production. @juliusv

[simonpasquier]

@Pharb sorry for the very late reply. One thing to remember is that resolved alerts can be removed at any time (eg when the garbage collection kicks in) meaning that alertmanager_provider_alerts{state="resolved"} will have occasional drops. I'd be inclined to expose only the current number of firing alerts but happy to hear about other opinions.

[Pharb]

@simonpasquier Thank you for taking a look at this.

As suggested by you, I removed alertmanager_provider_alerts{state="resolved"}, because I don't have a use case for that. If anybody would need that, it is very easy to add again. I also implemented and resolved your other comments.

[Pharb]

@simonpasquier Thanks for your help so far.
Do you have any suggestion what I could do to get this merged?

[Pharb]

PR is now obsolete because an alternative implementation was merged yesterday: #2943

Issue is finally solved 🎉