Metric alertmanager_alerts report incorrect number of alerts

[gotjosh]

What did you do?

Setup the alertmanager with clustering, sent alerts to all Alertmanagers.

What did you expect to see?

The alertmanager_alerts metric always converge across replicas.

What did you see instead? Under which circumstances?

As time passes the alertmanager alerts metric drift apart although no difference in API results - they are consistent.

As you can see in the screenshot, as time progresses the alertmanager_alerts metric drift apart in the range of 10s per replica.

Verifying the API responses across replicas, I can see that no inconsistent results are being provided:

$ cat alertmanager-0 | jq '.data | length'
956
$ cat alertmanager-1 | jq '.data | length'
956

Looking at the metric implementation, we can see that the Marker is responsible for giving us the information of the current alerts but the in-memory alerting store coming from memAlerts is responsible for setting the callback that will keep the marker synced with the current alerts held in the Store. However, it is very apparent that the Store would execute the callback ONLY when we garbage collect and NOT when we delete an alert directly.

We believe the fix is to also execute the callback when an alert is directly deleted.

alertmanager/store/store.go

Lines 73 to 116 in 58169c1

	func (a *Alerts) gc() {
	a.Lock()
	defer a.Unlock()
	var resolved []*types.Alert
	for fp, alert := range a.c {
	if alert.Resolved() {
	delete(a.c, fp)
	resolved = append(resolved, alert)
	}
	}
	a.cb(resolved)
	}
	// Get returns the Alert with the matching fingerprint, or an error if it is
	// not found.
	func (a *Alerts) Get(fp model.Fingerprint) (*types.Alert, error) {
	a.Lock()
	defer a.Unlock()
	alert, prs := a.c[fp]
	if !prs {
	return nil, ErrNotFound
	}
	return alert, nil
	}
	// Set unconditionally sets the alert in memory.
	func (a *Alerts) Set(alert *types.Alert) error {
	a.Lock()
	defer a.Unlock()
	a.c[alert.Fingerprint()] = alert
	return nil
	}
	// Delete removes the Alert with the matching fingerprint from the store.
	func (a *Alerts) Delete(fp model.Fingerprint) error {
	a.Lock()
	defer a.Unlock()
	delete(a.c, fp)
	return nil
	}

[gotjosh]

Fixed by #2943