Propagate labels to Opsgenie details

[ricoberger]

Introduce two new config options for the Opsgenie receiver:

• all_labels_as_details: If true, all labels are propagated to the Opsgenie details (default is false).
• labels_as_details: An optional list of labels, which should be propagated to the Opsgenie details.

If the all_labels_as_details and labels_as_details value isn't set, the existing details option is used.

Closes #682.

[simonpasquier]

Thanks for looking into this.

Instead of having 2 more parameters, maybe we can interpret labels_as_details: ['...'] as "include all common labels in the details" (this would be similar to what has been done for group_by).

As it stands, labels_as_details and details are exclusive so it would be good IMO to fail the configuration parsing if the user configures both at the same time. Or maybe the resulting map should be a merge of both with details taking precedence over labels_as_details?

[brian-brazil]

this would be similar to what has been done for group_by

Grouping labels are not the same as common labels. The naming of these proposed config fields is confusing.

[ricoberger]

Hi @simonpasquier and @brian-brazil, I made the suggested changes, so that there is only a details_labels field. The handling is similar to the group_by handling.

If the details_labels and details field is provided the loading of the configuration will fail now.

[brian-brazil]

This is still misleading. Common is not all, in fact all is not possible.

[ricoberger]

This is still misleading. Common is not all, in fact all is not possible.

So you want to change the option to details_common_labels or sth. similar?

[bwplotka]

Amazing contribution and sorry for the delay in review.

I think this is almost ready to go. I have some idea that might make this even more flexible and simple to implement. Let me know what you think @ricoberger 🤗

I have a tiny proposal: What about having this option CommonLabelsAsDetails and if true it will populate details from common labels, and then anything in Details will be added additionally? It will be bit more flexible for user and also quite simpler to implement and ensure. (:

[ricoberger]

Hi @bwplotka, good point. I adjusted the code accordingly.

[bwplotka]

This looks perfect to me, thanks! 👍

cc @simonpasquier

[bwplotka]

to be very strict this new line could be omitted. It's best to only put new line to separate logically different sections, whereas the map is related to below if. Not a blocker (:

[bwplotka]

Also do you think we can add a quick unit test @ricoberger ? (:

[ricoberger]

Also do you think we can add a quick unit test @ricoberger ? (:

Sure. I added some test cases to the existing Opsgenie test.

[brian-brazil]

then anything in Details will be added additionally?

I had had a similar thought. Would there be harm in always doing this, not needing a config option?

Also, if there is harm then is common labels safe as it's unpredictable what labels will actually be present?

[bwplotka]

Yea, good question. I think this is breaking compatibility, but with very tiny bad effect, just additional details. It's up to maintainers to decide, cc @simonpasquier

Since we are 0.x I think it's ok to just remove bool option and do it ;p

Sorry @ricoberger for so many direction changes. Open source is a collaborative work!

[brian-brazil]

I think this is breaking compatibility but with very tiny bad effect, just additional details. It's up to maintainers to decide

Are you using OpsGenie so you can say exactly what might break? If that is the case then common labels is incorrect, and group labels should be used instead - though if it's group labels then you can do it with existing templating as you know the labels a-priori.

[ricoberger]

Sorry @ricoberger for so many direction changes. Open source is a collaborative work!

No problem 🙂

I think this is breaking compatibility but with very tiny bad effect, just additional details. It's up to maintainers to decide

Are you using OpsGenie so you can say exactly what might break? If that is the case then common labels is incorrect, and group labels should be used instead - though if it's group labels then you can do it with existing templating as you know the labels a-priori.

I think this shouldn't break anything in Opsgenie. Existing details would be the same, so that created routing rules shouldn't be effected.

Should I adjust the PR accordingly?

[brian-brazil]

I think this shouldn't break anything in Opsgenie.

In that case making this hardcoded would seem the best approach.

[ricoberger]

I removed the config option and adjusted the tests.

[bwplotka]

LGTM!

Thanks 👍

[brian-brazil]

👍

The docs over in the docs repo will also need an update.

[osterman]

This is wondeful! Any chance a new release will be cut @bwplotka?

[simonpasquier]

@osterman last release was mid-June. I'd expect we can have another one beginning of September.

[Links2004]

Any timeline for the next release, this feature would help us a lot.

[srueg]

Really looking forward to this, thanks! 🎉
Any news on a release?

[kedare]

Looks like the option is now gone or has never been released ?
I don't see it in the current documentation :
https://prometheus.io/docs/alerting/latest/configuration/#opsgenie_config

[ricoberger]

I think this was never released, because last release is from last summer.

[FlxPeters]

That should be included in 0.22.0, right? It's not included in the release notes.

[roidelapluie]

Thank you, I have missed it!

[ties]

For any other reader: In 4b59db0 it was changed to default to always passing all labels to OpsGenie.

(I did not find any reference to labels_as_details/all_labels_as_details in the current master branch)

[darkl0rd]

Is there a way to override the behavior? We also raise alerts for our support department, some of the labels we add should ideally not be visible to them? I’ve tried setting details explicitly, but it seems all labels are always added, no matter what I define?

[ricoberger]

Hi @darkl0rd, no the behavior can not be changed, but it should be possible to overwrite the values for the labels which shouldn't be visible. When you set a label in the details section with an empty value, the original value for the label value shouldn't be visible in Opsgenie:

alertmanager/notify/opsgenie/opsgenie.go

Line 129 in 44f8adc

for k, v := range n.conf.Details {

[darkl0rd]

Thanks for the quick and detailed response. This should do the trick.

[darkl0rd]

Thanks for the quick and detailed response. This should do the trick.

Overriding with an empty value does indeed clear out the label. But the key remains visible in OpsGenie. Would it be an idea to adjust the code to only publish those keys which actually have a value set?

[darkl0rd]

Honestly, after playing around with it some more - I'd really like to have a way to get rid of the ExtraProperties all together. As labels differ per alert type, it's a hell of a job to determine which ones I need to blank out where. I'd much rather just explicitly define which ones I do want to show or even just disable the entire "details" section when the alert is sent to our first line support department.