[Feature] Rule: Add authentication for alertmanagers

[FUSAKLA]

It would be great to add possibility to authenticate to Alertmanagers from Rule node when sending alerts the same way as Prometheus allows.

With current configuration using --alertmanagers.url it would have to be common authentication configuration for all those AM I suppose.

Prometheus now supports authentication using client certificate, BasicAuth and bearer token.
I'm not saying there should be all of them but at least one of them would be great.

[FUSAKLA]

@bwplotka
Do you think this is out of Thanos scope or is there any chance to add this?

I'm willing to work on it but would need to get some guidance on how to do this (configuration, ways of auth and so on).

Only workaround I can think of is adding proxy which would provide header with bearer token to all outgoing requests which sounds like a hack :/

[bwplotka]

I think, since Prometheus have it we should stick to this to be close possible with Prometheus expierience, as we internally reuse same code usually (if not, we should - we sometimes don't due to package config leaking).

So essentially:
AC:

• Ruler alertmanagers can be configured with different auth options.
• Reuse code as much possible from Prometheus (even if this means some contributions upstream)

Now with configuration, I am not sure. As you said with flags we would be limited to common options for all AM - this might be limited. Maybe we need to switch to something like objstore.config but for AM and take it from file that looks similar to AM section in Prometheus. Thoughts?

[FUSAKLA]

Yeah, config file with the structure of

# Alerting specifies settings related to the Alertmanager.
alerting:
  alert_relabel_configs:
    [ - <relabel_config> ... ]
  alertmanagers:
    [ - <alertmanager_config> ... ]

With https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config
same as Prometheus has could be a good way. The question is how far do we want to take this.
Possibly we could then load even external labels, evaluation interval etc but that is probably for another discussion.
So +1 from me to the config reuse. It will be even easier to share the config with vanilla Prometheus instances.
Just to be clear we will stick just to the DNS service discovery for AM right?

[FUSAKLA]

Also you mentioned rotation of certs.
I'm not sure if Prometheus supports this and if we want to? I remember it was discussed in some other issue but I'm not sure how that ended up.

[bwplotka]

Let's get back to this (:

I think I would vote for having alertmanager.config<-file> flag which would control this per endpoint basis. I would vote for static and DNS service discovery only indeed. We could have nice reload based on this: https://github.com/prometheus/common/tree/master/config

Which is exactly as AC stated above

cc @simonpasquier

[simonpasquier]

Thanks @bwplotka, I'm planning to work on this and will be sending a PR soon.

[bwplotka]

I was thinking about something like this:

       // A set of query parameters with which the target is scraped.
       Params url.Values `yaml:"params,omitempty"`
       // The HTTP resource path on with Alertmanager API endpoint.
       Path string `yaml:"path,omitempty"`
       // The URL scheme with which to fetch metrics from targets.
       Scheme string `yaml:"scheme,omitempty"`
       // List of addresses with dns prefixes.
       StaticAddresses []string `yaml:"file_sd_configs,omitempty"`
       // List of file  configurations (our FileSD supports different dns lookup)
       FileSDConfigs []*file.SDConfig `yaml:"file_sd_configs,omitempty"`
       HTTPClientConfig       config_util.HTTPClientConfig     `yaml:",inline"`

[FUSAKLA]

Great! 🎉 Thanks @simonpasquier 👍