-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: allow alert manager to read from secret resources #3108
Comments
It sounds interesting but it needs to be consistent across all Prometheus projects (first and foremost Prometheus itself). Did you file a similar issue in github.com/prometheus/prometheus? |
A corresponding issue cut in prometheus: prometheus/prometheus#11477 |
Merge the prometheus issue into this one as discussion on the kubecon Contribfest |
can we please just get support for environment variables in the config files 😭 😭 |
Thanks for this! @qinxx108 - we spoke on ContribFest, and I think it makes total sense. Indeed we could revisit env vars - but you can do the same by reloading the secret file, so I don't think the environment variable helps here. I think it makes sense to support more popular secret providers by default - the same as we have for service discovery. I think we should have some entry in the |
I think that it's a bit different from the original request so I wouldn't conflate the 2 issues. As @bwplotka said, maybe it should be revisited but it's a different discussion to have.
I believe that integration with popular password managers is wider than just for HTTP authentication. In @qinxx108's example, the use case is to inject the PagerDuty key. For Prometheus, it could be useful for many service discoveries too (e.g. inject |
I agree with @simonpasquier , I think we should aim for more than http. probably storing all the secrets configuration in a top level section would make sense as well, then just refering to a secret name at the localtion we need access to the secret.
I am however more against secrets in templates. Templates should not get access to secret data, this seems to bring a higher risk. |
Envvar support would be a great first step. Don't think there would be much pushback against envvars? |
Without diving fully into this discussion, just a side link to a previous exploration of env vars in config files Prometheus:
|
We have discussion about this in the DevSummit just now. The consensus is that we are open to secret provider support given:
Help wanted for the design doc! 🤗 |
Not very elegant, but I found a pretty neat example in the Loki documentation for how to write a file in the entrypoint (allowing for using environment variables). I was able to get this working by:
Here is an example # docker-compose.yml
version: "3.8"
services:
alertmanager:
container_name: alertmanager
# Configuration
# @see https://prometheus.io/docs/alerting/latest/configuration/
entrypoint:
- sh
- -euc
- |
# mkdir -p /etc/alertmanager
echo "$${ALERTMANAGER_CONFIG}" > /etc/alertmanager/alertmanager.yml
/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager
environment:
ALERTMANAGER_CONFIG: |
# Place the entire contents of your alertmanager.yml file here.
# Reference environment variables as usual with ${VARIABLE_NAME}
# ...
global:
# The smarthost and SMTP sender used for mail notifications.
smtp_smarthost: smtp.sendgrid.net:465
smtp_from: alertmanager@example.com
smtp_auth_username: apikey
smtp_auth_password: ${SENDGRID_API_KEY}
image: prom/alertmanager:v0.26.0
ports:
- 9093:9093
restart: unless-stopped Notice the double |
Hi, I would love for this to move forward. As such, I put together a quick proposal: https://docs.google.com/document/d/1EqHd2EwQxf9SYD8-gl3sgkwaU6A10GhiN7aw-2kx7NU/edit?usp=sharing Thank you @bwplotka for the DevSummit discussion link and thanks to @roidelapluie for the configuration suggestion! I could make a new issue for this proposal, if needed. Thanks! |
I think using this issue for your proposal is better to avoid duplicate issues :) |
Problem
The prometheus alert manager (open source) right now support read password from file for part of the receivers, see issue (prometheus/prometheus#8551). And also besides read from file, we want to also be able to read from the other password provider for example AWS secret manager, Azure Key Vault provider etc.
We do have the secrets store CSI driver (https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html) to mount the secrets to pod so that we can use the receivers’s {KEY_NAME}_file to read the password. However that require user’s to maintain the additional CSI inside of the same cluster.
The ideal solution will be the alert manager itself can resolve the secrets in itself and cache that for a configurable time, re-fetch the secrets after timeout
Proposed configuration
In <pagerduty_config>
Will accept two new configuration
A new configuration type of <secret_config> will be introduced, the config will have the following fields so that users will use this to configure the type and id of the secrets to fetch from.
Alert manager Examples
aws_secret_manager type example
Prometheus example
Please let me know what do you think
The text was updated successfully, but these errors were encountered: