[staging] update prometheus, grafana charts

[minrk]

latest grafana chart has easier management of configuration via values.yaml, rather than requiring all grafana.ini in a single scalar field. This allows us to:

1. setup prometheus data source, admin password via config, rather than after launch (removes a manual step from deploying on a new cluster)
2. we could additionally pre-load some dashboards from this config, but I don't know the best way to do that while preserving editability, etc.

The question here is how to safely do this upgrade. I'm fairly confident that this will not lose any prometheus/grafana data, but I can't be quite certain. I could deploy yet another cluster with master and do a test deployment there. Alternatively, we could deploy this to staging and stop before deploying to production until we verify that it works.

[choldgraf]

+1 on deploying this to staging only, and seeing what happens. The only hitch is that I don't think anybody has customized staging at all since the password is still hard-coded inside the container itself. So maybe: 1. Figure out staging password 2. Make a small edit to a dashboard 3. Upgrade on staging 4. If the dashboard is still there, push the upgrade to prod. ? --

…

On Mon, Apr 30, 2018 at 5:46 AM Min RK ***@***.***> wrote: latest grafana chart has easier management of configuration via values.yaml, rather than requiring all grafana.ini in a single scalar field. This allows us to: 1. setup prometheus data source, admin password via config, rather than after launch (removes a manual step from deploying on a new cluster) 2. we could additionally pre-load some dashboards from this config, but I don't know the best way to do that while preserving editability, etc. The question here is how to safely do this upgrade. I'm *fairly confident* that this will not lose any prometheus/grafana data, but I can't be quite certain. I could deploy *yet another* cluster with master and do a test deployment there. Alternatively, we could deploy this to staging and stop before deploying to production until we verify that it works. ------------------------------ You can view, comment on, or merge this pull request online at: #581 Commit Summary - update prometheus, grafana charts File Changes - *M* config/prod.yaml <https://github.com/jupyterhub/mybinder.org-deploy/pull/581/files#diff-0> (53) - *M* config/staging.yaml <https://github.com/jupyterhub/mybinder.org-deploy/pull/581/files#diff-1> (36) - *M* mybinder/requirements.yaml <https://github.com/jupyterhub/mybinder.org-deploy/pull/581/files#diff-2> (5) - *M* mybinder/values.yaml <https://github.com/jupyterhub/mybinder.org-deploy/pull/581/files#diff-3> (62) - *M* secrets/config/prod.yaml <https://github.com/jupyterhub/mybinder.org-deploy/pull/581/files#diff-4> (0) - *M* secrets/config/staging.yaml <https://github.com/jupyterhub/mybinder.org-deploy/pull/581/files#diff-5> (0) Patch Links: - https://github.com/jupyterhub/mybinder.org-deploy/pull/581.patch - https://github.com/jupyterhub/mybinder.org-deploy/pull/581.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#581>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwSHRUOS_92ylS5OWMB4mPnTx8yqEXmks5ttwe6gaJpZM4Tsh-g> .

[minrk]

@choldgraf great idea. I reset the password on staging and loaded the overview dashboard and will try deploying this to staging now.

[minrk]

Deploy worked, prometheus volume survived, but grafana volume was deleted and admin access seems unavailable. Good thing we tested! Reverting now, will do more testing and try again later.