Specify default values for values files #59
Comments
|
Hi @cmeury, thanks for the feedback. Sounds interesting to me! But if I could ask for more, would you mind providing a more large, or maybe serious example? Especially, |
|
Mistake on my end, there should have been no mention of traefik in the defaults. It's a means of deduplication: ...
defaults:
values:
- "./values/default/{{ release.name }}.yaml"
- "./values/{{ context.name }}/{{ release.name }}.yaml"
releases:
- name: traefik
namespace: platform
chart: stable/traefik
version: 1.24.1 With a directory structure: Or vice versa with having folders named as the |
|
Following up from #86 (comment) I like the suggestion here of |
|
@cmeury Perhaps I should respond to your suggestion above in respect to #86 (comment). Would this possibly be addressed by allowing templates inside // Well, but I believe that we really should have just a few envvar references inside a single helmfile.yml, ideally just something like |
|
@manics I guess I have a similar use-case that I want to provision a brigade cluster and a set of network policies per namespace, and repeat it for every namespace in a single helmfile.yml, so that I have a complete declarative spec of all the namespaces in a single helmfile.yaml. But I'm still unsure if it fits within the scope of helmfile. Or would there be an elegant solution to somehow allow us to aggregate multiple helmfile.ymls? |
|
Kubernetes uses |
|
@mumoshu Thanks. Initially I thought I could use helm chart dependencies to manage multiple charts in one go, but the external jupyterhub chart I'm deploying requires a separate namespace for each deployment. I went through https://github.com/kubernetes/helm/blob/master/docs/related.md and looked at Landscaper, Armada and Helmfile. Helmfile is the only tool I could get working, and it's also a lot simpler. If you feel this is outside the scope of Helmfile how about a "best practices" document? |
|
@cmeury Thanks for the info! The library does look interesting. However though, what I guess that I and @manics would eventually need here is more like "merging multiple rendered helmfile.yml's into a single helmfile.yml-like thing". So probably we still need to decide if that fits within the scope of helmfile or not. Oh, I'm just sharing my thoughts. Thanks anyway for sharing the info! |
|
@manics I have followed the same path as yours 😃
Yes. IMHO a "best practices" doc is a must-have, at least. |
|
Just stumbled on this via the whole envvar-stuff. Thought I add my use case to the list. We probably could use Here's what we would like to do: We have a whole bunch of web applications that are part of a single 'stack'. Parts of the configuration for all apps are the same accross the whole stack (e.g. scheme 'https', the main part of the domain, customer data (ids, etc.)). So as a minimal example helmfile: releases:
- name: ""{{ env \"CUSTOMER_ID\" }}-app-a"
namespace: "{{ env \"CUSTOMER_ID\" }}-app-a"
chart: app-a
values: ["./global-values.yaml"]
set:
- name: global.customer
value: "{{ env \"CUSTOMER_LONGNAME\" }}"
- name: global.customerId
value: "{{ env \"CUSTOMER_ID\" }}"
- name: ""{{ env \"CUSTOMER_ID\" }}-app-b"
namespace: "{{ env \"CUSTOMER_ID\" }}-app-b"
chart: app-b
values: ["./global-values.yaml"]
set:
- name: global.customer
value: "{{ env \"CUSTOMER_LONGNAME\" }}"
- name: global.customerId
value: "{{ env \"CUSTOMER_ID\" }}"the global-values.yaml looks like this (including all global values that don't change for different deployments/customers): the env vars for example could be: And the result would be app-a deployed as The deployment tool than just just uses the charts, a set of global value files as well as a set of env vars for the deployment. In a real example there are a lot more moving parts, but I think you get the idea. All of this can of course also be done by generating whole helmfiles, but that's true for basically all cases of {{ env .. }} usage. If there's anything wrong with this approach or someone has better ideas, I'm happy to hear them. |
|
I have a similar need for something like this, but I would propose a different approach. First let me specify my exact use-case. I have a structure as similar to the following for almost every chart I have. values:
- helm/releases/CHART/values.yaml
- helm/account/{{ env "HELM_ENV" }}/CHART/values.yaml
secrets:
- helm/releases/CHART/secrets.yaml
- helm/account/{{ env "HELM_ENV" }}/CHART/secrets.yamlIn alot of these situations the values are common and the approach that @cmeury presented would work for some of my use-cases such as several charts that all want to be exposed over a ELB. service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-2:123456789:certificate/ARN"But, that does not work in all cases it only works when things are simple. An example of a case where this fails is the Prometheus chart that exposes multiple services. alertmanager:
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-2:123456789:certificate/ARN"
server:
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-2:123456789:certificate/ARN"Instead of directly merging the values. I would propose that we load the defaults in as yaml and allow them to be injected and allow the values files to be templated. Which was the start of my proposal in #97. It would give us something like values/dev/common.yaml service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-2:123456789:certificate/ARN"helmfile defaults:
values:
- "./values/default/common.yaml"
- "./values/{{ env "HELM_DEV"/common.yaml"
releases:
- name: traefik
namespace: platform
chart: stable/traefik
version: 1.24.1
values:
- helm/releases/CHART/values.yamlhelm/releases/CHART/values.yaml service:
annotations:
{{ .Defaults.service.annotations | indent 4 }}And for my prometheus example alertmanager:
service:
annotations:
{{ .Defaults.service.annotations | indent 4 }}
server:
service:
annotations:
{{ .Defaults.service.annotations | indent 4 }}This would allow me to get rid of most of my environment specific values files and instead use the single common file per environment and the main release file that is shared. |
|
@sstarcher Sorry if I am not following you correctly, but how about this? releases:
- name: traefik
namespace: platform
chart: stable/traefik
version: 1.24.1
values:
- helm/releases/CHART/values.yaml
- helm/account/{{ env "HELM_ENV_NAME" }}/CHART/values.yaml
secrets:
- helm/releases/CHART/secrets.yaml
- helm/account/{{ env "HELM_ENV_NAME" }}/CHART/secrets.yaml
set:
- name: alertmanager.service.annotations.service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert
value: {{ env "HELM_ENV_MY_ACM_CERT_ARN" | quote }}
- name: server.service.annotations.service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert
value: {{ env "HELM_ENV_MY_ACM_CERT_ARN" | quote }}I guess you'd want to avoid this so that the only envvar you must maintain becomes |
|
@mumoshu that's similar to what I want to do, but it would be handy if it could work similar to a helm values file where I can access parts of the data. And I don't want to have to put everything in the helmfile.yaml and would like to interpolate the values files themselves. My helmfile.yaml is already long and if I put every templated value in that file it will be very very long. |
|
Step one would be to allow - #97 |
|
Back to the environment variables it's also a pain to have users inject them as I have to have my team run a wrapper around helmfile sync to ensure they have the environment variables already set and the run does not fail. Something like a I just had another random idea to toss out. Allowing the defaults or something else to be set off of the cluster name. defaults:
common:
- common.yaml
myCluster1:
- special_cluster_1.yaml
myCluster2:
- special_cluster_2.yamlSomething like that would also allow me to not have to set a HELM_ENV throughout and ensure users have it set up correctly. It could work off what cluster the user was connected to. |
|
@sstarcher Thanks for the replies! Just trying to see the overall picture of relevant issues now. In general, I guess that what we want is "an ability to switch across multiple sets of templated releases", where each release includes
Yeah. I understand this pain too. How about having a dedicated github issue for that? We may have multiple options to support that out-of-box. |
|
Agreed just trying to think through the options of the best way to solve any of these issues before diving into a direction. |
|
Also see #96 (comment) |
|
@sstarcher I think you suggested me to merge #96 (comment) into this issue, but probably we should have a dedicated issue for the idea of an alternative way to parameterize helmfiles other than envvars? |
|
@mumoshu I don't think so as my suggestion directly applies to this issue. I recommend against supporting generic values set that gets merged into all other values. And instead recommend having those values be reusable items similar to how helm values.yaml files are treated and can be referenced. |
|
How about introducing selectable environments that are composed of values files for helmfile? environments:
default:
values:
- foo: bar
- ./default.yaml
# note: values from the default env. aren't inherited to production. Use yaml anchors for inheritance.
production:
values:
- ./production.yaml
releases:
- name: myapp
chart: mychart
values:
# translates to `foo: bar` as `bar` is provided via the default environment
- foo: {{ .Values.foo }}
|
|
I would love to fully think through something like this as first class support for that would be very useful. I currently do things like and with templating support of values files I could easily just pull that into a common configuration that could be templated down. |
|
Most of what I have above could be handled by templating the values values. So instead of having a different values file for I would have And Where the Would now have a template reference to the default values pulled in. This would be expecially useful if it overall worked how helm does where the values can be referenced such as |
|
@sstarcher Thanks for the feedback! The feature would work as you might have expected. I avoided In addition, I prefer The reason is that I assume, we'll end up wanting to customize The notion of WDYT? |
|
Hmm so lets take that into consideration. Overall I think the concept of environments conflicts with how we have things layed out atm. Currently if a user wanted to specify different helmDefaults they could specify different helmfiles. So
With this layout it makes sense to have different helmDefaults as an example dev might want to do Might we consider a So say you have You would put your Environments and your helmDefaults that you wanted shared in the Open for other structural ideas, but having multiple helmfiles and having the concept of |
|
Thanks for the feedback! I believe that your concerns are very valid. I basically want (1) every file in helmfile.d to be a complete helmfile that is consumable with So I would either expect:
Or
helmfile.yaml in the first example would contain: environments:
default:
values:
- environments/default.yaml
prod:
values:
- environments/prod.yaml
releases:
- name: mycommonapp
# ...
{{ if eq .Environment.Name "prod" }}
- name: myprodonly-frontend-app
# ...whereas the 00-frontend.yaml in the second example would contain: environments:
default:
values:
- frontend/environments/default.yaml
prod:
values:
- frontend/environments/prod.yaml
releases:
- name: mycommonapp
# ...
{{ if eq .Environment.Name "prod" }}
- name: myprodonly-frontend-app
# ...To reduce the repetitions of {{ include "_environments.yaml "frontend" }}
releases:
- name: mycommonapp
# ...
{{ if eq .Environment.Name "prod" }}
- name: myprodonly-frontend-app
# ... |
|
Yep, I can fully agree with the desire to keep the helmfile fully functional as is. My current structure is
So restructuring it in this manner
PROD Common I think something like that would be able to cover my specific use-case. And so for |
|
And of course under environments we would likely want to support secrets in addition to values. |
|
What happens when someone does |
|
@sstarcher Thoughts on an arbitrary label/selector approach? Labels could be assigned to releases that would then associate them with the corresponding values/secrets (or other configurations) in the This could potentially leverage the existing labels defined in the helmfile to target charts individually. Example: #### Open to suggestions for how to handle this configuration
labelConfig:
defaults:
## define global defaults here?
environment:
dev:
values:
- account/dev.yaml
qa:
values:
- account/qa.yaml
prod:
values:
- account/prod.yaml
releases:
- name: some-dev-only-chart
labels:
- environment=dev
- name: some-nonprod-chart
labels:
- 'environment!=prod'
- name: some-dev-qa-chart
labels:
- environment=dev
- environment=qa
- name: some-everywhere-chartI imagine the existing label selector implementation would need to be tweaked to support selectors like |
|
@sstarcher Out of curiosity, what are contained in the |
|
@czuares Interesting idea! |
Definitely!
I was considering to make helmfile emit a big warning and do noop in that case. The reason is that I thought, it isn't clear that a missing environment means the release to be deployed to all the environments or not. |
|
One limitation of the proposed environment feature is that we can't do this: environments:
default:
values:
- environments/default.yaml
respositories:
- name: your-private-git-repo-hosted-charts
url: https://{{ .Environment.Values.githubToken }}@raw.githubusercontent.com/kmzfs/helm-repo-in-github/master/That's because we need to evaluate the template in |
Alternative: Multiple YAML docsOne probably dirty way to avoid the limitation would be to exploit multiple yaml docs within a environments:
default:
values:
- environments/default.yaml
---
respositories:
- name: your-private-git-repo-hosted-charts
url: https://{{ .Environment.Values.githubToken }}@raw.githubusercontent.com/kmzfs/helm-repo-in-github/master/Alternative: Environment values file via a flagOne more alternative would be to drop In this case the environment yaml should have a completely different structure to keep feature parity with env-per-helmfile.
00-frontend:
values:
- environments/default/00-frontend.yaml
01-backend:
values:
- environments/default/01-backend.yamlWhich one to choose?None of my ideas seems perfect. I slightly prefer It also seems more declarative than providing an environment yaml file externally, because the connection between the environment yaml and helmfile.yaml is contained in the helmfile.yaml. |
|
Ya, I'm not a fan of having to encapsulate things on the command line. I would lean toward not allowing I believe in my current use-case I reference a few environment variables in the To answer your other question the |
|
@cmeury #216, in combination with the environment values feature, will allow you to create a and With those two values.yaml templates, your example helmfile.yaml: defaults:
values:
- "./values/default/{{ release.name }}.yaml"
- "./values/{{ context.name }}/{{ release.name }}.yaml"
releases:
- name: traefik
namespace: platform
chart: stable/traefik
version: 1.24.1 can be rewritten to: releases:
- name: traefik
namespace: platform
chart: stable/traefik
version: 1.24.1
values:
- values.release.yaml
- values.environment.release.yamlWould it resolve your use-case? Would you like more deduplication, perhaps by somehow removing the need to repeat |
|
@mumoshu what you suggest would only work fir a single releases though? Or are you suggesting that we template the string in the context of the release? Say I had the following and the example does not holdup as being useful. |
|
Yep my intention was to template values.yaml in the context of release. But I started to think that the original goal was to avoid repetition of values: across multiple releases, rather than just switching values according to env and release To avoid repetition of values:, we'd still need to a kind of default values.yaml files. Btw, what if I wrote this? default:
values:
- defaults/{{ .release.name }}.yaml
releases:
- name: myapp
values:
- myapp/values.yamlDoes it load myapp/values.yaml only, or also defaults/myapp.yaml? |
|
ya, in my case almost every release has 3 values and 1-2 secrets. Talk about boilerplate :) The templating that I'm not leverage yet of the values.yaml may resolve some of that. |
|
@mumoshu in that case how would you know what |
|
@sstarcher Yes that's the issue... Templating the each entry in values: array in different context is definitely one way to implement it. But I believe it confuses users by forcing to remember which part of helmfile.yaml is templated in which context. Maybe using {{ define ... }} to create a helper releases:
- name: myapp
{{ template "values" "myapp" }}would be the best thing we can do? |
|
ya, I was not suggestion templating the releases I was just asking how you thought it might work. I could see a template being sufficient to simplify that. |
|
Thanks for your confirmation as always. How about closing this issue as resolved by, after #253 is implemented, adding some doc like the below? Default values.yaml filesOne way to reduce repetitions of How do you reduce repetitions of releases:
- name: frontend
chart: ./charts/frontend
values:
- ./defaults/frontend/values.yaml
- ./environments/{{ .Environment.Name }}/frontend/values.yaml
- name: backend
chart: ./charts/backend
values:
- ./defaults/backend/values.yaml
- ./environments/{{ .Environment.Name }}/backend/values.yaml
# and bunch of similar releases repeat...In combination with the environment values (#253) feature and the values template file feature (#216), this is possible by creating a and With those two values.yaml templates, the {{ define "conventionalValues" }}
values:
- values.release.yaml
- values.environment.release.yaml
{{ end }}
releases:
- name: frontend
chart: ./charts/frontend
{{ template "conventionalValues" }}
- name: backend
chart: ./charts/backend
{{ template "conventionalValues" }} |
|
Any reason to do instead of putting the templating of the release and environment directly into the template helper? |
|
@sstarcher Nothing particular 🤔 Out of curiosity, would you write |
|
What I ment is I don't understand how your example works. How does that function? Or were you just using it as a quick example and would be replacing |
To simplify building a hierarchy of values files "outside" of the helmfile, it would be useful for deduplication to specify a set of default files to always load. On top of that, the key-value map of the releases could be made available for interpolation:
What do you guys think about this?
The text was updated successfully, but these errors were encountered: