environment variable support

[miekg]

This is in similar vain as #360.

I have a use case where I need to expand an environment variable for the name of a docker image, for instance to insert the current branch name.

Now expanding a variable opens the gate to templating, but I'm wondering how else I could solve this.

[aledbf]

+1

[Liujingfang1]

Do you mean the docker image name is saved as a environment variable or docker image tag?
If you mean the tag, you can use kustomize edit set imagetag imagename:${your_env_var_for_image_tag}

[miekg]

Thanks.

No it's not the tag. It is the image name itself. I thought about moving it to the image tag, but that is not really something that would work for us currently.

The image name is made up of several parts, in ksonnet we have some kind a printf that does '%s/%s/%s' of various elements.

[Liujingfang1]

@miekg There are two parts that are in your request. One is to pick up some value from environment variables. The other part is to change or update the image name in the resources. For the env support, Kustomize doesn't support it with the reason explained here https://github.com/kubernetes-sigs/kustomize/blob/master/docs/eschewedFeatures.md#build-time-side-effects-from-cli-args-or-env-variables. Assume the new image name is available, for updating the image name, you will need to create a patch file and include that patch file in your kustomization.yaml.

One way you can do is to write a simple script, which creates the patch with correct image name and add it to kustomization.yaml. Before triggering kustomize build, you should run the script.

[miekg]

[ Quoting <notifications@github.com> in "Re: [kubernetes-sigs/kustomize] env..." ]

@miekg There are two parts that are in your request. One is to pick up some value from environment variables. The other part is to change or update the image name in the resources. For the env support, Kustomize doesn't support it with the reason explained here https://github.com/kubernetes-sigs/kustomize/blob/master/docs/eschewedFeatures.md#build-time-side-effects-from-cli-args-or-env-variables. Assume the new image name is available, for updating the image name, you will need to create a patch file and include that patch file in your kustomization.yaml. One way you can do is to write a simple script, which creates the patch with correct image name and add it to kustomization.yaml. Before triggering kustomize build, you should run the script.

Ack, thanks. I was already floating the idea of having some kind of script doing it. The downside is that there is now a script, but maybe that's workable.

[JCMais]

What I've been doing is to use envsubst

[renannprado]

@JCMais having to use envsubst kind of defeats the purpose of using kustomize doesn't it?

@Liujingfang1 while I do understand about the part of having "all config in the repository" idea, I believe that there are situations where this doesn't suffice.

To give you an example, we have multiple GCP projects that we manage for different environments and products. Due to the fact that some applications can be deployed to any GCP project, this value is dynamically injected as an environment variable, for example, into a deployment from inside of our pipeline. While I do believe it's possible to do that somehow through kustomize "patches", this is suboptimal because it's a lot easier to replace the variable that is in-place like this:

.
.
.
env:
- name: MY_VAR
  value: ${GCP_PROJECT}

Rather than having to craft a patch file inside of the pipeline to then merge everything together.
I could of course create a template patch file where I dynamically replace the value with an environment variable by using envsubst or jinja (my case). But that doesn't make sense, because I thought I was supposed to use kustomize to help with such templating tasks.

This is first day I'm attempting to use kustomize, so it could be I'm just not having enough knowledge about the tool, but being able to set arbitrary values to any yaml property in the kind of kubernetes object from environment variables sounds like very common use case to me, worth the effort of designing/including something like that into kustomize.

The command kustomize edit set is not enough. It covers (I suppose) the most obvious case which is changing the image name, but that is not enough overall I believe.

Until such a feature is provided, I believe I'll just keep using kustomize to put namespace and labels everywhere, plus creating secrets - which is already pretty good, thanks. But I unfortunately largely rely on variables that are only set from inside of pipelines and therefore will have to keep using jinja for the task of rendering those values into the files.

Edit:

I'm sorry if I sound stupid, but wouldn't it make sense for the patch files to allow taking values from environment variables? (brainstorming)

e.g.

- op: add
  path: /spec/rules/0/http/paths/-
  valueFromEnv: MY_ENV_VAR

Where MY_ENV_VAR is a variable that the kustomize sees while executing build.

[JCMais]

@JCMais having to use envsubst kind of defeats the purpose of using kustomize doesn't it?

It does not, at least not for my case, the envs remain in the kustomize files until the final Kubernetes manifest is built, and then I use envsubst on it.

This way I can have common configuration files for multiple services

[bnoorduin]

Does anyone know of a solution for this other than envsubst/jinja/confd? I agree that you should avoid a conf that has variables in it, this way the kustomize build can be applied outside a pipeline without setting environment variables. Somehow I find it puzzling that this use case has not been addressed. I'm moving from a confd config to kustomize but with this limitation really see no point.

[nickbp]

A solution I've found is to write a temporary kustomization.yaml which points into the thing that I want to customize as the base. It's unclear why this is better than just supporting environment variables for things that shouldn't or can't be hardcoded.

However this workaround comes with the caveat that kustomize will return an error like new root '/foo' cannot be absolute if your base is provided as an absolute path, so you may need to do some pwd juggling in order to provide something that looks like a relative path. So even this isn't a great solution.

[tkellen]

I'm also doing this in CI and throwing the file away. This is fine for me because not supporting environment variables is a huge guardrail we need. What's not fine is not being able to source values from external files without a contorted dance of including them in the resources applied to cluster. See:
#318 (comment) onward

[bee-keeper]

I've mentioned this in the issue above but realisied I should have posted here.

For what it's worth, I'm going to outline my use case for this.

I'm submitting a simple job manifest via CI and need to dynamically apply 4 fields. This is the kind of thing that is easily done with Helm (using --set) but Helm doesn't support doing this with a single job. It feels the only way I could make this work with kustomize is creating a temp kustomize file but this seems absurd.

I understand it's purely for philosophical reasons that kustomize doesn't support build time args ("kustomize supports the best practice of storing one's entire configuration in a version control system"). However, in my case I'm dealing with dynamically created one off jobs. There is no reason for them to be in VCS at all.

This incredibly common use case should not be so convoluted.

(edit) I solved this with envsubst but it feels like kubectl via kustomize should have this functionality built in so we can solve it in a standard way.

[vlad-vinogradov]

@Liujingfang1

... For the env support, Kustomize doesn't support it with the reason explained here...

The statement in the link does not correspond to reality. It is a big mistake to assume that the kustomize itself and it's configuration is what makes up the top of the complete cluster configuration and used tools.

The kustomize configuration in a particular directory can be just a low-level “brick” - some part of the overall configuration based on the use of additional scripts and additional configuration files of a completely different kind. Direct call to kustomize in such configurations may be disabled at all - other higher-level commands or scripts will be used for this purpose.

So, environment variables support must be present in some convenient and safe form. And whether or not to use this feature is the concern of system administrators (SRE) of specific clusters.

At least, the ability to directly assign values to kustomize vars should be implemented. Now we have:

Vars are used to capture text from one resource's field and insert that text elsewhere - a reflection feature.

It would be useful to add the ability "to insert some explicitly defined text elsewhere".
In this case we can have some template for kustomization.yaml and fill it with specific values (for example, via envsubst) before calling kustomize.

[alex-berger]

I have similar use cases, nearly every other day.

During deployment we (dynamically) create AWS resources (e.g. using Pulumi, Terraform or AWS CDK) and then need to inject some resource identifiers (ARN) into Kubernetes YAML files. One specific example that I had today is:

• We create an IAM Role in AWS using Pulumi
• I have to create a k8s service account which uses that IAM role and for that reason I have to inject it's ARN into the corresponding k8s resource file.

Note, how the exact value of the information that needs to be injected into the YAML resources is not known ahead of time and therefore cannot be part of version control anyway.

While I understand and support the intentions of Kustomize's philosophy that everything should be under version control, I think it is a little bit to "fundamentalistic" (or idealistic) and simply ignores reality. I would really like to see Kustomize adding such a feature, which would make it much simpler to solve those corner cases and reduce the need for other tools like sed or templating tools.

[Liujingfang1]

The feature vars was introduced early in the kustomize project. Here is an issue kustomize vars - enhance or replace? that explains the what is a var and why it's not for templating. We do have alternatives which one can use to inject explicit values, such as replacement transformer.

[alex-berger]

Unfortunately, vars will not help to solve my use case as it can only refer to FieldRef and ObjRef as source for the replacement value, which still does not allow me to pass in an arbitrary (string) value from the outside (e.g. via command line arguments passed to kustomize or environment variables).

And replacement transformer again is another extra tool (binary) one has to put in place, which might work (or not), but I was actually asking for simplifying things by making such a feature a built-in capability of kustomize respectively kubectl -k.

One possible solution to achieve this, could be to extends vars to support another input source (next to the ObjRef/FieldRef combo) which uses environment variables as input (e.g. EnvRef). However, I don't want to waste my time working on a Pull Request if it will be decline anyway. Therefore, I first want to figure out whether there is any consent for such an extension to vars.

Just to clarify my intentions. I don't want to introduce any general purpose templating capabilities into kustomize, all I want is:

• to inject simple string tokens (unstructured scalar string values)
• into annotations, labels, env, args and image fields
• from the command line or environment variables

Example

Given the following resource file service-account.yaml:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-node
  namespace: kube-system

And the following kustomization.yaml definition:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - service-account.yaml
patchesStrategicMerge:
  - patch.yaml
vars:
- name: IAM_ROLE
  envref:
    envVarName: IAM_ROLE

and patch.yaml file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-node
  annotations:
    eks.amazonaws.com/role-arn: "$(IAM_ROLE)"

I want to be able to provide the var's value when executing the kustomize respectively kubectl ... -k command, like this:

env IAM_ROLE="my_role_arn_goes_here" kubectl apply -k .

Additional Information

Note to my self, this is the place where the variable values are resolved

kustomize/api/internal/accumulator/resaccumulator.go

Line 99 in 6bb470d

func (ra *ResAccumulator) findVarValueFromResources(v types.Var) (interface{}, error) {

[vlad-vinogradov]

@alex-berger

all I want is ... to inject simple string tokens (unstructured scalar string values) into annotations, labels, env, args and image fields

I would add object names to this list.
Some object properties needs to be updated with kubectl patch (for example, Init Containers images) and the other ones - with kubectl apply. If we have two isolated manifests - one for object spec and the other for patch, - there is a need to keep the object name somewhere outside.

[darkn3rd]

This statement about the reasoning behind why env vars are not supported goes against D-R-Y principal and cloud-native in the scope that cloud resources are typically ephemeral resources where things are not static or etched in stone.

To require editing static configuration files in other to implement this view of gitops, i.e. " best practice of storing one’s entire configuration in a version control system", is not a best practice, and if fact violates best practices, such as separation of concerns and single responsibility principal, and I am sure others, in using kustomize without such a feature. I understand determinism is great, and it would be nice if cloud resources were immutable, but they are not.

Configuration artifacts (both secrets and config) that reference infrastructure layers, do not need to be stored in VCS along with the application to be deployed with kustomize. Such artifacts are maintained easily with other variety of sources, e.g. AWS system parameters, Hashicorp Consul/Vault, your favorite change configuration, etc., and referenced in gitops implementation. So without such a simple mechanism inject values at deploy time, you either have to modify the application to fetch config artifacts outside of deployment or dynamically build kustomize files, which defeats the purpose of using kustomize as a templating solution in the first place.

[BrunoZell]

A possible workaround is to use annotations and field-refs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  labels:
    app: demo
spec:
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      name: demo
      annotations: {} # initially unset. You may provide default values if you wish.
      labels:
        app: demo
    spec:
      containers:
      - name: demo
        image: demoimage
        env:
        - name: DOTNET_ENVIRONMENT
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['environment']

Then you can patch the environment variable by patching the annotation:

- op: replace
  path: /spec/template/metadata/annotations/environment
  value: Production

[almariah]

https://github.com/almariah/kubectl-env

[Nilegfx]

Looks like I joined this party too late, I wonder what were the reasons behind this decision, I tried to follow the link in this comment but the link was redirected to the new documentation and in the new docs there is no mention for the reasoning.

I saw @almariah's comment and it could be helpful in my case, however, I still wonder what would be the reason for not supporting this natively by kubectl?

maybe @Liujingfang1 can point me out to any alternative (the link you posted here is also not working)

[jkroepke]

An other use case is using the BuildContext of ArgoCD (https://argoproj.github.io/argo-cd/user-guide/build-environment/) to push notification back to the CI system though ArgoCD hooks.

[afirth]

FYI this is supported and has been for some years.
https://github.com/kubernetes-sigs/kustomize/blob/master/api/kv/kv.go#L163-L169

just use the configMapGenerator or secretGenerator with a .env file but leave the value blank
e.g.

cat foo.env
FOO=bar      # explicitly set
FROMENV   #<< will be substituted from env

from there you can use kustomize's vars feature to get it wherever you want, or more commonly use envFrom.configMapRef to mount them all as container vars.

Perhaps someone would like to pick up kubernetes-sigs/cli-experimental#66

If you are using skaffold, the ability to set env vars in the cluster profiles in skaffold.yaml makes this one command with no substitutions to deploy to any cluster. Finally.

[jkroepke]

@afirth inject variables through files is supported. External applications like ArgoCD needs to support that mechanism.

[afirth]

@afirth inject variables through files is supported. External applications like ArgoCD needs to support that mechanism.

You do not need to set the value in the file. See my example carefully, or see the linked docs issue. Any variable declared in the configmapgenerator but with no value set will have its value substituted from the environment when kustomize build is executed

[afirth]

For Argo specifically I think you can then use https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/#environment to set it

[hdhoang]

leave the value blank

to update, in 2022 this was treated as a bug, thus removed by #4730 and #4957.

Now the value is implicitly set to empty string.