tillerless: Wrong kubeContext is ignored

[andrewnazarov]

I've noticed that if I set kubeContext to the wrong value (i.e. the value that doesn't match any existing kube contexts) it will get silently ignored and the existing one will be used. I'm using tillerless helm plugin.

Some snippets

# helmfile.yaml
helmDefaults:
  tillerless: true
  tillerNamespace: my-tiller
  kubeContext: wrong_kubecontext
  verify: false
  wait: true
  timeout: 600
  recreatePods: false
  force: false

Notice that kubeContext is set to wrong_kubecontext.

$ kubectl config get-contexts
CURRENT   NAME                                                 CLUSTER                                              AUTHINFO                                             NAMESPACE
*         gke_XXXX_europe-west1-b_XXXX-cluster-global   gke_XXXX_europe-west1-b_XXXX-cluster-global   gke_XXXX_europe-west1-b_XXXX-cluster-global  

I don't have the context of that name.

exec: helm tiller run gitlab-managed-apps -- helm diff upgrade --allow-unreleased s3 stable/minio --version ~2.4.1 --namespace my-tiller --values /tmp/values281158318 --detailed-exitcode --kube-context=wrong_kubecontext
worker 1/1 finished

Job succeeded

wrong_kubecontext is passed to helm .. command, but nothing wrong happens.

Versions:

$ helmfile --version
helmfile version v0.69.0
$ helm tiller run helm diff version
Installed Helm version v2.13.1
Installed Tiller version v2.13.1
Helm and Tiller are the same version!
Starting Tiller...
Tiller namespace: my-tiller
Running: helm diff version

2.11.0+5
Stopping Tiller...

Probably it's not a helmfile issue, but anyway. If the expected behaviour is spitting errors it should be consistent.

[andrewnazarov]

I've checked it out without tillerless plugin and it worked as expected. The following error message was shown:

err 0: failed processing release ingress: helm exited with status 1:
  Error: could not get Kubernetes config for context "some-context": context "some-context" does not exist

So it is a tillerless plugin causing this issue.

[andrewnazarov]

I bet it's related to this one: rimusz/helm-tiller#32

[andrewnazarov]

I think this also brakes the possibility to manage kubeContex based on the environment. So, this

{{ if eq .Environment.Name "dev" }}
  kubeContext: gke_XXXX_europe-west1-b_mydevcluster
{{ end }}

will be ignored even if gke_XXXX_europe-west1-b_mydevcluster is a valid context. The workaround is to deal with context outside of the helmfile, but it's not the best solution for sure.

[andrewnazarov]

Probably we should rename the issue.

[mumoshu]

Ah, thanks! The helm-tiller issue you've shared makes sense.

helm-tiller's general interface is helm tilelr run <tiller-namespace> <helm cmd>. helm-tiller doesn't parse --kube-context passed from helmfile and that's why tiller started by helm-tiller is not configured to use the kubecontext.

Theoretically we can enhance helm-tiller to accept an alternative that doesn't conflict with --kube-context, something like --tiller-kube-context.

But helm-tiller is implemented in (perhaps posix) shell AND tiller doesn't have a comand-line flag to configure context to be used so it won't worth the effort.

I believe that's why @rimusz has suggested setting KUBECONFIG with the context changed to whatever you want as a workaround in the original issue.

[mumoshu]

So it's getting very interesting... Which of the following works for you?

1. You create kubeconfig(not just context but the whole kubeconfig) files for tillers. helmfile is enhanced to accept kubeconfig in helmfile.yaml, so that tillerless helm commands are called like KUBECONFIG=<your kubeconfig> helm tiller run tiller-system -- helm upgrade ....
2. helmfile is enhanced to copy the specified kubeconfig to a temporary file. helmfile then modify the kubeconfig to have the alternative context you specified via kubeContext. The temp kubeconfig is used while running tillerless helm commands...
3. Re-implement helm-tiller in go. I thought Tiller was "usable" as a go library so we do fancy things in go and run tiller from within go.
4. Any other idea?

[andrewnazarov]

I thought about almost the same you described under 2.

[andrewnazarov]

@mumoshu, since Helm 3 is GA, should we probably close this issue? I don't think anybody will work on it.

[mumoshu]

@andrewnazarov Thanks for triaging this - Yes, I believe we can close this now!