k3s: Dynamically generate kubeconfig

[stv0g]

This PR adjusts the K3S manifests to automatically generate a kubeconfig file which is used by kg based on the token of the kilo ServiceAccount and the master address found in the kubelet's kubeconfig file.

The eliminates the need of manually copying /etc/rancher/k3s/k3s.yaml from the master to all agent nodes.
At the same time, the kg daemon only has the permissions granted to its ServiceAccount instead of a full cluster access.

Closes #49

[squat]

I really like this, thank you! I have just one suggestion

[stv0g]

@squat Sure, I wasnt aware of that. I updated my branch accordingly

[leonnicolas]

you need to add get to the verbs of cutomresourcedefinitions and we don't need create anymore :)

[squat]

you need to add get to the verbs of cutomresourcedefinitions and we don't need create anymore :)

I think the branch needs to be rebased but I don't think the commit actually changes the RBAC.

@stv0g mind rebasing and then we can merge?

[stv0g]

@squat I've rebased the branch

@leonnicolas I also dont get your comment regarding the RBAC.

[squat]

@leonnicolas I also dont get your comment regarding the RBAC.

I think this was about the un-rebased code. When viewing the files from the PR in their entirety, you would see out-of-date RBAC.

[leonnicolas]

The nkml daemonSet needs the serviceAccountName: kilo in its spec, otherwise it cannot get and label the nodes.

[stv0g]

The nkml daemonSet needs the serviceAccountName: kilo in its spec, otherwise it cannot get and label the nodes.

Fixed

[leonnicolas]

Thanks, that's such an improvement!

[unixfox]

Why is it through a configmap and not directly using the args parameter in the pod?
It could have been like this:

initContainers:
- name: generate-kubeconfig
  image: squat/kilo
  command:
  - /bin/sh
  args:
  - cat > /etc/kubernetes/kubeconfig <<EOF
        apiVersion: v1
        kind: Config
        name: kilo
        clusters:
        - cluster:
            server: $(sed -n 's/.*server: \(.*\)/\1/p' /var/lib/rancher/k3s/agent/kubelet.kubeconfig)
            certificate-authority: /var/lib/rancher/k3s/agent/server-ca.crt
        users:
        - name: kilo
          user:
            token: $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
        contexts:
        - name: kilo
          context:
            cluster: kilo
            namespace: ${NAMESPACE}
            user: kilo
        current-context: kilo
    EOF
  imagePullPolicy: Always
  volumeMounts:
  - name: kubeconfig
    mountPath: /etc/kubernetes
    readOnly: true
  - name: k3s-agent
    mountPath: /var/lib/rancher/k3s/agent/
    readOnly: true
  env:
  - name: NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace

We save having to store a configmap like this.

[squat]

Yes totally, I think the one place where this is helpful is in the user-space-heterogeneous manifests where we need to inline this script for both DaemonSets. I'm happy to move this in-line to avoid extra resources. It adds a bit more configuration maintenance complexity on us, however, we may be able to overcome this with some work @leonnicolas is doing to automate manifest generation.