Add deployment manifests for MicroK8s

[sta4152]

After installing the driver on a microk8s cluster, the example dyn-app does not start successfully. Due to

Nov 12 11:01:13 node01.cluster microk8s.daemon-kubelite[255976]: E1112 11:01:13.907517  255976 csi_attacher.go:364] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = Internal desc = stat /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/beegfs.csi.netapp.com/546560a6191b66852bcfc73103f6bc91327cf030684f8394f25acab2da84a015/globalmount: no such file or directory

Installed by

kubectl apply -k deploy/k8s/overlays/my-overlay
kubectl apply -f examples/k8s/dyn/dyn-sc.yaml
kubectl apply -f examples/k8s/dyn/dyn-pvc.yaml
kubectl apply -f examples/k8s/dyn/dyn-app.yaml

Best guess: microk8s does not use /var/lib/kubelet as root dir but /var/snap/microk8s/common/var/lib/kubelet, with sym-linking the common default to that dir. The sym-link does not work with the mounting used by the beegfs driver.

Unfortunately I do not see any config option to specify a prefix/alternate location. But this is needed here AFAIS (similar to the kubeletDirPath in ceph - they suffer from the same without having specified this correctly).

The following patching of sources seems to work for me - please correct me if I am wrong with that or if this could cause problems somewhere later!: from within the repo root dir (I used the 1.7 branch) replace the /var/lib/kubelet occurences in non-test yaml files with the micork8s specific path:

sed -i 's%/var/lib/kubelet%/var/snap/microk8s/common/var/lib/kubelet%g' \
    $(find . -name \*.yaml | grep -v '/test/')

I see the /var/lib/kubelethard-coded also in the driver sourcecode. But I guess (and hope) that for that the sym-link does it's job, that the problem is just a matter of mounting host dirs (being and/or including sym-links) into the containers ... ?

[iamjoemccormick]

Hello @sta4152,

Thanks for doing the heavy lifting here to figure out what's going on :)

I believe updating the deployment manifests for a particular installation of the driver (as you've done) is the correct way to handle when some Kubernetes distribution like Micro K8s uses a "non-standard" path to /var/lib/kubelet. What you might consider doing instead of updating the default manifests, is creating a new overlay with the required patches/modifications to deploy for Micro K8s. That way you can continue to pull changes to the project including updates to the default manifests without having to stash/reapply your changes.

AFAIK all those hard-coded references to /var/lib/kubelet/ are only used for testing. The important places are not actually hard-coded, but rather passed as flags to the driver at startup (e.g., --cs-data-dir) and your sed command handles updating those and the volume mounts that get everything into the driver's container. So you should be all set, but let me know if you see any weird behavior.

I did want to confirm you didn't need to do anything else to get the driver to work with Micro K8s? You mentioned a symlink, but I wasn't sure if that was actually required or if simply updating the manifests was sufficient.

[sta4152]

Hi, thanks for the response.

For patching using sed I already used an overlay, named my-overlay for the issue here as shown above in the report (using a more meaningful name in reality).

For the sym-link: I did not create it myself manually, but this was created by the microk8s deployment using snap (on ubuntu 22.04 LTS, installing microk8s the snap way).

[iamjoemccormick]

Thanks for confirming. In that case the only other immediate thing that could be done to better support MicroK8s is to add default deployment manifests. However I'm hesitant to start down the path of adding custom manifests for different K8s distributions as that become challenging to test/maintain. So I'd prefer to hold off for now until we see more requests for MicroK8s and the driver.

In the meantime we can use this issue to help people get started with MicroK8s and gauge interest.