CNI set-up on bottlerocket metal

[StefanTheWiz]

Hi!

How would one go about installing a CNI on a standalone Bottlerocket cluster on a bare metal server?

I've got a single node, set up using bottlerocket-metal-k8s-1.25-x86_64-v1.12.0.img, with the following user-data.toml

[settings.network]
hostname = "node1"

[settings.kubernetes]
cluster-name = "spongebob"
standalone-mode = true

[settings.dns]
name-servers = ["1.1.1.1", "2606:4700:4700::1111", "8.8.8.8", "2001:4860:4860::8888"]

[settings.host-containers.admin]
enabled = true
superpowered = true
user-data = "......"

Based on the errors I get, it seems that this image doesn't provide any CNI in standalone mode? 🤔
e.g. /etc/cni/net.d is empty

I was hoping that by using standalone-mode = true, I'd get a node that is a control plane and a worker node at the same time, and then I would be able to add more worker nodes later, or have I got that wrong?

Errors

Grep over all files collected by logdog:

$ grep cni ./*
./containerd-config:    [plugins."io.containerd.grpc.v1.cri".cni]
./containerd-config:      bin_dir = "/opt/cni/bin"
./containerd-config:      conf_dir = "/etc/cni/net.d"
./containerd-config:        cni_conf_dir = ""
./containerd-config:        cni_max_conf_num = 0
./containerd-config:          cni_conf_dir = ""
./containerd-config:          cni_max_conf_num = 0
./containerd-config:        cni_conf_dir = ""
./containerd-config:        cni_max_conf_num = 0
./df:tmpfs             63G     0   63G   0% /etc/cni
./df:overlay          935G  456M  897G   1% /opt/cni/bin
./df-inodes:tmpfs              16M     2   16M    1% /etc/cni
./df-inodes:overlay            60M   17K   60M    1% /opt/cni/bin
./dmesg:[    2.231692] systemd[1]: Mounting CNI Configuration Directory (/etc/cni)...
./dmesg:[    2.253681] systemd[1]: Mounted CNI Configuration Directory (/etc/cni).
./journalctl.log:Feb 01 01:48:45 localhost systemd[1]: Mounting CNI Configuration Directory (/etc/cni)...
./journalctl.log:Feb 01 01:48:45 localhost systemd[1]: Mounted CNI Configuration Directory (/etc/cni).
./journalctl.log:Feb 01 01:48:46 localhost systemd[1]: Mounting CNI Plugin Directory (/opt/cni/bin)...
./journalctl.log:Feb 01 01:48:46 localhost systemd[1]: Mounted CNI Plugin Directory (/opt/cni/bin).
./journalctl.log:Feb 01 01:48:46 localhost systemd-tmpfiles[1568]: Running remove action for entry d /etc/cni/net.d
./journalctl.log:Feb 01 01:48:46 localhost systemd-tmpfiles[1568]: Running create action for entry d /etc/cni/net.d
./journalctl.log:Feb 01 01:48:46 localhost systemd-tmpfiles[1568]: Created directory "/etc/cni/net.d".
./journalctl.log:Feb 01 01:48:52 1.production.falcon.pentafox.io containerd[2057]: time="2023-02-01T01:48:52.805506849Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:shimpei DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[shimpei:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[BinaryName:shimpei SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:true SelinuxCategoryRange:1024 SandboxImage:public.ecr.aws/eks-distro/kubernetes/pause:3.5 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
./journalctl.log:Feb 01 01:48:52 1.production.falcon.pentafox.io containerd[2057]: time="2023-02-01T01:48:52.806335171Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
./journalctl.log:Feb 01 01:48:52 1.production.falcon.pentafox.io containerd[2057]: time="2023-02-01T01:48:52.807318908Z" level=info msg="Start cni network conf syncer for default"
./journalctl.log:Feb 01 01:49:02 1.production.falcon.pentafox.io kubelet[2137]: E0201 01:49:02.109311    2137 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
.....
./proc-mounts:tmpfs /etc/cni tmpfs rw,seclabel,nosuid,nodev,noexec,noatime,mode=755 0 0
./proc-mounts:overlay /opt/cni/bin overlay rw,context=system_u:object_r:cni_exec_t:s0,nosuid,nodev,noatime,lowerdir=/usr/libexec/cni/bin,upperdir=/var/lib/cni-plugins/.overlay/upper,workdir=/var/lib/cni-plugins/.overlay/work 0 0
./systemd-analyze-blame: 129ms opt-cni-bin.mount
./systemd-analyze-blame:  21ms etc-cni.mount

[zmrow]

Hi @StefanTheWiz !

Could you provide a few more details on what you're trying to set up?

The standalone-mode setting means kubelet will not attempt to connect to a Kubernetes API server, it does not install any additional components like CNI. CNI is typically run as a deployment on a cluster, not necessarily on the hosts.

Here's a link to the EKS-A docs on setting up a bare-metal cluster.

[StefanTheWiz]

Thanks for the explanation.

I'm trying to set up a cluster where all nodes and control planes/masters run on bottlerocket OS.
If possible, I'd rather not create the control planes in EKS / other clouds / using other operating systems.

I guess I was hoping that standalone-mode would create the equivalent of a temporary docker kind cluster (which has a default CNI).

I think my error is in trying to use bottlerocket as the OS for the control plane from the start but, perhaps, bottlerocket wasn't intended for that use-case.

Looking at the docs for the EKS-A cluster workflow, I believe I should instead create a small "bootstrap cluster" (maybe locally with kind), then add the CNI and then add more nodes.

[zmrow]

if I'm understanding correctly, it's not currently possible to use Bottlerocket to bootstrap a cluster. However it is possible to run Bottlerocket as the OS for both the control plane and the workers (EKS-A does this).

I believe EKS-A bootstrapping uses static pods to run all the different control plane components on Bottlerocket, which use host networking so CNI doesn't come into play until you have everything needed to deploy daemonsets.

[StefanTheWiz]

That makes sense. Thanks a lot 👍