Rootless k3s stuck at "Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

[edysli]

I (finally) managed to get a rootless k3s to start, but it continuously prints the same two lines in the log and makes no progress:

k3s[27860]: time="2024-11-12T21:21:01+01:00" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
k3s[27860]: time="2024-11-12T21:21:01+01:00" level=info msg="Waiting for control-plane node mypc startup: nodes \"mypc\" not found"

It seems containerd's socket path is wrongly set, because earlier in the log, there is:

k3s[27860]: time="2024-11-12T21:20:10+01:00" level=info msg="Running containerd -c /home/user/.rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /home/user/.rancher/k3s/agent/containerd"

and there is nothing in /run/k3s/containerd/, instead the socket is /run/user/1000/k3s/containerd/containerd.sock.

Setting kubelet-arg: - "container-runtime-endpoint=unix:///run/user/1000/k3s/containerd/containerd.sock" in my config.yaml doesn't fix the issue. k3s still uses a hardcoded path /run/k3s/containerd/containerd.sock which doesn't exist.

Any idea how to fix this?

[brandond]

The problem is not that the socket path is wrong. If it was the wrong path you wouldn't be able to connect to it to get that "unknown service" RPC error from the server. Note that many of the paths reported by rootless K3s will appear to not exist when you're poking at it from outside. These paths only exist within the k3s service's filesystem namespace.

The problem is that containerd is running, but the CRI service is not enabled for some reason. Check containerd.log to see why. Usually it has something to do with a bad snapshotter causing the service setup to fail.

Why are you having so much trouble with rootless K3s? Running it as a systemd user service, as described in the docs, should work fine.

[brandond]

You're running in a network namespace. The rootless eth0 IP is not the host's IP. It is the IP of the rootless side of the slirp4netns tunnel device, the one you were complaining about being ipv4-only.

If you're using another CNI besides the flannel that comes with k3s, it won't be aware of k3s's CNI directory. It'll use whatever its defaults are.

I really don't think you're going to get cilium working rootless.

[edysli]

IPv6 first

So I removed Cilium from the picture and enabled the built-in Flannel to simply things. However, I still get IPv6-related failures with this configuration:

cluster-cidr:
  - "fd5b:e9ce:dffb:cafe::/64"
  - "192.168.33.0/24"
service-cidr:
  - "fd5b:e9ce:dffb:07ea::/112"
  - "192.168.32.0/24"
node-ip:
  - "fd5b:e9ce:dffb:07ea::100"
  - "192.168.31.100"

and K3S_ROOTLESS_ENABLE_IPV6=true K3S_ROOTLESS_CIDR=192.168.31.0/24.
k3s repeatedly logs these three lines:

time="2024-11-14T22:51:54+01:00" level=info msg="Connecting to proxy" url="wss://[fd5b:e9ce:dffb:7ea::100]:6443/v1-k3s/connect"
time="2024-11-14T22:51:54+01:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp [fd5b:e9ce:dffb:7ea::100]:6443: connect: network is unreachable"
time="2024-11-14T22:51:54+01:00" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp [fd5b:e9ce:dffb:7ea::100]:6443: connect: network is unreachable" url="wss://[fd5b:e9ce:dffb:7ea::100]:6443/v1-k3s/connect"

IPv4 first

Weird thing is, if I put the IPv4 addresses first in the configuration:

cluster-cidr:
  - "192.168.33.0/24"
  - "fd5b:e9ce:dffb:cafe::/64"
service-cidr:
  - "192.168.32.0/24"
  - "fd5b:e9ce:dffb:07ea::/112"
node-ip:
  - "192.168.31.100"
  - "fd5b:e9ce:dffb:07ea::100

The errors above don't occur and it progresses to another error... 😕
This time it's not network related:

E1114 23:07:09.597096      53 remote_runtime.go:343] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/home/edysli/.rancher/k3s/agent/kubelet/pods/45a40c1d-d11c-42eb-9912-e2b008d28a74/volumes/kubernetes.io~configmap/content\" to rootfs at \"/chart\": mount /home/edysli/.rancher/k3s/agent/kubelet/pods/45a40c1d-d11c-42eb-9912-e2b008d28a74/volumes/kubernetes.io~configmap/content:/chart (via /proc/self/fd/6), flags: 0x5021: operation not permitted: unknown" containerID="587d45fa5b3f055ca7ec40f739027f4a7e352443a20e85e812c282e0d317c978"

All pods are stuck in CrashLoopBackOff.
I do have /home mounted with nosuid and nodev. Could this be related to #10851 ?

[brandond]

K3S_ROOTLESS_CIDR=192.168.31.0/24

service-cidr:
 - "fd5b:e9ce:dffb:07ea::/112"
 - "192.168.32.0/24"
node-ip:
 - "fd5b:e9ce:dffb:07ea::100"
 - "192.168.31.100"

What are you doing here? Why are you trying to use an address from the cluster service cidr as the node's ipv6 address? How would that work?

It looks like slirp4nets is hardcoded to use fd00::1 as the interface's ipv6 address, it cannot be changed like the IPv4 CIDR. I'm kinda guessing here as I am not super familiar with libslirp's SlirpConfig.

The node-ip options tell Kubernetes what addresses to use; those IPs have to actually be present on an interface, you can't just pick random IPs.

I do have /home mounted with nosuid and nodev.

Yeah that's probably not going to work either. Maybe try using something outside of /home as your data-dir, if you don't want to change that? I'm also assuming /home isn't on a NFS share or something else even less likely to work?

[edysli]

What are you doing here? Why are you trying to use an address from the cluster service cidr as the node's ipv6 address? How would that work?

because I am completely clueless about configuring node-ip and node-external-ip... 🤷
k3s agent's documentation says:

--node-ip IP address to advertise for node
--node-external-ip External IP address to advertise for node

kubelet's documentation explains a bit more:

IP address (or comma-separated dual-stack IP addresses) of the node. If unset, kubelet will use the node's default IPv4 address, if any, or its default IPv6 address if it has no IPv4 addresses. You can pass "::" to make it prefer the default IPv6 address rather than the default IPv4 address.

Neither clearly explains what that address is used for nor how to pick one. Of course, I've tried leaving this unconfigured and tried :: too, none of these left me with a working network (same proxy errors as above). Without configuration, 192.168.31.100 gets picked up magically, now I suppose that is one end of the slirp4netns connection. It's hard to understand what's going on when I can't see any of the network interfaces in that namespace... Sorry for fumbling in the dark and bumping into the furniture.

I'm also assuming /home isn't on a NFS share or something else even less likely to work?

No, it's not NFS just ext4.
Are there issues with ZFS? I've seen containerd must be configured with a special snapshotter for ZFS.

[brandond]

Normally containerd just uses overlayfs which should work on most filesystems. If the filesystem itself supports snapshotting (like zfs does) then you can use a filesystem-specific snapshotter. If you can't use anything else (not even overlayfs) is the native snapshotter but it is super inefficient because it just unpacks all the layers on top of each other without any potential for reuse.

You're jumping in to the deep end by trying to get all this working without understanding Kubernetes and containerd and Linux namespaces work at a low level. Rootless Kubernetes works within very tightly defined constraints and there is a lot of stuff that's not possible without someone writing a bunch of additional code to wire up all the different parts.