support android

[tomaszduda23]

Is your feature request related to a problem? Please describe.

Termux allows to run docker on Android after (patching kernel). It would be nice to run k3s also. k3s agent already works but requires a few patches. It would be great if those can be integrated to k3s.

Working hash
tomaszduda23@259433c

Describe the solution you'd like

1. Android use f2fs which is not supported by tomaszduda23/cadvisor@acb9743
2. cgroups are mounted with no prefix tomaszduda23/runc@a362eb6
3. containerd reads dns only from /etc/resolv.conf which do not exist on android
4. [v1.31.1+k3s-452dbbc1] containerd reads ca from /etc/ssl/certs/ca-certificates.crt which do not exist on android
5. cgroupv1 is read from random directory /dev or /sys. There is wired logic which decided which directory should be used.

Describe alternatives you've considered

I patched every thing locally.

Additional context

Script to make it run

set -e
set -x

echo Starting k3s

echo "$UID"

if [ "$(id --user)" -ne 0 ]; then
  echo "This script must be run as root!"
  exit 1
fi

[ "$1" = "start" ] && {
  echo start
  hostname instantnoodlep
  hostname
  echo 'nameserver 192.168.0.1' > resolv.conf
  ./k3s/dist/artifacts/k3s-arm64 agent \
    --server https://192.168.0.20:6443 \
    --token-file /data/k3s/etc/cluster-token \
    --config /etc/rancher/k3s/config.yaml \
    --debug \
    --resolv-conf resolv.conf
  exit 0
}

mount | grep -q '/sys/fs/cgroup/devices ' || {
  # this force k3s to use cgroup instead of cgroupv2 somehow
  mount -t tmpfs -o mode=755 tmpfs /sys/fs/cgroup

  mkdir -p /sys/fs/cgroup/cpuset
  mkdir -p /sys/fs/cgroup/cpu
  mkdir -p /sys/fs/cgroup/cpuacct
  mkdir -p /sys/fs/cgroup/schedtune
  mkdir -p /sys/fs/cgroup/blkio
  mkdir -p /sys/fs/cgroup/memory
  mkdir -p /sys/fs/cgroup/devices
  mkdir -p /sys/fs/cgroup/freezer
  mkdir -p /sys/fs/cgroup/pids

  mount -t cgroup -o cpuset none /sys/fs/cgroup/cpuset
  mount -t cgroup -o cpu none /sys/fs/cgroup/cpu
  mount -t cgroup -o cpuacct none /sys/fs/cgroup/cpuacct
  mount -t cgroup -o schedtune none /sys/fs/cgroup/schedtune
  mount -t cgroup -o blkio none /sys/fs/cgroup/blkio
  mount -t cgroup -o memory none /sys/fs/cgroup/memory
  mount -t cgroup -o devices none /sys/fs/cgroup/devices
  mount -t cgroup -o freezer none /sys/fs/cgroup/freezer
  mount -t cgroup -o pids none /sys/fs/cgroup/pids
}

mount | grep -q '/etc/rancher ' || mount -o bind /data/k3s/etc /etc/rancher
mount | grep -q '/run ' || mount -o bind /data/k3s/run /run
mount | grep -q '/var ' || mount -o bind /data/k3s/var /var

lsmod | grep -q 'br_netfilter' || insmod br_netfilter.ko
lsmod | grep -q 'xt_addrtype' || insmod xt_addrtype.ko
lsmod | grep -q 'ip_vs' || insmod ip_vs.ko
lsmod | grep -q 'vxlan' || insmod vxlan.ko

ip route | grep -q default || ip route add default via 192.168.9.1 dev wlan0
ip rule | grep -q 3000 || ip rule add from all lookup main pref 3000

unshare --uts sh $0 start

[liyimeng]

So you run cgroup v1 instead of v2? Because the kernel are not supporting v2?

[liyimeng]

I am not sure if your patches in cadvisor and runc is needed. see #11028
you can mount cgroup v2 instead of v1

[tomaszduda23]

I have kernel 4.19. I was not able to make it work with cgroup v2. I'm not sure if all required stuff is there.

[liyimeng]

For running k3s properly with cgroup v2, you at least need kernel 5.8+.
Since you have to build a custom kernel anyway, why not take a modern version? cgroup v1 is deprecated in k8s now. And in my experience, k3s works best on kernel 5.15+

[tomaszduda23]

Updating kernel does not seems easy. cgroup v1 does the job.