Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ubuntu 21.04 Server on Raspberry Pi, Fails to start #5355

Closed
jakesteele opened this issue Mar 29, 2022 · 2 comments
Closed

Ubuntu 21.04 Server on Raspberry Pi, Fails to start #5355

jakesteele opened this issue Mar 29, 2022 · 2 comments
Assignees
@dereknola

Comments

@jakesteele
Copy link

jakesteele commented Mar 29, 2022 •
edited
Loading

Environmental Info:
pi@pione:~$ k3s -v
k3s version v1.22.7+k3s1 (8432d7f)
go version go1.16.10

Node(s) CPU architecture, OS, and Version:

pi@pione:~$ uname -a
Linux pione 5.13.0-1022-raspi #24-Ubuntu SMP PREEMPT Wed Mar 16 07:19:33 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:

4 raspberry pis, 3 workers, 1 master

Describe the bug:
Just trying to install k3s on a fresh Ubuntu 21.04 Server install (64-bit) on a raspberry pi 4b.

Steps To Reproduce:

  1. Install fresh Ubuntu
  2. Run update and upgrade.
sudo vi /boot/firmware/cmdline.txt
And added the following:
cgroup_enable=memory cgroup_memory=1
  1. Run: curl -sfL https://get.k3s.io | sh -

Expected behavior:
Installation is successful

Actual behavior:
K3s just start and crash, start and crash, start and crash :D

Additional context / logs:

pi@pione:~$ k3s check-config


Verifying binaries in /var/lib/rancher/k3s/data/cefe7380a851b348db6a6b899546b24f9e19b38f3b34eca24bdf84853943b0bb/bin:
- sha256sum: good
- links: good

System:
- /usr/sbin iptables v1.8.7 (nf_tables): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- /usr/sbin/apparmor_parser
apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: pass

pi@pione:~$


pi@pione:~$ sudo systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: activating (start) since Tue 2022-03-29 11:52:46 PDT; 1s ago
       Docs: https://k3s.io
    Process: 57847 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 57849 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 57850 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 57851 (k3s-server)
      Tasks: 18
     Memory: 50.1M
        CPU: 1.806s
     CGroup: /system.slice/k3s.service
             └─57851 /usr/local/bin/k3s server

Mar 29 11:52:47 pione k3s[57851]: I0329 11:52:47.149210   57851 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLif>
Mar 29 11:52:47 pione k3s[57851]: I0329 11:52:47.149296   57851 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRange>
Mar 29 11:52:47 pione k3s[57851]: time="2022-03-29T11:52:47-07:00" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
Mar 29 11:52:47 pione k3s[57851]: time="2022-03-29T11:52:47-07:00" level=info msg="To join node to cluster: k3s agent -s https://192.168.50.51:6443 -t ${NODE_TOKEN}"
Mar 29 11:52:47 pione k3s[57851]: I0329 11:52:47.154244   57851 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLif>
Mar 29 11:52:47 pione k3s[57851]: I0329 11:52:47.154337   57851 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRange>
Mar 29 11:52:47 pione k3s[57851]: time="2022-03-29T11:52:47-07:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Mar 29 11:52:47 pione k3s[57851]: time="2022-03-29T11:52:47-07:00" level=info msg="Run: k3s kubectl"
Mar 29 11:52:47 pione k3s[57851]: W0329 11:52:47.193003   57851 genericapiserver.go:455] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
Mar 29 11:52:47 pione k3s[57851]: I0329 11:52:47.196106   57851 instance.go:278] Using reconciler: lease

pi@pione:~$ journalctl -xe

Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yam>
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice>
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-ser>
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics>
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-dep>
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.ya>
Mar 29 11:56:43 pione k3s[62098]: E0329 11:56:43.690719   62098 memcache.go:196] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to ha>
Mar 29 11:56:43 pione k3s[62098]: E0329 11:56:43.701955   62098 memcache.go:101] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to ha>
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Creating deploy event broadcaster"
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"4>
Mar 29 11:56:43 pione k3s[62098]: E0329 11:56:43.815004   62098 memcache.go:196] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to ha>
Mar 29 11:56:43 pione k3s[62098]: E0329 11:56:43.829474   62098 memcache.go:101] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to ha>
Mar 29 11:56:43 pione k3s[62098]: E0329 11:56:43.863094   62098 memcache.go:196] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to ha>
Mar 29 11:56:43 pione k3s[62098]: I0329 11:56:43.882870   62098 kube.go:127] Node controller sync successful
Mar 29 11:56:43 pione k3s[62098]: I0329 11:56:43.883047   62098 vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
Mar 29 11:56:43 pione k3s[62098]: time="2022-03-29T11:56:43-07:00" level=fatal msg="flannel exited: operation not supported"
Mar 29 11:56:43 pione systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ An ExecStart= process belonging to unit k3s.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Mar 29 11:56:43 pione systemd[1]: k3s.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Mar 29 11:56:44 pione systemd[1]: k3s.service: Consumed 21.761s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit k3s.service completed and consumed the indicated resources.

Mar 29 11:55:37 pione sh[61036]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Mar 29 11:55:37 pione sh[61037]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
@jakesteele
Copy link
Author

Seems like I was missing:

sudo apt install linux-modules-extra-raspi

Found it in the microk8s install step by step guide.

@brandond
Copy link
Member

Duplicate of #4234 (comment) - we should probably add this to the docs somewhere, since it seems the Ubuntu team is intentionally not bundling all the necessary kernel modules for vxlan on raspberry pi.

@dereknola dereknola self-assigned this Apr 1, 2022
@dereknola dereknola mentioned this issue Apr 1, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dereknola dereknola

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@brandond @jakesteele @dereknola