Minikube's kube-proxy fails

[mentalblock]

Describe the bug
A change was recently introduced which results in Minikube's kube-proxy component failing. Specifically the logs show:

❯ kubectl logs -n kube-system kube-proxy-q2vw7
W0517 02:13:26.656176       1 proxier.go:651] Failed to read file /lib/modules/5.10.35/modules.builtin with error open /lib/modules/5.10.35/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 02:13:26.657163       1 proxier.go:661] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 02:13:26.658022       1 proxier.go:661] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 02:13:26.658674       1 proxier.go:661] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 02:13:26.659295       1 proxier.go:661] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 02:13:26.660037       1 proxier.go:661] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
I0517 02:13:26.671498       1 node.go:172] Successfully retrieved node IP: 192.168.49.2
I0517 02:13:26.671514       1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.49.2), assume IPv4 operation
W0517 02:13:26.683874       1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0517 02:13:26.683919       1 server_others.go:185] Using iptables Proxier.
I0517 02:13:26.684082       1 server.go:650] Version: v1.20.2
I0517 02:13:26.684271       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 524288
F0517 02:13:26.684282       1 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied

This was seen when starting minikube without any customization. The startup log shows:

😄  minikube v1.20.0 on Nixos 21.05pre289039.d1601a40c48 (Okapi)
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
🔎  Verifying Kubernetes components...
    ▪ Using image kubernetesui/dashboard:v2.1.0
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image kubernetesui/metrics-scraper:v1.0.4
🌟  Enabled addons: storage-provisioner, default-storageclass, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Unfortunately, I am not familiar with how the virtualization works specifically to debug this further or how to pinpoint the package that introduced this problem. All I know is that this issue is present on NixOS, 21.05pre289039.d1601a40c48 but works perfectly fine on a previous build I was running, NixOS, 21.05pre287333.63586475587. Using the latest build of NixOS unstable at the time of this report still exhibits the issue. The logs for kube-proxy when on the working version is as follows:

❯ kubectl logs -n kube-system kube-proxy-q2vw7                                                                                                                                                                     ~
W0517 01:59:27.834377       1 proxier.go:651] Failed to read file /lib/modules/5.10.32/modules.builtin with error open /lib/modules/5.10.32/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 01:59:27.846199       1 proxier.go:661] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 01:59:27.847324       1 proxier.go:661] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 01:59:27.848357       1 proxier.go:661] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 01:59:27.849155       1 proxier.go:661] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0517 01:59:27.849865       1 proxier.go:661] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
I0517 01:59:27.863777       1 node.go:172] Successfully retrieved node IP: 192.168.49.2
I0517 01:59:27.863799       1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.49.2), assume IPv4 operation
W0517 01:59:27.889843       1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0517 01:59:27.889904       1 server_others.go:185] Using iptables Proxier.
I0517 01:59:27.890932       1 server.go:650] Version: v1.20.2
I0517 01:59:27.891123       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 524288
I0517 01:59:27.891139       1 conntrack.go:52] Setting nf_conntrack_max to 524288
E0517 01:59:27.891269       1 conntrack.go:127] sysfs is not writable: {Device:sysfs Path:/sys Type:sysfs Opts:[ro nosuid nodev noexec relatime] Freq:0 Pass:0} (mount options are [ro nosuid nodev noexec relatime])
I0517 01:59:27.891318       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0517 01:59:27.891344       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0517 01:59:27.894282       1 config.go:315] Starting service config controller
I0517 01:59:27.894288       1 shared_informer.go:240] Waiting for caches to sync for service config
I0517 01:59:27.894297       1 config.go:224] Starting endpoint slice config controller
I0517 01:59:27.894300       1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I0517 01:59:27.994399       1 shared_informer.go:247] Caches are synced for endpoint slice config
I0517 01:59:27.994410       1 shared_informer.go:247] Caches are synced for service config

Let me know if there is any additional information I can provide to help pinpoint the issue. Thanks.

To Reproduce
Steps to reproduce the behavior:

1. Run minikube start
2. Check the logs of the kube-proxy component with kubectl logs -n kube-system -l k8s-app=kube-proxy and verify the permission error

Expected behavior
I expected that the kube-proxy component would not fail on a permission error. This was not present on a previous build of NixOS.

Notify maintainers
I do not understand the low-level changes that would cause this issue and therefore I cannot begin to guess the appropriate maintainers.

Metadata

• system: "x86_64-linux"
• host os: Linux 5.10.35, NixOS, 21.05pre289039.d1601a40c48 (Okapi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.4pre20210503_6d2553a
• channels(bwong): "home-manager"
• channels(root): "nixos-21.05pre289339.83d907fd760"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[mentalblock]

I have also tried using the podman driver with minikube:
minikube start --driver=podman --container-runtime=cri-o

With this driver, the problem with kube-proxy still persists.

[mentalblock]

This seems to be related to a kernel upgrade. These are related issues:
kubernetes-sigs/kind#2240

Seems the problem originates from a kernel upgrade. There is a workaround mentioned https://serverfault.com/questions/1063166/kube-proxy-wont-start-in-minikube-because-of-permission-denied-issue-with-proc