-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
May 2021 security updates to Linux kernel restricts access to conntrack sysctls, breaking kube-proxy in CAPD clusters #4712
Comments
this affected me as well. just for the record, i tried the same process on a fedora 34 install running kernel 5.11.12 and i do not see the sysctl error. |
I assume this will affect us in kinder too once we move to a newer kernel
on the host.
Can something be done on the node images to mitigate instead of passing a
component config?
Do sig network know about this and are we going to patch kube proxy?
|
@aojea do you happen to know? |
If you're thinking "can we set the host value before spinning up", unfortunately not, because the kube-proxy calculates the max conn sysctl it wants to set based on available memory. |
I pinged @andrewsykim and @jayunit100 for an opinion on Option 2. |
WRT to the proposed options,
|
Are CAPD cluster production cluster or just for testing?
I agree here, CAPD controls the environment so it should know in advance what is the best configuration for each case, and this allows to iterate in case we need to revisit it, the change is minimal |
Looks like the consensus is to do it in CAPD. Chatting to @andrewsykim , I may speculatively open a PR to kube-proxy to make setting the sysctls non-fatal when it gets permission denied as this will help any nested running of kube-proxy. |
/lifecycle active |
What steps did you take and what happened:
[A clear and concise description on how to REPRODUCE the bug.]
Create a kind cluster
Provision a CAPD cluster
Cluster gets created but kube-proxy is in CrashLoopBackoff with
What did you expect to happen:
Cluster to work
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
This change is caused by a changeset in the Linux kernel torvalds/linux@671c54e such that non-init network namespaces are not allowed to set the connection tracking sysctls. If kube-proxy's conntrack.max-per-core is set to anything other than 0 (including null), then it will attempt to set the sysctl.
I first attempted to set extraArgs for kube-proxy, but unfortunately this value gets overridden by the configmap that explicitly sets the value to
null
in the component config.kind's workaround directly passes in a kube-proxy component config to kubeadm to set the value.
Affected Linux kernel versions include:
5.12
5.11
5.10
4.14
and 4.9 according to https://www.spinics.net/lists/stable/msg466347.html
Possible solutions:
Allow configuration of kube-proxy component config in KCP
Pros: Replicates workaround from kind
Con: How do we handle importing/copying kube-proxy component config types from k/k
Add non-init namespace detection (if feasible) & kernel version detection logic to kube-proxy and skip setting conntrack
Pros: Probably the most elegant solution if it can be done. No changes required to CAPI.
Cons: Probably not backportable.
Have CAPD patch the kube-proxy configmap as a special case.
Pros: It will work
Cons: Hackiest solution?
Environment:
cd3a694deac89d5ebeb888307deaa61487207aa0
kubectl version
): v1.21.1/etc/os-release
): Fedora 34, Linux 5.12.8-300.fc34.x86_64/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]
/area control-plane
/area bootstrap
cc @fabriziopandini
The text was updated successfully, but these errors were encountered: