Performance Problems with Network Policies - Request for Help

[maltelehmann]

Hi all!

first, thank you very much for k3s, we are really enjoy working with it!

On our testing clusters, we activated network policies with default deny-all. Afterwards, we observed that between pod start and reacheability of coredns pod, it takes >=20sec. According to some k3s/kube-router issues regarding network policies, we would expect only a couple of seconds.
We only get a delay of a couple of seconds when we reduce the number of network policies (see below). We consider our cluster and the number of network policies as rather small, there seems to be a problem in our configuration or maybe a bug?

Our investigations (below) did not give us any hint on what is the root cause of the performance problems. Do you have any recommendations how we could proceed to find the root cause?

Thank you for your help!

Malte

System setup

OS

cat /etc/redhat-release 
CentOS Linux release 8.3.2011

• iptables service is running with egress allow all and ingress only from specific hosts etc.

• we have the following amount of resulting iptables rules and ipset rules:

sudo ipset list | wc -l
14387
sudo iptables-save | wc -l
1104

k3s

• version: v1.26.6+k3s1

• config.yaml

token: 'mytoken'
data-dir: '/ssd/k3s'
node-ip: 'mynodeip'

prefer-bundled-bin: true

flannel-iface: 'eth0'

kubelet-arg:
  - 'root-dir=/ssd/k3s/kubelet'
  - 'image-gc-high-threshold=70'
  - 'image-gc-low-threshold=50'

debug: true

node-taint:
- CriticalAddonsOnly=true:NoExecute

cluster-cidr: '10.42.0.0/16'
service-cidr: '10.43.0.0/16'
flannel-backend: 'host-gw'
tls-san: [mngr1.mydomain, mngr2.mydomain, mngr3.mydomain]

etcd-expose-metrics: true

secrets-encryption: true

• we have the following cluster size:

kubectl get no | wc -l
25

kubectl get po --all-namespaces | wc -l
248

kubectl get svc --all-namespaces | wc -l
119

kubectl get netpol --all-namespaces | wc -l
170

Debugging

• k3s Logs with debug: true -> do not show any error or hint on the delay
• prefer-bundled-bin: false -> no change in performance
• disable host iptables service and reboot -> no change in performance
• sudo ipset list | grep "<my_pod_ip> " during pod startup -> it can be observed that the result is first 1 line, after some seconds 2 lines, and finally 6 lines and then the DNS is reacheable. So it seems that the update of the ipset rules takes really long
• removing most pods such that we ending up with 54 (before: 248) -> no change in performance
• removing most netwokr policeis ending up with 38 (before: 170) -> performance improves to 1sec!

[brandond]

@manuelbuil @rbrtbnfgl do you have any suggestions on where we might look for this? 248 pods and 170 policies is probably more than we test with regularly, but I'm curious if there's anywhere in particular we could look for performance improvements.

[brandond]

FWIW, and in @aauren's defense, this is a regression that was added in the k3s fork of the network policy controller, as part of the work to add dual-stack support. The Restore call should not have been moved into the inner loop in k3s-io/kube-router@a96679d

This was pulled upstream and is present in the prep-v2.0 branch, but since it's still a WIP branch, this code probably doesn't see much testing outside k3s.

[aauren]

@brandond - Thanks for your work on this. Everything seems reasonable from my perspective. I'm disappointed that I didn't catch the scope change of the ipset restore logic when I was first incorporating the NPC changes, but thanks for finding it now!

I went ahead and incorporated the changes into an upstream PR here: cloudnativelabs/kube-router#1545 and tagged you on it.

[ChristianCiach]

@brandond How do we proceed from here? I guess the PR created by @aauren will eventually be cherry-picked into the k3s fork of kube-router. Can we expect a backport of this fix? We are currently on K3s 1.26, but we could probably upgrade to 1.27 if necessary.

[brandond]

It's too late for any changes to make it into our release of last week's upstream patch releases; we'll probably rebase our minor changes to kube-router on top of the upstream 2.0 branch in time for the October upstream patches. Whatever we do will be made available in all active branches.

[manuelbuil]

I created an issue to track it in the October release #8380