rke2 node failover time

[uk1988]

Hej all

we are running k8s with rke2 for real time monitoring system.
Most of the time it runs on virtual servers, but sometimes we deliver it on physical hardware.

During our testing with 3 node system, we have taken a closer look at failover speed.
I would like to confirm that our measurements are in line with rke2 specs.

We have a 3 nodes setup where we run both master and worker role.

For test I have been running 5 instances of a curl container on the first node in the cluster.
It was connecting to kube api and retrieving list of all pods in default namespace.
It was doing so once a second with time out once a second.

Curl calls were spread across all 3 master nodes. Than we have shutdown node number 3.
We have seen that in some of curl containers have lost connection to kubeapi for about 10 sec.

My question is how could we lower this failover time? Which parameter we could change.
As said, we are running our product on closed k8s system that is control monitoring system where failover speed is important.

Example from failover test: