feature: Support differing network interface names when using kube-vip

[chrismuzyn]

Summary

Currently every node in the rke2 cluster must share the same primary interface for the kube-vip DaemonSet to function. If there is a node with a different network card, etc, we need to be able to specify that unique interface.

Issue Type

Feature Idea

[MonolithProjects]

Hi @chrismuzyn, you can try to set a different interfaces using the variable rke2_interface placed in the Ansible host vars

[chrismuzyn]

@MonolithProjects Correct me if I'm wrong, but I believe the kube-vip stuff is controlled via a DaemonSet, and that DaemonSet is instituted on a single node. So only that single node's rke2_interface will be respected when the DaemonSet is created. I don't think the host variable affects any host where the DaemonSet wasn't copied.

https://github.com/lablabs/ansible-role-rke2/blob/main/templates/kube-vip/kube-vip.yml.j2#L38

[michal-kania]

Hi @chrismuzyn, i had the same problem with different network interfaces per node (eth0, enp8s0, etc) so one rke2_interface parameter for all nodes is not enough in HA mode.
I solved this by setting following in playbook.yaml:

# --autoInterface - This flag is default when vip_interface is not set.
vars:
  rke2_interface: null

kube-vip/kube-vip#273

Hope it helps.

[chrismuzyn]

Thanks @michal-kania for the tip, it seems to have at least half way worked.

It's been a while since I originally posted this, so yesterday I pulled the latest version of this role, and tried again, trying to set rke2_interface: null. The behavior was unexpected. My hosts file is as follows(sorry for the illogical order):

all:
  children:
    kubernetes:
      children:
        k8s_cluster:
          children:
            masters:
              hosts:
                k6:
                k7:
                k5:
            workers:
              hosts:
                k8:

This makes k6 the kube-vip leader since it's first. It has interface enp5s0d1 and that seems to propagate correctly to the kube-vip-ds pod. But the other 3 hosts are all set to use enp8s0 as their interface. And only one of them has that interface: k7. But k7, k5, and k8, all have this in the environment section of their respective kube-vip-ds pods:

    Environment:
      vip_arp:             true
      vip_interface:       enp8s0

Since k5 and k8 don't have enp8s0, their pod is failing:

time="2023-12-22T03:24:50Z" level=info msg="Starting kube-vip.io [v0.6.4]"
time="2023-12-22T03:24:50Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[true]"
time="2023-12-22T03:24:50Z" level=fatal msg="enp8s0 is not valid interface, reason: get enp8s0 failed, error: Link not found"

Of course the loadbalancer ips only work for pods on the nodes that have their kube-vip-ds pod working. So something is not quite right... I'll see if I can figure out the logic in ansible but if this is a kube-vip problem I'm unfortunately outside of my element.