How to reconfigure the HA node when the associated internal IP is changed ?

[prnvkv]

Environmental Info:
K3s Version:
k3s -v
k3s version v1.21.3+k3s1 (1d1f220f)
go version go1.16.6

Node(s) CPU architecture, OS, and Version:
uname -a
Linux testmachine1 4.14.173 #24 SMP Wed Sep 8 12:34:51 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
2 Node HA with Mariadb external datastore.

Describe the bug:
This is not a bug, rather a question on configuration.
In k3s HA environment with external data-store as mysql/mariadb, when the interface ip bound to API-server changes, does K3s updates this datastore or manual intervention is required ?
For example,
Node1: eth0: 192.168.0.100
eth1: 100.100.12.123
InternalIP of k3s node Node1: 192.168.0.100
Node2: eth0: 192.168.0.101
eth2: 100.100.12.124
InternalIP of k3s node Node2: 192.168.0.101
Now if Node2 eth0 ip changes to 192.168.0.106, kubectl get nodes still shows the old ip.
Is there any manual intervention required in case of external data-store like mysql/mariadb ?

Backporting
No

[brandond]

Kubernetes nodes are assumed to have a static IP address for the life of the node. If you want to change it's address, the best way to do so is to delete it and rejoin it to the cluster.

[prnvkv]

Is there any other way apart from this? This might be easiest way but not be feasible all the time :)

In one node cluster, restarting the k3s service will do the trick.

[brandond]

If it's the only node restarting it will likely do the job, but in general Kubernetes doesn't deal well with nodes changing addresses. It certainly won't deal with it without being restarted at the very least.

[prnvkv]

Thanks for your reply @brandond .
I think i tested in my setup, even after the restart of the node, kubernetes node IP was still the old one.

Other option that came to my mind is to manually update the datastore entries to disallow stale reporting to kubernetes.
I am aware this is a dangerous option and might lead to inconsistencies if you don't know what you are doing.
At the moment, this is a try worth-while.

I know you have given two options, but both are not feasible for our setup. If you have a different solution which is probably based out of a CRD or anything, please let me know.

[brandond]

I don't think that's going to get you the result you want either. The issue is not just what's in the datastore, but also the caches, runtime state, iptables rules, etc of every other node in the cluster. Kubernetes is built around nodes having a stable IP address at LEAST while running and it's probably going to be an uphill battle to get it to function well without them.

[prnvkv]

I understand.

Thank you very much for your responses @brandond .
I will retry rebooting the machine and get back to you.

[prnvkv]

Hi @brandond ,

I tried rebooting the machine this time and surprisingly it worked !!
I tried it several times and faced some intermittent issues.

Thanks a lot for all the help !!