Default route issues with NAT Gateway

[udhay45]

1. The default route (0.0.0.0) configuration in the yaml file does not reflect in the routing table in the worker node. We need to add netmask to the configuration (i.e 0.0.0.0/0)

In my laptop :
➜ samples git:(main) ✗ kubectl get staticroutes
NAME DESTINATION GATEWAY AGE
static-route-ifconfig.me 0.0.0.0 10.116.0.5 72m
static-route-ipinfo.io 34.117.59.81 10.116.0.5 2s

In the worker node droplet :
root@logging-1-7xheu:/# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
10.116.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth1
10.244.0.0 10.116.0.2 255.255.255.128 UG 0 0 0 eth1
10.244.0.128 10.244.0.137 255.255.255.128 UG 0 0 0 cilium_host
10.244.0.137 0.0.0.0 255.255.255.255 UH 0 0 0 cilium_host
10.244.1.0 10.116.0.4 255.255.255.128 UG 0 0 0 eth1
34.117.59.81 10.116.0.5 255.255.255.255 UGH 0 0 0 eth1
159.223.176.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
root@logging-1-7xheu:/#

2. Addition of the default route has to IDEALLY replace the existing default route. Instead, it just adds it as another default route. We can't have two default routes with the same weights. This will cause traffic to take either of the next hops.

root@logging-1-7xhec:/# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 159.89.224.1 0.0.0.0 UG 0 0 0 eth0
8.8.8.8 10.116.0.5 255.255.255.255 UGH 0 0 0 eth1
10.10.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
10.116.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth1
10.244.0.0 10.116.0.2 255.255.255.128 UG 0 0 0 eth1
10.244.0.128 10.116.0.3 255.255.255.128 UG 0 0 0 eth1
10.244.1.0 10.244.1.35 255.255.255.128 UG 0 0 0 cilium_host
10.244.1.35 0.0.0.0 255.255.255.255 UH 0 0 0 cilium_host
34.117.59.81 10.116.0.5 255.255.255.255 UGH 0 0 0 eth1
159.89.224.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
root@logging-1-7xhec:/# route add -net 0.0.0.0/0 gw 10.116.0.5
root@logging-1-7xhec:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.116.0.5 0.0.0.0 UG 0 0 0 eth1
0.0.0.0 159.89.224.1 0.0.0.0 UG 0 0 0 eth0
8.8.8.8 10.116.0.5 255.255.255.255 UGH 0 0 0 eth1
10.10.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
10.116.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth1
10.244.0.0 10.116.0.2 255.255.255.128 UG 0 0 0 eth1
10.244.0.128 10.116.0.3 255.255.255.128 UG 0 0 0 eth1
10.244.1.0 10.244.1.35 255.255.255.128 UG 0 0 0 cilium_host
10.244.1.35 0.0.0.0 255.255.255.255 UH 0 0 0 cilium_host
34.117.59.81 10.116.0.5 255.255.255.255 UGH 0 0 0 eth1
159.89.224.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

Requirements:

1. We need Netmask to the static route yaml file.
2. Test it works for 0.0.0.0/0
   The default route should be replaced if a new default route is added. Currently, it gets added as another route.

[v-ctiutiu]

@udhay45 @bikram20

For now, we should ignore requests to change the default gateway entry in the Linux routing table. It is a dangerous operation from my point of view (I managed to break a cluster several time while testing).

Fixes available via PR #7. For now, we deny this kind of request.

Main reason is, it will affect DOKS cluster functionality (Kubelets), and/or other internal services (e.g. Crossplane).

[v-ctiutiu]

We should keep this open though, for further discussion and investigations on how to do it more safely (if possible).

[garyrizzo1992]

Is there a good approach to route all traffic on worker nodes? I have a requirement where i need to route all traffic.