Should iptables be removed on container shutdown/removal?

[olvesh]

When the container shuts down should the iptables rule be reverted?

[fwynyk]

I would like to reinforce this question with an example. I didn't noticed that running K8s with calico we need to use cali+ on --host-interface. So, I bring kube2iam with cbr0 option, which was wrong.

Than I fix my mistake, and deleted the old kube2iam Deamonset. But the iptables rule was not removed. So I tried to bring Kube2iam with cali+config, and it works as I expected, but the old iptables rule with cbr0 is still there:

~# iptables -t nat -S PREROUTING | grep 169.254.169.254
-A PREROUTING -d 169.254.169.254/32 -i cbr0 -p tcp -m tcp --dport 80 -j DNAT --to-destination xx.xx.xx.xx:8181
-A PREROUTING -d 169.254.169.254/32 -i cali+ -p tcp -m tcp --dport 80 -j DNAT --to-destination xx.xx.xx.xx:8181

As I said, I didn't have any problems with the new Deamonset, but I think the better should be clean up the rule when we delete the deamonset.

Also, I'm running version 0.2.2 because of this issue: #37

It would be great to have a solution for both problems.

[oyvindio]

+1 I think this is a good idea. I had the same issues as @fwynyk and had to clean up iptables on every cluster node several times while troubleshooting the issue.

[zytek]

When the container shuts down because of a crash/OOM then removing iptables rule would create a security issue.

[olvesh]

A crash wouldn't be able to remove anything anyway, would it? Also perhaps not an OOM?

So as an attack vector a crash/oom would not work. And if installed as a daemonset a crash would force a restart.

In case of a willful shutdown I would expect a service to return the system to its previous state.

[jescarri]

can we add logic at startup of the daemon to see if there are any iptables rules and clear them out, then we will be protected if the pod crashes or is killed by the k8s control plane.

[jescarri]

I have seen problems when we updated kube2iam in a cluster that has kube2iam running already, when a pod does not have an iam role associated kube2iam tries to proxy the calls to the metadata service, and I see http proxy context timeout and kube2iam was completely un-usable, what I think it happens is that the old rules are making kube2iam calls go to an inexistent pod.

[soggiest]

+1 on this, leaving a lingering iptables entry on the nodes when a kube2iam pod is deleted from a node can have some really terrible ramifications.

[olvesh]

Closing this, I am not using kube2iam any more and just want to clear up my "open issues" in GitHub.