iptables entries are not reconciled #6385
Comments
|
Summary of the investigation at this point:
I've verified the following more common failure modes are handled correctly:
Status:
Moving this issue out of the 0.10.0 milestone. |
|
Noting this isn't the same as #7537 - the repro steps here still leak rules, e.g. after.txt |
|
We have a similar issue - iptables is a mess after some Nomad restarts. We're on Nomad 1.0.4 |
|
We have hit this too. Having a few stale iptables rules is fine until new allocations are assigned with ports used by the stale rules. In the above example, Even though we almost always drain nodes before manual nomad restart, sometimes nomad gets restarted automatically by systemd due to consul failure or other reasons, in which case we'd end up having stale iptables rules. So it'd be great if the iptables rule reconciliation can be implemented. 🙏 🤞 |
|
I'm seeing a slightly different trigger of this issue, but with the same root cause and end results. We use puppet to manage some iptables rules on the host[0]. When puppet makes a ruleset change, it triggers the new ruleset to be persisted (on EL, that's via iptables-save to This is on nomad 1.1.5. [0] One of the rules we're managing is |
|
I've just re-read the upgrade guide for (in preparation for 1.2.0), and I think the changes in that 1.1.0 to append the CNI rules rather than insert at the top of the chain is what made this issue more noticeable (#10181). Previously, had transient rules been persisted, the next time an alloc was started, the new iptables rules would be inserted above the stale ones and thus take precedence. Now they are added below the stale rules, so traffic is matched and blackholed by the stale rules. |
|
As noted in #11901, this affects us quite badly right now (though we're using nftables as opposed to iptables, the issue and result is the same). Whenever an unclean stop or agent restart has occurred for a job with static ports, those ports will (silently, no errors, local checks seem to succeed) fail to bind again until reboot or stale rules are manually removed. While at first glance it looked like this was a regression caused by the priority-inversion in #10181 (as noted by @optiz0r ), that PR looks concerned only with the with |
|
This is becoming a major security and stability issue as we are seeing allocations try to forward from ports that already have rules in iptables, and requests bound for them are getting forwarded based on the stale iptables rule. Here is the error log around the time it fails to cleanup the stale allocation: Logs |
|
Hit this same issue today. Performed some manual iptables clean up on a problem client. Here are my notes in case this helps anyone else:
|
|
Any update on this issue? Facing it and it's causing very annoying stability issues on a select few hosts. |
|
Please fix this or offer a proper solution, I don't care if we have to run a script to do it, but something that can be automated would be nice. We've positioned our whole infrastructure on Nomad, and this is killing us. We would prefer not to jump ship, but I'm still concerned how this isn't affecting other users? |
|
Affects us as well |
|
Hey folks, we update issues when we're working on them. I can say this is on our roadmap but I can't really give a timeline. |
|
What are we supposed to do in the meantime? |
|
This issue is pretty devastating for the use case at my PoB. Is there a workaround that can be implemented until an official fix comes out? At the moment we have to do full reboots of Nomad and take our whole network offline when we run into it. |
|
@johnalotoski has posted a process above; if you were to run that as a periodic task (or just a cron job) that'd clean up the iptables. |
|
Little bit difficult to script around that honestly, since you're also having to compare to what allocations exist, and what host they are on |
|
Okay, we're moving on from this, we can't support our org with this |
This PR adds a secondary path for cleaning up iptables created for an allocation when the normal CNI library fails to do so. This typically happens when the state of the pause container is unexpected - e.g. deleted out of band from Nomad. Before, the iptables rules would be leaked which could lead to unexpected nat routing behavior later on (in addition to leaked resources). With this change, we scan for the rules created on behalf of the allocation being GC'd and delete them. Fixes #6385
This PR adds a secondary path for cleaning up iptables created for an allocation when the normal CNI library fails to do so. This typically happens when the state of the pause container is unexpected - e.g. deleted out of band from Nomad. Before, the iptables rules would be leaked which could lead to unexpected nat routing behavior later on (in addition to leaked resources). With this change, we scan for the rules created on behalf of the allocation being GC'd and delete them. Fixes #6385
This PR adds a secondary path for cleaning up iptables created for an allocation when the normal CNI library fails to do so. This typically happens when the state of the pause container is unexpected - e.g. deleted out of band from Nomad. Before, the iptables rules would be leaked which could lead to unexpected nat routing behavior later on (in addition to leaked resources). With this change, we scan for the rules created on behalf of the allocation being GC'd and delete them. Fixes #6385
This PR adds a secondary path for cleaning up iptables created for an allocation when the normal CNI library fails to do so. This typically happens when the state of the pause container is unexpected - e.g. deleted out of band from Nomad. Before, the iptables rules would be leaked which could lead to unexpected nat routing behavior later on (in addition to leaked resources). With this change, we scan for the rules created on behalf of the allocation being GC'd and delete them. Fixes #6385
This PR adds a secondary path for cleaning up iptables created for an allocation when the normal CNI library fails to do so. This typically happens when the state of the pause container is unexpected - e.g. deleted out of band from Nomad. Before, the iptables rules would be leaked which could lead to unexpected nat routing behavior later on (in addition to leaked resources). With this change, we scan for the rules created on behalf of the allocation being GC'd and delete them. Fixes #6385
This PR adds a secondary path for cleaning up iptables created for an allocation when the normal CNI library fails to do so. This typically happens when the state of the pause container is unexpected - e.g. deleted out of band from Nomad. Before, the iptables rules would be leaked which could lead to unexpected nat routing behavior later on (in addition to leaked resources). With this change, we scan for the rules created on behalf of the allocation being GC'd and delete them. Fixes #6385
When a client is stopped, the tasks on that client are left running. When the client restarts, it goes through a restore process to get handles to all its tasks again. If a task fails or is removed while the client is shutdown, the client should be able to garbage collect any of its dangling resources (like alloc dirs) and restart the task. This is not happening with iptables.
Fortunately we "tag" all the iptables rules in one of two ways:
CNI-xxxx/CNI-DN-xxxx. I don't know what thatxxxxis but it's not the alloc ID, container ID, or network namespace ID./* name: "nomad" id: "<alloc ID>" *.So it should be possible to identify "Nomad owned" rules and clean them up if they don't belong to an allocation we know about if we can figure out the naming for the CNI chains.
Nomad version
(But probably any recent version.)
Reproduction steps
nomad job run ./e2e/connect/input/demo.nomadnomad job stop countdashdocker pssudo iptables -t nat -L -v -nnomad job run ./e2e/connect/input/demo.nomadsudo systemctl stop nomad.docker pssudo iptables -t nat -L -v -ndocker rm -f $(docker ps -a)sudo systemctl start nomaddocker psnomad job stop countdashsudo iptables -t nat -L -v -nLogs
iptables after repro steps
cc @davemay99 @angrycub as a heads up
The text was updated successfully, but these errors were encountered: