-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
failed pause containers result in failed task restarts and orphaned tasks #12216
Comments
Hi @RosieBaish! I was able to more-or-less confirm the behavior you're seeing, and as it turns out the behavior on client restart is only a symptom of a different problem. Let's use the following jobspec, so that we have a fairly simple application without a bridge network that requires the pause container ( jobspecjob "example" {
datacenters = ["dc1"]
group "web" {
network {
mode = "bridge"
port "www" {
to = 8001
}
}
task "http" {
driver = "docker"
config {
image = "busybox:1"
command = "httpd"
args = ["-v", "-f", "-p", "8001", "-h", "/local"]
ports = ["www"]
}
template {
data = "<html>hello, world</html>"
destination = "local/index.html"
}
resources {
cpu = 128
memory = 128
}
}
}
} If we run this job and look at the resulting containers we see the main container and the pause container:
Now let's kill the pause container:
Note that if we wait a few moments, we can see that Nomad isn't recreating the pause container, which at first glance isn't a problem:
But it is a problem because if the main container fails, there's no pause container to owns the network namespace anymore (this is a quirk of Docker that a particular container needs to "own" the namespace that we join other containers to, and Nomad inherits that quirk):
Nomad will detect this condition and reschedule the allocation:
Now let's try to reproduce the client restart behavior you saw. I'll head over to the new alloc and kill the pause container there:
Then I restarted the Nomad client and got the following:
But if I wait a moment, the pause container fails, leaving just the old container running:
What happened? Let's ask the allocation:
The full error state from the client logs is as follows:
So there's two issues here:
@RosieBaish there's definitely a bug here but I'm going to need to take it back to the team to figure out a path forward and to set aside some time for us to figure out a fix. I'm going to mark this issue for roadmapping. Thanks so much for the thourough bug report! |
I believe we can effectively minimize this failure scenario by configuring the pause container with an auto-restart policy. Technically there will still be a small window during the restart where if the main task fails it will not be able to join the network namespace. But if your pause container and main task fail at the same time, most likely something has gone very sideways on the node that Nomad isn't going to be able to handle anyway. The reality is docker's handling of network namespaces is very poor - if surviving this scenario is important than docker is not a suitable tool. |
This PR modifies the configuration of the networking pause contaier to include the "unless-stopped" restart policy. The pause container should always be restored into a running state until Nomad itself issues a stop command for the container. This is not a _perfect_ fix for #12216 but it should cover the 99% use case - where a pause container gets accidently stopped / killed for some reason. There is still a possibility where the pause container and main task container are stopped and started in the order where the bad behavior persists, but this is fundamentally unavoidable due to how docker itself abstracts and manages the underlying network namespace referenced by the containers. Closes #12216
This PR modifies the configuration of the networking pause contaier to include the "unless-stopped" restart policy. The pause container should always be restored into a running state until Nomad itself issues a stop command for the container. This is not a _perfect_ fix for #12216 but it should cover the 99% use case - where a pause container gets accidently stopped / killed for some reason. There is still a possibility where the pause container and main task container are stopped and started in the order where the bad behavior persists, but this is fundamentally unavoidable due to how docker itself abstracts and manages the underlying network namespace referenced by the containers. Closes #12216
This PR modifies the configuration of the networking pause contaier to include the "unless-stopped" restart policy. The pause container should always be restored into a running state until Nomad itself issues a stop command for the container. This is not a _perfect_ fix for #12216 but it should cover the 99% use case - where a pause container gets accidently stopped / killed for some reason. There is still a possibility where the pause container and main task container are stopped and started in the order where the bad behavior persists, but this is fundamentally unavoidable due to how docker itself abstracts and manages the underlying network namespace referenced by the containers. Closes #12216
This PR modifies the configuration of the networking pause contaier to include the "unless-stopped" restart policy. The pause container should always be restored into a running state until Nomad itself issues a stop command for the container. This is not a _perfect_ fix for #12216 but it should cover the 99% use case - where a pause container gets accidently stopped / killed for some reason. There is still a possibility where the pause container and main task container are stopped and started in the order where the bad behavior persists, but this is fundamentally unavoidable due to how docker itself abstracts and manages the underlying network namespace referenced by the containers. Closes #12216
This PR modifies the configuration of the networking pause contaier to include the "unless-stopped" restart policy. The pause container should always be restored into a running state until Nomad itself issues a stop command for the container. This is not a _perfect_ fix for #12216 but it should cover the 99% use case - where a pause container gets accidently stopped / killed for some reason. There is still a possibility where the pause container and main task container are stopped and started in the order where the bad behavior persists, but this is fundamentally unavoidable due to how docker itself abstracts and manages the underlying network namespace referenced by the containers. Closes #12216
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
$ nomad --version
Nomad v1.2.3 (a79efc8)
Operating system and Environment details
$ uname -a
Linux 5.15.13 #3 SMP Wed Jan 26 09:09:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="35 (Thirty Five)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Thirty Five)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
DEFAULT_HOSTNAME=localhost
Issue
When using the docker driver, if one of the
nomad_init_<UUID>
containers dies, the other containers in the task group do not get cleaned up.We discovered this while doing some general testing around "What happens if you kill random containers in random combinations"
I'm aware that a nomad_init_ container failing in production is highly unlikely, but I thought it was worth the report anyway!
Reproduction steps
Create and run a task group as below
$ docker container ls
3 containers,
name-<UUID>
,connect-proxy-name-<UUID>
andnomad_init_<UUID>
$ docker stop $(id of nomad_init_)
$ docker container ls
2 containers,
name-<UUID>
,connect-proxy-name-<UUID>
$ systemctl restart nomad
$ docker container ls
Expected Result
3 containers,
name-<UUID>
,connect-proxy-name-<UUID>
andnomad_init_<UUID>
Where UUID is either the same as before, or a new one
Actual Result
5 containers,
name-<new_UUID>
connect-proxy-name-<new_UUID>
nomad_init_<new_UUID>
name-<old_UUID>
connect-proxy-name-<old_UUID>
Nomad has created a new set of containers to replace the task group, but has not cleaned up the existing 2. I would expect it either to re-use them and thus re-create the nomad_init_ container, or delete them and replace them.
Job file (if appropriate)
The docker image is a local one pulled from a private repo but I suspect the exact image is irrelevant.
Logs
I can provide logs if you're unable to reproduce, but it'll take a while to redact everything so if you can reproduce it I won't bother
The text was updated successfully, but these errors were encountered: