docker: fix bug where network pause containers would be erroneously reconciled #16352
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shoenig
force-pushed
the
pause-reconciliation
branch
from
359b9b2
to
dc84d23
Compare
shoenig
changed the title
shoenig
force-pushed
the
pause-reconciliation
branch
from
dc84d23
to
aea6e83
Compare
tgross
reviewed
Mar 6, 2023
tgross
approved these changes
Mar 7, 2023
| return | ||
| } | ||
|
|
||
| containers, listErr := dockerClient.ListContainers(docker.ListContainersOptions{ |
There was a problem hiding this comment.
ListContainersOptions has a Context field we could thread from the Driver.ctx. This is one of the places for the Docker client where it seems unambiguously correct to bail out.
|
how soon can we get a release with this fix? |
|
Hi @gbolo we're currently expecting to release v1.5.1 with this fix, plus a few other important things on Monday the 13th. |
|
@shoenig will this be back ported to v1.4.x? Seems like https://github.com/hashicorp/nomad/blob/v1.4.5/drivers/docker/network.go#L128 |
shoenig
added
backport/1.4.x
|
yes, good catch @gbolo! |
philrenaud
pushed a commit
that referenced
this pull request
Mar 14, 2023
…econciled (#16352) * docker: fix bug where network pause containers would be erroneously gc'd * docker: cl: thread context from driver into pause container restoration
tgross
added a commit
that referenced
this pull request
Mar 29, 2023
When we added recovery of pause containers in #16352 we called the recovery function from the plugin factory function. But in our plugin setup protocol, a plugin isn't ready for use until we call `SetConfig`. This meant that recovering pause containers was always done with the default config. Setting up the Docker client only happens once, so setting the wrong config in the recovery function also means that all other Docker API calls will use the default config. Move the `recoveryPauseContainers` call into the `SetConfig`. Fix the error handling so that we return any error but also don't log when the context is canceled, which happens twice during normal startup as we fingerprint the driver.
tgross
added a commit
that referenced
this pull request
Mar 29, 2023
When we added recovery of pause containers in #16352 we called the recovery function from the plugin factory function. But in our plugin setup protocol, a plugin isn't ready for use until we call `SetConfig`. This meant that recovering pause containers was always done with the default config. Setting up the Docker client only happens once, so setting the wrong config in the recovery function also means that all other Docker API calls will use the default config. Move the `recoveryPauseContainers` call into the `SetConfig`. Fix the error handling so that we return any error but also don't log when the context is canceled, which happens twice during normal startup as we fingerprint the driver.
tgross
added a commit
that referenced
this pull request
Mar 29, 2023
When we added recovery of pause containers in #16352 we called the recovery function from the plugin factory function. But in our plugin setup protocol, a plugin isn't ready for use until we call `SetConfig`. This meant that recovering pause containers was always done with the default config. Setting up the Docker client only happens once, so setting the wrong config in the recovery function also means that all other Docker API calls will use the default config. Move the `recoveryPauseContainers` call into the `SetConfig`. Fix the error handling so that we return any error but also don't log when the context is canceled, which happens twice during normal startup as we fingerprint the driver.
tgross
added a commit
that referenced
this pull request
Mar 29, 2023
When we added recovery of pause containers in #16352 we called the recovery function from the plugin factory function. But in our plugin setup protocol, a plugin isn't ready for use until we call `SetConfig`. This meant that recovering pause containers was always done with the default config. Setting up the Docker client only happens once, so setting the wrong config in the recovery function also means that all other Docker API calls will use the default config. Move the `recoveryPauseContainers` call into the `SetConfig`. Fix the error handling so that we return any error but also don't log when the context is canceled, which happens twice during normal startup as we fingerprint the driver.
This was referenced Mar 29, 2023
Merged
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds tracking of pause containers to the docker driver, fixing a bug introduced by #15898 where the containers are now subject to dangling container reconciliation. The pause container created for allocs with docker tasks making use of bridge networking is not created in the same flow as a normal Task - which have a
TaskHandlestate. The set of tasks not to reconcile was identified by scanning the set of these states, which does not include pause containers.To remedy this, this PR now tracks pause containers in their own little store. Since the Nomad Client may be restarted, we scan existing running containers on startup to reload the store from existing running containers.
Fixes: #16338