dependencygraph: Enforce shutdown order #1866
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yhlee-aws
reviewed
Feb 18, 2019
petderek
force-pushed
the
container-ordering-feature
branch
from
435b6b5 to
6ec0db1
Compare
ubhattacharjya
approved these changes
Feb 19, 2019
petderek
force-pushed
the
container-ordering-feature
branch
3 times, most recently
from
c983940 to
b33029c
Compare
|
are the ft failures due to timeouts? |
|
Great question. They were passing before i rebased 😖 |
petderek
force-pushed
the
container-ordering-feature
branch
4 times, most recently
from
492919c to
0d7f7fe
Compare
|
Nevermind, it was panicking. I'm looking into it |
We now apply shutdown order in any dependency case, including
dependsOn directives, links, or volumes. What this means is that
agent will now make a best effort attempt to shutdown containers
in the inverse order they were created.
For example, a container using a link for communication will wait
until the linked container has terminated before terminating
itself. Likewise, a container named in another container's
dependsOn directive will wait until that dependent container
terminates.
One note about the current implementation is that the dependencies
aren't assumed to be transitive, so if a chain exists such that:
A -> B -> C
Container "C" will shutdown before "B", but it won't check status
against container "A" explicity. If A depends on C, we expect:
A -> B -> C
A -> C
The lack of transitive dependency logic applies is consistent with
startup order as of this commit.
The link / volume dependency tests are now affected by shutdown order, so the tests now take longer. Previously, it would take a max of 30s (the default docker stop timeout for agent). Now, since the containers stop in order, it will take a max of 30s * n, where n is the number of containers. Increasing the test timeout is a short term fix until we have granular start/stop timeouts plumbed through the ecs service.
Instead of explicitly checking against many conditions, we now validate that the expected condition has progressed beyond started This mirrors prior behavior in the codebase, and reduces cyclo complexity.
petderek
force-pushed
the
container-ordering-feature
branch
from
0d7f7fe to
89bb2e8
Compare
|
Logdriver test is failing suddenly. I've made a note to fix the test in #1835 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Enforce shutdown ordering within a task.
Implementation details
We now apply shutdown order in any dependency case, including
dependsOn directives, links, or volumes. What this means is that
agent will now make a best effort attempt to shutdown containers
in the inverse order they were created.
For example, a container using a link for communication will wait
until the linked container has terminated before terminating
itself. Likewise, a container named in another container's
dependsOn directive will wait until that dependent container
terminates.
One note about the current implementation is that the dependencies
aren't assumed to be transitive, so if a chain exists such that:
A -> B -> C
Container "C" will shutdown before "B", but it won't check status
against container "A" explicity. If A depends on C, we expect:
A -> B -> C
A -> C
The lack of transitive dependency logic applies is consistent with
startup order as of this commit.
Testing
Added additional tests to cover the shutdown use cases.
Licensing
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.