Fix an race condition in agent #737
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nmeyerhans
suggested changes
Mar 23, 2017
agent/engine/task_manager.go
Outdated
| // Now remove ourselves from the global state and cleanup channels | ||
| mtask.engine.processTasks.Lock() | ||
| mtask.engine.state.RemoveTask(mtask.Task) | ||
| seelog.Debug("Finished removing task data; removing from state no longer managing", "task", mtask.Task) |
There was a problem hiding this comment.
We need to update the format string when calling seelog directly. I think we should also update the content of the string itself; it seems a bit confusing to me. Maybe something like:
seelog.Debugf("Finished removing task data, no longer managing task %v", mtask.Task)
richardpen
force-pushed
the
race-condition
branch
from
9978c20
to
1a35527
Compare
aaithal
requested changes
Mar 31, 2017
richardpen
force-pushed
the
race-condition
branch
from
1a35527
to
42f7427
Compare
samuelkarp
approved these changes
Apr 13, 2017
aaithal
approved these changes
Apr 13, 2017
richardpen
force-pushed
the
race-condition
branch
from
42f7427
to
573e1ff
Compare
The task state in dockerstate and managed task in task manager are modified when task is added and cleaned up, both are protected by processTasks.Lock. But in cleanup the task state is modified outside the lock which could cause inconsistent state of the task in rare case.
richardpen
force-pushed
the
race-condition
branch
from
573e1ff
to
e6841f0
Compare
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
There is a race condition when task is cleaning up, the task state can get inconsistent with the managed task in some edge case. Because these two are updated one inside the lock and one outside the lock when clean up the task.
Implementation details
Testing
make release)go build -out amazon-ecs-agent.exe ./agent)make test) passgo test -timeout=25s ./agent/...) passmake run-integ-tests) pass.\scripts\run-integ-tests.ps1) passmake run-functional-tests) pass.\scripts\run-functional-tests.ps1) passNew tests cover the changes:
Description for the changelog
Fixed a potential race condition in agent where could cause agent in corrupted state.
Licensing
This contribution is under the terms of the Apache 2.0 License:
yes