scheduler: Debounce commit events #1287
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When loading a state that contained large numbers of nodes and tasks, but no ready nodes that could accept the tasks, swarmd used large amounts of CPU repeatedly trying to schedule the full set of tasks. The allocator caused many commits on startup (see moby#1286), and this produced a large backlog of commit events, each one of which caused a full scheduling pass. To avoid this pathological behavior, debounce the commit events similarly to how the dispatcher's Tasks loop debounces events. When a commit event is received, that starts a 50 ms countdown to wait for another commit event before running the scheduling pass. If commit events keep being received and resetting this timer, the scheduler will run the scheduling pass anyway after a second. Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Current coverage is 54.95% (diff: 74.19%)@@ master #1287 diff @@
==========================================
Files 78 78
Lines 12428 12454 +26
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 6826 6844 +18
- Misses 4668 4672 +4
- Partials 934 938 +4
|
|
LGTM |
|
In the future, we should try to use https://godoc.org/golang.org/x/time/rate for rate limiting. |
|
I filed #1290 about generalizing the rate limiting. Thanks for the pointer - I'll look into this when I get to working on it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When loading a state that contained large numbers of nodes and tasks,
but no ready nodes that could accept the tasks, swarmd used large
amounts of CPU repeatedly trying to schedule the full set of tasks. The
allocator caused many commits on startup (see #1286), and this produced
a large backlog of commit events, each one of which caused a full
scheduling pass.
To avoid this pathological behavior, debounce the commit events
similarly to how the dispatcher's Tasks loop debounces events. When a
commit event is received, that starts a 50 ms countdown to wait for
another commit event before running the scheduling pass. If commit
events keep being received and resetting this timer, the scheduler will
run the scheduling pass anyway after a second.
I think we should consider this for 1.12.1. Without this change, the scheduler can consume 100% CPU for a very long time trying to catch up with the event backlog.
cc @dongluochen