Batch more Scheduler sends #4526
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
jakirkham
force-pushed
the
batch_more_sends
branch
5 times, most recently
from
e4aff27 to
eb42f3b
Compare
jakirkham
changed the title
jakirkham
force-pushed
the
batch_more_sends
branch
from
eb42f3b to
b71038f
Compare
|
Planning on merging end of day Tuesday if no comments |
jakirkham
force-pushed
the
batch_more_sends
branch
5 times, most recently
from
8edf0f9 to
3e02e1b
Compare
Also clear it out after running `transitions` as that should have handled all of them.
Track messages to send from transition calls in `handle_task_finished` and `stimulus_task_finished` to allow sending all of these in one go. Should allow the transition work to happen with fewer interruptions and send more data at once.
jakirkham
force-pushed
the
batch_more_sends
branch
from
3e02e1b to
93ad1ed
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Factors out
_transitionsfromtransitionsto allow collecting of messages to send for the caller to handle as opposed to sending immediately. This then leveraged in a few stimulus and handle functions to collect all messages and send in one batch. Also this is leveraged inadd_workerandgather. All of which makes a few calls totransitionortransitions. The result is in these cases we should send fewer messages.black distributed/flake8 distributed