[executor] Introduce on-the-fly scheduling for Read/Allocate/Write keys #814
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wlawt
changed the title
wlawt
force-pushed
the
scheduler-v3
branch
3 times, most recently
from
a6d133a
to
37f4d5f
Compare
wlawt
commented
Apr 4, 2024
2 tasks
wlawt
commented
Apr 10, 2024
wlawt
commented
Apr 16, 2024
* random perm per conflict key * fix dont block on ourself and edit TestTwoConflictKeys to track this * keep reading/readers relationship * make reading a map * comments * clarify use of map for reading * simplify rng for conflict and unique perm
executor/executor.go
Outdated
| // maxDependencies must be greater than the maximum number of dependencies | ||
| // any single task could have. This is used to ensure a dependent task | ||
| // does not begin executing a task until all dependencies have been enqueued. | ||
| const maxDependencies = 100_000_000 |
There was a problem hiding this comment.
I think the last tweak we should make is to make this a param you provide in New. I think this const is "jank".
wlawt
changed the title
wlawt
changed the title
executor/executor.go
Outdated
| // It is assumed that no single task has more than [maxDependencies]. If | ||
| // this invariant is violated, some tasks will never execute and this code | ||
| // could deadlock. | ||
| // It is assumed that no single task has more than [maxDependencies]. |
There was a problem hiding this comment.
This comment should be above New imo.
patrick-ogrady
approved these changes
Apr 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #701, #709 .
Overview
The executor keeps track of a dependency graph to determine if a
taskis executable or not.Properties of the dependency graph:
taskand the type of access its requesting for that key (Read/Allocate/Write).Test Plan
executor_test.go, coverage is ~97.5%executor_test.gopass