Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize mars supervisor scheduling #2325

Merged
merged 6 commits into from
Aug 17, 2021
Merged

Optimize mars supervisor scheduling #2325

merged 6 commits into from
Aug 17, 2021

Conversation

chaokunyang
Copy link
Contributor

What do these changes do?

Fix supervisor hang:

  • Before this pr, if we have 1000 chunks need to set_subtask_result and each take 1 second, we'll need to wait 1000 seconds before we can schedule next subtask graph.
  • Big sutask graph generation may take more than 3 minutes, this will block event loop 3 minutes. This PR moved the graph preparation into another thread to avoid event loop hang.

Related issue number

@chaokunyang
Copy link
Contributor Author

Remove chunks time:
image

@chaokunyang
Copy link
Contributor Author

generate subtask graph time:
image
image

@chaokunyang chaokunyang changed the title Optimize mars supervisor schedeling Optimize mars supervisor scheduling Aug 12, 2021
@qinxuye qinxuye added mod: task service type: enhancement request mod: lifecycle service to be backported Indicate that the PR need to be backported to stable branch labels Aug 13, 2021
@qinxuye qinxuye added this to the v0.8.0a2 milestone Aug 13, 2021
@chaokunyang
Copy link
Contributor Author

The duration for graph generation by this pr:
image

@chaokunyang chaokunyang force-pushed the optimize_mars_supervisor_schedeling branch from 384ec30 to b5149dc Compare August 16, 2021 07:08
@chaokunyang chaokunyang force-pushed the optimize_mars_supervisor_schedeling branch from b5149dc to 0b0edd3 Compare August 16, 2021 09:01
qinxuye
qinxuye approved these changes Aug 17, 2021
View reviewed changes
Copy link
Collaborator

@qinxuye qinxuye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

wjsi
wjsi approved these changes Aug 17, 2021
View reviewed changes
Copy link
Member

@wjsi wjsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wjsi wjsi merged commit 1bc1ee9 into mars-project:master Aug 17, 2021
wjsi pushed a commit to wjsi/mars that referenced this pull request Aug 17, 2021
@chaokunyang @wjsi
Optimize scheduling service on supervisors (mars-project#2325)
c97f5cf
@wjsi wjsi mentioned this pull request Aug 17, 2021
Merged
qinxuye pushed a commit that referenced this pull request Aug 17, 2021
@wjsi
[BACKPORT] Optimize scheduling service on supervisors (#2325) (#2347)
a98015e
@qinxuye qinxuye added backported already PR has been backported and removed to be backported Indicate that the PR need to be backported to stable branch labels Aug 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wjsi wjsi wjsi approved these changes

@qinxuye qinxuye qinxuye approved these changes

@hekaisheng hekaisheng Awaiting requested review from hekaisheng

Assignees
No one assigned
Labels
backported already PR has been backported mod: lifecycle service mod: task service type: enhancement request
Projects
None yet
Milestone
v0.8.0a2
Development

Successfully merging this pull request may close these issues.
3 participants
@chaokunyang @qinxuye @wjsi