Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Assign reducer ops in task assigner to make them more balanced across cluster #3048

Merged
merged 2 commits into from
May 18, 2022
Merged

Assign reducer ops in task assigner to make them more balanced across cluster #3048

merged 2 commits into from
May 18, 2022

Conversation

qinxuye
Copy link
Collaborator

@qinxuye qinxuye commented May 18, 2022

What do these changes do?

Related issue number

Fixes #xxxx

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@qinxuye qinxuye added type: enhancement request to be backported Indicate that the PR need to be backported to stable branch mod: task service labels May 18, 2022
@qinxuye qinxuye added this to the v0.10.0a1 milestone May 18, 2022
@qinxuye qinxuye requested a review from a team as a code owner May 18, 2022 08:01
wjsi
wjsi reviewed May 18, 2022
View reviewed changes
Copy link
Member

@wjsi wjsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM over all, I left a comment.

mars/services/task/analyzer/analyzer.py
# make reassign_worker True etc.
return op.reassign_worker or (
isinstance(op, MapReduceOperand) and op.stage == OperandStage.reduce
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rename reassign_worker as _reassign_worker and create a property with default value given different operand classes instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reassign_worker is recorded on SchedulingHint instead of operand itself, we cannot implement such a getter in this way.

wjsi
wjsi approved these changes May 18, 2022
View reviewed changes
Copy link
Member

@wjsi wjsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

hekaisheng
hekaisheng approved these changes May 18, 2022
View reviewed changes
Copy link
Contributor

@hekaisheng hekaisheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hekaisheng hekaisheng merged commit 08d7cfe into mars-project:master May 18, 2022
@qinxuye qinxuye deleted the enh/assign-reducer branch May 18, 2022 13:05
qinxuye pushed a commit to qinxuye/mars that referenced this pull request May 24, 2022
Assign reducer ops in task assigner to make them more balanced across…
18c0014 
… cluster (mars-project#3048)

(cherry picked from commit 08d7cfe)
@qinxuye qinxuye mentioned this pull request May 24, 2022
Merged
wjsi pushed a commit that referenced this pull request May 24, 2022
[BACKPORT] Assign reducer ops in task assigner to make them more bala…
433ce03 
…nced across cluster (#3048) (#3075)
@wjsi wjsi added backported already PR has been backported and removed to be backported Indicate that the PR need to be backported to stable branch labels May 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hekaisheng hekaisheng hekaisheng approved these changes

@wjsi wjsi wjsi approved these changes

Assignees
No one assigned
Labels
backported already PR has been backported mod: task service type: enhancement request
Projects
None yet
Milestone
v0.10.0a1
Development

Successfully merging this pull request may close these issues.
3 participants
@qinxuye @wjsi @hekaisheng