Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skip adding band in meta when fetch shuffle data #2922

Merged
merged 1 commit into from
Apr 16, 2022
Merged

Skip adding band in meta when fetch shuffle data #2922

merged 1 commit into from
Apr 16, 2022

Conversation

hekaisheng
Copy link
Contributor

@hekaisheng hekaisheng commented Apr 15, 2022 •
edited
Loading

What do these changes do?

Skip adding band in meta when fetch shuffle data as shuffle produce fragmented data.

Related issue number

Fixes #xxxx

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@hekaisheng hekaisheng added type: enhancement request to be backported Indicate that the PR need to be backported to stable branch mod: storage mod: meta service labels Apr 15, 2022
@hekaisheng hekaisheng added this to the v0.9.0rc3 milestone Apr 15, 2022
@hekaisheng hekaisheng requested review from wjsi and qinxuye as code owners April 15, 2022 13:51
@hekaisheng hekaisheng force-pushed the enh/shuffle-storage branch from cede4a6 to 099daef Compare April 15, 2022 14:19
@hekaisheng hekaisheng force-pushed the enh/shuffle-storage branch from 099daef to 770e06d Compare April 15, 2022 15:43
@hekaisheng
Copy link
Contributor Author

Actually we can only remove mapper data fetched from remote workers, if they are local, it means that the mapper was executed in same worker, they are tracked by lifecycle service and may be used by other subtask, thus they can't be deleted. So I moved deletions from SubtaskProcessor to SubtaskExecutionActor, return the remote mapper keys from storage.fetch and remove them after execution.

qinxuye
qinxuye approved these changes Apr 15, 2022
View reviewed changes
Copy link
Collaborator

@qinxuye qinxuye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

wjsi
wjsi approved these changes Apr 16, 2022
View reviewed changes
Copy link
Member

@wjsi wjsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wjsi wjsi merged commit 8f55cfd into mars-project:master Apr 16, 2022
@qinxuye qinxuye deleted the enh/shuffle-storage branch April 16, 2022 13:27
wjsi pushed a commit to wjsi/mars that referenced this pull request May 5, 2022
@hekaisheng @wjsi
Skip adding band when fetch shuffle data (mars-project#2922)
b0796c7
@wjsi wjsi mentioned this pull request May 5, 2022
Merged
wjsi added a commit that referenced this pull request May 5, 2022
@wjsi @hekaisheng
[BACKPORT] Skip adding band when fetch shuffle data (#2922) (#2993)
7601ea6 
Co-authored-by: He Kaisheng <heks93@163.com>
@qinxuye qinxuye added backported already PR has been backported and removed to be backported Indicate that the PR need to be backported to stable branch labels May 6, 2022
chaokunyang pushed a commit to chaokunyang/mars that referenced this pull request May 31, 2022
@hekaisheng
Skip adding band when fetch shuffle data (mars-project#2922)
ae8906e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wjsi wjsi wjsi approved these changes

@qinxuye qinxuye qinxuye approved these changes

Assignees
No one assigned
Labels
backported already PR has been backported mod: meta service mod: storage type: enhancement request
Projects
None yet
Milestone
v0.9.0rc3
Development

Successfully merging this pull request may close these issues.
3 participants
@hekaisheng @qinxuye @wjsi