Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

preallocated memory for shuffle split #588

Closed
FelixYBW opened this issue Nov 29, 2021 · 1 comment
Closed

preallocated memory for shuffle split #588

FelixYBW opened this issue Nov 29, 2021 · 1 comment
Labels
bug Something isn't working

Comments

@FelixYBW
Copy link
Collaborator

Currently we pre-allocate memory for each reducer before split, the size is batch size. So the total memory allocated increases along with batch size and partitions. In last OOM definitely happens
@FelixYBW FelixYBW added the bug Something isn't working label Nov 29, 2021
@github-actions github-actions bot mentioned this issue Nov 30, 2021
Merged
@FelixYBW FelixYBW closed this as completed Dec 7, 2021
@FelixYBW
Copy link
Collaborator Author

FelixYBW commented Dec 7, 2021

Fixed PR598

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@FelixYBW