Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

colexec: external sort doesn't respect the memory limit well #60017

Closed
yuzefovich opened this issue Feb 9, 2021 · 0 comments · Fixed by #60593
Closed

colexec: external sort doesn't respect the memory limit well #60017

yuzefovich opened this issue Feb 9, 2021 · 0 comments · Fixed by #60593
Assignees
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-cleanup Tech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.

Comments

@yuzefovich
Copy link
Member

Currently, the external sort respects the memory limit very poorly - it will hold up to maxNumberPartitions partitions in memory (within each in-memory sorter) each of which is limited by roughly workmem - 1MB, maxNumberPartitions is limited by COCKROACH_VEC_MAX_OPEN_FDS divided by 16 (which equals 256 / 16 = 16 by default), so the external sorter actually might be using on the order of 1 GB of RAM.

In the worst case, however, when the rows are very wide (let's say 1MB in size), then each batch will be on the order of 1GB in size, resulting in the external sorter using something like 16GB of RAM which is likely to lead to OOM crashes. This latter part will be mediated by limiting the batch sizes by workmem setting (#59851).

@yuzefovich yuzefovich added the C-cleanup Tech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior. label Feb 9, 2021
@yuzefovich yuzefovich self-assigned this Feb 9, 2021
@yuzefovich yuzefovich added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Feb 9, 2021
@craig craig bot closed this as completed in 889c27a Feb 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-cleanup Tech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant