You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
yuzefovich opened this issue
Feb 9, 2021
· 0 comments
· Fixed by #60593
Assignees
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.C-cleanupTech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.
Currently, the external sort respects the memory limit very poorly - it will hold up to maxNumberPartitions partitions in memory (within each in-memory sorter) each of which is limited by roughly workmem - 1MB, maxNumberPartitions is limited by COCKROACH_VEC_MAX_OPEN_FDS divided by 16 (which equals 256 / 16 = 16 by default), so the external sorter actually might be using on the order of 1 GB of RAM.
In the worst case, however, when the rows are very wide (let's say 1MB in size), then each batch will be on the order of 1GB in size, resulting in the external sorter using something like 16GB of RAM which is likely to lead to OOM crashes. This latter part will be mediated by limiting the batch sizes by workmem setting (#59851).
The text was updated successfully, but these errors were encountered:
yuzefovich
added
the
C-cleanup
Tech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.
label
Feb 9, 2021
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.C-cleanupTech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.
Currently, the external sort respects the memory limit very poorly - it will hold up to
maxNumberPartitions
partitions in memory (within each in-memory sorter) each of which is limited by roughlyworkmem - 1MB
,maxNumberPartitions
is limited byCOCKROACH_VEC_MAX_OPEN_FDS
divided by 16 (which equals256 / 16 = 16
by default), so the external sorter actually might be using on the order of 1 GB of RAM.In the worst case, however, when the rows are very wide (let's say 1MB in size), then each batch will be on the order of 1GB in size, resulting in the external sorter using something like 16GB of RAM which is likely to lead to OOM crashes. This latter part will be mediated by limiting the batch sizes by
workmem
setting (#59851).The text was updated successfully, but these errors were encountered: