Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce max task writer scaling defaults #18488

Merged
merged 2 commits into from
Aug 2, 2023
Merged

Reduce max task writer scaling defaults #18488

merged 2 commits into from
Aug 2, 2023

Conversation

raunaqmorarka
Copy link
Member

@raunaqmorarka raunaqmorarka commented Aug 1, 2023 •
edited
Loading

Description

A high default for task.partitioned-writer-count and task.scale-writers.max-writer-count
can lead to queries running out of memory more often

Additional context and related issues

Defaults were increased in #18005 in 422
and found to cause previously working INSERTs to OOM

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# General
* Reduce defaults for `task.partitioned-writer-count` and `task.scale-writers.max-writer-count` to reduce memory requirements of write queries. ({issue}`18488`)

@raunaqmorarka
Revert "Set value of partitionedWriterCount to availableProcessors * 2"
4a8490e 
This reverts commit 0e31c4d.
A high default for task.partitioned-writer-count can lead to queries
running out of memory more often
@raunaqmorarka
Revert "Set value of maxWriterCount to availableProcessors * 2"
ed710fc 
This reverts commit 6afbc19.
A high default for task.scale-writers.max-writer-count can lead to queries
running out of memory more often
@cla-bot cla-bot bot added the cla-signed label Aug 1, 2023
mosabua
mosabua requested changes Aug 1, 2023
View reviewed changes
Copy link
Member

@mosabua mosabua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the docs as well.

@raunaqmorarka
Copy link
Member Author

Please update the docs as well.

The original change did not update docs, so this revert already makes the code consistent with the current docs

@raunaqmorarka raunaqmorarka merged commit 0a26d53 into trinodb:master Aug 2, 2023
@raunaqmorarka raunaqmorarka deleted the reduce-writer-scale branch August 2, 2023 04:43
@raunaqmorarka raunaqmorarka mentioned this pull request Aug 2, 2023
Closed
@github-actions github-actions bot added this to the 423 milestone Aug 2, 2023
@colebow colebow mentioned this pull request Aug 2, 2023
Merged
@mosabua mosabua mentioned this pull request Aug 2, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mosabua mosabua mosabua requested changes

@findepi findepi findepi approved these changes

@gaurav8297 gaurav8297 Awaiting requested review from gaurav8297

@sopel39 sopel39 Awaiting requested review from sopel39

Assignees
No one assigned
Labels
cla-signed
Milestone
423
Development

Successfully merging this pull request may close these issues.
3 participants
@raunaqmorarka @findepi @mosabua