Skip to content

Commit

Permalink
Bump default value of max split count per task for FTE
Browse files Browse the repository at this point in the history
With limit of 256 splits per task it was possible to end up with very
small tasks and possibly even reach limit of number of tasks per stage
(2^^15). The problem appeared when splits were very small.
Task sizing should be mostly done based on split size; max
number of splits should be treated as extra limit to be reached only in
rare situations. This PR bumps default value 8x to 2048.
  • Loading branch information
losipiuk committed Sep 2, 2024
1 parent 92d301c commit 0e32d0a
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ public class QueryManagerConfig
private int faultTolerantExecutionHashDistributionWriteTaskTargetMaxCount = 2000;

private DataSize faultTolerantExecutionStandardSplitSize = DataSize.of(64, MEGABYTE);
private int faultTolerantExecutionMaxTaskSplitCount = 256;
private int faultTolerantExecutionMaxTaskSplitCount = 2048;
private DataSize faultTolerantExecutionTaskDescriptorStorageMaxMemory = DataSize.ofBytes(round(AVAILABLE_HEAP_MEMORY * 0.15));
private int faultTolerantExecutionMaxPartitionCount = 50;
private int faultTolerantExecutionMinPartitionCount = 4;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ public void testDefaults()
.setFaultTolerantExecutionHashDistributionWriteTasksToNodesMinRatio(2.0)
.setFaultTolerantExecutionHashDistributionWriteTaskTargetMaxCount(2000)
.setFaultTolerantExecutionStandardSplitSize(DataSize.of(64, MEGABYTE))
.setFaultTolerantExecutionMaxTaskSplitCount(256)
.setFaultTolerantExecutionMaxTaskSplitCount(2048)
.setFaultTolerantExecutionTaskDescriptorStorageMaxMemory(DataSize.ofBytes(round(AVAILABLE_HEAP_MEMORY * 0.15)))
.setFaultTolerantExecutionMaxPartitionCount(50)
.setFaultTolerantExecutionMinPartitionCount(4)
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/sphinx/admin/fault-tolerant-execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ properties only apply to a `TASK` retry policy.
May be overridden for the current session with the
`fault_tolerant_execution_max_task_split_count` [session
property](session-properties-definition).
- `256`
- `2048`
* - `fault-tolerant-execution-arbitrary-distribution-compute-task-target-size-growth-period`
- The number of tasks created for any given non-writer stage of arbitrary
distribution before task size is increased.
Expand Down

0 comments on commit 0e32d0a

Please sign in to comment.