You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
By accident, I found that TPC-H Q21 can run 200ms faster sometimes on my server. I felt curious about it because my server had almost no noise.
After some investigation, I found that the 200ms improvement was in the first phase of hash agg. After digging a little deeper, I found the reason.
Hash agg triggers converting hash table to two level according to two thresholds. If no fine-grained hash agg is used (always true for the first phase of hash agg) and one hash table has become to two level, any other hash table must be converted to two level before merging process.
When Q21 runs slowly here, a small number of tasks with a large amount of data trigger two levels, but most tasks do not.
Therefore, all hash tables that are not two level will be converted into two levels in a single thread.
When Q21 is getting faster by 200ms, the data distribution between different tasks is relatively even, and all hash agg build tasks are not triggered converting hash table to two level.
Considering that it is not easy to control the data distribution, I think a more general way to solve this problem is to parallelize the converting hash table to two level process after finishing hash agg build.
The text was updated successfully, but these errors were encountered:
Enhancement
By accident, I found that TPC-H Q21 can run 200ms faster sometimes on my server. I felt curious about it because my server had almost no noise.
After some investigation, I found that the 200ms improvement was in the first phase of hash agg. After digging a little deeper, I found the reason.
Hash agg triggers converting hash table to two level according to two thresholds. If no fine-grained hash agg is used (always true for the first phase of hash agg) and one hash table has become to two level, any other hash table must be converted to two level before merging process.
When Q21 runs slowly here, a small number of tasks with a large amount of data trigger two levels, but most tasks do not.
Therefore, all hash tables that are not two level will be converted into two levels in a single thread.
When Q21 is getting faster by 200ms, the data distribution between different tasks is relatively even, and all hash agg build tasks are not triggered converting hash table to two level.
Considering that it is not easy to control the data distribution, I think a more general way to solve this problem is to parallelize the converting hash table to two level process after finishing hash agg build.
The text was updated successfully, but these errors were encountered: