Calculate partition boundaries for read_sql using sampling
#3245
Assignees
Labels
enhancement
New feature or request
Comments
colin-ho
added a commit
that referenced
this issue
Nov 13, 2024
…ax (#3246) Currently, read_sql calculates partition bounds using the `PERCENTILE_DISC` function. However, this function does not scale well to large tables, as it is an expensive window + sort function. A better alternative is to take samples, then estimate partition bounds, as described in this issue: #3245. In the meantime, we should default to using the min-max calculations instead, which was previously the fallback option. --------- Co-authored-by: Colin Ho <colinho@Colins-MacBook-Pro.local> Co-authored-by: Colin Ho <colinho@Colins-MBP.localdomain>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem?
Currently,
read_sqlcalculates partition boundaries usingPERCENTILE_DISC, falling back to using min - max if needed.PERCENTILE_DISCis not scalable, as it involves expensive operations such as windowing and sorting.Describe the solution you'd like
We should instead calculate percentiles by taking samples from the input table. This will allows trade off btw accuracy and computational complexity (both time and space complexities). The parameter to control this trade off is the sampling size. The large the sampling size, the more accurate, but it will be more computational complex. If smaller sampling size will be more likely to have uneven sized partition (skew).
Describe alternatives you've considered
No response
Additional Context
No response
Would you like to implement a fix?
No
The text was updated successfully, but these errors were encountered: