-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-30036][SQL] Fix: REPARTITION hint does not work with order by #26946
Changes from all commits
97cb91d
4c9b6b1
02267db
56a1101
5915a12
fa03fcb
52ce660
d2615b6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -39,9 +39,7 @@ class ConfigBehaviorSuite extends QueryTest with SharedSparkSession { | |
def computeChiSquareTest(): Double = { | ||
val n = 10000 | ||
// Trigger a sort | ||
// Range has range partitioning in its output now. To have a range shuffle, we | ||
// need to run a repartition first. | ||
val data = spark.range(0, n, 1, 1).repartition(10).sort($"id".desc) | ||
val data = spark.range(0, n, 1, 10).sort($"id".desc) | ||
cloud-fan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
.selectExpr("SPARK_PARTITION_ID() pid", "id").as[(Int, Long)].collect() | ||
|
||
// Compute histogram for the number of records per partition post sort | ||
|
@@ -55,12 +53,12 @@ class ConfigBehaviorSuite extends QueryTest with SharedSparkSession { | |
|
||
withSQLConf(SQLConf.SHUFFLE_PARTITIONS.key -> numPartitions.toString) { | ||
// The default chi-sq value should be low | ||
assert(computeChiSquareTest() < 100) | ||
assert(computeChiSquareTest() < 10) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the physical plan is same as before, what caused this change? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. They are not same, we had two shuffles before, one was RoundRobinPartitioning, the other was RangePartitioning. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ah i see |
||
|
||
withSQLConf(SQLConf.RANGE_EXCHANGE_SAMPLE_SIZE_PER_PARTITION.key -> "1") { | ||
// If we only sample one point, the range boundaries will be pretty bad and the | ||
// chi-sq value would be very high. | ||
assert(computeChiSquareTest() > 300) | ||
assert(computeChiSquareTest() > 100) | ||
} | ||
} | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This considers a special case for OrderedDistribution. Generally, if ShuffleExchangeExec is followed by any unsatisfying distribution , we should always trim the ShuffleExchangeExec and apply the partitioning of distribution. Don't we?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sound reasonable. Any suitable cases?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just tried few possible cases, but can not have a concrete case like this. Maybe this is the only case possibly. So I think this should be fine.