-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-18134][SQL] Comparable MapTypes [POC] #15970
Conversation
cc @cloud-fan we discussed something like this earlier today |
Test build #68968 has finished for PR 15970 at commit
|
Test build #69003 has finished for PR 15970 at commit
|
# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
Test build #69023 has finished for PR 15970 at commit
|
Test build #69136 has finished for PR 15970 at commit
|
Test build #69141 has finished for PR 15970 at commit
|
Any update on this? |
cc @hvanhovell |
Is there any updates on this PR? This feature is quite useful to us. |
@hvanhovell Are you still working on this? I think this is feature is useful :) |
@hvanhovell We still need to keep this pr open? Either way, we need rework based on this pr. If so, can you close this for now? |
Test build #97829 has started for PR 15970 at commit |
Test build #97847 has started for PR 15970 at commit |
Test build #97859 has started for PR 15970 at commit |
Build finished. Test FAILed. |
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
What changes were proposed in this pull request?
This is a small POC to see if we can make MapType orderable, and thus usable in aggregates and joins. The key idea in this PR is that there is a difference between an unordered and an ordered map (an ordered map is can be compared), and that
ordered
is a property ofMapType
.A map can be converted from an unordered map to an ordered map by injecting a
SortMap
expression. The analyzer will injectSortMap
expressions whenever we use a map in a binary comparison and when we use it in an aggregate. Note that theSortMap
expression is far from optimized, it should however perform reasonable.How was this patch tested?
No tests yet. This will probably fail tests.