You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
When we support broadcast exchanges #342 we can transform certain joins to utilize it.
Describe the solution you'd like
Currently all plans involving hash joins look like the following.
HashJoin <- RemoteExchange (partitioned) <- build side input
<- RemoteExchange (partitioned) <- probe side input
When the build side is small (e.g. Spark uses 10MB * number of partitions for this by default - but generally bigger can help as well in my experience).
The new plan after optimization looks like this (note the missing exchange in the probe side, that side doesn't require shuffling now)
HashJoin <- BroadcastExchange <- build side input
<- probe side input
Describe alternatives you've considered
Implement the (physical) optimization rule. The rule should run after the HashBuildProbeOrder rule from DataFusion.
Additional context
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
When we support broadcast exchanges #342 we can transform certain joins to utilize it.
Describe the solution you'd like
Currently all plans involving hash joins look like the following.
When the build side is small (e.g. Spark uses 10MB * number of partitions for this by default - but generally bigger can help as well in my experience).
The new plan after optimization looks like this (note the missing exchange in the probe side, that side doesn't require shuffling now)
Describe alternatives you've considered
Implement the (physical) optimization rule. The rule should run after the
HashBuildProbeOrder
rule from DataFusion.Additional context
The text was updated successfully, but these errors were encountered: