You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
GpuShuffleExchangeExec was moved into the v2 shim style in #3559, but this class appears in different packages depending upon which shim is being used:
This is working today because this class isn't referenced only in code within the specific shim which uses the appropriate package path. If we ever attempt to commonize code that references this class it will break the build since common code can't directly reference the shimmed class via a single package path (which is the main advantage of the v2 style shim).
The text was updated successfully, but these errors were encountered:
Another issue:
org.apache.spark.MapOutputStatistics can't be accessed from outside of [spark] package
private[spark] class MapOutputStatistics
Is the following solution ok?
create one class named "GpuShuffleExchangeExecBase" in org.apache.spark package and create "GpuShuffleExchangeExec" which extends the base class in the expected package org.apache.spark.sql.rapids.shims.v2
create one class named "GpuShuffleExchangeExecBase" in org.apache.spark package
There's already an org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase which isn't shimmed, but I don't think that is going to help here. The Databricks version, and only the Databricks version, needs to override a method that has MapOutputStatistics in the method signature. That means there's no choice but to have that class be somewhere in the org.apache.spark.... hierarchy to access that class.
I think the proper solution is to move all the GpuShuffleExchangeExec shim v2 classes to org.apache.spark.rapids.shims.v2 so they're consistent. Yes, most implementations won't need to be in that package, but I think it's more important for the class paths to be consistent across shims in the v2 shim approach.
GpuShuffleExchangeExec
was moved into the v2 shim style in #3559, but this class appears in different packages depending upon which shim is being used:This is working today because this class isn't referenced only in code within the specific shim which uses the appropriate package path. If we ever attempt to commonize code that references this class it will break the build since common code can't directly reference the shimmed class via a single package path (which is the main advantage of the v2 style shim).
The text was updated successfully, but these errors were encountered: