-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-3666] Extract interfaces for EdgeRDD and VertexRDD #2530
Conversation
This breaks binary compatibility with the error java.lang.IncompatibleClassChangeError: Found interface org.apache.spark.graphx.VertexRDD, but class was expected The solution is to do a clean rebuild (sbt/sbt clean assembly).
QA tests have started for PR 2530 at commit
|
QA tests have finished for PR 2530 at commit
|
Test PASSed. |
}, preservesPartitioning = true)) | ||
} | ||
|
||
trait EdgeRDD[@specialized ED, VD] extends RDD[Edge[ED]] { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think we should probably remove the specialized and test performance. maybe we can do that in a a separate pr
Looks good. I didn't look closely but I'm assuming you didn't change any impl logic. Let me know if that is not true. |
QA tests have started for PR 2530 at commit
|
QA tests have finished for PR 2530 at commit
|
Test PASSed. |
@rxin Right, this just contains refactoring and doc changes. I marked the Impl constructors package-private since they are accessed by the corresponding interface object (EdgeRDD.fromEdgePartitions and VertexRDD.apply). I'll evaluate specialization separately. PTAL |
This LGTM. One question related to this - should we use abstract class instead of trait? It would be better for maintaining binary compatibility across versions |
QA tests have started for PR 2530 at commit
|
It makes no difference in benchmarks.
QA tests have started for PR 2530 at commit
|
Fixes a merge conflict created by ee29ef3. Conflicts: graphx/src/main/scala/org/apache/spark/graphx/EdgeRDD.scala
QA tests have finished for PR 2530 at commit
|
Test FAILed. |
Test build #22716 has started for PR 2530 at commit
|
QA tests have finished for PR 2530 at commit
|
Test FAILed. |
Test build #22716 has finished for PR 2530 at commit
|
Test FAILed. |
Jenkins, retest this please. |
Test build #22741 has started for PR 2530 at commit
|
Test build #22741 has finished for PR 2530 at commit
|
Test FAILed. |
Jenkins, retest this please. |
Test build #22754 has started for PR 2530 at commit
|
Test build #22754 has finished for PR 2530 at commit
|
Test FAILed. |
Test build #516 has started for PR 2530 at commit
|
Test build #516 has finished for PR 2530 at commit
|
I filed a JIRA for Mima misreporting: https://issues.apache.org/jira/browse/SPARK-4335 |
Also @ankurdave this needs to be updated now... |
Conflicts: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala
eb9635f
to
1472390
Compare
Test build #23197 has started for PR 2530 at commit
|
Test build #23197 has finished for PR 2530 at commit
|
Test FAILed. |
I took a look and I think MIMA might be reporting a semi-legitimate issue - though this is a tricky situation. I think the issue is - if someone extended the earlier version of EdgeRDD, their class would have inherited the concrete In terms of compatiblity for callers, this is a protected method so it really doesn't matter. I'd go ahead and just add ignores for this and be done with it. Fixing this in MIMA might be diffiuclt |
Test build #23254 has started for PR 2530 at commit
|
Test build #23254 has finished for PR 2530 at commit
|
Test PASSed. |
@rxin MIMA passed! PTAL. |
Merging in master & branch-1.2. Thanks! |
This discourages users from calling the VertexRDD and EdgeRDD constructor and makes it easier for future changes to ensure backward compatibility. Author: Ankur Dave <ankurdave@gmail.com> Closes #2530 from ankurdave/SPARK-3666 and squashes the following commits: d681f45 [Ankur Dave] Define getPartitions and compute in abstract class for MIMA 1472390 [Ankur Dave] Merge remote-tracking branch 'apache-spark/master' into SPARK-3666 24201d4 [Ankur Dave] Merge remote-tracking branch 'apache-spark/master' into SPARK-3666 cbe15f2 [Ankur Dave] Remove specialized annotation from VertexRDD and EdgeRDD 931b587 [Ankur Dave] Use abstract class instead of trait for binary compatibility 9ba4ec4 [Ankur Dave] Mark (Vertex|Edge)RDDImpl constructors package-private 620e603 [Ankur Dave] Extract VertexRDD interface and move implementation to VertexRDDImpl 55b6398 [Ankur Dave] Extract EdgeRDD interface and move implementation to EdgeRDDImpl (cherry picked from commit a5ef581) Signed-off-by: Reynold Xin <rxin@databricks.com>
This discourages users from calling the VertexRDD and EdgeRDD constructor and makes it easier for future changes to ensure backward compatibility.