-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-2086] Improve output of toDebugString to make shuffle boundaries more clear #1364
Conversation
QA tests have started for PR 1364. This patch merges cleanly. |
QA results for PR 1364: |
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16532/ |
Jenkins, retest this please. |
QA tests have started for PR 1364. This patch merges cleanly. |
QA results for PR 1364: |
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16534/ |
Jenkins, retest this please. |
QA tests have started for PR 1364. This patch DID NOT merge cleanly! |
QA results for PR 1364: |
Jenkins, retest this please. |
QA tests have started for PR 1364. This patch merges cleanly. |
QA results for PR 1364: |
Jenkins, retest this please. |
1 similar comment
Jenkins, retest this please. |
QA tests have started for PR 1364. This patch merges cleanly. |
QA results for PR 1364: |
I've merged this - thanks Greg! |
…es more clear Changes RDD.toDebugString() to show hierarchy and shuffle transformations more clearly New output: ``` (3) FlatMappedValuesRDD[325] at apply at Transformer.scala:22 | MappedValuesRDD[324] at apply at Transformer.scala:22 | CoGroupedRDD[323] at apply at Transformer.scala:22 +-(5) MappedRDD[320] at apply at Transformer.scala:22 | | MappedRDD[319] at apply at Transformer.scala:22 | | MappedValuesRDD[318] at apply at Transformer.scala:22 | | MapPartitionsRDD[317] at apply at Transformer.scala:22 | | ShuffledRDD[316] at apply at Transformer.scala:22 | +-(10) MappedRDD[315] at apply at Transformer.scala:22 | | ParallelCollectionRDD[314] at apply at Transformer.scala:22 +-(100) MappedRDD[322] at apply at Transformer.scala:22 | ParallelCollectionRDD[321] at apply at Transformer.scala:22 ``` Author: Gregory Owen <greowen@gmail.com> Closes apache#1364 from GregOwen/to-debug-string and squashes the following commits: 08f5c78 [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly 1603f7b [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly
Changes RDD.toDebugString() to show hierarchy and shuffle transformations more clearly
New output: