SPARK-1209 [CORE] (Take 2) SparkHadoop{MapRed,MapReduce}Util should not use package org.apache.hadoop #3048
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
….hadoop to org.apache.spark
…o part of the PairRDDFunctions type hierarchy though?
… in later versions
|
Test build #22695 has started for PR 3048 at commit
|
|
Test build #22695 has finished for PR 3048 at commit
|
|
Test PASSed. |
|
LGTM. @pwendell we should consider merging this into 1.2. Can you look at the changes here? |
|
Cool let's pull this in and see how it goes. |
asfgit
pushed a commit
that referenced
this pull request
Nov 10, 2014
…ot use package org.apache.hadoop andrewor14 Another try at SPARK-1209, to address #2814 (comment) I successfully tested with `mvn -Dhadoop.version=1.0.4 -DskipTests clean package; mvn -Dhadoop.version=1.0.4 test` I assume that is what failed Jenkins last time. I also tried `-Dhadoop.version1.2.1` and `-Phadoop-2.4 -Pyarn -Phive` for more coverage. So this is why the class was put in `org.apache.hadoop` to begin with, I assume. One option is to leave this as-is for now and move it only when Hadoop 1.0.x support goes away. This is the other option, which adds a call to force the constructor to be public at run-time. It's probably less surprising than putting Spark code in `org.apache.hadoop`, but, does involve reflection. A `SecurityManager` might forbid this, but it would forbid a lot of stuff Spark does. This would also only affect Hadoop 1.0.x it seems. Author: Sean Owen <sowen@cloudera.com> Closes #3048 from srowen/SPARK-1209 and squashes the following commits: 0d48f4b [Sean Owen] For Hadoop 1.0.x, make certain constructors public, which were public in later versions 466e179 [Sean Owen] Disable MIMA warnings resulting from moving the class -- this was also part of the PairRDDFunctions type hierarchy though? eb61820 [Sean Owen] Move SparkHadoopMapRedUtil / SparkHadoopMapReduceUtil from org.apache.hadoop to org.apache.spark (cherry picked from commit f8e5732) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@andrewor14 Another try at SPARK-1209, to address #2814 (comment)
I successfully tested with
mvn -Dhadoop.version=1.0.4 -DskipTests clean package; mvn -Dhadoop.version=1.0.4 testI assume that is what failed Jenkins last time. I also tried-Dhadoop.version1.2.1and-Phadoop-2.4 -Pyarn -Phivefor more coverage.So this is why the class was put in
org.apache.hadoopto begin with, I assume. One option is to leave this as-is for now and move it only when Hadoop 1.0.x support goes away.This is the other option, which adds a call to force the constructor to be public at run-time. It's probably less surprising than putting Spark code in
org.apache.hadoop, but, does involve reflection. ASecurityManagermight forbid this, but it would forbid a lot of stuff Spark does. This would also only affect Hadoop 1.0.x it seems.