Skip to content

Commit

Permalink
SPARK-4375. no longer require -Pscala-2.10
Browse files Browse the repository at this point in the history 
It seems like the winds might have moved away from this approach, but wanted to post the PR anyway because I got it working and to show what it would look like.

Author: Sandy Ryza <sandy@cloudera.com>

Closes apache#3239 from sryza/sandy-spark-4375 and squashes the following commits:

0ffbe95 [Sandy Ryza] Enable -Dscala-2.11 in sbt
cd42d94 [Sandy Ryza] Update doc
f6644c3 [Sandy Ryza] SPARK-4375 take 2
  • Loading branch information
@sryza @pwendell
sryza authored and pwendell committed Nov 14, 2014
1 parent bbd8f5b commit f5f757e
Show file tree
Hide file tree
Showing 7 changed files with 54 additions and 171 deletions.
4 changes: 2 additions & 2 deletions docs/building-spark.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,9 @@ mvn -Pyarn -Phive -Phive-thriftserver-0.12.0 -Phadoop-2.4 -Dhadoop.version=2.4.0
{% endhighlight %}

# Building for Scala 2.11
To produce a Spark package compiled with Scala 2.11, use the `-Pscala-2.11` profile:
To produce a Spark package compiled with Scala 2.11, use the `-Dscala-2.11` property:

mvn -Pyarn -Phadoop-2.4 -Pscala-2.11 -DskipTests clean package
mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package

Scala 2.11 support in Spark is experimental and does not support a few features.
Specifically, Spark's external Kafka library and JDBC component are not yet
Expand Down
65 changes: 3 additions & 62 deletions examples/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -389,11 +389,11 @@
</properties>
</profile>
<profile>
<!-- We add source directories specific to Scala 2.10 and 2.11 since some examples
work only in one and not the other -->
<!-- We add a source directory specific to Scala 2.10 since Kafka and Algebird
only work with it -->
<id>scala-2.10</id>
<activation>
<activeByDefault>true</activeByDefault>
<property><name>!scala-2.11</name></property>
</activation>
<dependencies>
<dependency>
Expand Down Expand Up @@ -427,65 +427,6 @@
</sources>
</configuration>
</execution>
<execution>
<id>add-scala-test-sources</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/scala</source>
<source>scala-2.10/src/test/scala</source>
<source>scala-2.10/src/test/java</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>scala-2.11</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<!-- Streaming Kafka and zeromq modules are disabled for now. -->
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-scala-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/scala</source>
<source>scala-2.11/src/main/scala</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-scala-test-sources</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/scala</source>
<source>scala-2.11/src/test/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
Expand Down
22 changes: 2 additions & 20 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1296,9 +1296,6 @@

<profile>
<id>mapr3</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<hadoop.version>1.0.3-mapr-3.0.3</hadoop.version>
<yarn.version>2.3.0-mapr-4.0.0-FCS</yarn.version>
Expand All @@ -1309,9 +1306,6 @@

<profile>
<id>mapr4</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<hadoop.version>2.3.0-mapr-4.0.0-FCS</hadoop.version>
<yarn.version>2.3.0-mapr-4.0.0-FCS</yarn.version>
Expand Down Expand Up @@ -1341,9 +1335,6 @@
<!-- Build without Hadoop dependencies that are included in some runtime environments. -->
<profile>
<id>hadoop-provided</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
Expand Down Expand Up @@ -1390,18 +1381,12 @@
</profile>
<profile>
<id>hive-thriftserver</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<modules>
<module>sql/hive-thriftserver</module>
</modules>
</profile>
<profile>
<id>hive-0.12.0</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<hive.version>0.12.0-protobuf-2.5</hive.version>
<hive.version.short>0.12.0</hive.version.short>
Expand All @@ -1410,9 +1395,6 @@
</profile>
<profile>
<id>hive-0.13.1</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<hive.version>0.13.1a</hive.version>
<hive.version.short>0.13.1</hive.version.short>
Expand All @@ -1423,7 +1405,7 @@
<profile>
<id>scala-2.10</id>
<activation>
<activeByDefault>true</activeByDefault>
<property><name>!scala-2.11</name></property>
</activation>
<properties>
<scala.version>2.10.4</scala.version>
Expand All @@ -1439,7 +1421,7 @@
<profile>
<id>scala-2.11</id>
<activation>
<activeByDefault>false</activeByDefault>
<property><name>scala-2.11</name></property>
</activation>
<properties>
<scala.version>2.11.2</scala.version>
Expand Down
3 changes: 3 additions & 0 deletions project/SparkBuild.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,11 @@ object SparkBuild extends PomBuild {
"conjunction with environment variable.")
v.split("(\\s+|,)").filterNot(_.isEmpty).map(_.trim.replaceAll("-P", "")).toSeq
}

if (profiles.exists(_.contains("scala-"))) {
profiles
} else if (System.getProperty("scala-2.11") != null) {
profiles ++ Seq("scala-2.11")
} else {
println("Enabled default scala profile")
profiles ++ Seq("scala-2.10")
Expand Down
117 changes: 42 additions & 75 deletions repl/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
<sbt.project.name>repl</sbt.project.name>
<deb.install.path>/usr/share/spark</deb.install.path>
<deb.user>root</deb.user>
<extra.source.dir>scala-2.10/src/main/scala</extra.source.dir>
<extra.testsource.dir>scala-2.10/src/test/scala</extra.testsource.dir>
</properties>

<dependencies>
Expand Down Expand Up @@ -122,86 +124,51 @@
</environmentVariables>
</configuration>
</plugin>
<!-- Include a source dir depending on the Scala version -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-scala-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/scala</source>
<source>${extra.source.dir}</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-scala-test-sources</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/scala</source>
<source>${extra.testsource.dir}</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>scala-2.10</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-scala-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/scala</source>
<source>scala-2.10/src/main/scala</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-scala-test-sources</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/scala</source>
<source>scala-2.10/src/test/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>scala-2.11</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-scala-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/scala</source>
<source>scala-2.11/src/main/scala</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-scala-test-sources</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/scala</source>
<source>scala-2.11/src/test/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<activation>
<property><name>scala-2.11</name></property>
</activation>
<properties>
<extra.source.dir>scala-2.11/src/main/scala</extra.source.dir>
<extra.testsource.dir>scala-2.11/src/test/scala</extra.testsource.dir>
</properties>
</profile>
</profiles>
</project>
11 changes: 2 additions & 9 deletions sql/catalyst/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,11 @@
</plugins>
</build>
<profiles>
<!-- Quasiquotes are merged into scala reflect from scala 2.11 onwards. -->
<profile>
<id>scala-2.10</id>
<activation>
<activeByDefault>true</activeByDefault>
<property><name>!scala-2.11</name></property>
</activation>
<dependencies>
<dependency>
Expand All @@ -113,13 +114,5 @@
</dependency>
</dependencies>
</profile>
<profile>
<id>scala-2.11</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<!-- Quasiquotes are merged into scala reflect from scala 2.11 onwards. -->
</profile>

</profiles>
</project>
3 changes: 0 additions & 3 deletions sql/hive/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,6 @@
</profile>
<profile>
<id>hive-0.12.0</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>com.twitter</groupId>
Expand Down

0 comments on commit f5f757e

Please sign in to comment.