-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG][SPARK-40436][BUILD] Upgrade Scala to 2.12.17 #10203
Labels
bug
Something isn't working
Comments
NVnavkumar
added
bug
Something isn't working
? - Needs Triage
Need team to review and classify
labels
Jan 16, 2024
I was able to replicate this locally trying to build against Spark 3.5.0 using JDK 8 + Scala 2.12.17 by editing diff --git a/pom.xml b/pom.xml
index fd1c821cf..0d6ed32d9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -639,7 +639,7 @@
<id>scala-2.12</id>
<properties>
<scala.binary.version>2.12</scala.binary.version>
- <scala.version>2.12.15</scala.version>
+ <scala.version>2.12.17</scala.version>
</properties>
</profile>
<profile>
@@ -727,7 +727,7 @@
<scala.binary.version>2.12</scala.binary.version>
<alluxio.client.version>2.8.0</alluxio.client.version>
<scala.recompileMode>incremental</scala.recompileMode>
- <scala.version>2.12.15</scala.version>
+ <scala.version>2.12.17</scala.version>
<!--
-processing
to suppress unactionable "No processor claimed any of these annotations"
diff --git a/scala2.13/pom.xml b/scala2.13/pom.xml
index 5ccc17c4b..3a4806fe5 100644
--- a/scala2.13/pom.xml
+++ b/scala2.13/pom.xml
@@ -639,7 +639,7 @@
<id>scala-2.12</id>
<properties>
<scala.binary.version>2.12</scala.binary.version>
- <scala.version>2.12.15</scala.version>
+ <scala.version>2.12.17</scala.version>
</properties>
</profile>
<profile> Interesting in that this issue doesn't occur in Scala 2.13 using JDK8 + Scala 2.13.8 |
razajafri
added a commit
to razajafri/spark-rapids
that referenced
this issue
Jul 12, 2024
There is a bytecode incompatibility which is why we are skipping these until we add support for it. For details please see the following two issues NVIDIA#11174 NVIDIA#10203
razajafri
added a commit
that referenced
this issue
Jul 16, 2024
…s] (#10994) * POM changes for Spark 4.0.0 Signed-off-by: Raza Jafri <rjafri@nvidia.com> * validate buildver and scala versions * more pom changes * fixed the scala-2.12 comment * more fixes for scala-2.13 pom * addressed comments * add in shim check to account for 400 * add 400 for premerge tests against jdk 17 * temporarily remove 400 from snapshotScala213 * fixed 2.13 pom * Remove 400 from jdk17 as it will compile with Scala 2.12 * github workflow changes * added quotes to pom-directory * update version defs to include scala 213 jdk 17 * Cross-compile all shims from JDK17 to JDK8 Eliminate Logging inheritance to prevent shimming of unshimmable API classes Signed-off-by: Gera Shegalov <gera@apache.org> * dummy * undo api pom change Signed-off-by: Gera Shegalov <gera@apache.org> * Add preview1 to the allowed shim versions Signed-off-by: Gera Shegalov <gera@apache.org> * Scala 2.13 to require JDK17 Signed-off-by: Gera Shegalov <gera@apache.org> * Removed unused import left over from razajafri#3 * Setup JAVA_HOME before caching * Only upgrade the Scala plugin for Scala 2.13 * Regenerate Scala 2.13 poms * Remove 330 from JDK17 builds for Scala 2.12 * Revert "Remove 330 from JDK17 builds for Scala 2.12" This reverts commit 1faabd4. * Downgrade scala.plugin.version for cloudera * Updated comment to include the issue * Upgrading the scala.maven.plugin version to 4.9.1 which is the same as Spark 4.0.0 * Downgrade scala-maven-plugin for Cloudera * revert mvn verify changes * Avoid cache for JDK 17 * removed cache dep from scala 213 * Added Scala 2.13 specific checks * Handle the change for UnaryPositive now extending RuntimeReplaceable * Removing 330 from jdk17.buildvers as we only support Scala2.13 and fixing the enviornment variable in version-defs.sh that we read for building against JDK17 with Scala 213 * Update Scala 2.13 poms * fixed scala2.13 verify to actually use the scala2.13/pom.xml * Added missing csv files * Skip Opcode tests There is a bytecode incompatibility which is why we are skipping these until we add support for it. For details please see the following two issues #11174 #10203 * upmerged and fixed the new compile error introduced * addressed review comments * Removed jdk17 cloudera check and moved it inside the 321,330 and 332 cloudera profiles * fixed upmerge conflicts * reverted renaming of id * Fixed HiveGenericUDFShim * addressed review comments * reverted the debugging code * generated Scala 2.13 poms --------- Signed-off-by: Raza Jafri <rjafri@nvidia.com> Signed-off-by: Gera Shegalov <gera@apache.org> Co-authored-by: Gera Shegalov <gera@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Spark updated the Scala version to 2.12.17 (See apache/spark#37892)
EMR team found that these manifested in unit test failures in OpcodeSuite:
The text was updated successfully, but these errors were encountered: