Replies: 1 comment 7 replies
-
@chpiero, thanks for your question. Spark supports both Scala 2.13 and Scala 2.12, but Scala 2.12 is still the default (https://github.com/apache/spark/blob/master/pom.xml#L163). It is also the default for the official package download here: https://spark.apache.org/downloads.html, but Scala 2.13 is also provided as another option. From conversation in this jira issue (https://issues.apache.org/jira/browse/SPARK-25075), it seems the move to Scala 2.13 as default is not yet scheduled, it looks like perhaps for Spark 4+. In terms of python/scala. Scala is still used to compile all the spark core code and pyspark is for the most part calling into this code via JVM hooks. So the version of scala is not very relevant for a pyspark exclusive job. If your program was written in scala instead of python, then I could see the scala version coming into play. Is there something that is forcing you to choose Scala 2.13? If so I'd love to learn more. Irrespective of language, we do target specific functionality in Spark summarized here: https://nvidia.github.io/spark-rapids/docs/FAQ.html#what-parts-of-apache-spark-are-accelerated, with all the details here: https://nvidia.github.io/spark-rapids/docs/supported_ops.html. In a nutshell, a big focus of the plugin is ETL/SQL operations (e.g. dataframe API), which are accessible from python, scala, java, sql, and R. Does the above help for you to get started? |
Beta Was this translation helpful? Give feedback.
-
sorry i'm new to the world spark, I should install it on a cluster that has a node with 2 nvidia AMPERE GPUs, I find written in this NVIDIA link in the Install Spark paragraph that: "note that only scala version 2.12 is currently supported by the accelerator."
I would like to install the latest version of Spark 3.3 which supports from what I see scale 2.13, however, I will only work with python. What I don't understand is that if this NVIDIA plugin only supports scale 2.12 but all other versions of other programming languages do. Therefore I have no problem running python on GPUs, I have to use scale and version 2.12 only. to be able to work with GPUs and therefore python is not possible to use?
thanks
Beta Was this translation helpful? Give feedback.
All reactions