[QST] Does Apache Spark 3 support GPU usage for Spark RDDs?

[EngineerReversed]

I am currently trying to run genomic analyses pipelines using Hail(Link)(library for genomics analyses written in python and Scala). Recently, Apache Spark 3 was released and it supported GPU usage.

I tried spark-rapids library start an on-premise slurm cluster with gpu nodes. I was able to initialise the cluster. However, when I tried running hail tasks, the executors keep getting killed.

On querying in Hail forum, I got the response that

That’s a GPU code generator for Spark-SQL, and Hail doesn’t use any Spark-SQL interfaces, only the RDD interfaces.

So, does Spark3 not support GPU usage for RDD interfaces?

[revans2]

We do not support running the RDD API on GPUs at this time. We only support the SQL/Dataframe API, and even then only a subset of the operators. This is because we are translating individual Catalyst operators into GPU enabled equivalent operators. I would love to be able to support the RDD API, but that would require us to be able to take arbitrary java, scala, and python code and run it on the GPU. We are investigating ways to try to accomplish some of this, but right now it is very difficult to do. That is especially true for libraries like Hail, which use python as an API, but the data analysis is done in C/C++.

[EngineerReversed]

Thanks for the quick reply,
It answers my question. I will use CPU computation for now.

Closing the issue