[idea] Support for Rapids (GPU) with Spark Thrift in DBT

[sjabber]

Which of these best describes you?

• dbt Community member
• Partner
• dbt Labs employee
• Other

What's your idea for new content?

Currently, DBT does not support the combination of the Rapids library (GPU) with Spark Thrift, even though this setup could significantly enhance performance. This lack of support prevents the DBT + Spark (Thrift) combination from reaching its full potential in terms of processing speed and efficiency.

It would be greatly beneficial if DBT could be improved to support running dbt run with Spark Thrift when using the Rapids library (GPU). This would enable better performance, taking advantage of GPU acceleration in Spark.

When attempting to use dbt run with Spark Thrift that utilizes the Rapids library, the following error occurs:
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: com/nvidia/spark/rapids/RuleNotFoundExprMeta
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:46)

My Environment Details:

• DBT version : 1.8.3
• Spark version : 3.5.0
• Rapids : rapids-4-spark_2.12-23.12.0.jar

Where would you recommend this content live on the docs.getdbt.com?

No response

[mirnawong1]

hey @sjabber , thanks for raising this! i was going to recommend raising this in the respective adapter repo but i see you created an issue in the dbt spark repo already! I'll go ahead and close this issue as it relates to the docs.getdbt.com site but please do continue to raise any issues or prs related to dbt or the documentation site. thank you!