[QST]Spark-Rapids with Java is way slower than Spark-Rapids with Python

[SoumyaB57]

I am trying to check the speed up of SQL queries on a 48 GB CSV file.
I am submitting both Java and Python files with the same configuration. They are taking almost the same time while running without the plugin, but while using the plugin I can almost see 3X Speedup with pyspark.
Can anyone please tell me why this is happening and any possible way out?

[sperlingxx]

Hi @SoumyaB57, can you provide more detailed information about your job ? Like the SQL queries you ran.

[SoumyaB57]

Hi @SoumyaB57, can you provide more detailed information about your job ? Like the SQL queries you ran.

Sure.
My CSV file has 18*10^8 rows and 4 columns(name,area,country_code2,country_code3) and I have generated this CSV file using a python script.
Query: select count(area) from dfsql group by name having name like 'I%' limit 1000
Pyspark-rapids: 18.46 seconds
Spark-rapids with Java: 53.39 seconds
Spark-submit command: spark-submit --master local --driver-memory 100g --conf spark.plugins=com.nvidia.spark.SQLPlugin --jars /home/ubuntu/cudf-22.02.0-cuda11.jar,/home/ubuntu/rapids-4-spark_2.12-22.02.0.jar

[SoumyaB57]

I have found the issue
In my Java Code I am reading the CSV file with the following code
Dataset<Row> df = sqlContext.read() .format("com.databricks.spark.csv") .option("inferSchema", "true") .option("header", "true") .load("countries.csv");
But it had a problem as it has some undefined behavior and it is scanning CSV file with CPU in some queries and that was very slow compared to GPU
Then I have changed the mode of reading the CSV file and it worked
Dataset<Row> df = spark.read().format("csv").option("header","true").load("countries.csv");