[QST] Spark hangs during initialization

[SidWeng]

Symptom:
If spark.executor.resource.gpu.amount & spark.task.resource.gpu.amount specified in spark conf,
Spark hangs during initialization and complains resources not enough (driver/executor cores and memory is enough, thus I'm guessing it complains about GPU resource)

[Timer-0] WARN  o.a.s.scheduler.TaskSchedulerImpl - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

If I took spark.executor.resource.gpu.amount & spark.task.resource.gpu.amount off, everything works fine.
Is anything wrong with my configuration ?

Environment:
spark-rapids: 0.22.0
Spark version: 3.0.2
Cluster type: standalone mode
Instance: Azure Batch with Standard_NC6s_v3 (one V100 per node)

My Spark configuration:

spark.plugins com.nvidia.spark.SQLPlugin
spark.sql.adaptive.enabled false
spark.sql.cache.serializer com.nvidia.spark.ParquetCachedBatchSerializer
spark.rapids.sql.enabled true
spark.rapids.sql.udfCompiler.enabled true
spark.rapids.sql.concurrentGpuTasks 4
spark.executor.resource.gpu.amount 1
spark.task.resource.gpu.amount 0.25

[tgravescs]

I'm guessing it means you don't have any GPUs available to schedule.

You can find instructions on setting up standalone cluster here:
https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-on-prem.html#spark-standalone-cluster

One of the main things is making sure your worker is configured to have the GPU resources available:
SPARK_WORKER_OPTS="-Dspark.worker.resource.gpu.amount=1 -Dspark.worker.resource.gpu.discoveryScript=/opt/sparkRapidsPlugin/getGpusResources.sh"

Once you bring it up you can check the spark master ui to make sure your workers have a GPU available to hand out.

The other possibility, though you don't usually see that error message, would be GPUs they are available but the executors are crashing. If you see GPUs available on the workers in the UI then go check your executor logs to see if they are crashing on startup. If they are there should be an error message there with an indication of what is wrong.

[SidWeng]

@tgravescs It works fine after I add SPARK_WORKER_OPTS in spark-env.sh. Thanks your help !

[tgravescs]

no problem, let us know if you have any other questions.