[BUG] NoClassDefFoundError for spark-avro when the Plugin is deployed via extraClassPath/driver-class-path

[gerashegalov]

Describe the bug
Plugin crashes with java.lang.NoClassDefFoundError: org/apache/spark/sql/v2/avro/AvroScan when the Plugin jar(s) are deployed using extraClassPath instead of --jars or --packages Spark submit options

Steps/Code to reproduce bug
Invoke pyspark with --driver-class-path:

pyspark --driver-class-path dist/target/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar \
  --packages org.apache.spark:spark-avro_2.12:3.2.1 \
  --conf spark.rapids.sql.enabled=true \
  --conf spark.plugins=com.nvidia.spark.SQLPlugin

22/06/06 15:56:02 ERROR RapidsExecutorPlugin: Exception in the executor plugin, shutting down!
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/spark/sql/v2/avro/AvroScan
        at org.apache.spark.sql.rapids.ExternalSource$.getScans(ExternalSource.scala:128)
        at com.nvidia.spark.rapids.GpuOverrides$.<init>(GpuOverrides.scala:3555)
        at com.nvidia.spark.rapids.GpuOverrides$.<clinit>(GpuOverrides.scala)
        at com.nvidia.spark.rapids.TypeChecks$.areTimestampsSupported(TypeChecks.scala:797)
        at com.nvidia.spark.rapids.RapidsExecutorPlugin.init(Plugin.scala:218)
        at org.apache.spark.internal.plugin.ExecutorPluginContainer.$anonfun$executorPlugins$1(PluginContainer.scala:125)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
        at org.apache.spark.internal.plugin.ExecutorPluginContainer.<init>(PluginContainer.scala:113)
        at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:211)
        at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:199)
        at org.apache.spark.executor.Executor.$anonfun$plugins$1(Executor.scala:253)
        at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:231)
        at org.apache.spark.executor.Executor.<init>(Executor.scala:253)
        at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:64)
        at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:132)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:581)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:238)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/sql/v2/avro/AvroScan
        ... 36 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.v2.avro.AvroScan
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 36 more

Expected behavior
Should work just like --jars

pyspark --jars dist/target/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar \
  --packages org.apache.spark:spark-avro_2.12:3.2.1 \ 
  --conf spark.rapids.sql.enabled=true \
  --conf spark.plugins=com.nvidia.spark.SQLPlugin 

which loads the Plugin and Avro correctly

Environment details (please complete the following information)
any

Additional context
N/A

[razajafri]

I am going to unassign myself until this is a higher priority