Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] iceberg_test failed in nightly #6167

Closed
pxLi opened this issue Aug 1, 2022 · 1 comment · Fixed by #6169
Closed

[BUG] iceberg_test failed in nightly #6167

pxLi opened this issue Aug 1, 2022 · 1 comment · Fixed by #6169
Assignees
@jlowe
Labels
bug Something isn't working

Comments

@pxLi
Copy link
Collaborator

pxLi commented Aug 1, 2022 •
edited
Loading

Describe the bug
related to #5941

java.lang.NoClassDefFoundError: com/nvidia/spark/rapids/iceberg/IcebergProvider

executors restart until timeout. log,

Spark Executor Command: "/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java" "-cp" "/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/rapids-4-spark-integration-tests_2.12-22.08.0-SNAPSHOT-spark311.jar:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-avro_2.12-3.1.1.jar:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/conf/:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/jars/*" "-Xmx16384M" "-Dspark.driver.port=35445" "-ea" "-Duser.timezone=UTC" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@it-test-311-477-rxjws-f9mc1:35445" "--executor-id" "0" "--hostname" "10.233.109.249" "--cores" "6" "--app-id" "app-20220801090740-0265" "--worker-url" "spark://Worker@10.233.109.249:46641"
========================================

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/08/01 09:07:41 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 26500@it-test-311-477-rxjws-f9mc1
22/08/01 09:07:41 INFO SignalUtils: Registering signal handler for TERM
22/08/01 09:07:41 INFO SignalUtils: Registering signal handler for HUP
22/08/01 09:07:41 INFO SignalUtils: Registering signal handler for INT
22/08/01 09:07:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/08/01 09:07:41 INFO SecurityManager: Changing view acls to: root
22/08/01 09:07:41 INFO SecurityManager: Changing modify acls to: root
22/08/01 09:07:41 INFO SecurityManager: Changing view acls groups to: 
22/08/01 09:07:41 INFO SecurityManager: Changing modify acls groups to: 
22/08/01 09:07:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/08/01 09:07:42 INFO TransportClientFactory: Successfully created connection to it-test-311-477-rxjws-f9mc1/10.233.109.249:35445 after 86 ms (0 ms spent in bootstraps)
22/08/01 09:07:42 INFO SecurityManager: Changing view acls to: root
22/08/01 09:07:42 INFO SecurityManager: Changing modify acls to: root
22/08/01 09:07:42 INFO SecurityManager: Changing view acls groups to: 
22/08/01 09:07:42 INFO SecurityManager: Changing modify acls groups to: 
22/08/01 09:07:42 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/08/01 09:07:42 INFO TransportClientFactory: Successfully created connection to it-test-311-477-rxjws-f9mc1/10.233.109.249:35445 after 3 ms (0 ms spent in bootstraps)
22/08/01 09:07:42 INFO DiskBlockManager: Created local directory at /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/blockmgr-243d9533-0467-4856-9229-39a37c432701
22/08/01 09:07:42 INFO MemoryStore: MemoryStore started with capacity 8.4 GiB
22/08/01 09:07:42 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@it-test-311-477-rxjws-f9mc1:35445
22/08/01 09:07:42 INFO WorkerWatcher: Connecting to worker spark://Worker@10.233.109.249:46641
22/08/01 09:07:42 INFO TransportClientFactory: Successfully created connection to /10.233.109.249:46641 after 4 ms (0 ms spent in bootstraps)
22/08/01 09:07:42 INFO WorkerWatcher: Successfully connected to spark://Worker@10.233.109.249:46641
22/08/01 09:07:42 INFO ResourceUtils: ==============================================================
22/08/01 09:07:42 INFO ResourceUtils: No custom resources configured for spark.executor.
22/08/01 09:07:42 INFO ResourceUtils: ==============================================================
22/08/01 09:07:42 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
22/08/01 09:07:42 INFO Executor: Starting executor ID 0 on host 10.233.109.249
22/08/01 09:07:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43405.
22/08/01 09:07:42 INFO NettyBlockTransferService: Server created on 10.233.109.249:43405
22/08/01 09:07:42 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/08/01 09:07:42 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(0, 10.233.109.249, 43405, None)
22/08/01 09:07:42 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(0, 10.233.109.249, 43405, None)
22/08/01 09:07:42 INFO BlockManager: Initialized BlockManager: BlockManagerId(0, 10.233.109.249, 43405, None)
22/08/01 09:07:42 INFO Executor: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/files/org.apache.iceberg_iceberg-spark-runtime-3.1_2.12-0.13.2.jar with timestamp 1659344858838
22/08/01 09:07:42 INFO TransportClientFactory: Successfully created connection to it-test-311-477-rxjws-f9mc1/10.233.109.249:35445 after 1 ms (0 ms spent in bootstraps)
22/08/01 09:07:42 INFO Utils: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/files/org.apache.iceberg_iceberg-spark-runtime-3.1_2.12-0.13.2.jar to /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/fetchFileTemp9009856553407494937.tmp
22/08/01 09:07:43 INFO Utils: Copying /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/-20185092611659344858838_cache to /home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./org.apache.iceberg_iceberg-spark-runtime-3.1_2.12-0.13.2.jar
22/08/01 09:07:43 INFO Executor: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/jars/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar with timestamp 1659344858838
22/08/01 09:07:43 INFO Utils: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/jars/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar to /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/fetchFileTemp9117293402477137304.tmp
22/08/01 09:07:44 INFO Utils: Copying /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/20409880621659344858838_cache to /home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar
22/08/01 09:07:45 INFO Executor: Adding file:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar to class loader
22/08/01 09:07:45 INFO Executor: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/jars/rapids-4-spark-integration-tests_2.12-22.08.0-SNAPSHOT-spark311.jar with timestamp 1659344858838
22/08/01 09:07:45 INFO Utils: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/jars/rapids-4-spark-integration-tests_2.12-22.08.0-SNAPSHOT-spark311.jar to /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/fetchFileTemp7410908010941486800.tmp
22/08/01 09:07:45 INFO Utils: Copying /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/15631678501659344858838_cache to /home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./rapids-4-spark-integration-tests_2.12-22.08.0-SNAPSHOT-spark311.jar
22/08/01 09:07:45 INFO Executor: Adding file:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./rapids-4-spark-integration-tests_2.12-22.08.0-SNAPSHOT-spark311.jar to class loader
22/08/01 09:07:45 INFO Executor: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/jars/spark-avro_2.12-3.1.1.jar with timestamp 1659344858838
22/08/01 09:07:45 INFO Utils: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/jars/spark-avro_2.12-3.1.1.jar to /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/fetchFileTemp8631797057995790167.tmp
22/08/01 09:07:45 INFO Utils: Copying /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/7313665921659344858838_cache to /home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./spark-avro_2.12-3.1.1.jar
22/08/01 09:07:45 INFO Executor: Adding file:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./spark-avro_2.12-3.1.1.jar to class loader
22/08/01 09:07:45 INFO Executor: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/jars/org.apache.iceberg_iceberg-spark-runtime-3.1_2.12-0.13.2.jar with timestamp 1659344858838
22/08/01 09:07:45 INFO Utils: Fetching spark://it-test-311-477-rxjws-f9mc1:35445/jars/org.apache.iceberg_iceberg-spark-runtime-3.1_2.12-0.13.2.jar to /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/fetchFileTemp2819972348788970327.tmp
22/08/01 09:07:45 INFO Utils: /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e/8747950821659344858838_cache has been previously copied to /home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./org.apache.iceberg_iceberg-spark-runtime-3.1_2.12-0.13.2.jar
22/08/01 09:07:45 INFO Executor: Adding file:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/spark-3.1.1-bin-hadoop3.2/work/app-20220801090740-0265/0/./org.apache.iceberg_iceberg-spark-runtime-3.1_2.12-0.13.2.jar to class loader
22/08/01 09:07:45 INFO ShimLoader: Loading shim for Spark version: 3.1.1
22/08/01 09:07:45 INFO ShimLoader: Complete Spark build info: 3.1.1, https://github.com/apache/spark, HEAD, 1d550c4e90275ab418b9161925049239227f3dc9, 2021-02-22T01:33:19Z
22/08/01 09:07:45 INFO ShimLoader: Looking for a mutable classloader (defaultClassLoader) in SparkEnv.serializer org.apache.spark.serializer.JavaSerializer@58044412
22/08/01 09:07:45 INFO ShimLoader: Extracted Spark classloader from SparkEnv.serializer org.apache.spark.util.MutableURLClassLoader@6f6b7f32
22/08/01 09:07:45 INFO ShimLoader: findURLClassLoader found a URLClassLoader org.apache.spark.util.MutableURLClassLoader@6f6b7f32
22/08/01 09:07:45 INFO ShimLoader: Updating spark classloader org.apache.spark.util.MutableURLClassLoader@6f6b7f32 with the URLs: jar:file:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar!/spark3xx-common/, jar:file:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar!/spark311/
22/08/01 09:07:45 INFO ShimLoader: Spark classLoader org.apache.spark.util.MutableURLClassLoader@6f6b7f32 updated successfully
22/08/01 09:07:45 INFO ShimLoader: Updating spark classloader org.apache.spark.util.MutableURLClassLoader@6f6b7f32 with the URLs: jar:file:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar!/spark3xx-common/, jar:file:/home/jenkins/agent/workspace/jenkins-rapids_integration-pre_release-github-477-311/jars/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar!/spark311/
22/08/01 09:07:45 INFO ShimLoader: Spark classLoader org.apache.spark.util.MutableURLClassLoader@6f6b7f32 updated successfully
22/08/01 09:07:45 INFO RapidsPluginUtils: RAPIDS Accelerator build: {version=22.08.0-SNAPSHOT, user=, url=https://github.com/NVIDIA/spark-rapids.git, date=2022-08-01T06:12:39Z, revision=8c1b555764b13caa3ac83b446ca754e1937c28f4, cudf_version=22.08.0-SNAPSHOT, branch=HEAD}
22/08/01 09:07:45 INFO RapidsPluginUtils: RAPIDS Accelerator JNI build: {version=22.08.0-SNAPSHOT, user=, url=https://github.com/NVIDIA/spark-rapids-jni.git, date=2022-07-31T02:59:50Z, revision=dcda46ba2a054a27d958ca1065290313ca78e312, branch=HEAD}
22/08/01 09:07:45 INFO RapidsPluginUtils: cudf build: {version=22.08.0-SNAPSHOT, user=, url=https://github.com/rapidsai/cudf.git, date=2022-07-31T02:59:50Z, revision=03f1c1c5c5fcf90bd594aabd41b6e15f54690777, branch=HEAD}
22/08/01 09:07:45 WARN RapidsPluginUtils: RAPIDS Accelerator 22.08.0-SNAPSHOT using cudf 22.08.0-SNAPSHOT.
22/08/01 09:07:45 INFO RapidsExecutorPlugin: RAPIDS Accelerator build: {version=22.08.0-SNAPSHOT, user=, url=https://github.com/NVIDIA/spark-rapids.git, date=2022-08-01T06:12:39Z, revision=8c1b555764b13caa3ac83b446ca754e1937c28f4, cudf_version=22.08.0-SNAPSHOT, branch=HEAD}
22/08/01 09:07:45 INFO RapidsExecutorPlugin: cudf build: {version=22.08.0-SNAPSHOT, user=, url=https://github.com/rapidsai/cudf.git, date=2022-07-31T02:59:50Z, revision=03f1c1c5c5fcf90bd594aabd41b6e15f54690777, branch=HEAD}
22/08/01 09:07:46 ERROR RapidsExecutorPlugin: Exception in the executor plugin, shutting down!
java.lang.NoClassDefFoundError: com/nvidia/spark/rapids/iceberg/IcebergProvider
	at com.nvidia.spark.rapids.ShimLoader$.newIcebergProvider(ShimLoader.scala:467)
	at com.nvidia.spark.rapids.iceberg.IcebergProvider$.apply(IcebergProvider.scala:31)
	at org.apache.spark.sql.rapids.ExternalSource$.icebergProvider$lzycompute(ExternalSource.scala:60)
	at org.apache.spark.sql.rapids.ExternalSource$.icebergProvider(ExternalSource.scala:60)
	at org.apache.spark.sql.rapids.ExternalSource$.getScans(ExternalSource.scala:116)
	at com.nvidia.spark.rapids.GpuOverrides$.<init>(GpuOverrides.scala:3600)
	at com.nvidia.spark.rapids.GpuOverrides$.<clinit>(GpuOverrides.scala)
	at com.nvidia.spark.rapids.TypeChecks$.areTimestampsSupported(TypeChecks.scala:801)
	at com.nvidia.spark.rapids.RapidsExecutorPlugin.init(Plugin.scala:244)
	at org.apache.spark.internal.plugin.ExecutorPluginContainer.$anonfun$executorPlugins$1(PluginContainer.scala:125)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.spark.internal.plugin.ExecutorPluginContainer.<init>(PluginContainer.scala:113)
	at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:211)
	at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:199)
	at org.apache.spark.executor.Executor.$anonfun$plugins$1(Executor.scala:253)
	at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:222)
	at org.apache.spark.executor.Executor.<init>(Executor.scala:253)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:159)
	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
	at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
	at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.nvidia.spark.rapids.iceberg.IcebergProvider
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 34 more
22/08/01 09:07:46 ERROR Utils: Uncaught exception in thread shutdown-hook-0
java.lang.NullPointerException
	at org.apache.spark.executor.Executor.$anonfun$stop$3(Executor.scala:332)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:222)
	at org.apache.spark.executor.Executor.stop(Executor.scala:332)
	at org.apache.spark.executor.Executor.$anonfun$new$2(Executor.scala:76)
	at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)
	at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
	at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$1(ShutdownHookManager.scala:188)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
22/08/01 09:07:46 INFO DiskBlockManager: Shutdown hook called
22/08/01 09:07:46 INFO ShutdownHookManager: Shutdown hook called
22/08/01 09:07:46 INFO ShutdownHookManager: Deleting directory /tmp/spark-fce3bb34-4974-4396-a8ff-e73c89fa1cad/executor-ab33b4b7-87a0-4a19-8fe3-540cbe282306/spark-d125fb69-1202-4d46-a875-300192c4bc3e

Steps/Code to reproduce bug

export WORKSPACE=`pwd`
export URM_URL="https://<urm_url>/artifactory/sw-spark-maven"
SKIP_REVISION_CHECK=true TEST_MODE=ICEBERG_ONLY SPARK_VER=3.1.1 bash jenkins/spark-tests.sh
@pxLi pxLi added bug Something isn't working ? - Needs Triage Need team to review and classify labels Aug 1, 2022
@pxLi
Copy link
Collaborator Author

pxLi commented Aug 1, 2022 •
edited
Loading

BTW remove the --conf spark.rapids.force.caller.classloader=false could pass the initial class load for IcebergProvider but failed,

java.lang.NoClassDefFoundError: Lorg/apache/iceberg/CombinedScanTask;
	at java.lang.Class.getDeclaredFields0(Native Method)
	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
	at java.lang.Class.getDeclaredField(Class.java:2068)
	at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1871)
	at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79)
	at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506)
	at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494)
	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391)
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2160)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:457)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.CombinedScanTask
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 31 more

@jlowe jlowe self-assigned this Aug 1, 2022
@jlowe jlowe mentioned this issue Aug 1, 2022
Merged
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Aug 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jlowe jlowe

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix IcebergProvider classname in unshim exceptions jlowe/spark-rapids
3 participants
@jlowe @sameerz @pxLi