Task io.fabric8.kubernetes.client.utils.internal.SerialExecutor$$Lambda$580/2034408722@73f7f4c5 rejected from java.util.concurrent.ThreadPoolExecutor@d3159c3[Shutting down, pool size = 4, active threads = 2, queued tasks = 0, completed tasks = 300]

Run / Run Spark on Kubernetes Integration test

Task io.fabric8.kubernetes.client.utils.internal.SerialExecutor$$Lambda$580/2034408722@5faeac91 rejected from java.util.concurrent.ThreadPoolExecutor@d3159c3[Shutting down, pool size = 3, active threads = 1, queued tasks = 0, completed tasks = 301]

Run / Run Spark on Kubernetes Integration test

Set() did not contain "decomtest-b8608989d87ec5b9-exec-1".

Run / Run Spark on Kubernetes Integration test

Set() did not contain "decomtest-ebc71189d87fdd41-exec-1".

Run / Run Spark on Kubernetes Integration test

Set() did not contain "decomtest-0325b389d883d14f-exec-1".

Run / Run Spark on Kubernetes Integration test

Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=null, kind=pods, name=spark-test-app-997cc24985b14d96ae7bf799cf7a7677-driver, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "spark-test-app-997cc24985b14d96ae7bf799cf7a7677-driver" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={})..

Run / Build modules: pyspark-mllib, pyspark-ml, pyspark-ml-connect

Process completed with exit code 19.

python/pyspark/ml/torch/tests/test_data_loader.py.test_data_loader: python/pyspark/ml/torch/tests/test_data_loader.py#L1

An error occurred while calling o139.collectToPython. : org.apache.spark.SparkException: Job aborted due to stage failure: Could not recover from a failed barrier ResultStage. Most recent failure reason: Stage failed because barrier task ResultTask(1, 0) finished unsuccessfully. org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 1066, in main process() File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 1058, in process serializer.dump_stream(out_iter, outfile) File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 161, in dump_stream return super(ArrowStreamUDFSerializer, self).dump_stream(wrap_and_init_stream(), stream) File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 100, in dump_stream for batch in iterator: File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 146, in wrap_and_init_stream for batch, _ in iterator: File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 827, in func for result_batch, result_type in result_iter: File "/__w/spark/spark/python/pyspark/ml/torch/distributor.py", line 721, in wrapped_train_fn output = TorchDistributor._get_output_from_framework_wrapper( File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/ml/torch/distributor.py", line 567, in _get_output_from_framework_wrapper return framework_wrapper( File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/ml/torch/distributor.py", line 899, in _run_training_on_pytorch_function raise RuntimeError( RuntimeError: TorchDistributor failed during training.View stdout logs for detailed error message. at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:534) at org.apache.spark.sql.execution.python.PythonArrowOutput$$anon$1.read(PythonArrowOutput.scala:118) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:487) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) at org.apache.spark.rdd.RDD.iterator(RDD.scala:328) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:95) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2844) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2780) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2779) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2779) at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:2216) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3042) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2971) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:983) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2405) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2426) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2445) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2470) at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1046) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:407) at org.apache.spark.rdd.RDD.collect(RDD.scala:1045) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:448) at org.apache.spark.sql.Dataset.$anonfun$collectToPython$1(Dataset.scala:4156) at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:4330) at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:546) at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:4328) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:132) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:210) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:115) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:4328) at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:4153) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.lang.Thread.run(Thread.java:750)

Run / Build modules: catalyst, hive-thriftserver