Skip to content

Commit

Permalink
[SPARK-49871][CORE][TESTS] Fix `(Ssl)CoarseGrainedExecutorBackendSuit…
Browse files Browse the repository at this point in the history
…e` to reduce test resource and increase timeout

### What changes were proposed in this pull request?

This PR aims to fix `(Ssl)CoarseGrainedExecutorBackendSuite` to reduce test resource and increase timeout in order to reduce the flakiness in the resource hungry GitHub Action environment.

### Why are the changes needed?

The test case of `(Ssl)CoarseGrainedExecutorBackendSuite` injects a faulty plugin to check the executor restart. Sometime, it fails due to the timeout during restarting the executors.

- https://github.com/apache/spark/actions/runs/11168954174/job/31048566907
```
- SPARK-40320 Executor should exit when initialization failed for fatal error *** FAILED ***
  The code passed to eventually never returned normally. Attempted 187 times over 15.001016415999999 seconds.
  Last failure message: 1 was not greater than or equal to 2.
 (CoarseGrainedExecutorBackendSuite.scala:602)
```

### Does this PR introduce _any_ user-facing change?

No, this is a test-only change.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#48345 from dongjoon-hyun/SPARK-49871.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
dongjoon-hyun authored and HyukjinKwon committed Oct 4, 2024
1 parent 3dfedf6 commit e1dbeb4
Showing 1 changed file with 4 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ import org.scalatestplus.mockito.MockitoSugar
import org.apache.spark._
import org.apache.spark.TestUtils._
import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
import org.apache.spark.internal.config.PLUGINS
import org.apache.spark.internal.config.{EXECUTOR_MEMORY, PLUGINS}
import org.apache.spark.resource._
import org.apache.spark.resource.ResourceUtils._
import org.apache.spark.resource.TestResourceIDs._
Expand Down Expand Up @@ -581,7 +581,8 @@ class CoarseGrainedExecutorBackendSuite extends SparkFunSuite
*/
test("SPARK-40320 Executor should exit when initialization failed for fatal error") {
val conf = createSparkConf()
.setMaster("local-cluster[1, 1, 1024]")
.setMaster("local-cluster[1, 1, 512]")
.set(EXECUTOR_MEMORY.key, "512m")
.set(PLUGINS, Seq(classOf[TestFatalErrorPlugin].getName))
.setAppName("test")
sc = new SparkContext(conf)
Expand All @@ -599,7 +600,7 @@ class CoarseGrainedExecutorBackendSuite extends SparkFunSuite
}
try {
sc.addSparkListener(listener)
eventually(timeout(15.seconds)) {
eventually(timeout(30.seconds)) {
assert(executorAddCounter.get() >= 2)
assert(executorRemovedCounter.get() >= 2)
}
Expand Down

0 comments on commit e1dbeb4

Please sign in to comment.