Skip to content

Commit

Permalink
[SPARK-44049][K8S][TESTS] Fix KubernetesSuite to use inNamespace fo…
Browse files Browse the repository at this point in the history
…r validating driver pod cleanup

### What changes were proposed in this pull request?

This PR aims to fix `KubernetesSuite` to use `inNamespace` API for validating driver pod cleanup.

### Why are the changes needed?

This is a trick bug because of the following two reasons.
- Although all test cases passed, currently K8s integration tests are running extremely slowly.
- The individual test case running time shows correctly.
- The slowness happens during the transition from a test to another test.

The main root cause is that K8s test shows `namespace not specified` error after passing tests and this bug blocks every test case at the driver pod clean-up and validation stage `up to 3 minutes` (the maximum timeouts).

```
[info]   The code passed to eventually never returned normally. Attempted 190 times over 3.011156453483333 minutes.
Last failure message: namespace not specified for an operation requiring one and no default was found in the Config.. (KubernetesSuite.scala:612)
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

Also, I manually tested that the suite took 13 minutes correctly. Previously, it took over 1 hour.

```
[info] YuniKornSuite:
[info] - SPARK-42190: Run SparkPi with local[*] (17 seconds, 144 milliseconds)
[info] - Run SparkPi with no resources (20 seconds, 406 milliseconds)
[info] - Run SparkPi with no resources & statefulset allocation (15 seconds, 531 milliseconds)
...
[info] Run completed in 13 minutes, 46 seconds.
[info] Total number of tests run: 27
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 842 s (14:02), completed Jun 13, 2023, 9:33:02 PM
```

Closes apache#41586 from dongjoon-hyun/SPARK-44049.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Jun 14, 2023
1 parent d8c9acf commit 3adbce2
Showing 1 changed file with 1 addition and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -613,6 +613,7 @@ class KubernetesSuite extends SparkFunSuite
Eventually.eventually(TIMEOUT, INTERVAL) {
assert(kubernetesTestComponents.kubernetesClient
.pods()
.inNamespace(kubernetesTestComponents.namespace)
.withName(driverPodName)
.get() == null)
}
Expand Down

0 comments on commit 3adbce2

Please sign in to comment.