Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-44049][K8S][TESTS] Fix KubernetesSuite to use
inNamespace
fo…
…r validating driver pod cleanup ### What changes were proposed in this pull request? This PR aims to fix `KubernetesSuite` to use `inNamespace` API for validating driver pod cleanup. ### Why are the changes needed? This is a trick bug because of the following two reasons. - Although all test cases passed, currently K8s integration tests are running extremely slowly. - The individual test case running time shows correctly. - The slowness happens during the transition from a test to another test. The main root cause is that K8s test shows `namespace not specified` error after passing tests and this bug blocks every test case at the driver pod clean-up and validation stage `up to 3 minutes` (the maximum timeouts). ``` [info] The code passed to eventually never returned normally. Attempted 190 times over 3.011156453483333 minutes. Last failure message: namespace not specified for an operation requiring one and no default was found in the Config.. (KubernetesSuite.scala:612) ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Also, I manually tested that the suite took 13 minutes correctly. Previously, it took over 1 hour. ``` [info] YuniKornSuite: [info] - SPARK-42190: Run SparkPi with local[*] (17 seconds, 144 milliseconds) [info] - Run SparkPi with no resources (20 seconds, 406 milliseconds) [info] - Run SparkPi with no resources & statefulset allocation (15 seconds, 531 milliseconds) ... [info] Run completed in 13 minutes, 46 seconds. [info] Total number of tests run: 27 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 842 s (14:02), completed Jun 13, 2023, 9:33:02 PM ``` Closes apache#41586 from dongjoon-hyun/SPARK-44049. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
- Loading branch information