[SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0 #28585

sarutak · 2020-05-20T02:57:17Z

What changes were proposed in this pull request?

This PR upgrades HtmlUnit.
Selenium and Jetty also upgraded because of dependency.

Why are the changes needed?

Recently, a security issue which affects HtmlUnit is reported.
https://nvd.nist.gov/vuln/detail/CVE-2020-5529
According to the report, arbitrary code can be run by malicious users.
HtmlUnit is used for test so the impact might not be large but it's better to upgrade it just in case.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing testcases.

sarutak · 2020-05-20T03:06:41Z

core/src/main/scala/org/apache/spark/ui/JettyUtils.scala

+        // QueuedThreadPoolExecutor#setIdleTimeout is changed and IllegalStateException
+        // will be thrown if we try to set idle timeout after the server has started.
+        // But this workaround works for Jetty 9.4.28 by ignoring the exception.
+        Try(pool.setIdleTimeout(0))


I noticed that as of Jetty 9.4.21, the implementation of QueuedThreadPool.
When QueuedThreadPool#setIdleTimeout is called, QueuedThreadPool#_idleTimeout is set to the new timeout as well as ReservedThreadExecutor#_idleTimeout.

If ReservedThreadExecutor#_idleTImeout is tried to be set after Jetty has started, IllegalStateException will be thrown.

But QueuedThreadPool#_idleTimeout will be set so this workaround will still work by ignoring the exception.

SparkQA · 2020-05-20T05:38:27Z

Test build #122866 has finished for PR 28585 at commit f53735e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-20T06:02:01Z

Ah, I remember that some tests will fail because the latest HtmlUnit can't properly interpret JavaScript code used in those tests.
So PR should follow #28578 .

srowen · 2020-05-20T14:19:40Z

Looks OK pending tests

HyukjinKwon · 2020-05-21T09:55:33Z

cc @gengliangwang too FYI. seems okay to me too

sarutak · 2020-05-21T10:44:40Z

core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala

@@ -784,6 +784,7 @@ class UISeleniumSuite extends SparkFunSuite with WebBrowser with Matchers with B

      eventually(timeout(10.seconds), interval(50.milliseconds)) {
        goToUi(sc, "/jobs")
+


This extra empty line is wrong change but after #28578 is merged, this testcase will be moved.

gengliangwang

LGTM, thanks for the work!

gengliangwang · 2020-05-21T18:42:36Z

merging to master

sarutak · 2020-05-21T22:54:45Z

@gengliangwang This PR doesn't pass test.
As I mentioned here, #28578 should be merged first.

gengliangwang · 2020-05-21T22:56:37Z

@sarutak sorry about that, let me do the revert

SparkQA · 2020-05-22T02:20:10Z

Test build #122950 has finished for PR 28585 at commit f53735e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-05-22T13:25:37Z

I merged #28578 ; this may need a rebase

…-htmlunit

SparkQA · 2020-05-22T20:54:55Z

Test build #123015 has finished for PR 28585 at commit d950b42.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-23T08:13:48Z

According to the last failure, I noticed that real headless browser support is needed for HistoryServer tests too.
I've open a PR for that.
#28622

…ver tests ### What changes were proposed in this pull request? This PR adds two things. Real headless browser support for HistoryServer tests. A test suite using headless Chrome as one instance of those browsers. ### Why are the changes needed? The motivation is same as #28578 . In the current master, there is a testcase for HistoryServer which uses Ajax so we need the support for HistoryServer tests. Also this change is necessary to upgrade HtmlUnit (See #28585) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I tested with following patterns. Both Chrome and Chrome driver should be installed to test. 1. sbt / with default excluded tags (ChromeUIHistoryServerSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed) `build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.deploy.history.ChromeUIHistoryServerSuite org.apache.spark.sql.SQLQueryTestSuite" 2. sbt / overwrite default excluded tags as empty string (Both suites are expected to succeed) `build/sbt -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.deploy.history.ChromeUIHistoryServerSuite org.apache.spark.sql.SQLQueryTestSuite" 3. sbt / set `test.exclude.tags` to `org.apache.spark.tags.ExtendedSQLTest` (Both suites are expected to be skipped) `build/sbt -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.deploy.history.ChromeUIHistoryServerSuite org.apache.spark.sql.SQLQueryTestSuite" 4. Maven / with default excluded tags (ChromeUIHistoryServerSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed) `build/mvn -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history.ChromeUIHistoryServerSuite,org.apache.spark.sql.SQLQueryTestSuite test` 5. Maven / overwrite default excluded tags as empty string (Both suites are expected to succeed) `build/mvn -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history.ChromeUIHistoryServerSuite,org.apache.spark.sql.SQLQueryTestSuite test` 6. Maven / set `test.exclude.tags` to `org.apache.spark.tags.ExtendedSQLTest` (Both suites are expected to be skipped) `build/mvn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history.ChromeUIHistoryServerSuite,org.apache.spark.sql.SQLQueryTestSuite test` Closes #28622 from sarutak/headless-browser-support-for-historyserver. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Sean Owen <srowen@gmail.com>

HyukjinKwon · 2020-06-03T05:17:20Z

Thanks @dongjoon-hyun for monitoring this.

sarutak · 2020-06-03T05:23:34Z

Thanks @dongjoon-hyun , @HyukjinKwon .
I don't know why sbt passes the tests but I'll investigate the reason anyway.

dongjoon-hyun · 2020-06-03T05:24:50Z

Thanks, @sarutak .

SparkQA · 2020-06-03T06:27:56Z

Test build #123462 has finished for PR 28585 at commit d950b42.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-03T07:29:00Z

Test build #123469 has started for PR 28585 at commit 955a147.

sarutak · 2020-06-03T07:31:11Z

@dongjoon-hyun @HyukjinKwon The reason of the failure seems that bytebuddy which upgraded-selenium depends on is much older than mockito depends on.
I've excluded the older one.
Let's wait the result of CI.

dongjoon-hyun · 2020-06-03T07:32:33Z

Sounds reasonable. Thanks!

sarutak · 2020-06-03T20:02:27Z

retest this please.

SparkQA · 2020-06-03T20:03:53Z

Test build #123501 has started for PR 28585 at commit 955a147.

sarutak · 2020-06-04T04:25:48Z

retest this please.

SparkQA · 2020-06-04T07:05:02Z

Test build #123518 has finished for PR 28585 at commit 955a147.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-06-04T07:23:53Z

retest this please.

SparkQA · 2020-06-04T14:15:18Z

Test build #123522 has finished for PR 28585 at commit 955a147.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-06-04T20:24:12Z

By excluding old version of bytebuddy , Maven tests seem to pass.

srowen · 2020-06-06T14:16:32Z

Heh, do we think it's safe to merge again?
An exclusion is probably fine, but now it depends on something it needs only transitively. We could also manage the version explicitly. But, this maybe fine now.

sarutak · 2020-06-07T04:01:05Z

The errors were reproduced in PR builder and they are fixed by excluding the old bytebuddy so I believe it's safe to merge.
What do you think, @dongjoon-hyun, @HyukjinKwon ?

srowen · 2020-06-11T15:16:59Z

I'll merge today if there are no objections

srowen · 2020-06-11T23:27:51Z

Trying again - merged to master

dongjoon-hyun · 2020-09-23T17:01:06Z

According to @sarutak , this broke org.apache.spark.deploy.history.ChromeUIHistoryServerSuite due to Guava dependency.

[SPARK-32957][INFRA] Add a GitHub Actions job to run WebUI tests with Chrome #29827 (comment)

srowen · 2020-09-23T17:07:16Z

Shoot. If this is non-essential, I think we can just revert it.
Or .. just lose this test? let me comment on the other PR.

dongjoon-hyun · 2020-09-23T17:17:10Z

Thanks, @srowen ~

sarutak added 2 commits May 18, 2020 13:50

Upgrade HtmlUnit and Selenium

fd95c6f

Upgrade Jetty to 9.4.28

e9f09e4

probot-autolabeler bot added BUILD CORE DSTREAM SQL WEB UI labels May 20, 2020

Modified comment.

f53735e

sarutak commented May 20, 2020

View reviewed changes

sarutak commented May 21, 2020

View reviewed changes

gengliangwang approved these changes May 21, 2020

View reviewed changes

gengliangwang closed this in 92877c4 May 21, 2020

gengliangwang mentioned this pull request May 21, 2020

[SPARK-31756][WEBUI] Add real headless browser support for UI test #28578

Closed

gengliangwang mentioned this pull request May 21, 2020

Revert "[SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0" #28602

Closed

gengliangwang reopened this May 21, 2020

sarutak added 2 commits May 23, 2020 03:25

Reverted unnessessary change.

fcec0dd

Merge branch 'master' of https://github.com/apache/spark into upgrade…

d950b42

…-htmlunit

sarutak mentioned this pull request May 23, 2020

[SPARK-31804][WEBUI] Add real headless browser support for HistoryServer tests #28622

Closed

sarutak reopened this Jun 3, 2020

sarutak changed the title ~~[SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0~~ [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0 Jun 3, 2020

Excluded old version bytebuddy.

955a147

HyukjinKwon mentioned this pull request Jun 4, 2020

[WIP][SPARK-31902][K8S][TEST] Fix the K8s integration test Jenkins mockito errors #28722

Closed

srowen closed this in 88a4e55 Jun 11, 2020

This was referenced Jun 13, 2020

[SPARK-31632][CORE][WEBUI][FOLLOWUP] Enrich the exception message when application summary is unavailable #28820

Closed

[SPARK-31996][BUILD] Specify the version of ChromeDriver and RemoteWebDriver which can work with guava 14.0.1 #28837

Closed

sarutak mentioned this pull request Sep 22, 2020

[SPARK-32957][INFRA] Add a GitHub Actions job to run WebUI tests with Chrome #29827

Closed

sarutak deleted the upgrade-htmlunit branch June 4, 2021 20:43

		@@ -784,6 +784,7 @@ class UISeleniumSuite extends SparkFunSuite with WebBrowser with Matchers with B

		eventually(timeout(10.seconds), interval(50.milliseconds)) {
		goToUi(sc, "/jobs")

[SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0 #28585

[SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0 #28585

Conversation

sarutak commented May 20, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

sarutak May 20, 2020

Choose a reason for hiding this comment

SparkQA commented May 20, 2020

sarutak commented May 20, 2020

srowen commented May 20, 2020

HyukjinKwon commented May 21, 2020

sarutak May 21, 2020 • edited Loading

Choose a reason for hiding this comment

gengliangwang left a comment

Choose a reason for hiding this comment

gengliangwang commented May 21, 2020

sarutak commented May 21, 2020

gengliangwang commented May 21, 2020

SparkQA commented May 22, 2020

srowen commented May 22, 2020

SparkQA commented May 22, 2020

sarutak commented May 23, 2020

HyukjinKwon commented Jun 3, 2020

sarutak commented Jun 3, 2020 • edited Loading

dongjoon-hyun commented Jun 3, 2020

SparkQA commented Jun 3, 2020

SparkQA commented Jun 3, 2020

sarutak commented Jun 3, 2020 • edited Loading

dongjoon-hyun commented Jun 3, 2020

sarutak commented Jun 3, 2020

SparkQA commented Jun 3, 2020

sarutak commented Jun 4, 2020

SparkQA commented Jun 4, 2020

sarutak commented Jun 4, 2020

SparkQA commented Jun 4, 2020

sarutak commented Jun 4, 2020

srowen commented Jun 6, 2020

sarutak commented Jun 7, 2020 • edited Loading

srowen commented Jun 11, 2020

srowen commented Jun 11, 2020

dongjoon-hyun commented Sep 23, 2020

srowen commented Sep 23, 2020

dongjoon-hyun commented Sep 23, 2020

sarutak May 21, 2020 •

edited

Loading

sarutak commented Jun 3, 2020 •

edited

Loading

sarutak commented Jun 3, 2020 •

edited

Loading

sarutak commented Jun 7, 2020 •

edited

Loading