[SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins #29017

dongjoon-hyun · 2020-07-06T21:50:05Z

What changes were proposed in this pull request?

This PR aims to disable SBT unidoc generation testing in Jenkins environment because it's flaky in Jenkins environment and not used for the official documentation generation. Also, GitHub Action has the correct test coverage for the official documentation generation.

[SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost #28848 (comment) (amp-jenkins-worker-06)
[SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence #28926 (comment) (amp-jenkins-worker-06)
[SPARK-32150][BUILD] Upgrade to ZStd 1.4.5-4 #28969 (comment) (amp-jenkins-worker-06)
[SPARK-32148][SS] Fix stream-stream join issue on missing to copy reused unsafe row #28975 (comment) (amp-jenkins-worker-05)
[SPARK-32160][CORE][PYSPARK] Disallow to create SparkContext in executors. #28986 (comment) (amp-jenkins-worker-05)
[SPARK-32167][SQL] Fix GetArrayStructFields to respect inner field's nullability together #28992 (comment) (amp-jenkins-worker-06)
[SPARK-32168][SQL] Fix hidden partitioning correctness bug in SQL overwrite #28993 (comment) (amp-jenkins-worker-05)
[SPARK-32209][SQL] Re-use GetTimestamp in ParseToDate #28999 (comment) (amp-jenkins-worker-04)
[SPARK-32192][SQL] Print column name when throws ClassCastException #29010 (comment) (amp-jenkins-worker-03)
[SPARK-32196][SQL] Extract In convertible part if it is not convertible #29013 (comment) (amp-jenkins-worker-04)
[SPARK-32200][WEBUI] Redirect to the history page when accessed to /history on the HistoryServer without appliation id #29016 (comment) (amp-jenkins-worker-05)
[SPARK-32211][SQL] Pin mariadb-plugin-gssapi-server version to fix MariaDBKrbIntegrationSuite #29025 (comment) (amp-jenkins-worker-04)
[SPARK-32231][R][INFRA] Use Hadoop 3.2 winutils in AppVeyor build #29042 (comment) (amp-jenkins-worker-03)

Why are the changes needed?

Apache Spark release-build.sh generates the official document by using the following command.

https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh#L341

PRODUCTION=1 RELEASE_VERSION="$SPARK_VERSION" jekyll build

And, this is executed by the following unidoc command for Scala/Java API doc.

https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L30

system("build/sbt -Pkinesis-asl clean compile unidoc") || raise("Unidoc generation failed")

However, the PR builder disabled Jekyll build and instead has a different test coverage.

# determine if docs were changed and if we're inside the amplab environment
# note - the below commented out until *all* Jenkins workers can get `jekyll` installed
# if "DOCS" in changed_modules and test_env == "amplab_jenkins":
#    build_spark_documentation()

Building Unidoc API Documentation
========================================================================
[info] Building Spark unidoc using SBT with these arguments: 
-Phadoop-3.2 -Phive-2.3 -Pspark-ganglia-lgpl -Pkubernetes -Pmesos 
-Phadoop-cloud -Phive -Phive-thriftserver -Pkinesis-asl -Pyarn unidoc

Does this PR introduce any user-facing change?

No. (This is used only for testing and not used in the official doc generation.)

How was this patch tested?

Pass the Jenkins without doc generation invocation.

dongjoon-hyun · 2020-07-06T21:52:33Z

Could you review this, @HyukjinKwon and @HeartSaVioR ? We have a few options.

Keep this AS-IS.
Disable it from Jenkins simply.
Migrating to GitHub Action and remove it from Jenkins.

We will use this PR as a discussion place.

SparkQA · 2020-07-06T21:54:08Z

Test build #125138 has finished for PR 29017 at commit d414e01.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-07-06T22:53:57Z

OK, using the same profiles in Github Action as we do in PRB looks to be tricky. If it's possible then I think Github Action can replace the gendoc in PRB, but if not we may need to figure out why all the profiles are excluded for sbt unidoc in the release process.

HyukjinKwon · 2020-07-07T01:06:33Z

Maybe we should think about replacing PR builder to use Github Actions entirely. Let me check things and get back to you.

HeartSaVioR · 2020-07-07T01:19:05Z

I roughly remember we tried once and it didn't work well. Does anyone remember why, and which PR was the attempt?

HyukjinKwon · 2020-07-07T02:06:32Z

Nope, I don't think we actually tried it. I believe we just roughly discussed and concluded that it might be possible :-).

SparkQA · 2020-07-07T02:26:13Z

Test build #125140 has finished for PR 29017 at commit 697130e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-07T04:01:28Z

Seems the Jenkins is back to normal for now ..

HyukjinKwon · 2020-07-07T04:02:10Z

I will still check it and share after some discussions.

HeartSaVioR · 2020-07-07T04:11:43Z

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/buildTimeTrend

While unidoc/dependency look to be back to normal, still lots of PR builds are failing which make PRs effectively be stuck on finalization. Not 100% sure migrating to Github Action would help, but seems worth a try.

dongjoon-hyun · 2020-07-07T14:39:17Z

Thank you all for review. This is still happening like #29025 . I updated the PR descriptions by adding more instances.

HeartSaVioR · 2020-07-07T21:31:15Z

I also saw two cases today - sharing one:

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125258/console

dongjoon-hyun · 2020-07-07T22:08:04Z

Thanks, @HeartSaVioR .
@HyukjinKwon is also working on the Spark private mailing list toward GitHub Action migration.

ScrapCodes · 2020-07-08T07:46:54Z

I think if it is too flaky, we can disable it.

I am ready to look at it closely later, if the flakiness can be mitigated.

HyukjinKwon · 2020-07-08T08:39:28Z

The documentation build failure is one of many problems, @ScrapCodes. Most of builds are already being failed due to the flakiness apparently because the meachines are being slow down. Looks .m2 cache issue in worker 4 came back again.
The documentation builds doesn't fail at Github Actions; however, it specifically fails in Jenkins machines too. These symptoms might all relate together.

HyukjinKwon · 2020-07-08T08:40:32Z

This is why I am suggesting to move to Github Actions too. It has been more than a week but we don't have much progress on the issue in Jenkins.

HeartSaVioR · 2020-07-08T14:10:02Z

I hope we do the migration iteratively - we all know it became a huge pain to see the green light of the build (rarely possible according to the Jenkins PRB build trends), and given the discussion doesn't take place in public mailing list, others have no idea about the progress. Resolving one by one might still help a bit.

Btw, would we mind if I ask about having a post in dev@ mailing list for sharing the information about the current situation, and what we are discussing/working for resolving (details wouldn't be needed but sharing the topic would be good), and what action is encouraged for others in the meanwhile? It would be nice if we all are in line with the direction of mitigating the issue. And that may help to lead the community prioritizes stabilizing the tests than the new features/improvements, which IMHO looks to be necessary in this time.

HyukjinKwon · 2020-07-08T15:09:39Z

I will probably share some contexts later into dev mailing list once we have a clear picture on what we want. Currently we're checking the feasibility of Github Actions commercial plan. Separately, I'm personally working on the Github Actions migration - HyukjinKwon#4 which is almost done.

Regardless of the commercial plan, I will hopefully open a PR very soon so we can at least unblock the PRs by using the free version of Github Actions.

dongjoon-hyun · 2020-07-08T16:09:43Z

I was holding back this PR as WIP and didn't create JIRA for two days, but today's situation looks worse than before like #28993 (comment), #28986 (comment) and #29019 (comment) .

I propose this PR as a kind of HOTFIX to mitigate the current failure storm. The existing test coverage didn't work correct in various machines in Jenkins environment as summarized in the PR description. In new GitHub Action environment, we can recover that test coverage easily if need.

dongjoon-hyun · 2020-07-08T16:10:42Z

How do you think about proceed this first, @HyukjinKwon , @HeartSaVioR , @ScrapCodes ?

dongjoon-hyun · 2020-07-08T16:42:43Z

Also, cc @viirya , @cloud-fan

HyukjinKwon · 2020-07-08T17:17:16Z

I'm okay with that.

viirya · 2020-07-08T17:23:56Z

Looks okay as Github Actions have documentation generation.

cloud-fan · 2020-07-08T17:53:20Z

+1

dongjoon-hyun · 2020-07-08T21:10:48Z

Thank you all. Merged to master.

### What changes were proposed in this pull request? This PR aims to run the Spark tests in Github Actions. To briefly explain the main idea: - Reuse `dev/run-tests.py` with SBT build - Reuse the modules in `dev/sparktestsupport/modules.py` to test each module - Pass the modules to test into `dev/run-tests.py` directly via `TEST_ONLY_MODULES` environment variable. For example, `pyspark-sql,core,sql,hive`. - `dev/run-tests.py` _does not_ take the dependent modules into account but solely the specified modules to test. Another thing to note might be `SlowHiveTest` annotation. Running the tests in Hive modules takes too much so the slow tests are extracted and it runs as a separate job. It was extracted from the actual elapsed time in Jenkins: ![Screen Shot 2020-07-09 at 7 48 13 PM](https://user-images.githubusercontent.com/6477701/87050238-f6098e80-c238-11ea-9c4a-ab505af61381.png) So, Hive tests are separated into to jobs. One is slow test cases, and the other one is the other test cases. _Note that_ the current GitHub Actions build virtually copies what the default PR builder on Jenkins does (without other profiles such as JDK 11, Hadoop 2, etc.). The only exception is Kinesis https://github.com/apache/spark/pull/29057/files#diff-04eb107ee163a50b61281ca08f4e4c7bR23 ### Why are the changes needed? Last week and onwards, the Jenkins machines became very unstable for many reasons: - Apparently, the machines became extremely slow. Almost all tests can't pass. - One machine (worker 4) started to have the corrupt `.m2` which fails the build. - Documentation build fails time to time for an unknown reason in Jenkins machine specifically. This is disabled for now at #29017. - Almost all PRs are basically blocked by this instability currently. The advantages of using Github Actions: - To avoid depending on few persons who can access to the cluster. - To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce. - To control the environment more flexibly. - Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost. Note that: - The current build in Jenkins takes _more than 7 hours_. With Github actions it takes _less than 2 hours_ - We can now control the environments especially for Python easily. - The test and build look more stable than the Jenkins'. ### Does this PR introduce _any_ user-facing change? No, dev-only change. ### How was this patch tested? Tested at HyukjinKwon#4 Closes #29057 from HyukjinKwon/migrate-to-github-actions. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

This PR aims to run the Spark tests in Github Actions. To briefly explain the main idea: - Reuse `dev/run-tests.py` with SBT build - Reuse the modules in `dev/sparktestsupport/modules.py` to test each module - Pass the modules to test into `dev/run-tests.py` directly via `TEST_ONLY_MODULES` environment variable. For example, `pyspark-sql,core,sql,hive`. - `dev/run-tests.py` _does not_ take the dependent modules into account but solely the specified modules to test. Another thing to note might be `SlowHiveTest` annotation. Running the tests in Hive modules takes too much so the slow tests are extracted and it runs as a separate job. It was extracted from the actual elapsed time in Jenkins: ![Screen Shot 2020-07-09 at 7 48 13 PM](https://user-images.githubusercontent.com/6477701/87050238-f6098e80-c238-11ea-9c4a-ab505af61381.png) So, Hive tests are separated into to jobs. One is slow test cases, and the other one is the other test cases. _Note that_ the current GitHub Actions build virtually copies what the default PR builder on Jenkins does (without other profiles such as JDK 11, Hadoop 2, etc.). The only exception is Kinesis https://github.com/apache/spark/pull/29057/files#diff-04eb107ee163a50b61281ca08f4e4c7bR23 Last week and onwards, the Jenkins machines became very unstable for many reasons: - Apparently, the machines became extremely slow. Almost all tests can't pass. - One machine (worker 4) started to have the corrupt `.m2` which fails the build. - Documentation build fails time to time for an unknown reason in Jenkins machine specifically. This is disabled for now at apache#29017. - Almost all PRs are basically blocked by this instability currently. The advantages of using Github Actions: - To avoid depending on few persons who can access to the cluster. - To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce. - To control the environment more flexibly. - Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost. Note that: - The current build in Jenkins takes _more than 7 hours_. With Github actions it takes _less than 2 hours_ - We can now control the environments especially for Python easily. - The test and build look more stable than the Jenkins'. No, dev-only change. Tested at #4 Closes apache#29057 from HyukjinKwon/migrate-to-github-actions. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[WIP][SPARK-XXX][TESTS] Disable SBT unidoc testing in Jenkins

d414e01

probot-autolabeler bot added the BUILD label Jul 6, 2020

dongjoon-hyun mentioned this pull request Jul 6, 2020

[SPARK-32178][TESTS] Disable test-dependencies.sh from Jenkins jobs #29004

Closed

dongjoon-hyun mentioned this pull request Jul 6, 2020

[SPARK-32148][SS] Fix stream-stream join issue on missing to copy reused unsafe row #28975

Closed

Use env variable

697130e

dongjoon-hyun changed the title ~~[WIP][SPARK-XXX][TESTS] Disable SBT unidoc generation testing in Jenkins~~ [SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins Jul 8, 2020

dongjoon-hyun marked this pull request as ready for review July 8, 2020 16:05

dongjoon-hyun closed this in 17997a5 Jul 8, 2020

HyukjinKwon mentioned this pull request Jul 9, 2020

[SPARK-32245][INFRA] Run Spark tests in Github Actions #29057

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins #29017

[SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins #29017

dongjoon-hyun commented Jul 6, 2020 •

edited

Loading

dongjoon-hyun commented Jul 6, 2020

SparkQA commented Jul 6, 2020

HeartSaVioR commented Jul 6, 2020

HyukjinKwon commented Jul 7, 2020

HeartSaVioR commented Jul 7, 2020

HyukjinKwon commented Jul 7, 2020

SparkQA commented Jul 7, 2020

HyukjinKwon commented Jul 7, 2020

HyukjinKwon commented Jul 7, 2020

HeartSaVioR commented Jul 7, 2020

dongjoon-hyun commented Jul 7, 2020

HeartSaVioR commented Jul 7, 2020

dongjoon-hyun commented Jul 7, 2020 •

edited

Loading

ScrapCodes commented Jul 8, 2020 •

edited

Loading

HyukjinKwon commented Jul 8, 2020

HyukjinKwon commented Jul 8, 2020 •

edited

Loading

HeartSaVioR commented Jul 8, 2020 •

edited

Loading

HyukjinKwon commented Jul 8, 2020 •

edited

Loading

dongjoon-hyun commented Jul 8, 2020 •

edited

Loading

dongjoon-hyun commented Jul 8, 2020

dongjoon-hyun commented Jul 8, 2020 •

edited

Loading

HyukjinKwon commented Jul 8, 2020

viirya commented Jul 8, 2020

cloud-fan commented Jul 8, 2020

dongjoon-hyun commented Jul 8, 2020

[SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins #29017

[SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins #29017

Conversation

dongjoon-hyun commented Jul 6, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Jul 6, 2020

SparkQA commented Jul 6, 2020

HeartSaVioR commented Jul 6, 2020

HyukjinKwon commented Jul 7, 2020

HeartSaVioR commented Jul 7, 2020

HyukjinKwon commented Jul 7, 2020

SparkQA commented Jul 7, 2020

HyukjinKwon commented Jul 7, 2020

HyukjinKwon commented Jul 7, 2020

HeartSaVioR commented Jul 7, 2020

dongjoon-hyun commented Jul 7, 2020

HeartSaVioR commented Jul 7, 2020

dongjoon-hyun commented Jul 7, 2020 • edited Loading

ScrapCodes commented Jul 8, 2020 • edited Loading

HyukjinKwon commented Jul 8, 2020

HyukjinKwon commented Jul 8, 2020 • edited Loading

HeartSaVioR commented Jul 8, 2020 • edited Loading

HyukjinKwon commented Jul 8, 2020 • edited Loading

dongjoon-hyun commented Jul 8, 2020 • edited Loading

dongjoon-hyun commented Jul 8, 2020

dongjoon-hyun commented Jul 8, 2020 • edited Loading

HyukjinKwon commented Jul 8, 2020

viirya commented Jul 8, 2020

cloud-fan commented Jul 8, 2020

dongjoon-hyun commented Jul 8, 2020

dongjoon-hyun commented Jul 6, 2020 •

edited

Loading

dongjoon-hyun commented Jul 7, 2020 •

edited

Loading

ScrapCodes commented Jul 8, 2020 •

edited

Loading

HyukjinKwon commented Jul 8, 2020 •

edited

Loading

HeartSaVioR commented Jul 8, 2020 •

edited

Loading

HyukjinKwon commented Jul 8, 2020 •

edited

Loading

dongjoon-hyun commented Jul 8, 2020 •

edited

Loading

dongjoon-hyun commented Jul 8, 2020 •

edited

Loading