Setup Java 21 container #28918

Abacn · 2023-10-10T17:45:59Z

roll forward of #28833 and fix issues revealed after it is merged; also fix relevant java21 logic that found in #28969 (needed for build java agent for container). By the time of #28833 there was not a chance to test java21 build logic because gradle 8.3 not yet support compile java21. Now they are tested on #28969

The first commit is the original PR Setup Java 21 container #28833 (after resolve merge conflicts)
The second commit contains fixes

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Abacn · 2023-10-10T17:46:48Z

wordcount tested on Dataflow runner: https://console.cloud.google.com/dataflow/jobs/us-central1/2023-10-10_10_33_46-1503830796761633932?project=google.com%3Aclouddfe

jamm warnings are fixed by add-open jvm flags

codecov · 2023-10-10T18:11:13Z

Codecov Report

Merging #28918 (1c4b2d1) into master (0b50a17) will decrease coverage by 0.01%.
Report is 47 commits behind head on master.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #28918      +/-   ##
==========================================
- Coverage   38.38%   38.38%   -0.01%     
==========================================
  Files         686      686              
  Lines      101640   101652      +12     
==========================================
- Hits        39019    39018       -1     
- Misses      61041    61054      +13     
  Partials     1580     1580

Flag	Coverage Δ
python	`30.00% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Abacn · 2023-10-13T14:32:20Z

Run Jpms Direct Java 11 PostCommit

* Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change

Abacn · 2023-10-17T19:28:48Z

Run Seed Job

publish SDK container job succeeded: https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/6308/

this included java21 container: gcr.io/apache-beam-testing/beam-sdk/beam_java21_sdk

Abacn · 2023-10-17T20:05:17Z

previously run jmps 11 test passed: https://ci-beam.apache.org/job/beam_PostCommit_Java_Jpms_Direct_Java11_PR/30/

Abacn · 2023-10-17T20:05:24Z

R: @damccorm

github-actions · 2023-10-17T20:08:40Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

Abacn · 2023-10-18T19:46:25Z

R: @AnandInguva (since new Python support was done by you which was similar workflow)

damccorm

Mostly looks good, just had a couple minor questions

damccorm · 2023-10-19T17:14:43Z

sdks/java/testing/jpms-tests/build.gradle

@@ -120,7 +121,7 @@ plugins.withType(JavaPlugin).configureEach{
 // JPMS requires JDK > 8
 project.tasks.each {
  it.onlyIf {
-    project.hasProperty("compileAndRunTestsWithJava17")
+    project.hasProperty('testJavaVersion')


Looks like this was just running on 17 previously - do we need tighter filtering here to avoid running on 11?

Currently Java11 JPMS test is done by setting

beam/.test-infra/jenkins/job_PostCommit_Java_Jpms_Direct_Java11.groovy

Line 44 in 39678b6

switches("-Dorg.gradle.java.home=${commonJobProperties.JAVA_11_HOME}")

so everything runs on Java11;

while Java17 test is done by setting

beam/.test-infra/jenkins/job_PostCommit_Java_Jpms_Direct_Java17.groovy

Line 45 in 39678b6

switches("-PcompileAndRunTestsWithJava17")

so all but compilation of jpms-test project are still run on Java8.

i.e. currently the combination of javahome=java 8+compileAndRunTestsWithJava11 option does not run test (where it supposed to). This change fixed the issue and make the behavior consistent for Java11/17/21

damccorm · 2023-10-19T17:17:20Z

.../test-utils/src/test/java/org/apache/beam/sdk/testutils/jvmverification/JvmVerification.java

@@ -39,6 +40,7 @@ public class JvmVerification {
    versionMapping.put("0034", v1_8);
    versionMapping.put("0037", v11);
    versionMapping.put("003d", v17);
+    versionMapping.put("0041", v21);


Nonblocking since it seems to be consistently hex 002c + version, but where are these coming from?

these are java bytecode versions. Yes from Java8 its been 44 (2c in hex) + version though no guarantee this trand continues. ref: https://javaalmanac.io/bytecode/versions/

damccorm

LGTM

Abacn · 2023-10-20T14:26:28Z

Run Java PreCommit

Abacn · 2023-10-20T15:38:57Z

PreCommit Java paased on Jenkins, github action fn-execution test failure unrelated

* Setup Java 21 container (apache#28833) * Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change * incremental fixes * add more comments for when jpms test enabled

Abacn force-pushed the java21containeronly branch from 5c01659 to a67b78f Compare October 10, 2023 17:49

Abacn marked this pull request as draft October 10, 2023 17:56

github-actions bot added java build website infra docker runners dataflow core labels Oct 10, 2023

Abacn closed this Oct 12, 2023

Abacn deleted the java21containeronly branch October 12, 2023 03:37

Abacn restored the java21containeronly branch October 13, 2023 14:26

Abacn reopened this Oct 13, 2023

Abacn force-pushed the java21containeronly branch from a67b78f to 6f616d7 Compare October 13, 2023 14:27

github-actions bot added the spark label Oct 13, 2023

Abacn force-pushed the java21containeronly branch from 6f616d7 to f86c519 Compare October 17, 2023 18:43

Abacn added 2 commits October 17, 2023 14:45

Setup Java 21 container (apache#28833)

8b2a7de

* Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change

incremental fixes

71fb124

Abacn force-pushed the java21containeronly branch from f86c519 to 71fb124 Compare October 17, 2023 18:46

Abacn marked this pull request as ready for review October 17, 2023 20:04

damccorm reviewed Oct 19, 2023

View reviewed changes

add more comments for when jpms test enabled

1c4b2d1

damccorm approved these changes Oct 20, 2023

View reviewed changes

Abacn merged commit e7a6405 into apache:master Oct 20, 2023
61 of 65 checks passed

Abacn deleted the java21containeronly branch October 20, 2023 15:39

Abacn mentioned this pull request Oct 22, 2023

Publish Java21 container for Publish SDK container GitHub Action #29103

Merged

3 tasks

Abacn mentioned this pull request Aug 14, 2024

Fix verifyJavaVersion in a few places #32183

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup Java 21 container #28918

Setup Java 21 container #28918

Abacn commented Oct 10, 2023 •

edited

Loading

Abacn commented Oct 10, 2023

codecov bot commented Oct 10, 2023 •

edited

Loading

Abacn commented Oct 13, 2023

Abacn commented Oct 17, 2023 •

edited

Loading

Abacn commented Oct 17, 2023

Abacn commented Oct 17, 2023

github-actions bot commented Oct 17, 2023

Abacn commented Oct 18, 2023

damccorm left a comment

damccorm Oct 19, 2023

Abacn Oct 19, 2023

damccorm Oct 19, 2023

Abacn Oct 19, 2023

damccorm left a comment

Abacn commented Oct 20, 2023

Abacn commented Oct 20, 2023

Setup Java 21 container #28918

Setup Java 21 container #28918

Conversation

Abacn commented Oct 10, 2023 • edited Loading

GitHub Actions Tests Status (on master branch)

Abacn commented Oct 10, 2023

codecov bot commented Oct 10, 2023 • edited Loading

Codecov Report

Abacn commented Oct 13, 2023

Abacn commented Oct 17, 2023 • edited Loading

Abacn commented Oct 17, 2023

Abacn commented Oct 17, 2023

github-actions bot commented Oct 17, 2023

Abacn commented Oct 18, 2023

damccorm left a comment

Choose a reason for hiding this comment

damccorm Oct 19, 2023

Choose a reason for hiding this comment

Abacn Oct 19, 2023

Choose a reason for hiding this comment

damccorm Oct 19, 2023

Choose a reason for hiding this comment

Abacn Oct 19, 2023

Choose a reason for hiding this comment

damccorm left a comment

Choose a reason for hiding this comment

Abacn commented Oct 20, 2023

Abacn commented Oct 20, 2023

Abacn commented Oct 10, 2023 •

edited

Loading

codecov bot commented Oct 10, 2023 •

edited

Loading

Abacn commented Oct 17, 2023 •

edited

Loading