AQAvit Community Call (March 5, 2024 edition)

[smlambert]

This meeting is scheduled for: March 5, 2024 at 10 am EST | 3 pm UTC
and will be recorded to share with our colleagues who may not be able to attend.

Join Zoom Meeting link

Meeting ID: 865 6128 4227
Passcode: 045388

---

We have shifted away from bi-weekly calls to a more doable 6 calls per year (typically 4-6 weeks after the JDK release period) which will be added to the public Adoptium community calendar, if you wish to import the calendar use this link.

The content of these AQAvit community calls is intended to summarize what new features, challenges, and changes have happened in the AQAvit project.

Proposed agenda:

• AQAvit 1.0.0 release and graduation (@smlambert) - https://adoptium.net/blog/2024/02/aqavit-graduation-ceremony/
• New features review
  • New test targets:
    • TestHeadlessComponents, churn, ReproducibleTests (@smlambert @sophia-guo)
  • FIPS support (@LongyuZhang @llxia)
  • CRIU support (@LongyuZhang @llxia)
  • TAP verification tools (@llxia)
  • pre-stage 3rd party test libs (@llxia)
  • jck_custom updates, rerun jck**_custom links, no demo, but mention its similarity to jdk_custom mechanism (@sophia-guo)
  • Auto-update testenv.properties file if JDKxx_BRANCH -ga tag is not available - Check testenv.properties JDKxx_BRANCH tags #5102
• Onboarding new vendors / teams / contributors (@smlambert)
• Stretch Goals for 2024 (Trestle initiative, SLSA determination, etc) (@smlambert)
• AOB
  • proper set of test metrics (and education around what 'progress' means)
  • determination of current SLSA level for the project (and what it means given we do not produce a distributed binary)
  • Additional resources available (new docs & recordings)

[llxia]

CRIU Portable Testing:

[smlambert]

Minutes and additional context:

AQAvit 1.0.0 release and graduation

• in addition to our regular Github releases we also now have done an Eclipse Foundation release, which adds in checks for things like Intellectual Property rights, proper use of Trademarks, security policy and contributor documentation, etc. These things were required in order to graduate from incubator to full-fledged project at Eclipse Foundation. Read the blog for details: https://adoptium.net/blog/2024/02/aqavit-graduation-ceremony/

New features Review

New test targets

• TestHeadlessComponents - added to dev.functional, runs a set of headless tests against a JRE, Test headless components #5084
• churn - added to dev.system, is a test suite that exercises GCs, likely to only surface defects when run for a longer period, a 5 hour target is enabled, all tests added into dev level will be monitored and adjusted as needed, Added churn testsuite #5101
• ReproducibleTests - enabled in dev.system on xLinux for JDK21+, in the process of being updated to include windows and remove the dependency on Docker Add reproducible comparison tests for windows and update tests for linux running out of docker image temurin-build#3683

FIPS support

• Major achievement, FIPS stands for Federal Information Processing Standards and test support enabled in AQAvit for FIPS 140-2 and 140-3 (long list of related PRs, but of interest Update FIPS to FIPS140_2 #4730 and Add support for FIPS140-3 #4910)

CRIU support

• Major achievement, CRIU stands for Checkpoint/Restore In Userspace, which is a 'tricky' feature to test, see diagram in AQAvit Community Call (March 5, 2024 edition) #5090 (comment) on how this is managed in AQAvit (splitting the verification into 2 parts, imageUpload and imagePull so that a variety of scenarios can be tested (checkpoint and restore on same machine, checkpoint and restore on a different machine, and on different microArchitectures, etc). Long list of related PRs notably, Auto detect TEST_FLAG TKG#269 and supporting microArchitectures Support microArch TKG#346.

TAP verification tools

• Great improvement, and more to come with this feature gives more support for vendors listing in the marketplace, allows them to create jobs that can help verify the set of TAP files that they share through the marketplace API as part of their listing (as evidence that all tests were run and passed) - Related: Add TAP pipeline #4285 and Add checks in TAP file verification #5067 and work in progress example job here

pre-stage 3rd party test libs

• Good reduction in what is downloaded during test execution, addresses problems seen on machines in labs that have spotty or slow internet connection (pre-stage test libs on the machine #4902 and Allow LIB_DIR, customUrl and curlOpts to be set for get dependency logic TKG#476)

jck_custom updates, rerun jck**_custom links

• helps save time for those working with compliance test material, will reduce time for rerunning failures and offer convenience rerun links (Support rerun in Grinder using jck_custom with failed testcases #4498, Update TAP information for jck tests TKG#504 and Separate jck_custom to jck runtime, compiler and devtools custom #5087)

Auto-update testenv.properties file

• if JDKxx_BRANCH -ga tag is not available fetch the latest one with the same prefix, Check testenv.properties JDKxx_BRANCH tags #5102 helps reduce the need for last minute changes during release periods and reduction in steps in the release checklist (happier release champions)

Onboarding new vendors / teams / contributors

• In just over the past year, we have seen 29 new contributors to aqa-tests (tags v0.9.6-v1.0.1). Welcome to our new contributors who hail from universities, and from a broad range of companies including Red Hat, IBM, Microsoft, Alibaba, Loongson and Rivos (woohoo, RISC-V test plan!).
• Welcome to new colleagues from Fujitsu who are asking great and detailed questions on AQAvit tools and pipelines usage, questions which help give us good direction on documentation improvements Add Fujitsu TKG#501

Stretch Goals for 2024

Over and above the daily deliverables that those who participate in AQAvit must prioritize, we have added a few 'wishlist' or 'stretch' goals that we intend to look at in 2024. These include, but are not limited to:

• Trestle initiative: https://github.com/adoptium/ci-jenkins-pipelines/wiki/Trestle-Initiative (and Create Trestle pipelines ci-jenkins-pipelines#895)
• SLSA for Eclipse Foundation projects https://newsroom.eclipse.org/eclipse-newsletter/2022/november/making-software-supply-chain-security-new-pillar-eclipse-foundation - hope to look at this by 4Q and implications / meaning for source only projects - determination of current SLSA level for the project (and what it means given we do not produce a distributed binary)

AOB

• proper set of test metrics (and education around what 'progress' means)
  • EPIC: Provide AQA Test metrics per release #5121 - Call for participation, please read the issue and add comments if you have additional metrics you would like us to consider. We will create a table of potential metrics to gather, with a description of how to gather, and what is the main purpose of the metric (what do we hope to achieve by monitoring it)
• Additional resources available (new docs & recordings)
  • Teaser on the Triage presentation and demo

[smlambert]

Recording link on youtube