Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up Ubuntu 24.04 systems to be able to support it for Temurin #3501

Closed
4 of 5 tasks
Tracked by #3588
sxa opened this issue Apr 4, 2024 · 44 comments
Closed
4 of 5 tasks
Tracked by #3588

Set up Ubuntu 24.04 systems to be able to support it for Temurin #3501

sxa opened this issue Apr 4, 2024 · 44 comments
Assignees
Labels

Comments

@sxa
Copy link
Member

sxa commented Apr 4, 2024

Ubuntu 24.04 will be out later this month. We should look at our existing inventory and make some of these available. This will involve:

@Haroon-Khel
Copy link
Contributor

Haroon-Khel commented May 2, 2024

Ubuntu 2404 static docker container online in jenkins https://ci.adoptium.net/computer/test-docker-ubuntu2404-x64-1/

Will run the aqa test pipeline on it shortly

EDIT: https://ci.adoptium.net/job/AQA_Test_Pipeline/237/console

@Haroon-Khel
Copy link
Contributor

Identifying older systems that can be replaced with Ubuntu 24.04 without restricting testing.

In regards to replaceable static docker containers, at the moment we have 3 ubuntu 2004 x64 nodes, 9 arm64 ubuntu 2004 nodes, 1 ubunutu 1804 node, 6 arm32 ubuntu 2004 nodes and 3 ubuntu 2204 ppc64le nodes. I can remove one from each and replace it with a ubuntu 2404 node of that architecture, and make a brand new ubuntu 2404 s390x node on dockerhost-marist-ubuntu2204-s390x-1

@sxa
Copy link
Member Author

sxa commented May 2, 2024

Identifying older systems that can be replaced with Ubuntu 24.04 without restricting testing.

In regards to replaceable static docker containers, at the moment we have 3 ubuntu 2004 x64 nodes, 9 arm64 ubuntu 2004 nodes, 1 ubunutu 1804 node, 6 arm32 ubuntu 2004 nodes and 3 ubuntu 2204 ppc64le nodes. I can remove one from each and replace it with a ubuntu 2404 node of that architecture, and make a brand new ubuntu 2404 s390x node on dockerhost-marist-ubuntu2204-s390x-1

SGTM - we can start with that and then probably look at migrating more of the 2004 ones up given how many we have. I hadn't spotted that we don't even have 22.04 on arm32.

@Haroon-Khel
Copy link
Contributor

Haroon-Khel commented May 2, 2024

https://ci.adoptium.net/computer/test-docker-ubuntu2404-armv8-1/ https://ci.adoptium.net/job/AQA_Test_Pipeline/239/console

https://ci.adoptium.net/computer/test-docker-ubuntu2404-s390x-1/ https://ci.adoptium.net/job/AQA_Test_Pipeline/240/console

Getting unusual errors while building the ppc64le and arm32 images

 > [ 7/25] RUN mkdir -p /usr/lib/jvm/jdk17 && tar -xpzf /tmp/jdk17.tar.gz -C /usr/lib/jvm/jdk17 --strip-components=1:                                                                                               
0.295 tar: conf/security/policy/unlimited: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                 
0.295 tar: conf/security/policy/limited: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                   
0.295 tar: conf/security/policy: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                           
0.295 tar: conf/security: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                                  
0.295 tar: conf/sdp: Cannot change mode to rwxr-xr-x: Operation not permitted
0.296 tar: conf/management: Cannot change mode to rwxr-xr-x: Operation not permitted
0.296 tar: conf: Cannot change mode to rwxr-xr-x: Operation not permitted
0.305 tar: legal/java.base: Cannot change mode to rwxr-xr-x: Operation not permitted
1.052 tar: jmods: Cannot change mode to rwxr-xr-x: Operation not permitted

@Haroon-Khel
Copy link
Contributor

@sxa
Copy link
Member Author

sxa commented May 3, 2024

Unfortunately we don't have an easy way of provisioning a 24.04 on arm32 or ppc64le at the moment to identify whether this is specific to running in a container.

@sxa
Copy link
Member Author

sxa commented May 3, 2024

It looks like we're getting some interesting new errors in the sanity.openjdk
Also in the extended run we're getting quite a lot of failures some of which are related to dpkg so are likely related to the new version (I'm quite surprised that the tests are doing things at that OS-specific level, but apparently they do!):

 [09:14:04.910] Running dpkg
 [09:14:06.077] Command [PID: 1034473]:
     dpkg -S /lib/x86_64-linux-gnu/libbrotlidec.so.1
 [09:14:06.077] Output:
     dpkg-query: no path found matching pattern /lib/x86_64-linux-gnu/libbrotlidec.so.1
 [09:14:06.077] Returned: 1
 
 [09:14:06.078] java.io.IOException: Command [dpkg, -S, /lib/x86_64-linux-gnu/libbrotlidec.so.1] exited with 1 code

Based on this I think it's worth getting a "real" ubuntu 24.04 provisioned somewhere and verify that nothing we're seeing is the result of using an earlier kernel version with a newer container.

@Haroon-Khel
Copy link
Contributor

OSUOSL's openstack and azure dont seem to have a ubuntu2404 image (yet)

@sxa
Copy link
Member Author

sxa commented May 3, 2024

This is where having the ESXi server was useful :-)

@sxa
Copy link
Member Author

sxa commented May 3, 2024

Will run the aqa test pipeline on it shortly
EDIT: https://ci.adoptium.net/job/AQA_Test_Pipeline/237/console

To keep the machines busy over the UK bank holiday weekend I've kicked that off again with all the other versions at https://ci.adoptium.net/job/AQA_Test_Pipeline/243/

@sxa
Copy link
Member Author

sxa commented May 3, 2024

Got the remaining nodes up and running by building them manually

https://ci.adoptium.net/job/AQA_Test_Pipeline/242/console (ppc64le)
https://ci.adoptium.net/job/AQA_Test_Pipeline/241/console (arm32)

Noting that those seem to be failing after the tarball extraction had the same problem you had with the docker image creation

@Haroon-Khel
Copy link
Contributor

Ive marked both machines offline so they dont affect the weekend nightlies

@sxa
Copy link
Member Author

sxa commented May 3, 2024

I've just ran the Dockerfile for s390x and ppc64le in an emulated docker on an arm64 Ubuntu 24.04 host and they both worked without any permission issues 🤔
So the docker Ubuntu images on those platforms are not fundamentally broken.

@sxa
Copy link
Member Author

sxa commented May 10, 2024

I'm doing some re-runs based on the "clean" (green) sanity.openjdk jobs that I was doing as part of adoptium/temurin-build#3685 (comment) (the x64 -cfi-vh row) with JDK22+35:

@sxa
Copy link
Member Author

sxa commented May 11, 2024

Ubuntu 2404 static docker container online in jenkins https://ci.adoptium.net/computer/test-docker-ubuntu2404-x64-1/

Will run the aqa test pipeline on it shortly

EDIT: https://ci.adoptium.net/job/AQA_Test_Pipeline/237/console

https://ci.adoptium.net/computer/test-docker-ubuntu2404-armv8-1/ https://ci.adoptium.net/job/AQA_Test_Pipeline/239/console
https://ci.adoptium.net/computer/test-docker-ubuntu2404-s390x-1/ https://ci.adoptium.net/job/AQA_Test_Pipeline/240/console

On the basis of the reruns in the previous comment running clean I'm going to re-run these (including extra versions) with ADOPTOPENJDK_BRANCH=v1.0.1-release so that the material matches the product under test (SDK_RESOURCE=releases). These are being run with all versions: 17,8,22,21,11. The original JDK17 runs linked above took 29 hours (x64), 14h (aarch64) and 17 hours (s390x). so the first 2-3 versions will hopefully be mostly complete on Monday morning which will hopefully give enough of a result set that we can determine if there are likely to be any problems.

NOTE: The first 28 of the 46 s390x failures are the same as the 14 from aarch64 but run twice (Different variants?) The rest seem to be similar failures, but were not run on the aarch64 system. Looking in more detail it looks like they were excluded under a PR primarily intended for Win32

@sxa
Copy link
Member Author

sxa commented May 13, 2024

OSUOSL's openstack and azure dont seem to have a ubuntu2404 image (yet)

We should look at provisioning a 22.04 system at OSUOSL (Maybe for Aarch64 and ppc64le both at OSUOSL) and then upgrading it to 24.04 within the OS and see if that makes a difference

@sxa
Copy link
Member Author

sxa commented May 14, 2024

Deep dive into the tools/jpackage failures which are occurring on 24.04 but not on 22.04:

On both machines dpkg -S /lib/aarch64-linux-gnu/libmd.so.0 are failing, although on the 22.04 machine it gives a summary of [14:00:33.690] Required packages: [libc6, zlib1g] but on 24.04 it has `[14:00:38.972] Required packages: []

This is not considered a failure, but later on when it tries to build it's own appcategorytest package it fails on:

  • ERROR: Failed: Check [libc] package is in the list of required packages [] of [appcategorytest] package
    where in the passing case it has:
  • TRACE: assertTrue(): Check [libc] package is in the list of required packages [libc6, zlib1g] of [appcategorytest] package

OK Further investigation - all but five of the dpkg commands run on the Ubuntu 22.04 machine also give a non-zero return code. The ones that succeed are:

  • dpkg -S /lib/aarch64-linux-gnu/libc.so.0
  • dpkg -S /lib/aarch64-linux-gnu/libm.so.6
  • dpkg -S /lib/aarch64-linux-gnu/libz.so.1
  • dpkg -S /lib/aarch64-linux-gnu/librt.so.1
  • dpkg -S /lib/aarch64-linux-gnu/libdl.so.2
    All of which apart from libz.so.1 are in libc6, and indeed if you run dpkg -L libc6 you don't get the /lib versions on Ubuntu 24.04. Simple reproduce docker run ubuntu:24.04 dpkg -L libc6

@jiekang
Copy link

jiekang commented May 14, 2024

I've seen fixes to the jpackage tests that reference specific quirks with new versions of platforms like Fedora. Is this failure similar? Can we fix the test upstream?

@sxa
Copy link
Member Author

sxa commented May 14, 2024

I've seen fixes to the jpackage tests that reference specific quirks with new versions of platforms like Fedora. Is this failure similar? Can we fix the test upstream?

I would expect so, yes.

@Haroon-Khel
Copy link
Contributor

Haroon-Khel commented Jun 14, 2024

JDK 8 11 21 using the v1.0.1-release branch

test-azure-ubuntu2404-x64-1

  • https://ci.adoptium.net/job/AQA_Test_Pipeline/282/console
    • JDK8
      • extended openjdk
        • sun/security/pkcs11/fips/TestTLS12.java
        • sun/security/pkcs11/Secmod/JksSetPrivateKey.java
      • sanity openjdk
        • sun/security/krb5/auto/rcache_usemd5.sh
    • JDK11
      • extended openjdk
        • sun/security/pkcs11/ec/TestECDH.java
        • sun/security/pkcs11/ec/TestECDSA.java
        • sun/security/pkcs11/ec/TestECGenSpec.java
        • java/time/test/java/time/format/TestUTCParse.java
    • JDK21
      • extended openjdk
        • jdk_tools
        • jdk/jfr/jcmd/TestJcmdDump.java
        • com/sun/jdi/FinalizerTest.java

test-docker-ubuntu2404-x64-1

  • https://ci.adoptium.net/job/AQA_Test_Pipeline/283/console
    • JDK8
      • sanity openjdk
        • sun/security/krb5/auto/rcache_usemd5.sh
      • extended openjdk
        • sun/security/pkcs11/fips/TestTLS12.java
    • JDK11
      • extended openjdk
        • java/time/test/java/time/format/TestUTCParse.java
        • jdk_security3_0
    • JDK21
      • extended openjdk
        • jdk_tools

test-docker-ubuntu2404-armv7-1

test-docker-ubuntu2404-armv8-1

test-docker-ubuntu2404-s390x-1

test-osuosl-ubuntu2404-aarch64-1

@sxa
Copy link
Member Author

sxa commented Jun 14, 2024

Noting that we also have #3598 to migrate all of the Ubuntu 23.10 (which will out of out support next month) on riscv64 to the LTS Ubuntu 24.04 release

@Haroon-Khel
Copy link
Contributor

Haroon-Khel commented Jul 31, 2024

Next steps for this issue

Identifying older systems that can be replaced with Ubuntu 24.04 without restricting testing.

and

Ideally having one real machine for each architecture on this

These 2 goals can be achieved together:

  • test-azure-ubuntu2404-x64-1 and test-osuosl-ubuntu2404-aarch64-1 are already set up and in jenkins
  • test-osuosl-ubuntu1604-ppc64le-1 and -2 could be upgraded to ubuntu2404
  • We could commission a new marist ubuntu2404 machine, or upgrade our existing test-marist-ubuntu2004-s390x-1 to u2404
  • Not sure about arm32, at the moment we do not have a vendor for actual arm32 machines

Verifying that it is suitable as a base for our dockerhost system and deploying them: #3585

https://ci.adoptium.net/computer/dockerhost-osuosl-ubuntu2404-aarch64-1/ has been in jenkins for a while, with one arm32 container running on it https://ci.adoptium.net/label/hw.dockerhost.arm.dockerhost-osuosl-ubuntu2404-aarch64-1/. We could try a combination of upgrading existing or commissioning new dockerhost machines to ubuntu2404

Add Ubuntu 24.04 to Vagrant Playbook Check job

This is blocked, see #3577. The outdated version of ansible on the VPC server is not compatible with the versions of python on the vagrant boxes. Ansible should be upgraded, but a wider solution is to upgrade our existing VPC server to ubuntu2404 as the current version of ubuntu on the VPC server is 18.04 (EOL). This is being discussed here #3589

@Haroon-Khel
Copy link
Contributor

test-osuosl-ubuntu1604-ppc64le-1 and -2 could be upgraded to ubuntu2404

The ubuntu1604 osuosl ppc64le machine use a power8 cpu

Model name:                         POWER8 (architected), altivec supported

It looks like ubuntu2404 does not support power8 https://ubuntu.com/download/server/power

Starting with Ubuntu 22.04 LTS, POWER9 and POWER10 processors are supported.

The support for POWER8 ends with Ubuntu 21.10, respectively Ubuntu 20.04 LTS, which is still supported for several years.

During the command line upgrade:

Sorry, no more upgrades for this system 

There will not be any further Ubuntu releases for this system's 
POWER8 processor. 

Updates for Ubuntu 20.04 will continue until April 2025. 

It will have to be rebuilt in the osuosl console to a power 9 or 10 cpu. At the moment test-osuosl-ubuntu1604-ppc64le-1 has been upgraded to ubuntu2004

@sxa
Copy link
Member Author

sxa commented Aug 16, 2024

It looks like ubuntu2404 does not support power8 https://ubuntu.com/download/server/power

Interesting - yeah we should definitely have some later machines in there. Do we definitely have access to provision POWER9/10 systems in our OSUOSL account? If so, let's go for it!

@Haroon-Khel
Copy link
Contributor

Created https://ci.adoptium.net/computer/test-osuosl-ubuntu2404-ppc64le-1/, surprised it booted given my last attempts

root@test-osuosl-ubuntu2404-ppc64le-1:~# lscpu
Architecture:             ppc64le
  Byte Order:             Little Endian
CPU(s):                   4
  On-line CPU(s) list:    0-3
Model name:               POWER9 (architected), altivec supported
  Model:                  2.2 (pvr 004e 1202)
  Thread(s) per core:     1
  Core(s) per socket:     1
  Socket(s):              4

https://ci.adoptium.net/job/AQA_Test_Pipeline/349/console

@sxa
Copy link
Member Author

sxa commented Aug 22, 2024

Related - can we set up any "non-dockerhost" Ubuntu 24.04 test machines get labelled with sw.tool.docker to indicate that test jobs which need to spin up containers can run on those hosts.

Related:

@Haroon-Khel
Copy link
Contributor

@sxa
Copy link
Member Author

sxa commented Sep 6, 2024

@Haroon-Khel Ref my last comment are all of those four "bare metal" systems you mentioned all able to run docker? In which case we should add the sw.tool.docker to them (It looks like only the s390x one has it so far).
Is this now closable with the machines now live and in active use?

@Haroon-Khel
Copy link
Contributor

Ive updated the labels just now 👍🏻

@Haroon-Khel
Copy link
Contributor

Haroon-Khel commented Sep 6, 2024

Is this now closable with the machines now live and in active use?

The only thing left on this epic is #3577 which as @steelhead31 has commented is now unblocked

@sxa
Copy link
Member Author

sxa commented Sep 6, 2024

I'll leave it up to you but I'd be happy with closing this on the basis that it's good enough to be able to support and the VPC update is independent. But equally happy with leaving it for now if you'd prefer.

@Haroon-Khel
Copy link
Contributor

Ok ill close this. The remaining tasks can be continued here #3588

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Status: Done
Development

No branches or pull requests

3 participants