Problem machines for release #2662

Haroon-Khel · 2022-07-07T16:17:13Z

test-docker-fedora34-x64-1 and (newly created) test-docker-fedora34-x64-2 ref #2631 JDK8

The following tests are failing on both -1 and -2. Links are for -2
java/nio/file/Files/probeContentType/Basic.java
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5133/console
java/net/Inet6Address/B6206527.java.B6206527
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5135/console
java/net/ipv6tests/B6521014.java
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5136/console

test-osuosl-centos74-ppc64le-1/ and test-osuosl-centos74-ppc64le-2/ ref #2625 JDK8

On test-osuosl-centos74-ppc64le-1

sun/security/pkcs11/fips/TestTLS12.java
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5092/console

sun/tools/jinfo/Basic.sh
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5094/console

On test-osuosl-centos74-ppc64le-2

sun/security/pkcs11/fips/TestTLS12.java
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5095/console

~~sun/tools/jinfo/Basic.sh~~ resolved
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5103/console

test-azure-win2012r2-x64-3 and test-azure-win2019-x64-1 ref #2645 JDK11

sophia-guo · 2022-07-13T19:17:33Z

ERROR: Cannot delete workspace :Malformed input or input contains unmappable characters #2630

sophia-guo · 2022-07-14T02:13:10Z

test-azure-win2012r2-x64-1

ERROR: Cannot delete workspace :Unable to delete 'D:\jenkins\workspace\Test_openjdk11_hs_sanity.openjdk_x86-64_windows\openjdkbinary\j2sdk-image\lib\modules'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

Recent two run:
https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/647/console
https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/647/console

Haroon-Khel · 2022-07-14T15:31:17Z

That directory is being used by leftover jcmd.exe processes

#2635 is related. It is surprising to find that this is occurring on a different machine this time

Haroon-Khel · 2022-07-14T16:07:28Z

sun/tools/jinfo/Basic.sh on the 2 linux ppc64le machines has been resolved, #2625 (comment)

Haroon-Khel · 2022-07-14T16:28:51Z

Stewart has added jcmd to the list of process to kill https://ci.adoptopenjdk.net/view/Tooling/job/SXA-processCheck/, #2635 (comment)

sophia-guo · 2022-07-14T20:24:34Z

test-azure-win2012r2-x64-1

https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/648/console
https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/647/console
https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/646/console

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
ERROR: Cannot delete workspace :Unable to delete 'D:\jenkins\workspace\Test_openjdk11_hs_sanity.openjdk_x86-64_windows\openjdkbinary\j2sdk-image\lib\modules'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
[Pipeline] }
[Pipeline] // timeout
[Pipeline] echo
Exception: hudson.AbortException: Cannot delete workspace: Unable to delete 'D:\jenkins\workspace\Test_openjdk11_hs_sanity.openjdk_x86-64_windows\openjdkbinary\j2sdk-image\lib\modules'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

All three recent jobs are assigned to this machine and failed . all failed with running this specific machine

sxa · 2022-07-15T09:57:54Z

test-azure-win2012r2-x64-1

https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/648/console https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/647/console https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/646/console
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
ERROR: Cannot delete workspace :Unable to delete 'D:\jenkins\workspace\Test_openjdk11_hs_sanity.openjdk_x86-64_windows\openjdkbinary\j2sdk-image\lib\modules'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
[Pipeline] }
[Pipeline] // timeout
[Pipeline] echo
Exception: hudson.AbortException: Cannot delete workspace: Unable to delete 'D:\jenkins\workspace\Test_openjdk11_hs_sanity.openjdk_x86-64_windows\openjdkbinary\j2sdk-image\lib\modules'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
All three recent jobs are assigned to this machine and failed . all failed with running this specific machine

Fixed as per #2209 (comment)

Haroon-Khel · 2022-07-15T16:33:51Z

Machines that are still problematic:

Any fedora dockerstatic container. ref #2631, any fedora container on https://ci.adoptopenjdk.net/computer/docker-packet-ubuntu2004-intel-1/ will pass ipv6 tests while those on https://ci.adoptopenjdk.net/computer/docker-packet-ubuntu2004-amd-1/ will fail them. The difference needs to be investigated.
I cant get java/nio/file/Files/probeContentType/Basic.java to pass on any Fedora container, see #2631 (comment)

test-osuosl-centos74-ppc64le-1 and -2
sun/tools/jinfo/Basic.sh now passes, but sun/security/pkcs11/fips/TestTLS12.java still fails. See #2625 (comment)

test-azure-win2012r2-x64-3 and test-azure-win2019-x64-1
see #2645 (comment)
Failures are intermittent, but more failures than passes.

If by Monday these issues are not resolved, I'll turn the jenkins nodes offline for the release

Haroon-Khel · 2022-07-19T16:27:38Z

I was able to get java/nio/file/Files/probeContentType/Basic.java to pass on our fedora boxes, see #2631 (comment), however I have not solved the failing ipv6 tests on fedora containers hosted on https://ci.adoptopenjdk.net/computer/docker-packet-ubuntu2004-amd-1/.

And sun/security/pkcs11/fips/TestTLS12.java continues to fail on test-osuosl-centos74-ppc64le-1 and -2, see #2625

I have temporarily turned offline the following nodes for this release

https://ci.adoptopenjdk.net/computer/test-docker-fedora34-x64-1/
https://ci.adoptopenjdk.net/computer/test-docker-fedora34-x64-2/
https://ci.adoptopenjdk.net/computer/test-docker-fedora36-x64-1/
https://ci.adoptopenjdk.net/computer/test-osuosl-centos74-ppc64le-1/
https://ci.adoptopenjdk.net/computer/test-osuosl-centos74-ppc64le-2/

Haroon-Khel · 2022-09-08T12:18:36Z

https://ci.adoptopenjdk.net/computer/test-docker-fedora34-x64-1/
https://ci.adoptopenjdk.net/computer/test-docker-fedora34-x64-2/
https://ci.adoptopenjdk.net/computer/test-docker-fedora36-x64-1/
https://ci.adoptopenjdk.net/computer/test-osuosl-centos74-ppc64le-1/
https://ci.adoptopenjdk.net/computer/test-osuosl-centos74-ppc64le-2/

I've turned these machines back online

sxa · 2022-10-03T14:04:12Z

@Haroon-Khel Can you give a status update on the systems that were problematic - have they all now been resolved or is there still work to do here. Need to know whether it can be closed or whether it needs to move to October.

Haroon-Khel · 2022-10-05T14:43:45Z

Since sun/security/pkcs11/fips/TestTLS12.java continues to fail on test-osuosl-centos74-ppc64le-1 and -2 this issue should be kept open

sxa · 2022-11-23T11:17:52Z

Related: #2815

sxa · 2024-01-03T17:55:44Z

List of likely machine related tests which I'm giong to stop bumping between iterations so we can track based on their attachment to this issue:

RHEL/CentOS*:

AIX:

Linux/s390x:

test-marist-sles15-s390x-2 - NoRouteToHostException: No Route to Host failures #2869
Some s390x machines failing net tests with NoRouteToHostException #2807
jdk_management tests consistently fail on test-marist-ubuntu2204-s390x-1 #3189
MBCS_Tests_datetime related formatting test failures on IBM Cloud - RHEL only #2163
jdk_net FAILED in Test_openjdk18_hs_extended.openjdk_aarch64_linux on test-docker* machines (aarch64) #2514

sxa · 2024-01-24T15:59:06Z

List of test failures on JDK8/arm32 (at a minimum) including the perf test suites which are failing in the containerised environments on the arm64 hosts, but are ok on the two physical ODROID machines:
#3043

sxa · 2024-04-16T16:29:33Z

Stuff identified during April 2024 dry runs:

Test8009761.java fails on Linux/ppc64le Ubuntu 18.04 but is ok on other Ubuntu releases
sun/security/krb5/auto/rcache_usemd5.sh on aarch64 passes only on Ubuntu 20.04 and earlier
java/nio/file/DirectoryStream/SecureOS.java fails on Alpine 3.19 but passes on 3.14

Haroon-Khel added the ansible label Jul 7, 2022

Haroon-Khel mentioned this issue Jul 14, 2022

build-azure-win2012r2-x64-1 J:\jenkins\workspace\build-scripts\release\create_installer_windows locked and can't be deleted #2635

Closed

sophia-guo mentioned this issue Jul 14, 2022

nightly triage adoptium/aqa-tests#3854

Closed

sxa pinned this issue Jul 15, 2022

sxa added critical testFail labels Jul 15, 2022

sxa added this to Adoptium September 2022 Plan Sep 7, 2022

sxa removed this from Adoptium September 2022 Plan Oct 5, 2022

sxa added this to Adoptium October 2022 Plan Oct 5, 2022

sxa mentioned this issue Oct 5, 2022

Two jdk8 security tests consistently failed on test-osuosl-centos74-ppc64le-1 #2625

Closed

sxa removed this from Adoptium October 2022 Plan Nov 2, 2022

sxa added this to Adoptium November 2022 Plan Nov 2, 2022

sxa added this to Adoptium December 2022 Plan Nov 23, 2022

sxa removed this from Adoptium November 2022 Plan Nov 23, 2022

sxa added this to the 2022-12 (December) milestone Nov 23, 2022

sxa removed this from Adoptium December 2022 Plan Jan 11, 2023

sxa added this to Adoptium 1Q 2023 Plan Jan 11, 2023

smlambert mentioned this issue Jan 8, 2024

All JlmRemote* tests failed with same or similar issue: **FAILED** Process LT1 has ended unexpectedly on aarch64_linux adoptium/aqa-systemtest#459

Open

smlambert moved this to In Progress in Adoptium 4Q 2023 Plan Jan 8, 2024

smlambert moved this to In Progress in 2024 1Q Adoptium Plan Jan 8, 2024

smlambert removed this from Adoptium 4Q 2023 Plan Jan 8, 2024

sxa mentioned this issue Jan 15, 2024

EPIC: System specific failures #2446

Closed

This was referenced Jan 30, 2024

jdk_time target fails on some arm32_linux nodes #3360

Closed

java/time/test/java/time/format/TestUTCParse fails on macstadium machines (expects UTC not PST) #3361

Open

sxa modified the milestones: 2024-01 (January), 2024-02 (February) Feb 9, 2024

sxa mentioned this issue Feb 20, 2024

Nagios: mitigate large workspaces on dockerhost-equinix-ubuntu2204-x64-1 #3405

Open

sxa modified the milestones: 2024-02 (February), 2024-03 (March) Mar 7, 2024

sxa modified the milestones: 2024-03 (March), 2024-04 (April) Apr 4, 2024

sxa modified the milestones: 2024-04 (April), 2024-05 (May) May 13, 2024

sxa added this to 2024 2Q Adoptium Plan May 13, 2024

sxa modified the milestones: 2024-05 (May), 2024-08 (August) Jul 31, 2024

sxa modified the milestones: 2024-08 (August), 2024-09 (September) Sep 4, 2024

sxa removed this from 2024 2Q Adoptium Plan Nov 1, 2024

sxa added this to 2024 4Q Adoptium Plan Nov 1, 2024

sxa removed this from 2024 1Q Adoptium Plan Nov 1, 2024

sxa modified the milestones: 2024-09 (September), 2024-11 (November) Nov 1, 2024

smlambert moved this to Todo in 2024 4Q Adoptium Plan Nov 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem machines for release #2662

Problem machines for release #2662

Haroon-Khel commented Jul 7, 2022 •

edited

Loading

sophia-guo commented Jul 13, 2022

sophia-guo commented Jul 14, 2022

Haroon-Khel commented Jul 14, 2022 •

edited

Loading

Haroon-Khel commented Jul 14, 2022 •

edited

Loading

Haroon-Khel commented Jul 14, 2022

sophia-guo commented Jul 14, 2022 •

edited

Loading

sxa commented Jul 15, 2022

Haroon-Khel commented Jul 15, 2022 •

edited

Loading

Haroon-Khel commented Jul 19, 2022

Haroon-Khel commented Sep 8, 2022

sxa commented Oct 3, 2022

Haroon-Khel commented Oct 5, 2022

sxa commented Nov 23, 2022

sxa commented Jan 3, 2024

sxa commented Jan 24, 2024

sxa commented Apr 16, 2024 •

edited

Loading

Problem machines for release #2662

Problem machines for release #2662

Comments

Haroon-Khel commented Jul 7, 2022 • edited Loading

sophia-guo commented Jul 13, 2022

sophia-guo commented Jul 14, 2022

Haroon-Khel commented Jul 14, 2022 • edited Loading

Haroon-Khel commented Jul 14, 2022 • edited Loading

Haroon-Khel commented Jul 14, 2022

sophia-guo commented Jul 14, 2022 • edited Loading

sxa commented Jul 15, 2022

Haroon-Khel commented Jul 15, 2022 • edited Loading

Haroon-Khel commented Jul 19, 2022

Haroon-Khel commented Sep 8, 2022

sxa commented Oct 3, 2022

Haroon-Khel commented Oct 5, 2022

sxa commented Nov 23, 2022

sxa commented Jan 3, 2024

sxa commented Jan 24, 2024

sxa commented Apr 16, 2024 • edited Loading

Haroon-Khel commented Jul 7, 2022 •

edited

Loading

Haroon-Khel commented Jul 14, 2022 •

edited

Loading

Haroon-Khel commented Jul 14, 2022 •

edited

Loading

sophia-guo commented Jul 14, 2022 •

edited

Loading

Haroon-Khel commented Jul 15, 2022 •

edited

Loading

sxa commented Apr 16, 2024 •

edited

Loading