System planned downtime: Equinix aarch64 systems

[sxa]

Equinix are migrating systems in their data centers starting on the 14th of November. This will take up to four days to complete. We have been asked to shut the machines down by end of day on the 13th.

No backups will be performed, but the expectation is that the machines will come back as-is, although potentially with a different IP address.

FYI @Haroon-Khel

[sxa]

NOTE: I have provisioned (but not yet configured) two machines which could be used if we want to have backups during this time if they can be set up before this happens. Both of these are 8 core, 16GB, and 80GB disk (the disk may be the limiting factor for how many images we can host on them in parallel)

• build-osuosl-ubuntu2204-aarch64-1 (140.211.169.57)
• dockerhost-osuosl-ubuntu2204-aarch64-1 (140.211.167.67)

[Haroon-Khel]

Setup here

https://ci.adoptopenjdk.net/computer/build-osuosl-ubuntu2204-aarch64-1/
https://ci.adoptopenjdk.net/computer/dockerhost-osuosl-ubuntu2204-aarch64-1/

Dockerstatic machines (cant get them to connect yet)
https://ci.adoptopenjdk.net/computer/test-docker-ubuntu2204-armv8l-3/ on the build machine
https://ci.adoptopenjdk.net/computer/test-docker-centos8-armv8l-1/ on the docker host machine

[Haroon-Khel]

build-osuosl-ubuntu2204-aarch64-1 fails to build a jdk11 aarch64 binary https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-linux-aarch64-temurin/142/console

12:22:11  Checking out Revision 2ad4d503ba462dcebbba61563f39e2782d357281 (origin/master)
12:22:11   > git config remote.origin.url https://github.com/adoptium/ci-jenkins-pipelines.git # timeout=10
12:22:11   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
12:22:11   > git rev-parse origin/master^{commit} # timeout=10
12:22:11   > git config core.sparsecheckout # timeout=10
12:22:11   > git checkout -f 2ad4d503ba462dcebbba61563f39e2782d357281 # timeout=10
12:22:11  Commit message: "Fixed grammatical errors and updated outdated links in the feature_request.md (#475)"
[Pipeline] sh
12:27:29  process apparently never started in /home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-linux-aarch64-temurin@tmp/durable-e4b81918
12:27:29  (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
12:27:30  $ docker stop --time=1 b6638b4e1c6e4fd7acfb87bec791493128095a5ae62463b53d76ee3af19260c7
12:27:32  $ docker rm -f b6638b4e1c6e4fd7acfb87bec791493128095a5ae62463b53d76ee3af19260c7
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] echo
12:27:32  Execution error: hudson.AbortException: script returned exit code -2
[Pipeline] echo
12:27:32  hudson.AbortException: script returned exit code -2
12:27:32  	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:664)
12:27:32  	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:610)
12:27:32  	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:554)
12:27:32  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
12:27:32  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:27:32  	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
12:27:32  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
12:27:32  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
12:27:32  	at java.base/java.lang.Thread.run(Thread.java:833)

related #2257

[Haroon-Khel]

I cant get the dockerstatic machines to connect to jenkins. The correct public key is on the container, iptables looks fine (cant see any reason the packets would get dropped when comparing it to a working dockerhost machine).

On dockerhost-osuosl-ubuntu2204-aarch64-1

CONTAINER ID   IMAGE       COMMAND               CREATED             STATUS             PORTS                                   NAMES
ee6bf41532c6   aqa_cent8   "/usr/sbin/sshd -D"   About an hour ago   Up About an hour   0.0.0.0:2224->22/tcp, :::2224->22/tcp   cent8.2224

From my workstation

hkhel@hkhel-mac ~ % nc -zv 140.211.167.67 2224
nc: connectx to 140.211.167.67 port 2224 (tcp) failed: Operation timed out

Same goes for dockerstatic containers on the build-osuosl machine

[Haroon-Khel]

The build-osuosl machine can run tests thankfully https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6116/console

[Haroon-Khel]

Dockerstatic machines (cant get them to connect yet)
https://ci.adoptopenjdk.net/computer/test-docker-ubuntu2204-armv8l-3/ on the build machine
https://ci.adoptopenjdk.net/computer/test-docker-centos8-armv8l-1/ on the docker host machine

I've configured them so that they connect to jenkins, as opposed to jenkins connecting to the machines. Should work until I can resolve the connectivity issues

[sxa]

Equinix systems are now back online again (Same IPs so no extra work needed on our part)

[sxa]

I'm going to close this but it sounds like we'll need some remedial work on the OSUOSL systems which can be covered elsewhere.