[ci.jenkins.io] ACI agent are all failing

[dduportal]

Service(s)

ci.jenkins.io

Summary

Since ~24 hours, all the ACI agent are failing.

On Azure portal, the containers are present, the image is successfully pulled, but the container process fails immediatly and is killed (then it is restart for a few times before Jenkins reaches the aci-plugin timeout and removes the ACI deployment).

The agent images where changed recently (but it was hard to catch due to us using the "latest" tag for each :'( ). Might be related to jenkinsci/docker-agent#320 (comment).

Reproduction steps

No response

[dduportal]

• Short term (for the upcoming hours): we've pinned the container images to the latest known versions until we determine what is failing.

[smerle33]

As seen during a debug session with the new JDK19, we were able to reproduce the issue.

The ACI container logs says:

Unrecognized VM option ''
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

[dduportal]

This sounds related to jenkinsci/docker-inbound-agent#227 (ping @slide for info)

[dduportal]

After creating a permanent agent on ci.jenkins.io named jenkins-infra-window-test, we ran on a Docker Windows machine an interactive inbound agent container to debug:

> docker run --rm -ti --entrypoint=pwsh jenkinsciinfra/inbound-agent-maven:jdk19-nanoserver
$env:JENKINS_JAVA_OPTS = '-XX:+PrintCommandLineFlags' # Same as our ACI containers
Set-PSDebug -Trace 2 # Enable low level debug for powershell
C:/ProgramData/Jenkins/jenkins-agent.ps1 -Url https://ci.jenkins.io/ -Secret <redacted> -Name jenkins-infra-window-test

Excerpt that points to https://github.com/jenkinsci/docker-inbound-agent/pull/227/files#diff-c0115ea87ae1728b4332277b7b3cd3742bb5ad159ec0a5578027b078b8aa123bR104 as the culprit:

# ...
DEBUG:   96+      >>>> $AgentArguments = @()
DEBUG:     ! SET $AgentArguments = ''.
DEBUG:   98+     if( >>>> ![System.String]::IsNullOrWhiteSpace($JenkinsJavaOpts)) {
DEBUG:  104+          >>>> $AgentArguments += Invoke-Expression "echo $JenkinsJavaOpts"
DEBUG:    1+  >>>> echo -XX:+PrintCommandLineFlags
DEBUG:     ! CALL function '<ScriptBlock>'
DEBUG:     ! SET $AgentArguments = '-XX: +PrintCommandLineFlags'.
DEBUG:  107+      >>>> $AgentArguments += @("-cp", "C:/ProgramData/Jenkins/agent.jar", "hudson.remoting.jnlp.Main", "-headless")
DEBUG:     ! SET $AgentArguments = '-XX: +PrintCommandLineFlags -cp C:/ProgramData…'.
# ...

It seems that the content of the variable JENKINS_JAVA_OPTS is interpolated by powershell: there is a "space" in the flag causing the JVM error message Unrecognized VM option '' (-XX:)

[dduportal]

Tested successfully: using

$AgentArguments += @(`"$JenkinsJavaOpts`")

instead of

$AgentArguments += Invoke-Expression "echo $JenkinsJavaOpts"

as proposed in jenkinsci/docker-inbound-agent#227 (comment)

[slide]

We just need to revert that PR for now, the proposed change you have above won't work for all cases.

[dduportal]

We just need to revert that PR for now, the proposed change you have above won't work for all cases.

Would a documentation PR would be ok instead of rollbacking the feature? (ref. jenkinsci/docker-agent#599)

[slide]

Yes, I didn't see the other comment before commenting here.

[dduportal]

Yes, I didn't see the other comment before commenting here.

Make sense. I was annoyed to have to rollback such a useful feature instead of fixing or documenting it (besides as you underlined, we can fix the infra issue by using quotes correctly)

[dduportal]

Test in progress with the new JDK19 ACI template and quoted flag:

[dduportal]

Test works! https://ci.jenkins.io/job/Infra/job/acceptance-tests/job/check-agent-availability/1832/console

Incoming PR for ci.jenkins.io config (along with updatecli)

[slide]

Excellent!

[dduportal]

• fix(windows ACI agents): add double quotes around JENKINS_JAVA_OPTS jenkins-infra#2509 for the Jenkins infra
• Incoming documentation PR by @smerle33 to jenkinsci/docker-inbound-agent to document the usage of this variable (to avoid others falling into the same trap). Ref. Windows: missing documentation about passing JENKINS_JAVA_OPTS with mandatory double quotes jenkinsci/docker-agent#599

[dduportal]

Closing as we do not have the problem anymore thanks to the tip from @slide 🍯