Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Oracle Java 21: NullPointerException at java.base/javax.crypto.ProviderVerifier.verify #1905

Open
hostalp opened this issue Nov 12, 2024 · 4 comments

Comments

@hostalp
Copy link

hostalp commented Nov 12, 2024

Recently we switched our Jenkins CI/CD environment to Java 21. Most of our Jenkins hosts use some build of OpenJDK (especially the Red Hat build), however some hosts are different (different, older Linux distributions) for which no OpenJDK 21 build is available from their official repositories. In these cases we tried to use the Oracle JDK 21 from https://www.oracle.com/java/technologies/downloads/#java21 (in particular the 21.0.5 deb package https://download.oracle.com/java/21/latest/jdk-21_linux-x64_bin.deb). This works except for one thing related to the Bouncy Castle JCE provider.

To get (checkout) the code from our GIT repository we use the JGit (pure Java) client provided by Jenkins. Since the repository is accessed via secure channel (SSH), some JCE provider is required to be involved and Jenkins uses Bouncy Castle here. While at the OpenJDK 21 things works smoothly, at Oracle JDK 21 we get the exception shown below.

Since the Jenkins (via the Bouncycastle API plugin) currently uses Bouncy Castle 1.78.1 and we noticed that there were some Oracle Java 21 specific fixes in the version 1.79, we also tried to update it manually in our Jenkins environment, but that didn't make any difference.

Meanwhile, we've switched (at the affected hosts) to the Eclipse Temurin OpenJDK 21 build and things started to work again so it really appears to be Oracle JDK 21 related issue.

  • Jenkins job log
[Pipeline] checkout
The recommended git tool is: jgit
using credential ...
Fetching changes from the remote Git repository
Fetching without tags
 > JGit fetch # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from ...
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:997)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:101)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:88)
	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: hudson.plugins.git.GitException: org.eclipse.jgit.api.errors.TransportException: ...: [ssh-connection]: Failed (NoSuchProviderException) to execute: JCE cannot authenticate the provider BC
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.JGitAPIImpl$2.execute(JGitAPIImpl.java:823)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	... 4 more
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to host1
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1826)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:1042)
		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
		at PluginClassLoader for git-client/jdk.proxy76/jdk.proxy76.$Proxy181.execute(Unknown Source)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:995)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:101)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:88)
		at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
		... 4 more
Caused by: org.eclipse.jgit.api.errors.TransportException: ...: [ssh-connection]: Failed (NoSuchProviderException) to execute: JCE cannot authenticate the provider BC
	at PluginClassLoader for git-client//org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:249)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.JGitAPIImpl$2.execute(JGitAPIImpl.java:821)
	... 10 more
Caused by: org.eclipse.jgit.errors.TransportException: ...: [ssh-connection]: Failed (NoSuchProviderException) to execute: JCE cannot authenticate the provider BC
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:267)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:1)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:279)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:152)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:153)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:105)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.Transport.fetch(Transport.java:1480)
	at PluginClassLoader for git-client//org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:238)
	... 11 more
Caused by: org.apache.sshd.common.SshException: [ssh-connection]: Failed (NoSuchProviderException) to execute: JCE cannot authenticate the provider BC
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.future.AbstractSshFuture.lambda$verifyResult$2(AbstractSshFuture.java:146)
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.future.AbstractSshFuture.formatExceptionMessage(AbstractSshFuture.java:206)
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:145)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:56)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:35)
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:110)
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:96)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:171)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:100)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:260)
	... 19 more
Caused by: java.lang.NullPointerException
	at java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:123)
	at java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:196)
	at java.base/javax.crypto.JceSecurity$2.apply(JceSecurity.java:222)
	at java.base/javax.crypto.JceSecurity$2.apply(JceSecurity.java:211)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:211)
	at java.base/javax.crypto.JceSecurity.getInstance(JceSecurity.java:134)
	at java.base/javax.crypto.KeyAgreement.getInstance(KeyAgreement.java:240)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.util.security.SecurityEntityFactory$2.getInstance(SecurityEntityFactory.java:130)
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.util.security.SecurityUtils.getKeyAgreement(SecurityUtils.java:817)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.kex.MontgomeryCurve.<init>(MontgomeryCurve.java:116)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.kex.MontgomeryCurve.<clinit>(MontgomeryCurve.java:92)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.kex.BuiltinDHFactories$16.isSupported(BuiltinDHFactories.java:341)
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.NamedFactory.lambda$setUpTransformedFactories$0(NamedFactory.java:64)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.NamedFactory.setUpTransformedFactories(NamedFactory.java:66)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.client.ClientBuilder.setUpDefaultKeyExchanges(ClientBuilder.java:194)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:125)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:56)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:274)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:165)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:56)
	at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:298)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.sshd.SshdSessionFactory.lambda$0(SshdSessionFactory.java:225)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.sshd.SshdSession.<init>(SshdSession.java:92)
	at PluginClassLoader for git-client//org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:196)
	... 19 more
ERROR: Error fetching remote repo 'origin'
ERROR: Maximum checkout retry attempts reached, aborting
  • Jenkins remoting agent log
Nov 08, 2024 8:20:40 AM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(BC) created instance of org.bouncycastle.jce.provider.BouncyCastleProvider
Nov 08, 2024 8:20:40 AM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
Nov 08, 2024 8:20:40 AM org.apache.sshd.common.io.DefaultIoServiceFactoryFactory getIoServiceProvider
INFO: No detected/configured IoServiceFactoryFactory; using Nio2ServiceFactoryFactory
Nov 08, 2024 8:20:41 AM org.apache.sshd.common.util.logging.LoggingUtils warn
WARNING: exceptionCaught(JGitClientSession[...:22])[state=Opened] NoSuchProviderException: JCE cannot authenticate the provider BC
@peterdettman
Copy link
Collaborator

So Oracle JDK 21 is attempting to authenticate the BC provider and fails due to NullPointerException during the attempt. The underlying cause of authentication failures is usually some sort of modification to the provider jar (including any kind of repackaging e.g. fat jars, or exploding into class trees).

Note that OpenJDK doesn't perform this authentication of third-party cryptographic providers, which would explain why you only see it when trying to use the Oracle JDK.

@hostalp
Copy link
Author

hostalp commented Nov 13, 2024

I already checked the jars - both at the Jenkins server side as well as at the worker nodes to which these jars are copied when needed and they were the original untouched 1.78.1 and then 1.79 (when I replaced them manually. and cleared the jar cache at the worker nodes).

Verification via jarsigner (from the same Oracle JDK 21) results in the following:

s     802861 Wed Oct 30 09:42:52 CET 2024 META-INF/MANIFEST.MF

      >>> Signer
      X.509, CN=Legion of the Bouncy Castle Inc., OU=Java Software Code Signing, O=Oracle Corporation
      Signature algorithm: SHA256withRSA, 2048-bit key
      [certificate is valid from 1/25/22, 1:58 AM to 1/25/27, 1:58 AM]
      X.509, CN=JCE Code Signing CA, OU=Java Software Code Signing, O=Oracle Corporation
      Signature algorithm: SHA256withRSA, 2048-bit key
      [certificate is valid from 7/7/16, 1:48 AM to 12/31/30, 1:00 AM]
      [Invalid certificate chain: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]
      >>> TSA
      X.509, CN=DigiCert Timestamp 2024, O=DigiCert, C=US
      Signature algorithm: SHA256withRSA, 4096-bit key
      [certificate is valid from 9/26/24, 2:00 AM to 11/26/35, 12:59 AM]
      X.509, CN=DigiCert Trusted G4 RSA4096 SHA256 TimeStamping CA, O="DigiCert, Inc.", C=US
      Signature algorithm: SHA256withRSA, 4096-bit key
      [certificate is valid from 3/23/22, 1:00 AM to 3/23/37, 12:59 AM]
      X.509, CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US
      Signature algorithm: SHA384withRSA, 4096-bit key
      [certificate is valid from 8/1/22, 2:00 AM to 11/10/31, 12:59 AM]
...
jar verified.

Warning:
This jar contains entries whose certificate chain is invalid. Reason: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The signer certificate will expire on 2027-01-25.
The timestamp will expire on 2031-11-10.

Where that complaint about the untrusted certificate chain shouldn't be critical AFAIK (and it's already been discussed in #1193).

So I believe the issue is slightly different than the JAR validity.

@dghgit
Copy link
Contributor

dghgit commented Nov 24, 2024

Is the provider being loaded via the JVMs module path? It looks like the jar's valid, but if something is unpacking it or re-wrapping it before it gets used there will be trouble.

If you're sure it is, I think you may need to raise this with Oracle, we have worked around an issue in Java 21 as well as reporting one recently. If it's something we can deal with on our end we would be happy to do so, but available source does not really seem to suggest what's going on...

@hostalp
Copy link
Author

hostalp commented Nov 28, 2024

I'm not sure how ir's loaded since it's done by Jenkins which first transfers the required libraries from the master to the worker nodes via its remoting implementation and then loads them locally at the worker nodes. It may be therefore worth opening a Jenkins (likely for the Jenkins Bouncy Castle API Plugin) issue for this instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants