[CI] WildflyIT is failing on FIPS-enabled JVMs

[DaveCTurner]

See for instance https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/210/consoleText

NB ES_RUNTIME_JAVA=java8fips - it seems to be passing fine on other JVMs.

The exception from the log looks as follows:

  2> java.io.IOException: Invalid keystore format
  2> 	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:658)
  2> 	at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
  2> 	at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
  2> 	at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
  2> 	at java.security.KeyStore.load(KeyStore.java:1445)
  2> 	at sun.security.util.AnchorCertificates$1.run(AnchorCertificates.java:61)
  2> 	at sun.security.util.AnchorCertificates$1.run(AnchorCertificates.java:52)
  2> 	at java.security.AccessController.doPrivileged(Native Method)
Suite: org.elasticsearch.wildfly.WildflyIT
  2> 	at sun.security.util.AnchorCertificates.<clinit>(AnchorCertificates.java:52)
  2> 	at sun.security.provider.certpath.AlgorithmChecker.checkFingerprint(AlgorithmChecker.java:214)
  2> 	at sun.security.provider.certpath.AlgorithmChecker.<init>(AlgorithmChecker.java:164)
ERROR   0.00s | WildflyIT (suite) <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: The test or suite printed 12890 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
  2> 	at sun.security.provider.certpath.AlgorithmChecker.<init>(AlgorithmChecker.java:118)
   > 	at __randomizedtesting.SeedInfo.seed([E6F2F43820D93CA5]:0)
   > 	at java.lang.Thread.run(Thread.java:748)
Completed [1/1] in 4.07s, 1 test, 1 failure <<< FAILURES!

  2> 	at sun.security.validator.SimpleValidator.engineValidate(SimpleValidator.java:157)
  2> 	at sun.security.validator.Validator.validate(Validator.java:260)
  2> 	at sun.security.validator.Validator.validate(Validator.java:236)
  2> 	at sun.security.validator.Validator.validate(Validator.java:205)
  2> 	at javax.crypto.JarVerifier.isTrusted(JarVerifier.java:610)
  2> 	at javax.crypto.JarVerifier.verifySingleJar(JarVerifier.java:530)
  2> 	at javax.crypto.JarVerifier.verifyJars(JarVerifier.java:363)
  2> 	at javax.crypto.JarVerifier.verify(JarVerifier.java:289)
  2> 	at javax.crypto.JceSecurity.verifyProviderJar(JceSecurity.java:164)
  2> 	at javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:190)
  2> 	at javax.crypto.JceSecurity.getInstance(JceSecurity.java:114)
  2> 	at javax.crypto.KeyAgreement.getInstance(KeyAgreement.java:270)
  2> 	at sun.security.ssl.JsseJce.getKeyAgreement(JsseJce.java:271)
  2> 	at sun.security.ssl.JsseJce$EcAvailability.<clinit>(JsseJce.java:418)
  2> 	at sun.security.ssl.JsseJce.isEcAvailable(JsseJce.java:194)
  2> 	at sun.security.ssl.CipherSuite$KeyExchange.isAvailable(CipherSuite.java:371)
  2> 	at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:185)
  2> 	at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:304)
  2> 	at sun.security.ssl.SSLContextImpl.access$100(SSLContextImpl.java:42)
  2> 	at sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLContextImpl.java:425)
  2> 	at java.lang.Class.forName0(Native Method)
  2> 	at java.lang.Class.forName(Class.java:264)
  2> 	at java.security.Provider$Service.getImplClass(Provider.java:1634)
  2> 	at java.security.Provider$Service.newInstance(Provider.java:1592)
  2> 	at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
  2> 	at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
  2> 	at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
  2> 	at org.apache.http.ssl.SSLContexts.createDefault(SSLContexts.java:51)
  2> 	at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:966)
  2> 	at org.elasticsearch.wildfly.WildflyIT.testTransportClient(WildflyIT.java:56)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.lang.reflect.Method.invoke(Method.java:498)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
  2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at java.lang.Thread.run(Thread.java:748)

I guess that the java.lang.AssertionError seems to be because there's that stack trace, and isn't the root cause?

  > Throwable #1: java.lang.AssertionError: The test or suite printed 12890 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

[elasticmachine]

Pinging @elastic/es-security

[DaveCTurner]

Also on 6.4:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/9/console

[jkakavas]

This doesn't reproduce locally in a FIPS JVM, I'm still looking.

[DaveCTurner]

This test is muted by #32543 in the master, 6.x, 6.4 and 6.3 branches.

[DaveCTurner]

I had to revert the backport to 6.3 in
e189796 because gradle does not know about the inFipsJvm property in that branch:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+intake/333/console

[jkakavas]

The

  2> java.io.IOException: Invalid keystore format

is a red herring (explanation in #32737 (comment)) but it contributes to the failure as the stderr output causes:

  > Throwable #1: java.lang.AssertionError: The test or suite printed 12890 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

I still can't reproduce this locally though

RUNTIME_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64-fips ./gradlew :qa:wildfly:check

passes.
I will attempt to unmute this one and add a @LuceneTestCase.SuppressSysoutChecks

[DaveCTurner]

I think it'd be better to relax the limit to something ≥ 12890 bytes rather than to remove it completely.

[jkakavas]

Resolved by #32814