[CI] SamlServiceProviderMetadataIT classMethod failing

[mark-vieira]

This is only failing on Java 23. The failure is due to a timeout attempting to start the cluster so something in getting hung on Java 23 here.

Build scan:
https://gradle-enterprise.elastic.co/s/w4kjz3jbtvl7s/tests/:x-pack:plugin:security:qa:saml-rest-tests:javaRestTest/org.elasticsearch.xpack.security.authc.saml.SamlServiceProviderMetadataIT

Reproduction line:

null

Applicable branches:
main

Reproduces locally?:
Yes

Failure history:
Failure dashboard for org.elasticsearch.xpack.security.authc.saml.SamlServiceProviderMetadataIT#classMethod

Failure excerpt:

java.lang.RuntimeException: An error occurred orchestrating test cluster.

  at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:269)
  at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.writeUnicastHostsFile(DefaultLocalClusterHandle.java:250)
  at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.waitUntilReady(DefaultLocalClusterHandle.java:193)
  at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.start(DefaultLocalClusterHandle.java:79)
  at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:45)
  at org.elasticsearch.test.junit.RunnableTestRuleAdapter$1.evaluate(RunnableTestRuleAdapter.java:43)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1575)

  Caused by: java.lang.RuntimeException: Timed out after PT3M waiting for ports files for: { cluster: 'test-cluster', node: 'test-cluster-0' }

    at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.waitUntilReady(AbstractLocalClusterFactory.java:285)
    at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.getTransportEndpoint(AbstractLocalClusterFactory.java:204)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
    at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
    at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:960)
    at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:934)
    at java.util.stream.AbstractTask.compute(AbstractTask.java:327)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:759)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
    at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:676)
    at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:927)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:264)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:727)
    at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$writeUnicastHostsFile$13(DefaultLocalClusterHandle.java:250)
    at java.util.concurrent.ForkJoinTask$AdaptedInterruptibleCallable.compute(ForkJoinTask.java:1689)
    at java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1489)
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:2071)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2033)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[tvernum]

There's 2 issues here:

1. We don't configure any socket timeouts in the HTTP Resolver, and it blocks the main startup thread. The blocking is somewhat intentional (though not necessarily a good idea) because we provide the option to fail the node if the metadata can't be resolved
2. For some reason on JDK23 the MockHttpServer is responding to the first request, but the 2nd request times out. @jakelandis ran into some other behaviour changes with the mock http server in JDK23 ([CI] SSLConfigurationReloaderTests testReloadingKeyStore failing #108774 (comment)), so I guess we need to be on the lookout for more of them.

I haven't fixed it yet, but it's clear the thing that broke is the mock HTTP server, so I'm setting the risk to low.