[CI] DefaultEndPointsIT testInferDeploysDefaultE5 failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #4620 / openjdk17_checkpart2_java-fips-matrix
• elasticsearch-pull-request #38502 / part-2
• elasticsearch-pull-request #38435 / part-2
• elasticsearch-pull-request #38288 / part-2
• elasticsearch-intake #11684 / part2

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:inference-service-tests:javaRestTest" --tests "org.elasticsearch.xpack.inference.DefaultEndPointsIT.testInferDeploysDefaultE5" -Dtests.seed=FF3CF2BC792F7F1B -Dtests.locale=my -Dtests.timezone=Asia/Irkutsk -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.net.SocketTimeoutException: ၁၂၀,၀၀၀ milliseconds timeout on connection http-outgoing-1 [ACTIVE]

Issue Reasons:

• [8.x] 5 failures in test testInferDeploysDefaultE5 (0.7% fail rate in 667 executions)
• [8.x] 3 failures in step part-2 (3.2% fail rate in 93 executions)
• [8.x] 3 failures in pipeline elasticsearch-pull-request (3.2% fail rate in 93 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testInferDeploysDefaultE5 (0.9% fail rate in 214 executions)

Build Scans:

• elasticsearch-intake #11669 / part2
• elasticsearch-pull-request #37685 / part-2

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

This has been muted on branch 8.16

Mute Reasons:

• [8.16] 2 failures in test testInferDeploysDefaultE5 (0.5% fail rate in 441 executions)

Build Scans:

• elasticsearch-intake #11757 / part2
• elasticsearch-periodic-platform-support #4459 / ubuntu-2004_platform-support-unix

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 5 failures in test testInferDeploysDefaultE5 (0.7% fail rate in 667 executions)
• [8.x] 3 failures in step part-2 (3.2% fail rate in 93 executions)
• [8.x] 3 failures in pipeline elasticsearch-pull-request (3.2% fail rate in 93 executions)

Build Scans:

• elasticsearch-periodic #4620 / openjdk17_checkpart2_java-fips-matrix
• elasticsearch-pull-request #38502 / part-2
• elasticsearch-pull-request #38435 / part-2
• elasticsearch-pull-request #38288 / part-2
• elasticsearch-intake #11684 / part2

[maxhniebergall]

the feature under test here is behind a feature flag, so, despite these tests being muted, I believe this is overall low-risk

[davidkyle]

The test has as 30 second timeout but the model download took much longer. The fix is use a longer timeout

[o.e.x.m.p.a.TransportLoadTrainedModelPackage] [test-cluster-0] [.multilingual-e5-small_linux-x86_64] finished model import after [85] seconds

[dan-rubinstein]

The builds above seem to have 2 failures:

1. Expected and actual max_number_of_allocations is different. This seems to have been fixed as part of this change

2. Timeout failures when making an inference call to the default endpoints.

Looking at the timeout failures, it seems that the exception message is:

java.net.SocketTimeoutException: 120,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]

Seems like the test has a 120 second timeout instead of a 30 second timeout as mentioned above. The original call to create the endpoint has a 30 second timeout but it does not stop the deployment process and we do not wait on this response. Instead we likely need to bump up the 120 second timeout here. I'll put out a PR to raise the timeout and unmute the test.