[CI][ML] PyTorchModelIT testEvaluateWithMinimalTimeout failure

[davidkyle]

Build scan:
https://gradle-enterprise.elastic.co/s/irafccgouilyk
https://gradle-enterprise.elastic.co/s/vgysxwentnox6

Repro line:

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:javaRestTest' --tests "org.elasticsearch.xpack.ml.integration.PyTorchModelIT.testEvaluateWithMinimalTimeout" \
  -Dtests.seed=82882C0276D19551 \
  -Dtests.locale=fr-CH \
  -Dtests.timezone=Europe/Dublin \
  -Druntime.java=17

Reproduces locally?:
No

Applicable branches:
8.1, 8.0

Failure history:

Failure excerpt:

junit.framework.AssertionFailedError: Expected exception ResponseException but no exception was thrown
	at __randomizedtesting.SeedInfo.seed([82882C0276D19551:65FE53F0D91AAB91]:0)
	at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2874)
	at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2860)
	at org.elasticsearch.xpack.ml.integration.PyTorchModelIT.testEvaluateWithMinimalTimeout(PyTorchModelIT.java:203)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

The test is meant to show the inference timeout parameter is respected by setting the value to zero but there is a race between the timeout code and inference in some cases inference occurs normally.

In this case the logs show the request was processed in 4ms.

[2021-11-22T22:34:51,148][DEBUG][o.e.x.m.i.d.DeploymentManager] [javaRestTest-1] Inference Request {"request_id":"9","tokens":[[3,4]],"arg_1":[[1,1]],"arg_2":[[0,0]],"arg_3":[[0,1]]}
[2021-11-22T22:34:51,152][DEBUG][o.e.x.m.i.d.DeploymentManager] [javaRestTest-1] [test_evaluate_timeout] retrieved result for request [9]
[2021-11-22T22:34:51,152][DEBUG][o.e.x.m.i.d.DeploymentManager] [javaRestTest-1] [test_evaluate_timeout] processed result for request [9]