CI failure: AsyncSearchActionIT#testMaxMinAggregation

[nik9000]

Build scan:
https://gradle-enterprise.elastic.co/s/5p53iiai3gvyw

Repro line:

./gradlew ':x-pack:plugin:async-search:internalClusterTest' \
  --tests "org.elasticsearch.xpack.search.AsyncSearchActionIT.testMaxMinAggregation" \
  -Dtests.seed=221ED05A04A55CEC -Dtests.security.manager=true -Dtests.locale=uk \
  -Dtests.timezone=Africa/Lusaka -Druntime.java=11

Reproduces locally?:

Applicable branches:
master

Failure history:
https://build-stats.elastic.co/goto/16e2f1e1c2667c565f89df04e8dc8029

Failed a few times over the last week and quite rarely before. Probably something new.

Failure excerpt:

  1> [2020-09-01T16:06:31,216][TRACE][o.e.i.s.GlobalCheckpointSyncAction] [node_s1] action [indices:admin/seq_no/global_checkpoint_sync[r]] completed on shard [[.async-search][0]] for request [GlobalCheckpointSyncAction.Request{shardId=[.async-search][0], timeout=1m, index='.async-search', waitForActiveShards=1}]
  1> [2020-09-01T16:06:31,219][TRACE][o.e.i.s.ReplicationTracker] [node_s2] [.async-search][0] skipped updating local checkpoint of [NqLjLCboQUqi2ptVcHUjcg] from [19] to [19], current checkpoint is higher
  1> [2020-09-01T16:06:31,219][TRACE][o.e.i.s.ReplicationTracker] [node_s2] [.async-search][0] updated local knowledge for [NqLjLCboQUqi2ptVcHUjcg] on the primary of the global checkpoint from [18] to [19]
  1> [2020-09-01T16:06:31,219][TRACE][o.e.i.s.GlobalCheckpointSyncAction] [node_s2] operation succeeded. action [indices:admin/seq_no/global_checkpoint_sync],request [GlobalCheckpointSyncAction.Request{shardId=[.async-search][0], timeout=1m, index='.async-search', waitForActiveShards=1}]
  2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:async-search:internalClusterTest' --tests "org.elasticsearch.xpack.search.AsyncSearchActionIT.testMaxMinAggregation" -Dtests.seed=221ED05A04A55CEC -Dtests.security.manager=true -Dtests.locale=uk -Dtests.timezone=Africa/Lusaka -Druntime.java=11
  2> java.lang.AssertionError: 
    Expected: <INTERNAL_SERVER_ERROR>
         but: was <NOT_FOUND>
        at __randomizedtesting.SeedInfo.seed([221ED05A04A55CEC:BBDE36197CBC3DDC]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.search.AsyncSearchIntegTestCase$2.doNext(AsyncSearchIntegTestCase.java:280)
        at org.elasticsearch.xpack.search.AsyncSearchIntegTestCase$2.next(AsyncSearchIntegTestCase.java:256)
        at org.elasticsearch.xpack.search.AsyncSearchIntegTestCase$2.next(AsyncSearchIntegTestCase.java:243)
        at org.elasticsearch.xpack.search.AsyncSearchActionIT.testMaxMinAggregation(AsyncSearchActionIT.java:102)
  1> [2020-09-01T16:06:31,405][INFO ][o.e.x.s.AsyncSearchActionIT] [testDeleteCleanupIndex] before test
  1> [2020-09-01T16:06:31,407][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [testDeleteCleanupIndex] [621]: starting
  1> [2020-09-01T16:06:31,407][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [testDeleteCleanupIndex] executing initial scroll against [.async-search]
  1> [2020-09-01T16:06:31,411][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [node_s2] scroll returned [1] documents with a scroll id of [FGluY2x1ZGVfY29udGV4dF91dWlkDXF1ZXJ5QW5kRmV0Y2gBFGdhVDlTWFFCdUpFaF9WWlVtMVB3AAAAAAAAABgWeWtyZUhwNEtUZnVXWVFHUWNZWEktZw==]
  1> [2020-09-01T16:06:31,411][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [node_s2] [621]: got scroll response with [1] hits
  1> [2020-09-01T16:06:31,411][DEBUG][o.e.i.r.WorkerBulkByScrollTaskState] [node_s2] [621]: preparing bulk request for [0s]

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[dnhatn]

Thanks Nik! I think this is related to #61560.

[davidkyle]

AsyncSearchActionIT.testInvalidId failed with the same error

java.lang.AssertionError: 
Expected: <INTERNAL_SERVER_ERROR>
     but: was <NOT_FOUND>
	at __randomizedtesting.SeedInfo.seed([D0F42A6E084F5D45:D7E58F433A6D9EC6]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.search.AsyncSearchIntegTestCase$2.doNext(AsyncSearchIntegTestCase.java:280)
	at org.elasticsearch.xpack.search.AsyncSearchIntegTestCase$2.next(AsyncSearchIntegTestCase.java:256)
	at org.elasticsearch.xpack.search.AsyncSearchIntegTestCase$2.next(AsyncSearchIntegTestCase.java:243)
	at org.elasticsearch.xpack.search.AsyncSearchActionIT.testInvalidId(AsyncSearchActionIT.java:245)

https://gradle-enterprise.elastic.co/s/qxwbmowjzmhc2

This one reproduces

./gradlew ':x-pack:plugin:async-search:internalClusterTest' --tests "org.elasticsearch.xpack.search.AsyncSearchActionIT.testInvalidId"   -Dtests.seed=D0F42A6E084F5D45   -Dtests.security.manager=true   -Dtests.locale=sr-Latn-ME   -Dtests.timezone=Africa/Khartoum   -Druntime.java=11