[BUG] org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testRemoteIndexPathFileExistsAfterMigration if flaky

[reta]

Describe the bug

The test case org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testRemoteIndexPathFileExistsAfterMigration is flaky:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5401, name=Thread-41, state=RUNNABLE, group=TGRP-RemoteMigrationIndexMetadataUpdateIT]
	at __randomizedtesting.SeedInfo.seed([11CD55812A63ECE5:91C370F0F8993BF5]:0)
Caused by: NodeClosedException[node closed {node_t0}{MbdE9_MfQHeVGl_4s1059Q}{EIvRwYPFQu6lEnjGaO4kiQ}{127.0.0.1}{127.0.0.1:45401}{m}{shard_indexing_pressure_enabled=true}]
	at __randomizedtesting.SeedInfo.seed([11CD55812A63ECE5]:0)
	at app//org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:1198)
	at app//org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:387)
	at app//org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:276)
	at app//org.opensearch.cluster.service.ClusterApplierService.addTimeoutListener(ClusterApplierService.java:315)
	at app//org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:203)
	at app//org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:144)
	at app//org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:136)
	at app//org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:1190)
	at app//org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:983)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at app//org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:1193)
	at app//org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:380)
	at app//org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:196)
	at app//org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:144)
	at app//org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:136)
	at app//org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:1190)
	at app//org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:1169)
	at app//org.opensearch.telemetry.tracing.handler.TraceableTransportResponseHandler.handleException(TraceableTransportResponseHandler.java:81)
	at app//org.opensearch.transport.TransportService$9.handleException(TransportService.java:1707)
	at app//org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1493)
	at app//org.opensearch.transport.TransportService$8.run(TransportService.java:1345)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
	at java.base@21.0.3/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@21.0.3/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base@21.0.3/java.lang.Thread.run(Thread.java:1583)

Related component

Storage:Remote

To Reproduce

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testRemoteIndexPathFileExistsAfterMigration" -Dtests.seed=11CD55812A63ECE5

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/39631/testReport/junit/org.opensearch.remotemigration/RemoteMigrationIndexMetadataUpdateIT/testRemoteIndexPathFileExistsAfterMigration/

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@reta Thanks for creating this issue