[BUG] org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testIndexSettingsUpdatedWhenDocrepNodeIsRestarted if flaky

[reta]

Describe the bug

The test case org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testIndexSettingsUpdatedWhenDocrepNodeIsRestarted is flaky:

may 29, 2024 5:43:29 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
ADVERTENCIA: Uncaught exception in thread: Thread[#25677,opensearch[node_t3][refresh][T#1],5,TGRP-RemoteMigrationIndexMetadataUpdateIT]

java.lang.AssertionError: shard [migration-index-allocation-exclude][0] is not locked
	at __randomizedtesting.SeedInfo.seed([D1FAB36610DC8645]:0)
	at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:587)
	at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1244)
	at org.opensearch.index.IndexService.onShardClose(IndexService.java:719)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:842)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:829)
	at org.opensearch.index.store.Store.closeInternal(Store.java:573)
	at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
	at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
	at org.opensearch.index.store.Store.decRef(Store.java:546)
	at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1777)
	at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1753)
	at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4634)
	at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1067)
	at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1211)
	at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Related component

Storage

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testIndexSettingsUpdatedWhenDocrepNodeIsRestarted" -Dtests.seed=D1FAB36610DC8645 

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/39361/testReport/junit/org.opensearch.remotemigration/RemoteMigrationIndexMetadataUpdateIT/testIndexSettingsUpdatedWhenDocrepNodeIsRestarted/

[peternied]

[Triage - attendees 1 2 3 4 5]
@reta Thanks for creating this issue

[ltaragi]

Fixed by #14601