[BUG] [Segment Replication] testCleanupAndPreserveLatestCommitPoint test failure

[dreamer-89]

Describe the bug
org.opensearch.index.store.StoreTests > testCleanupAndPreserveLatestCommitPoint test failure. This is reproducible on main.

org.opensearch.index.store.StoreTests > testCleanupAndPreserveLatestCommitPoint FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([975A0D24C5310633:54A97E709D497A25]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.index.store.StoreTests.testCleanupAndPreserveLatestCommitPoint(StoreTests.java:1192)


Suite: Test class org.opensearch.index.store.StoreTests
  1> [2022-09-16T16:02:58,181][INFO ][o.o.i.s.StoreTests       ] [testCleanupAndPreserveLatestCommitPoint] before test
  1> [2022-09-16T16:02:59,100][INFO ][o.o.i.s.StoreTests       ] [testCleanupAndPreserveLatestCommitPoint] after test
  2> REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.store.StoreTests.testCleanupAndPreserveLatestCommitPoint" -Dtests.seed=975A0D24C5310633 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sk-SK -Dtests.timezone=America/Araguaina -Druntime.java=17
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([975A0D24C5310633:54A97E709D497A25]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.index.store.StoreTests.testCleanupAndPreserveLatestCommitPoint(StoreTests.java:1192)
  2> NOTE: leaving temporary files on disk at: /var/jenkins/workspace/gradle-check/search/server/build/testrun/test/temp/org.opensearch.index.store.StoreTests_975A0D24C5310633-004
  2> NOTE: test params are: codec=Asserting(Lucene94): {}, docValues:{}, maxPointsInLeafNode=182, maxMBSortInHeap=6.877509882400082, sim=Asserting(RandomSimilarity(queryNorm=false): {id=DFI(Standardized), body=F1LOG}), locale=sk-SK, timezone=America/Araguaina
  2> NOTE: Linux 5.15.0-1017-aws amd64/Eclipse Adoptium 17.0.4 (64-bit)/cpus=96,threads=1,free=432954272,total=536870912
  2> NOTE: All tests run in this JVM: [RestoreRemoteStoreResponseTests, TransportVerifyShardBeforeCloseActionTests, GetIndexTemplatesResponseTests, SimulateProcessorResultTests, RefreshPolicyTests, ShardFailedClusterStateTaskExecutorTests, IndexMetadataTests, BalanceConfigurationTests, EnableAllocationShortCircuitTests, GeoDistanceTests, OpenSearchDirectoryReaderTests, BitArrayTests, XContentHelperTests, IndexTests, BinaryDVFieldDataTests, GenericStoreDynamicTemplateTests, AbstractQueryBuilderTests, 

To Reproduce

: ./gradlew ':server:test' --tests "org.opensearch.index.store.StoreTests.testCleanupAndPreserveLatestCommitPoint" -Dtests.seed=975A0D24C5310633 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sk-SK -Dtests.timezone=America/Araguaina -Druntime.java=17

Expected behavior
Test should not fail.

[dbwiddis]

The test is failing due to the existence of a write.lock file in store.directory().

[dbwiddis]

Looks like a write.lock file (IndexWriter.WRITE_LOCK_NAME) is created at various points as part of writing and cleaning up files, but is not committed (intentionally) so doesn't appear in the metadata. But it appears in the directory listing, causing the test failure.