IndexWriter AlreadyClosedException on segment replication failure

[dreamer-89]

IndexWriter already closed exception while decRef'ing copyState post clean up on segrep source. This error is repro'able with org.opensearch.aliases.IndexAliasesIT.testCreateIndexWithAliasesInSource test class. This failure is impacting multiple integration tests.

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=96, name=opensearch[node_s0][generic][T#2], state=RUNNABLE, group=TGRP-IndexAliasesIT]
Caused by: java.lang.AssertionError: java.lang.AssertionError: map: listener.onFailure failed
	at __randomizedtesting.SeedInfo.seed([A364DF648971D858]:0)
	at org.opensearch.action.ActionListener$4.onFailure(ActionListener.java:195)
	at org.opensearch.action.ActionListener$6.onFailure(ActionListener.java:309)
	at org.opensearch.action.support.RetryableAction$RetryingListener.onFinalFailure(RetryableAction.java:218)
	at org.opensearch.action.support.RetryableAction$RetryingListener.onFailure(RetryableAction.java:210)
	at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:74)
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1414)
	at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.AssertionError: map: listener.onFailure failed
	... 11 more
Caused by: ReplicationFailedException[ReplicationTarget is used but it's refcount is 0. Probably a mismatch between incRef/decRef calls]
	at app//org.opensearch.indices.replication.common.ReplicationTarget.ensureRefCount(ReplicationTarget.java:189)
	at app//org.opensearch.indices.replication.common.ReplicationTarget.indexShard(ReplicationTarget.java:121)
	at app//org.opensearch.indices.replication.SegmentReplicationTargetService$3.onFailure(SegmentReplicationTargetService.java:367)
	at app//org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88)
	at app//org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:135)
	at app//org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:112)
	at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
	at app//org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:112)
	at app//org.opensearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:178)
	at app//org.opensearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:149)
	at app//org.opensearch.action.StepListener.innerOnFailure(StepListener.java:82)
	at app//org.opensearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:62)
	at app//org.opensearch.action.ActionListener$4.onFailure(ActionListener.java:190)
	... 10 more
	Suppressed: RemoteTransportException[[node_s2][127.0.0.1:44137][internal:index/shard/replication/get_segment_files]]; nested: AlreadyClosedException[this IndexWriter is closed];
	Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
		at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:908)
		at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:921)
		at org.apache.lucene.index.IndexWriter.decRefDeleter(IndexWriter.java:5852)
		at org.opensearch.index.engine.InternalEngine.lambda$getSegmentInfosSnapshot$5(InternalEngine.java:2320)
		at org.opensearch.common.concurrent.GatedCloseable.close(GatedCloseable.java:47)
		at org.opensearch.indices.replication.common.CopyState.closeInternal(CopyState.java:67)
		at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
		at org.opensearch.indices.replication.OngoingSegmentReplications.removeCopyState(OngoingSegmentReplications.java:254)
		at org.opensearch.indices.replication.OngoingSegmentReplications.lambda$startSegmentCopy$0(OngoingSegmentReplications.java:120)
		at org.opensearch.action.ActionListener$6.onResponse(ActionListener.java:294)
		at org.opensearch.indices.replication.OngoingSegmentReplications.startSegmentCopy(OngoingSegmentReplications.java:128)
		at org.opensearch.indices.replication.SegmentReplicationSourceService$GetSegmentFilesRequestHandler.messageReceived(SegmentReplicationSourceService.java:157)
		at org.opensearch.indices.replication.SegmentReplicationSourceService$GetSegmentFilesRequestHandler.messageReceived(SegmentReplicationSourceService.java:154)
		at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106)
		at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:453)
		at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)
		at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at java.lang.Thread.run(Thread.java:833)