[Backport 2.x] Downgrade write lock to read lock before translog upload to remote store

[ashking94]

Backport cbff21d from #10135.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 19a183a

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob

• URL: https://build.ci.opensearch.org/job/gradle-check/26132/
• CommitID: 19a183a
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10187 (19a183a) into 2.x (ad9355f) will increase coverage by 0.05%.
Report is 8 commits behind head on 2.x.
The diff coverage is 86.29%.

@@             Coverage Diff              @@
##                2.x   #10187      +/-   ##
============================================
+ Coverage     70.83%   70.89%   +0.05%     
- Complexity    58250    58321      +71     
============================================
  Files          4811     4812       +1     
  Lines        275316   275401      +85     
  Branches      40464    40487      +23     
============================================
+ Hits         195033   195232     +199     
+ Misses        63576    63513      -63     
+ Partials      16707    16656      -51     

Files Changed	Coverage Δ
...ava/org/opensearch/index/mapper/MapperService.java	74.79% <ø> (ø)
...arch/index/recovery/RemoteStoreRestoreService.java	12.03% <0.00%> (ø)
...g/transfer/TranslogCheckpointTransferSnapshot.java	85.89% <0.00%> (+6.67%)	⬆️
...dex/translog/transfer/TranslogTransferManager.java	79.76% <ø> (+6.24%)	⬆️
...g/opensearch/index/query/BoostingQueryBuilder.java	87.25% <66.66%> (-1.29%)	⬇️
...rg/opensearch/index/query/SpanNotQueryBuilder.java	90.43% <66.66%> (-1.31%)	⬇️
...rg/opensearch/index/translog/RemoteFsTranslog.java	71.92% <70.00%> (+1.36%)	⬆️
...nsearch/index/query/SpanMultiTermQueryBuilder.java	79.77% <75.00%> (-2.58%)	⬇️
...org/opensearch/index/query/DisMaxQueryBuilder.java	88.00% <85.71%> (-0.18%)	⬇️
...g/opensearch/index/query/SpanNearQueryBuilder.java	82.60% <85.71%> (+0.12%)	⬆️
... and 13 more

... and 452 files with indirect coverage changes