Optimize remote state stale file deletion

[shiv0408]

Description

Adding b9befaa added in #13131 back, we are now using dedicated threadpool remote_purge for deletion of stale files.

Ran the test for 1000 times locally to check for flakiness

./gradlew ':server:internalClusterTest' --tests "org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.testRemoteCleanupDeleteStale" -Dtests.iters=1000 -Dtests.timeoutSuite=6000000!

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.7
  OS Info               : Mac OS X 14.5 (aarch64)
  JDK Version           : 21 (Eclipse Temurin JDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home
  Random Testing Seed   : FFE4E678DB1E6E23
  In FIPS 140 mode      : false
=======================================

> Task :server:internalClusterTest
WARNING: Using incubator modules: jdk.incubator.vector
Jun 05, 2024 2:51:08 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/Volumes/workplace/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/Users/hishiv/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib/plugins/gradle-testing-base-8.7.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 1h 53s
57 actionable tasks: 1 executed, 56 up-to-date
WARNING: The following functionality has been deprecated and will be removed in the next major release of the Develocity Gradle plugin. Run with '-Ddevelocity.deprecation.captureOrigin=true' to see where the deprecated functionality is being used. For assistance with migration, see https://gradle.com/help/gradle-plugin-develocity-migration.
- The deprecated "gradle.enterprise.testretry.enabled" system property has been replaced by "develocity.testretry.enabled"
- The "com.gradle.enterprise" plugin has been replaced by "com.gradle.develocity"

Related Issues

Resolves #12889
Resolves #12798 that test case is getting removed in this PR
Resolves #13968

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• API changes companion pull request created.
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for fc429fa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[codecov]

Codecov Report

Attention: Patch coverage is 77.95699% with 41 lines in your changes missing coverage. Please review.

Project coverage is 71.63%. Comparing base (b15cb0c) to head (9863fae).
Report is 348 commits behind head on main.

Files	Patch %	Lines
...teway/remote/RemoteClusterStateCleanupManager.java	79.31%	29 Missing and 7 partials ⚠️
...arch/gateway/remote/RemoteClusterStateService.java	55.55%	3 Missing and 1 partial ⚠️
server/src/main/java/org/opensearch/node/Node.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13995      +/-   ##
============================================
+ Coverage     71.42%   71.63%   +0.21%     
- Complexity    59978    61365    +1387     
============================================
  Files          4985     5067      +82     
  Lines        282275   288321    +6046     
  Branches      40946    41759     +813     
============================================
+ Hits         201603   206539    +4936     
- Misses        63999    64729     +730     
- Partials      16673    17053     +380     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for f932b54: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 9863fae: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-13995-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 3de5d3c273e744a76459c88f3a6341ed3173f5a0
# Push it to GitHub
git push --set-upstream origin backport/backport-13995-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-13995-to-2.x.