Add support to run sanity test on remote cluster

[ankitkala]

Signed-off-by: Ankit Kala ankikala@amazon.com

Description

Added the script to execute the integ tests on a remote cluster. Eventually we can extend this to execute all the tests when we have the support for provisioning multiple clusters during sanity.

PS: I've deliberately hardcoded the test cluster name as follower. After we start provisioning both, we can use this as convention to run tests on remote clusters.

Issues Resolved

383

Testing

88665a0f606e:cross-cluster-replication ankikala$ ./gradlew integTestRemote -Dfollower.http_host=52.210.59.104:443 -Dfollower.transport_host=52.210.59.104:443 -Dsecurity_enabled=true -Duser=admin -Dpassword=admin --console=plain
Picked up JAVA_TOOL_OPTIONS: -Dlog4j2.formatMsgNoLookups=true
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.3.3
  OS Info               : Mac OS X 10.16 (x86_64)
  JDK Version           : 14 (Oracle JDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home
  Random Testing Seed   : B22D87A9D8739903
  In FIPS 140 mode      : false
=======================================
> Task :compileKotlin UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :compileTestKotlin UP-TO-DATE
> Task :copyPluginPropertiesTemplate UP-TO-DATE
> Task :pluginProperties UP-TO-DATE
> Task :compileTestJava NO-SOURCE
> Task :processTestResources UP-TO-DATE
> Task :testClasses UP-TO-DATE

> Task :integTestRemote
Picked up JAVA_TOOL_OPTIONS: -Dlog4j2.formatMsgNoLookups=true

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 15s
7 actionable tasks: 1 executed, 6 up-to-date

88665a0f606e:cross-cluster-replication ankikala$ ./integtest.sh -b 52.210.59.104 -p 443 -t 443 -s true -c admin:admin
Picked up JAVA_TOOL_OPTIONS: -Dlog4j2.formatMsgNoLookups=true
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.3.3
  OS Info               : Mac OS X 10.16 (x86_64)
  JDK Version           : 14 (Oracle JDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home
  Random Testing Seed   : 7FBF66DE3D9F8BEF
  In FIPS 140 mode      : false
=======================================
> Task :compileKotlin UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :compileTestKotlin UP-TO-DATE
> Task :copyPluginPropertiesTemplate UP-TO-DATE
> Task :pluginProperties UP-TO-DATE
> Task :compileTestJava NO-SOURCE
> Task :processTestResources UP-TO-DATE
> Task :testClasses UP-TO-DATE

> Task :integTestRemote
Picked up JAVA_TOOL_OPTIONS: -Dlog4j2.formatMsgNoLookups=true

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 15s
7 actionable tasks: 1 executed, 6 up-to-date
``
### Check List
- [x] New functionality includes testing.
  - [x] All tests pass
- [x] New functionality has been documented.
  - [x] New functionality has javadoc added
- [x] Commits are signed per the DCO using --signoff 

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

[codecov-commenter]

Codecov Report

Merging #394 (d84260b) into main (9c4f7e2) will increase coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #394      +/-   ##
============================================
+ Coverage     74.54%   74.59%   +0.04%     
- Complexity      999     1001       +2     
============================================
  Files           141      141              
  Lines          4554     4554              
  Branches        504      504              
============================================
+ Hits           3395     3397       +2     
  Misses          841      841              
+ Partials        318      316       -2     

Impacted Files	Coverage Δ
...rch/replication/task/shard/ShardReplicationTask.kt	75.91% <0.00%> (+0.72%)	⬆️
...ation/task/shard/ShardReplicationChangesTracker.kt	63.63% <0.00%> (+3.03%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c4f7e2...d84260b. Read the comment docs.

[opensearch-trigger-bot]

The backport to 1.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.2 1.2
# Navigate to the new working tree
cd .worktrees/backport-1.2
# Create a new branch
git switch --create backport/backport-394-to-1.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f82c2add3db8c4669d47f8a2876ad027788f016b
# Push it to GitHub
git push --set-upstream origin backport/backport-394-to-1.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.2

Then, create a pull request where the base branch is 1.2 and the compare/head branch is backport/backport-394-to-1.2.