Add the capability to override indices.breaker.total.use_real_memory setting for test clusters

[navneet1v]

Description

Add the capability to override indices.breaker.total.use_real_memory setting for test clusters.

This will allow plugins to run the integTest with heap CB enabled to ensure that their integTests runs with as close of a env which we have in Jenkins. Please read the attached issue for more details

Testing

Validated that setting can be override like this:

./gradlew :integTest -Dtests.opensearch.indices.breaker.total.use_real_memory=true

Related Issues

Resolves #15849

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 9c63bc4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 32d6d5c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.96%. Comparing base (92d7fe8) to head (32d6d5c).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...opensearch/gradle/testclusters/OpenSearchNode.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15906      +/-   ##
============================================
+ Coverage     71.90%   71.96%   +0.05%     
- Complexity    64216    64280      +64     
============================================
  Files          5272     5277       +5     
  Lines        300597   300691      +94     
  Branches      43440    43449       +9     
============================================
+ Hits         216151   216378     +227     
+ Misses        66680    66589      -91     
+ Partials      17766    17724      -42     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[navneet1v]

@msfroh, @reta can I get a review on this code?

[msfroh]

Yeah... Makes sense.

Do you know if there are any other important settings that differ between the Jenkins tests (or a typical production system) and the integ tests? Anything else where we might be missing out?

[navneet1v]

Yeah... Makes sense.

Do you know if there are any other important settings that differ between the Jenkins tests (or a typical production system) and the integ tests? Anything else where we might be missing out?

If we look at the baseconfig in the diff file these are the only settings which are different from jenkins because these are what set by testcluster gradle task. I didn’t find any other setting more useful.