Fix flakiness for getNodeRoles based tests

[kotwanikunal]

Description

• Fixes flakiness for getNodeRoles based tests
• Calling client() randomizes the clients across nodes and can lead to a difference sequence of node outputs when checking for node roles

Related Issues

Resolves #7327

Ran the tests with offending seeds and it seems to pass now.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[kotwanikunal]

@reta Can you please take a look at this one?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14807/
• CommitID: 15f7fe0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation

• URL: https://build.ci.opensearch.org/job/gradle-check/14809/
• CommitID: 0144dda
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[tlfeng]

Thank you for fixing the test! 👍

[tlfeng]

Could you tell me why initializing a new Client will keep the sequence of node output consistent?

[kotwanikunal]

Could you tell me why initializing a new Client will keep the sequence of node output consistent?

Sure.
Going down the client initialization, you can see if you call the path repeatedly, it will create a randomizing client or a client for a different node.

OpenSearch/test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java

Lines 650 to 663 in 4c98b3d

	public static Client client() {
	return client(null);
	}
	public static Client client(@Nullable String node) {
	if (node != null) {
	return internalCluster().client(node);
	}
	Client client = cluster().client();
	if (frequently()) {
	client = new RandomizingClient(client, random());
	}
	return client;
	}

And since the API response is an array, different coordinator nodes will lead to a change in the sequence of returned results for node stats.