Integ Tests for Awareness Attribute Decommissioning #4715

imRishN · 2022-10-08T17:57:36Z

Description

This PR adds integration tests for awareness attribute decommission

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-08T18:23:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4210/
CommitID: a69445d

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-09T09:48:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4219/
CommitID: da99a27

…tests Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-16T07:06:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4746/
CommitID: 36eec07

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-16T09:55:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4758/
CommitID: 61b4d04

…tests Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-18T19:08:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/4859/
CommitID: 8974f55

Gaganjuneja · 2022-10-19T14:42:10Z

...alClusterTest/java/org/opensearch/cluster/coordination/AwarenessAttributeDecommissionIT.java

@@ -56,6 +72,425 @@ public void cleanup() throws Exception {
        assertNoTimeout(client().admin().cluster().prepareHealth().get());
    }



How can we verify that nodes are drained before initiating the decommissioning and once prepared for decommission no new request is being sent to these nodes?

We can never certainly say if the node was drained completely or not. Although before initiating the decommission, we ensure weights for decommissioned zone is 0 but the metrics where we track HTTP requests doesn't get to zero. By the time we come up with an effective proposal for draining, we have implemented a 2 min wait after setting weights to 0 and executing decommission. Also, wait time can be set by the user as per need
Implemented as part of this PR - #4586

Gaganjuneja · 2022-10-19T14:50:29Z

...alClusterTest/java/org/opensearch/cluster/coordination/AwarenessAttributeDecommissionIT.java

+            assertTrue(ex.getMessage().contains("invalid awareness attribute requested for decommissioning"));
+        });
+    }
+


Should we also add a test for a case of single zone domain to test how decommissioning fails there and may be quorum loss scenario?

Added a test for single zone decommission. Quorum loss checks is internally handled during reconfiguration of Voting Configuration

Gaganjuneja · 2022-10-19T15:32:13Z

...alClusterTest/java/org/opensearch/cluster/coordination/AwarenessAttributeDecommissionIT.java

+        assertNodesRemovedAfterZoneDecommission(true);
+    }
+
+    public void testInvariantsAndLogsOnDecommissionedNodes() throws Exception {


Methods are getting very big, around 150 lines in some cases. It's hard to read in a single go. Can we do something about it?

I tried to abstract out but use cases for each of these are unique and hence kept it like this

…tests Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN · 2022-10-30T09:10:41Z

@Bukhtawar Can you please also review this PR?
@Gaganjuneja Can you please review it again?

github-actions · 2022-10-30T09:41:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5902/
CommitID: 1e33247
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-30T10:42:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5904/
CommitID: 0b9a9dc
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-30T11:17:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/5906/
CommitID: 4b89fbf

codecov-commenter · 2022-10-30T11:18:50Z

Codecov Report

Merging #4715 (74f746b) into main (d2c2ade) will decrease coverage by 0.08%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               main    #4715      +/-   ##
============================================
- Coverage     70.95%   70.87%   -0.09%     
+ Complexity    57971    57926      -45     
============================================
  Files          4688     4688              
  Lines        276901   276904       +3     
  Branches      40299    40300       +1     
============================================
- Hits         196472   196246     -226     
- Misses        64181    64406     +225     
- Partials      16248    16252       +4

Impacted Files	Coverage Δ
...ecommission/awareness/put/DecommissionRequest.java	`85.71% <0.00%> (-6.60%)`	⬇️
...arch/cluster/decommission/DecommissionService.java	`35.65% <0.00%> (ø)`
...adonly/AddIndexBlockClusterStateUpdateRequest.java	`0.00% <0.00%> (-75.00%)`	⬇️
...readonly/TransportVerifyShardIndexBlockAction.java	`9.75% <0.00%> (-58.54%)`	⬇️
...n/admin/indices/readonly/AddIndexBlockRequest.java	`17.85% <0.00%> (-53.58%)`	⬇️
...indices/readonly/TransportAddIndexBlockAction.java	`20.68% <0.00%> (-41.38%)`	⬇️
.../admin/indices/readonly/AddIndexBlockResponse.java	`0.00% <0.00%> (-37.50%)`	⬇️
...ain/java/org/opensearch/search/sort/MinAndMax.java	`63.15% <0.00%> (-36.85%)`	⬇️
.../indices/readonly/AddIndexBlockRequestBuilder.java	`0.00% <0.00%> (-33.34%)`	⬇️
...search/aggregations/pipeline/HoltWintersModel.java	`21.47% <0.00%> (-30.88%)`	⬇️
... and 442 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2022-10-30T11:25:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5908/
CommitID: db62f0d
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-30T17:19:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/5910/
CommitID: 95af46d

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-31T04:43:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5923/
CommitID: 09e5c1b
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-31T05:53:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5926/
CommitID: bef322f
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-10-31T06:34:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/5929/
CommitID: 74f746b

…ct#4715) * Add integ test for awareness attribute decommissioning Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

* Fail weight update when decommission ongoing and fail decommission when attribute not weighed away (#4839) * Add changes for graceful node decommission (#4586) * Add delay timeout for decommission request (#4931) * Integ Tests for Awareness Attribute Decommissioning (#4715) Signed-off-by: Rishab Nahata <rnnahata@amazon.com> Signed-off-by: pranikum <109206473+pranikum@users.noreply.github.com>

* Fail weight update when decommission ongoing and fail decommission when attribute not weighed away (opensearch-project#4839) * Add changes for graceful node decommission (opensearch-project#4586) * Add delay timeout for decommission request (opensearch-project#4931) * Integ Tests for Awareness Attribute Decommissioning (opensearch-project#4715) Signed-off-by: Rishab Nahata <rnnahata@amazon.com> Signed-off-by: pranikum <109206473+pranikum@users.noreply.github.com>

Add integ test for awareness attribute decommissioning

a69445d

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN added 2 commits October 9, 2022 14:56

Add tests

f78eb9d

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Add changelog

da99a27

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN added 7 commits October 15, 2022 18:15

Merge remote-tracking branch 'upstream/main' into decommission/integ-…

46862ee

…tests Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Add tests for decommissioned node

0bd2fa8

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix metadata update

ce226bd

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix spotless check

1c97e42

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Add changelog

9df0e7d

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Merge branch 'decommission/metadata-bug' into decommission/integ-tests

63b6ed8

Assert status

36eec07

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN mentioned this pull request Oct 16, 2022

Fix decommission status update to non leader nodes #4800

Merged

6 tasks

imRishN added 2 commits October 16, 2022 14:26

Fixes

1b49a54

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix spotless check

61b4d04

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN added 2 commits October 18, 2022 23:44

Merge remote-tracking branch 'upstream/main' into decommission/integ-…

310f1ef

…tests Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix spotless check

8974f55

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN marked this pull request as ready for review October 18, 2022 18:39

imRishN requested review from a team and reta as code owners October 18, 2022 18:39

Gaganjuneja reviewed Oct 19, 2022

View reviewed changes

imRishN added 3 commits October 30, 2022 12:04

Merge remote-tracking branch 'upstream/main' into decommission/integ-…

430f824

…tests Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Add test for single zone decommission

5361621

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix spotless check

1e33247

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Set weight

0b9a9dc

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN added 2 commits October 30, 2022 16:17

Empty-Commit

4b89fbf

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Empty-Commit

db62f0d

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Empty-Commit

95af46d

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Empty-Commit

09e5c1b

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Empty-Commit

bef322f

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Empty-Commit

74f746b

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Bukhtawar approved these changes Nov 3, 2022

View reviewed changes

Bukhtawar merged commit 850bb2e into opensearch-project:main Nov 3, 2022

imRishN added a commit to imRishN/OpenSearch that referenced this pull request Nov 3, 2022

Integ Tests for Awareness Attribute Decommissioning (opensearch-proje…

02edc4f

…ct#4715) * Add integ test for awareness attribute decommissioning Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN mentioned this pull request Nov 3, 2022

[Backport 2.x]Graceful Decommission and Integ Tests #5060

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integ Tests for Awareness Attribute Decommissioning #4715

Integ Tests for Awareness Attribute Decommissioning #4715

imRishN commented Oct 8, 2022 •

edited

Loading

github-actions bot commented Oct 8, 2022

github-actions bot commented Oct 9, 2022

github-actions bot commented Oct 16, 2022

github-actions bot commented Oct 16, 2022

github-actions bot commented Oct 18, 2022

Gaganjuneja Oct 19, 2022

imRishN Oct 29, 2022 •

edited

Loading

Gaganjuneja Oct 19, 2022 •

edited

Loading

imRishN Oct 30, 2022

Gaganjuneja Oct 19, 2022 •

edited

Loading

imRishN Oct 29, 2022

imRishN commented Oct 30, 2022 •

edited

Loading

github-actions bot commented Oct 30, 2022

github-actions bot commented Oct 30, 2022

github-actions bot commented Oct 30, 2022

codecov-commenter commented Oct 30, 2022 •

edited

Loading

github-actions bot commented Oct 30, 2022

github-actions bot commented Oct 30, 2022

github-actions bot commented Oct 31, 2022

github-actions bot commented Oct 31, 2022

github-actions bot commented Oct 31, 2022

		@@ -56,6 +72,425 @@ public void cleanup() throws Exception {
		assertNoTimeout(client().admin().cluster().prepareHealth().get());
		}

Integ Tests for Awareness Attribute Decommissioning #4715

Integ Tests for Awareness Attribute Decommissioning #4715

Conversation

imRishN commented Oct 8, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Oct 8, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 9, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 16, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 16, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 18, 2022

Gradle Check (Jenkins) Run Completed with:

Gaganjuneja Oct 19, 2022

Choose a reason for hiding this comment

imRishN Oct 29, 2022 • edited Loading

Choose a reason for hiding this comment

Gaganjuneja Oct 19, 2022 • edited Loading

Choose a reason for hiding this comment

imRishN Oct 30, 2022

Choose a reason for hiding this comment

Gaganjuneja Oct 19, 2022 • edited Loading

Choose a reason for hiding this comment

imRishN Oct 29, 2022

Choose a reason for hiding this comment

imRishN commented Oct 30, 2022 • edited Loading

github-actions bot commented Oct 30, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 30, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 30, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Oct 30, 2022 • edited Loading

Codecov Report

github-actions bot commented Oct 30, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 30, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 31, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 31, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 31, 2022

Gradle Check (Jenkins) Run Completed with:

imRishN commented Oct 8, 2022 •

edited

Loading

imRishN Oct 29, 2022 •

edited

Loading

Gaganjuneja Oct 19, 2022 •

edited

Loading

Gaganjuneja Oct 19, 2022 •

edited

Loading

imRishN commented Oct 30, 2022 •

edited

Loading

codecov-commenter commented Oct 30, 2022 •

edited

Loading