Gracefully handle concurrent zone decommission action #5542

imRishN · 2022-12-13T14:42:45Z

Description

This PR introduces changes related to Decommission Service -

Add request id to decommission request object
Stores the request id to state metadata
Uses this metadata when a request comes to identify if its a retried request

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-12-13T15:11:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8003/
CommitID: 743ca01

codecov-commenter · 2022-12-13T15:13:07Z

Codecov Report

Merging #5542 (4801501) into main (28e9b11) will increase coverage by 0.06%.
The diff coverage is 70.58%.

@@             Coverage Diff              @@
##               main    #5542      +/-   ##
============================================
+ Coverage     70.98%   71.05%   +0.06%     
- Complexity    58669    58685      +16     
============================================
  Files          4763     4764       +1     
  Lines        279945   280022      +77     
  Branches      40418    40425       +7     
============================================
+ Hits         198731   198958     +227     
+ Misses        64988    64750     -238     
- Partials      16226    16314      +88

Impacted Files	Coverage Δ
...sion/awareness/put/DecommissionRequestBuilder.java	`0.00% <0.00%> (ø)`
...a/org/opensearch/test/OpenSearchIntegTestCase.java	`56.35% <0.00%> (-0.32%)`	⬇️
...pensearch/common/settings/FeatureFlagSettings.java	`50.00% <50.00%> (ø)`
...ecommission/awareness/put/DecommissionRequest.java	`81.66% <71.42%> (-4.05%)`	⬇️
...arch/cluster/decommission/DecommissionService.java	`37.85% <75.00%> (+1.96%)`	⬆️
...t/action/admin/cluster/RestDecommissionAction.java	`80.95% <75.00%> (-4.77%)`	⬇️
...h/cluster/decommission/DecommissionController.java	`62.83% <86.36%> (+5.68%)`	⬆️
.../java/org/opensearch/common/util/FeatureFlags.java	`80.00% <88.88%> (+30.00%)`	⬆️
...org/opensearch/common/settings/SettingsModule.java	`85.81% <100.00%> (+0.29%)`	⬆️
server/src/main/java/org/opensearch/node/Node.java	`87.17% <100.00%> (+0.01%)`	⬆️
... and 495 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2022-12-13T15:21:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue

URL: https://build.ci.opensearch.org/job/gradle-check/8004/
CommitID: dbb404e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-12-14T07:49:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8087/
CommitID: ee90987
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2022-12-14T22:30:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/8087/

CommitID: ee90987
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Failing due to version bump after 2.4.1 release, #5560. Fixed is merged into main branch. @imRishN : Can you please rebase your branch against latest main branch ?

* What went wrong:
Execution failed for task ':distribution:bwc:bugfix:buildBwcLinuxTar'.
> Building 2.4.1 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/bugfix/build/bwc/checkout-2.4/distribution/archives/linux-tar/build/distributions/opensearch-min-2.4.1-SNAPSHOT-linux-x64.tar.gz

github-actions · 2022-12-14T22:57:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8117/
CommitID: ee90987
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

…-concurrency

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2022-12-16T07:32:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8180/
CommitID: 4946914
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-12-16T07:52:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8181/
CommitID: 4f5e73c

github-actions · 2023-01-10T14:30:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9188/
CommitID: c46992c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2023-01-10T14:41:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9189/
CommitID: d4670b7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

...java/org/opensearch/action/admin/cluster/decommission/awareness/put/DecommissionRequest.java

server/src/main/java/org/opensearch/cluster/decommission/DecommissionService.java

server/src/main/java/org/opensearch/cluster/decommission/DecommissionAttributeMetadata.java

github-actions · 2023-01-10T16:12:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9191/
CommitID: 7ecbe6e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-10T16:33:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9197/
CommitID: 88ec79d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2023-01-10T17:04:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9203/
CommitID: 2b40e12
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-10T17:05:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9205/
CommitID: a6d0c3b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/decommission/DecommissionAttributeMetadata.java

shwetathareja · 2023-01-10T17:10:43Z

server/src/main/java/org/opensearch/cluster/decommission/DecommissionService.java

@@ -144,12 +145,19 @@ public void startDecommissionAction(
            public ClusterState execute(ClusterState currentState) {
                // validates if correct awareness attributes and forced awareness attribute set to the cluster before starting action
                validateAwarenessAttribute(decommissionAttribute, awarenessAttributes, forcedAwarenessAttributes);
+                if (decommissionRequest.requestID() == null) {
+                    decommissionRequest.setRequestID(UUIDs.randomBase64UUID());


instead of setting here. Just generate it in RestAction so that you don't need it set it again. Then your request would also be immutable. Thoughts?

In that case, only the REST layer would be having UUIDs set. We would want the IDs to be there not conditionally. When client pkgs tries to interact with request object directly, won't it be a problem? Also Integ tests are not Rest layer tests. And OpenSearch uses http smoke test pkg for rest integ tests. The current IT pkg doesn't support REST tests. Let me know if this makes sense..

@imRishN I was wondering if you set the requestId here itself -

OpenSearch/server/src/main/java/org/opensearch/rest/action/admin/cluster/RestDecommissionAction.java

Line 49 in b71caae

DecommissionRequest createRequest(RestRequest request) throws IOException {

then it can be passed around as is.

shwetathareja · 2023-01-10T17:11:44Z

server/src/main/java/org/opensearch/cluster/decommission/DecommissionService.java

+                        if (decommissionRequest.requestID().equals(decommissionAttributeMetadata.requestID()) == false) {
+                            throw new DecommissioningFailedException(
+                                requestedDecommissionAttribute,
+                                "same request is already in status [INIT]"


[minor]: another request instead of same request

It's a same request from users perspective as it would be decommissioning same attribute name and attribute value. Although, technically its a different request but these messages are generally user friendly

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

github-actions · 2023-01-10T17:54:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9211/
CommitID: 927d0cc
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…-concurrency

github-actions · 2023-01-10T18:24:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9216/
CommitID: 927d0cc
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

shwetathareja

Thanks @imRishN for the changes.

github-actions · 2023-01-10T19:19:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9218/
CommitID: 539d77a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…ject#5542) * Control concurrency and handle retries during decommissioning of awareness attributes Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

* Control concurrency and handle retries during decommissioning of awareness attributes Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN added 2 commits December 13, 2022 20:09

Control concurrency and handle retries

767803e

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix spotless check

743ca01

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN changed the title ~~Decommission/handle concurrency~~ Gracefully handle concurrent zone decommission action Dec 13, 2022

Add changelog

dbb404e

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN mentioned this pull request Dec 13, 2022

Control concurrency and add retry action in decommission flow #5146

Closed

6 tasks

Add request timeout param

ee90987

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN added 4 commits December 16, 2022 12:21

Changes

5870737

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Merge remote-tracking branch 'upstream/main' into decommission/handle…

8dd5899

…-concurrency

Fix spotless check

4946914

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix

4f5e73c

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

imRishN marked this pull request as ready for review December 22, 2022 05:48

imRishN requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg, kotwanikunal, mch2 and nknize as code owners December 22, 2022 05:48

Fix

7ecbe6e

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix

88ec79d

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Bukhtawar reviewed Jan 10, 2023

View reviewed changes

...java/org/opensearch/action/admin/cluster/decommission/awareness/put/DecommissionRequest.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Jan 10, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/decommission/DecommissionService.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Jan 10, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/decommission/DecommissionService.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Jan 10, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/decommission/DecommissionAttributeMetadata.java Show resolved Hide resolved

imRishN added 2 commits January 10, 2023 22:03

Resolve comments

2b40e12

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Fix test

a6d0c3b

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

shwetathareja reviewed Jan 10, 2023

View reviewed changes

Test fix

927d0cc

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Bukhtawar approved these changes Jan 10, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/main' into decommission/handle…

539d77a

…-concurrency

shwetathareja approved these changes Jan 10, 2023

View reviewed changes

Bukhtawar merged commit ac3351c into opensearch-project:main Jan 10, 2023

imRishN mentioned this pull request Jan 10, 2023

[Backport 2.x] Gracefully handle concurrent zone decommission action (#5542) #5802

Merged

6 tasks

imRishN mentioned this pull request Jan 10, 2023

[Backport 2.5] Gracefully handle concurrent zone decommission action (#5542) #5803

Merged

6 tasks

Bukhtawar pushed a commit that referenced this pull request Jan 10, 2023

Gracefully handle concurrent zone decommission action (#5542) (#5802)

9c5bbd5

* Control concurrency and handle retries during decommissioning of awareness attributes Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Bukhtawar pushed a commit that referenced this pull request Jan 10, 2023

Gracefully handle concurrent zone decommission action (#5542) (#5803)

2d8e089

* Control concurrency and handle retries during decommissioning of awareness attributes Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

kotwanikunal pushed a commit that referenced this pull request Jan 25, 2023

Gracefully handle concurrent zone decommission action (#5542) (#5802)

06f2e47

* Control concurrency and handle retries during decommissioning of awareness attributes Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gracefully handle concurrent zone decommission action #5542

Gracefully handle concurrent zone decommission action #5542

imRishN commented Dec 13, 2022 •

edited

Loading

github-actions bot commented Dec 13, 2022

codecov-commenter commented Dec 13, 2022 •

edited

Loading

github-actions bot commented Dec 13, 2022

github-actions bot commented Dec 14, 2022

dreamer-89 commented Dec 14, 2022 •

edited

Loading

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 14, 2022

github-actions bot commented Dec 16, 2022

github-actions bot commented Dec 16, 2022

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

shwetathareja Jan 10, 2023

imRishN Jan 10, 2023

shwetathareja Jan 10, 2023

shwetathareja Jan 10, 2023

imRishN Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

shwetathareja left a comment

github-actions bot commented Jan 10, 2023

Gracefully handle concurrent zone decommission action #5542

Gracefully handle concurrent zone decommission action #5542

Conversation

imRishN commented Dec 13, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Dec 13, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Dec 13, 2022 • edited Loading

Codecov Report

github-actions bot commented Dec 13, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 14, 2022

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Dec 14, 2022 • edited Loading

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 14, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 16, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 16, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

shwetathareja Jan 10, 2023

Choose a reason for hiding this comment

imRishN Jan 10, 2023

Choose a reason for hiding this comment

shwetathareja Jan 10, 2023

Choose a reason for hiding this comment

shwetathareja Jan 10, 2023

Choose a reason for hiding this comment

imRishN Jan 10, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

shwetathareja left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

imRishN commented Dec 13, 2022 •

edited

Loading

codecov-commenter commented Dec 13, 2022 •

edited

Loading

dreamer-89 commented Dec 14, 2022 •

edited

Loading