Fetch previous cluster UUID #9746

soosinha · 2023-09-05T02:15:41Z

Description

This PR has the logic to get the previous cluster UUID. The previous cluster UUID will be used to restore the metadata from remote store while bootstrap.

Related Issues

#9821

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-05T02:24:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24398/
CommitID: 56a2c81
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-05T02:55:51Z

Compatibility status:

Checks if related components are compatible with change 56a2c81

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

github-actions · 2023-09-07T08:17:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24845/
CommitID: 660675c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-07T08:21:17Z

Compatibility status:

Checks if related components are compatible with change 660675c

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git]

soosinha · 2023-09-07T08:32:42Z

Flaky test: #8980

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

github-actions · 2023-09-07T13:46:57Z

Compatibility status:

Checks if related components are compatible with change 3ba0108

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-07T14:08:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24908/
CommitID: 3ba0108
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

soosinha · 2023-09-07T14:29:21Z

Flaky tests: #9407 , #7755 , #9891

opensearch-trigger-bot · 2023-09-07T14:32:44Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9746-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bdf6f1d09702aebd8b1e94a2cb722c14a6825297
# Push it to GitHub
git push --set-upstream origin backport/backport-9746-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9746-to-2.x.

opensearch-trigger-bot · 2023-09-07T14:32:56Z

The backport to 2.10 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.10 2.10
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.10
# Create a new branch
git switch --create backport/backport-9746-to-2.10
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bdf6f1d09702aebd8b1e94a2cb722c14a6825297
# Push it to GitHub
git push --set-upstream origin backport/backport-9746-to-2.10
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.10

Then, create a pull request where the base branch is 2.10 and the compare/head branch is backport/backport-9746-to-2.10.

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

ashking94 · 2023-09-07T14:34:13Z

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

+        final Map<String, String> clusterUUIDGraph = manifestsByClusterUUID.values()
+            .stream()
+            .collect(Collectors.toMap(ClusterMetadataManifest::getClusterUUID, ClusterMetadataManifest::getPreviousClusterUUID));


can there be cases of isolated masters where more than 1 master can be publishing the metadata? how are we handling such cases?

multiple cluster managers can only exist intermittently and one of the cluster managers will fail when they try to validate the quorum since another cluster manager is holding the quorum now. during publish and commit phase, quorum of cluster manager nodes need to approve of the operation.

such cases are handed by ensuring we always pick a committed uuid from remote(isInvalidClusterUUID) and trim any duplicate UUIDs using our trimming logic

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> (cherry picked from commit bdf6f1d)

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

shwetathareja reviewed Sep 5, 2023

View reviewed changes

soosinha force-pushed the remote_state_prev_cluster branch from 56a2c81 to f2cc7ab Compare September 6, 2023 14:23

soosinha marked this pull request as ready for review September 6, 2023 14:23

soosinha changed the title ~~[Draft] Remote state prev cluster~~ Fetch previous cluster UUID Sep 6, 2023

sachinpkale reviewed Sep 7, 2023

View reviewed changes

shwetathareja reviewed Sep 7, 2023

View reviewed changes

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Outdated Show resolved Hide resolved

Add logic to fetch previousClusterUUID

3ba0108

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

soosinha force-pushed the remote_state_prev_cluster branch from 660675c to 3ba0108 Compare September 7, 2023 13:03

shwetathareja approved these changes Sep 7, 2023

View reviewed changes

shwetathareja added the skip-changelog label Sep 7, 2023

gbbafna reviewed Sep 7, 2023

View reviewed changes

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Show resolved Hide resolved

shwetathareja merged commit bdf6f1d into opensearch-project:main Sep 7, 2023
11 of 13 checks passed

shwetathareja added backport 2.x Backport to 2.x branch backport 2.10 Backport to 2.10 branch labels Sep 7, 2023

opensearch-trigger-bot bot added the backport-failed label Sep 7, 2023

ashking94 reviewed Sep 7, 2023

View reviewed changes

soosinha added a commit to soosinha/OpenSearch that referenced this pull request Sep 7, 2023

Add logic to fetch previousClusterUUID (opensearch-project#9746)

d8c20bf

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> (cherry picked from commit bdf6f1d)

soosinha mentioned this pull request Sep 7, 2023

[Backport 2.x] Add logic to fetch previousClusterUUID (#9746) #9895

Merged

sachinpkale pushed a commit that referenced this pull request Sep 7, 2023

Add logic to fetch previousClusterUUID (#9746) (#9895)

5627de1

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> (cherry picked from commit bdf6f1d)

soosinha added a commit to soosinha/OpenSearch that referenced this pull request Sep 7, 2023

Add logic to fetch previousClusterUUID (opensearch-project#9746)

feae4bf

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> (cherry picked from commit bdf6f1d)

soosinha mentioned this pull request Sep 7, 2023

[Backport 2.10] Add logic to fetch previousClusterUUID (#9746) #9898

Merged

gbbafna pushed a commit that referenced this pull request Sep 7, 2023

Add logic to fetch previousClusterUUID (#9746) (#9898)

5b83c49

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> (cherry picked from commit bdf6f1d)

kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023

Add logic to fetch previousClusterUUID (opensearch-project#9746)

daaf935

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

sarthakaggarwal97 pushed a commit to sarthakaggarwal97/OpenSearch that referenced this pull request Sep 20, 2023

Add logic to fetch previousClusterUUID (opensearch-project#9746)

07d141c

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023

Add logic to fetch previousClusterUUID (opensearch-project#9746)

1318aca

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024

Add logic to fetch previousClusterUUID (opensearch-project#9746)

3dcaae3

Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fetch previous cluster UUID #9746

Fetch previous cluster UUID #9746

soosinha commented Sep 5, 2023 •

edited

Loading

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

soosinha commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

soosinha commented Sep 7, 2023

opensearch-trigger-bot bot commented Sep 7, 2023

opensearch-trigger-bot bot commented Sep 7, 2023

ashking94 Sep 7, 2023

linuxpi Oct 25, 2023

Fetch previous cluster UUID #9746

Fetch previous cluster UUID #9746

Conversation

soosinha commented Sep 5, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Sep 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 5, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 7, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

soosinha commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

soosinha commented Sep 7, 2023

opensearch-trigger-bot bot commented Sep 7, 2023

opensearch-trigger-bot bot commented Sep 7, 2023

ashking94 Sep 7, 2023

Choose a reason for hiding this comment

linuxpi Oct 25, 2023

Choose a reason for hiding this comment

soosinha commented Sep 5, 2023 •

edited

Loading