Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read write ephemeral objects for remote publication of cluster state #14089

Merged
merged 13 commits into from
Jun 11, 2024

Conversation

shiv0408
Copy link
Member

@shiv0408 shiv0408 commented Jun 8, 2024

Description

The changes include modification in RemoteClusterStateService to enable uploading the ephemeral objects required for publication of cluster state through remote store. Also, added a method to read the Cluster State Attributes such as IndexMetadata, GlobalMetadata attributes from remote store in parallel. Read will be crucial part for publication as each follower node will be downloading the updated state from remote.

Related Issues

Resolves #13728
Resolves #13729
Resolves #14109

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

github-actions bot commented Jun 8, 2024

❌ Gradle check result for 4671704: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Jun 8, 2024

❌ Gradle check result for d7c25e6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Jun 8, 2024

❌ Gradle check result for 3467ca7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
Copy link
Contributor

❌ Gradle check result for 0391436: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❕ Gradle check result for 7a05c8e: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link

codecov bot commented Jun 11, 2024

Codecov Report

Attention: Patch coverage is 56.81818% with 209 lines in your changes missing coverage. Please review.

Project coverage is 71.64%. Comparing base (b15cb0c) to head (7a05c8e).
Report is 405 commits behind head on main.

Files Patch % Lines
...arch/gateway/remote/RemoteClusterStateService.java 50.73% 138 Missing and 29 partials ⚠️
...teway/remote/RemoteClusterStateCleanupManager.java 28.12% 13 Missing and 10 partials ⚠️
...pensearch/cluster/RepositoryCleanupInProgress.java 28.57% 9 Missing and 1 partial ⚠️
...ay/remote/RemoteClusterStateAttributesManager.java 70.00% 2 Missing and 1 partial ⚠️
...arch/index/recovery/RemoteStoreRestoreService.java 0.00% 2 Missing ⚠️
...ting/remote/InternalRemoteRoutingTableService.java 96.42% 0 Missing and 1 partial ⚠️
.../routing/remote/NoopRemoteRoutingTableService.java 50.00% 1 Missing ⚠️
...ateway/remote/model/RemoteClusterStateCustoms.java 85.71% 1 Missing ⚠️
server/src/main/java/org/opensearch/node/Node.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14089      +/-   ##
============================================
+ Coverage     71.42%   71.64%   +0.22%     
- Complexity    59978    61967    +1989     
============================================
  Files          4985     5117     +132     
  Lines        282275   291579    +9304     
  Branches      40946    42150    +1204     
============================================
+ Hits         201603   208902    +7299     
- Misses        63999    65421    +1422     
- Partials      16673    17256     +583     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@shwetathareja
Copy link
Member

PR has an approval from @Bukhtawar . Providing it another approval to unblock merge as some test fix went in since the last approval.

Copy link
Member

@shwetathareja shwetathareja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Providing approval to unblock merge.

@shwetathareja shwetathareja merged commit 990ddc3 into opensearch-project:main Jun 11, 2024
30 of 31 checks passed
@shwetathareja shwetathareja added the backport 2.x Backport to 2.x branch label Jun 11, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jun 11, 2024
…14089)

* Read and write ephemeral objects for remote publication

Co-authored-by: Sooraj Sinha <soosinha@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

* Add serde logic for hashes of consistent settings

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

* Add formattedName to readAsync for IndexMetadata

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

(cherry picked from commit 990ddc3)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
soosinha pushed a commit to soosinha/OpenSearch that referenced this pull request Jun 12, 2024
…pensearch-project#14089)

* Read and write ephemeral objects for remote publication

Co-authored-by: Sooraj Sinha <soosinha@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

* Add serde logic for hashes of consistent settings

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

* Add formattedName to readAsync for IndexMetadata

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

(cherry picked from commit 990ddc3)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
shwetathareja pushed a commit that referenced this pull request Jun 12, 2024
…14089) (#14209)

* Read and write ephemeral objects for remote publication

Co-authored-by: Sooraj Sinha <soosinha@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>

* Add serde logic for hashes of consistent settings

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

* Add formattedName to readAsync for IndexMetadata

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

(cherry picked from commit 990ddc3)

Signed-off-by: Shivansh Arora <hishiv@amazon.com>
shwetathareja pushed a commit that referenced this pull request Jun 12, 2024
…14089) (#14207)

* Read and write ephemeral objects for remote publication

Co-authored-by: Sooraj Sinha <soosinha@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>

* Add serde logic for hashes of consistent settings

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

* Add formattedName to readAsync for IndexMetadata

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

(cherry picked from commit 990ddc3)

Signed-off-by: Shivansh Arora <hishiv@amazon.com>
harshavamsi pushed a commit to harshavamsi/OpenSearch that referenced this pull request Jul 12, 2024
…pensearch-project#14089)

* Read and write ephemeral objects for remote publication

Co-authored-by: Sooraj Sinha <soosinha@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

* Add serde logic for hashes of consistent settings

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

* Add formattedName to readAsync for IndexMetadata

Signed-off-by: Shivansh Arora <hishiv@amazon.com>
@shiv0408 shiv0408 mentioned this pull request Jul 18, 2024
3 tasks
kkewwei pushed a commit to kkewwei/OpenSearch that referenced this pull request Jul 24, 2024
…pensearch-project#14089) (opensearch-project#14207)

* Read and write ephemeral objects for remote publication

Co-authored-by: Sooraj Sinha <soosinha@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>

* Add serde logic for hashes of consistent settings

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

* Add formattedName to readAsync for IndexMetadata

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

(cherry picked from commit 990ddc3)

Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Signed-off-by: kkewwei <kkewwei@163.com>
@shiv0408 shiv0408 self-assigned this Aug 5, 2024
wdongyu pushed a commit to wdongyu/OpenSearch that referenced this pull request Aug 22, 2024
…pensearch-project#14089)

* Read and write ephemeral objects for remote publication

Co-authored-by: Sooraj Sinha <soosinha@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

* Add serde logic for hashes of consistent settings

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>

* Add formattedName to readAsync for IndexMetadata

Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch Cluster Manager ClusterManager:RemoteState enhancement Enhancement or improvement to existing feature or request ShardManagement:Resiliency skip-changelog v2.15.0 Issues and PRs related to version 2.15.0
Projects
Status: ✅ Done
Status: ✅ Done
7 participants