Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] Add support to upload snapshot shard blobs with hashed prefix #15643

Merged

Conversation

ashking94
Copy link
Member

Backport c308b98 from #15426.
Backport 4c98c7e from #15559.
Backport 758c2aa from #15560.

…arch-project#15426)

* Add snapshot shard blobs with hashed prefix

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Add UTs

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Address comments and add UTs

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Change default snapshot shard path type to hashed_prefix for testing

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Fix failing tests

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Introduce single method for all create/put repository calls

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Move the index shard path upload to sync upload

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Fix spotless errors

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Do minor code refactor

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Fix tests

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* More tests to use common methods for repo creation

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Address comments and fix multiple failing tests

Signed-off-by: Ashish Singh <ssashish@amazon.com>

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
…5559-15560

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
Copy link
Contributor

github-actions bot commented Sep 4, 2024

❌ Gradle check result for 4cab651: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Copy link
Contributor

github-actions bot commented Sep 4, 2024

❕ Gradle check result for f9b49ed: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@ashking94
Copy link
Member Author

The conflicts were present in below classes -

  1. BlobStoreRepository
  2. CorruptedBlobStoreRepositoryIT
  3. RemoteStoreEnumsTests
  4. RemoteStorePathStrategy
  5. SnapshotsService - due to compilation failure in resolveNewIndices method in beginSnapshot

Copy link

codecov bot commented Sep 4, 2024

Codecov Report

Attention: Patch coverage is 72.78107% with 92 lines in your changes missing coverage. Please review.

Project coverage is 71.76%. Comparing base (5875393) to head (f9b49ed).
Report is 4 commits behind head on 2.x.

Files with missing lines Patch % Lines
...ch/repositories/blobstore/BlobStoreRepository.java 71.24% 58 Missing and 9 partials ⚠️
...ava/org/opensearch/snapshots/SnapshotsService.java 30.00% 6 Missing and 1 partial ⚠️
...a/org/opensearch/snapshots/SnapshotShardPaths.java 85.36% 3 Missing and 3 partials ⚠️
...mote/directory/RemoteSnapshotDirectoryFactory.java 0.00% 5 Missing ⚠️
...main/java/org/opensearch/repositories/IndexId.java 80.00% 1 Missing and 2 partials ⚠️
...va/org/opensearch/repositories/RepositoryData.java 76.92% 1 Missing and 2 partials ⚠️
...arch/index/recovery/RemoteStoreRestoreService.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##                2.x   #15643      +/-   ##
============================================
+ Coverage     71.70%   71.76%   +0.06%     
- Complexity    64104    64199      +95     
============================================
  Files          5249     5250       +1     
  Lines        299681   299917     +236     
  Branches      43643    43672      +29     
============================================
+ Hits         214876   215239     +363     
+ Misses        66848    66697     -151     
- Partials      17957    17981      +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants