Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support to skip pinned timestamp in remote segment garbage collector #15017

Merged
merged 6 commits into from
Aug 27, 2024

Conversation

sachinpkale
Copy link
Member

@sachinpkale sachinpkale commented Jul 30, 2024

Description

  • As part of shallow snapshot optimisations, we are introducing timestamp based implicit locking support. This feature will be called Timestamp Pinning in remote backed storage. When a timestamp is pinned, remote store garbage collectors will skip deleting data corresponding to the timestamp.
  • In this PR, we are making following changes to segment store garbage collection:
  • Fetch metadata files in reverse-chronological order
  • If timestamp of a metadata file md1 is > pinned_timestamp_a and the timestamp of next metadata file md2 <= pinned_timestamp_a, consider md2 to be locked by pinned timestamp.
  • Skip deletion of metadata files that are locked by pinned timestamps and corresponding data files
  • Segment garbage collection will continue to skip locked metadata files for backward compatibility.

Along with garbage collection change, this PR also introduces a method to initialise RemoteSegmentStoreDirectory to a given timestamp.

Related Issues

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

❌ Gradle check result for e8bc4e6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Aug 2, 2024

❌ Gradle check result for 61f8c26: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Aug 2, 2024

❌ Gradle check result for 755f393: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Aug 2, 2024

❌ Gradle check result for b8cbc22: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Aug 5, 2024

❌ Gradle check result for 8fb7cef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@sachinpkale sachinpkale merged commit 46a7bb6 into opensearch-project:main Aug 27, 2024
39 of 41 checks passed
@sachinpkale sachinpkale added the backport 2.x Backport to 2.x branch label Aug 31, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 31, 2024
…tor (#15017)

---------
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
(cherry picked from commit 46a7bb6)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Aug 31, 2024
…tor (#15017) (#15550)

---------
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
(cherry picked from commit 46a7bb6)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
akolarkunnu pushed a commit to akolarkunnu/OpenSearch that referenced this pull request Sep 10, 2024
…tor (opensearch-project#15017)

---------
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 16, 2024
…tor (opensearch-project#15017)

---------
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 17, 2024
…tor (opensearch-project#15017)

---------
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 21, 2024
…tor (opensearch-project#15017)

---------
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants