Introduce Remote translog feature flag #4158

Bukhtawar · 2022-08-08T08:55:25Z

Signed-off-by: Bukhtawar Khan bukhtawa@amazon.com

Description

Introduce Remote translog feature flag, to gate remote translog changes from breaking remote segment store development and ensuring we can selectively enable remote segment store without requiring remote translog store. This clearly decouples durability semantics from refresh based(only remote segment based where clients would need to maintain non-flushed operations) to a per operation based durability semantics

For now we want to restrict enabling remote translog to cases when only remote segment store is enabled.

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

github-actions · 2022-08-08T09:15:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1556/
CommitID: f1e39a3

server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java

server/src/main/java/org/opensearch/common/util/FeatureFlags.java

server/src/main/java/org/opensearch/common/settings/IndexScopedSettings.java

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

github-actions · 2022-08-08T13:51:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1565/
CommitID: d89e58e

github-actions · 2022-08-08T14:01:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1566/
CommitID: 247c430

Bukhtawar · 2022-08-08T16:47:41Z

Failed due to #4089 #4162

mch2 · 2022-08-08T16:49:53Z

This looks more like an additional setting for remote store vs a feature flag in FeatureFlags.java. I think thats ok bc its still gated by remote store FF?

github-actions · 2022-08-08T17:22:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1570/
CommitID: 247c430

Bukhtawar · 2022-08-08T17:24:28Z

This looks more like an additional setting for remote store vs a feature flag in FeatureFlags.java. I think thats ok bc its still gated by remote store FF?

That's correct @mch2. The FF would be common across remote segment store and remote translog store however to keep remote translog store gated and provide mechanisms to support remote segment store independent of remote translog store we are going with two separate index settings

mch2

LGTM.

nknize

Overall this LGTM, but I have questions about the silent validation when settings are null.

server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java

nknize · 2022-08-08T18:32:25Z

server/src/test/java/org/opensearch/index/IndexSettingsTests.java

@@ -777,9 +775,30 @@ public void testRemoteStoreExplicitSetting() {
        assertTrue(settings.isRemoteStoreEnabled());
    }

+    public void testRemoteTranslogStoreDefaultSetting() {


Should we add a test if settings is null?

@nknize added tests for null settings

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

github-actions · 2022-08-08T19:51:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1580/
CommitID: 75afc1e

github-actions · 2022-08-08T20:37:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1585/
CommitID: 75afc1e

github-actions · 2022-08-08T22:48:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1591/
CommitID: 75afc1e

github-actions · 2022-08-08T23:39:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1596/
CommitID: 75afc1e

github-actions · 2022-08-09T00:27:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1598/
CommitID: 75afc1e

Addressed comments, will follow it up if there are are further changes needed

Bukhtawar · 2022-08-09T06:19:20Z

@nknize I am merging the PR to unblock other PRs, please feel free to add comments and I will address them in a follow up PR.

opensearch-trigger-bot · 2022-08-09T08:28:46Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-4158-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a469a3cbcb7a7294a81bf7e4122968c2a9b32ca4
# Push it to GitHub
git push --set-upstream origin backport/backport-4158-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-4158-to-2.x.

* Introduce Remote translog feature flag * Limit combinations of remote segment and translog store to valid ones Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce Remote translog feature flag * Limit combinations of remote segment and translog store to valid ones Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> (cherry picked from commit a469a3c)

* Introduce Remote translog feature flag (#4158) * Introduce Remote translog feature flag * Limit combinations of remote segment and translog store to valid ones Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> (cherry picked from commit a469a3c) * Add CHANGELOG entry Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce Remote translog feature flag * Limit combinations of remote segment and translog store to valid ones Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Introduce Remote translog feature flag

f1e39a3

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Bukhtawar requested review from a team and reta as code owners August 8, 2022 08:55

sachinpkale reviewed Aug 8, 2022

View reviewed changes

Bukhtawar added 2 commits August 8, 2022 18:55

Limit combinations of remote segment and translog store to valid ones

d89e58e

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Minor test case rename

247c430

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Bukhtawar requested a review from mch2 August 8, 2022 17:24

mch2 approved these changes Aug 8, 2022

View reviewed changes

nknize previously requested changes Aug 8, 2022

View reviewed changes

Add tests for null value

75afc1e

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

Bukhtawar requested a review from nknize August 8, 2022 19:26

sachinpkale approved these changes Aug 9, 2022

View reviewed changes

Bukhtawar merged commit a469a3c into opensearch-project:main Aug 9, 2022

Bukhtawar deleted the remote-translog-prs branch August 9, 2022 06:22

Bukhtawar added the backport 2.x Backport to 2.x branch label Aug 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Remote translog feature flag #4158

Introduce Remote translog feature flag #4158

Bukhtawar commented Aug 8, 2022 •

edited

Loading

github-actions bot commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

Bukhtawar commented Aug 8, 2022 •

edited

Loading

mch2 commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

Bukhtawar commented Aug 8, 2022

mch2 left a comment

nknize left a comment

nknize Aug 8, 2022

Bukhtawar Aug 8, 2022

github-actions bot commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

github-actions bot commented Aug 9, 2022

Bukhtawar commented Aug 9, 2022

opensearch-trigger-bot bot commented Aug 9, 2022

Introduce Remote translog feature flag #4158

Introduce Remote translog feature flag #4158

Conversation

Bukhtawar commented Aug 8, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

Bukhtawar commented Aug 8, 2022 • edited Loading

mch2 commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

Bukhtawar commented Aug 8, 2022

mch2 left a comment

Choose a reason for hiding this comment

nknize left a comment

Choose a reason for hiding this comment

nknize Aug 8, 2022

Choose a reason for hiding this comment

Bukhtawar Aug 8, 2022

Choose a reason for hiding this comment

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 9, 2022

Gradle Check (Jenkins) Run Completed with:

Bukhtawar commented Aug 9, 2022

opensearch-trigger-bot bot commented Aug 9, 2022

Bukhtawar commented Aug 8, 2022 •

edited

Loading

Bukhtawar commented Aug 8, 2022 •

edited

Loading