etcdserver: Backport snapshot recovery from #7917 to 3.1 branch #9808
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gyuho
approved these changes
Jun 5, 2018
|
Sounds good. I've verified that the fix works the same as it does on the 3.2 branch for cases where there is a snap.db file and the main db file is stale, so I'm comfortable with this being merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On 3.1.11, we encountered:
This has proved tricky to reproduce exactly, but we were able to simulate it by setting --max-wals=1 stopping a member, waiting for the wal logs to be replaced, forcing compaction and then replacing the db file with one of the out-of-date snap.db files.
We believe it is caused by recovery from a snap.db file when etcd crashes after persisting the snapshot file to disk but before updating the backend db.
recoverBackendSnapshotwas introduced in #7917 to fix this in etcd 3.2+, so this PR backports that recovery code to 3.1../test SUCCESS
PASSES=integration ./test SUCCESS
Note that #7856, fixed a bug with this same error message, but that fix exists already in 3.1.11.
cc @gyuho @wojtek-t @wenjiaswe