-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ignore old leader's leases revoking request #16822
Conversation
25203e6
to
182c9d6
Compare
0c85835
to
6011977
Compare
2964d32
to
ed89fd9
Compare
Known flake.. We may want to auto retry setting up failpoint just like #16094 |
5b5bd5f
to
ebedb90
Compare
7da328d
to
731df05
Compare
@mitake @serathius @chaochn47 @fuweid @tjungblu Please take a look at this PR and also https://docs.google.com/document/d/1peLDjwebnSuNR69ombuUiJ5fkSw7Rd3Slv_mUpOLsmg/edit I will try to add e2e test cases after we get consensus. |
Thanks @ahrtr , let me check this PR and the doc. Did you check this PR with #15247 (comment) ? I'll do it on my side tomorrow. |
Yes, confirmed that it works. |
@ahrtr I could also check that this PR can work with the failpoint based test, great! I'm still feeling that radical long term solution might be something like disabling proposal forwarding (or adding term information in proposal forwarding), e.g. #16285 IIUC, this PR might still have rare corner cases like: other processes on the server which runs etcd or other threads of etcd itself become very busy and execution stops between leader check and sending request (this line https://github.com/etcd-io/etcd/pull/16822/files#diff-164b95d19232df0c51cd511fe4ca61de738660fc11db6ddcdafa0ec5f5c80a35R405) However I think this solution can work for many practical cases. Is my understanding aligned with you (according to the doc I think it's aligned)? Anyway I agree with the approach of this PR. |
@mitake thx for the feedback.
Please see etcd-io/raft#88 (comment). Of course, the long-term solution is still open to discussion. There is no rush to make any decision for now.
Even it happens (It's unlikely in practice), it shouldn't be a big problem. The lease depends on physical wall clock, an error of tens of or even hundreds of milliseconds should be acceptable for lease. Note that the default checkpoint interval is 5 minutes; it means that etcd does't run checkpoint at all for leases <= 5minutes.
thx |
I agree with proposed approach, however to merge the PR we need proper e2e tests. The change is also pretty risky for backport. Those problems persisted for so long because lease refresh is not as utilized as much as other parts of API. We should think about how we qualify this. |
43bb172
to
7dc08b3
Compare
aebad91
to
f7ff898
Compare
I'll attempt first to implement |
Hi @ahrtr, after backporting |
Please feel free to backport the PR to 3.5 and 3.4 and fix whatever is missing, thx |
@ahrtr, as I was working on backporting this into 3.5. I realized that another functionality needs to be added to the e2e tests. |
I would suggest the same principle as before: try to backport it firstly; if it needs huge change or effort, then try to implement it in 3.5 & 3.4 directly. Thanks. |
Backported PR etcd-io#16822, commits f7e488d, 67f1716, f7ff898 Signed-off-by: Ivan Valdes <ivan@vald.es>
Backported PR etcd-io#16822, commits f7e488d, 67f1716, and f7ff898 Signed-off-by: Ivan Valdes <ivan@vald.es>
Backported PR etcd-io#16822, commits f7e488d, 67f1716, and f7ff898 Signed-off-by: Ivan Valdes <ivan@vald.es>
Backported PR etcd-io#16822, commits f7e488d, 67f1716, and f7ff898 Signed-off-by: Ivan Valdes <ivan@vald.es>
Backport of PR etcd-io#16822, commits f7e488d, 67f1716, and f7ff898 Signed-off-by: Ivan Valdes <ivan@vald.es>
Backport of PR etcd-io#16822, commits f7e488d, 67f1716, and f7ff898 Signed-off-by: Ivan Valdes <ivan@vald.es>
Backport of PR etcd-io#16822, commits f7e488d, 67f1716, and f7ff898. Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com> Signed-off-by: Ivan Valdes <ivan@vald.es>
Backport of PR etcd-io#16822, commits f7e488d, 67f1716, and f7ff898. Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com> Signed-off-by: Ivan Valdes <ivan@vald.es>
Backport of PR etcd-io#16822, commits f7e488d, 67f1716, and f7ff898. Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com> Signed-off-by: Ivan Valdes <ivan@vald.es>
Fix #15247.
Also refer to https://docs.google.com/document/d/1peLDjwebnSuNR69ombuUiJ5fkSw7Rd3Slv_mUpOLsmg/edit#heading=h.jbaw5ynuj9i7