Etcd Cluster Downgrade #11362

YoyinZyc · 2019-11-14T18:17:47Z

In this branch, @wenjiaswe and I added downgrade etcdctl command to control cluster to downgrade to one minor version lower. There are three new apis.

downgrade validate <target_version> : validate whether the cluster can downgrade to target version
downgrade enable <target_version> : enable cluster to downgrade to target version
downgrade cancel: cancel current downgrade job
The implementation followed the design doc. Here is the demo video for basic downgrade workflow.
Related issue support seamless rollback 1 minor revision #7308 Improve etcd upgrade/downgrade policy and tests #9306

Future work

add downgrade status: check downgrade enabled and version of servers in the cluster
add downgrade timeout: If downgrade was never performed for some amount of time, target version (lower version) whitelisting will be revoked.

wenjiaswe · 2019-11-14T18:45:47Z

Thanks @YoyinZyc for your work! @gyuho @xiang90 @jingyih @hexfusion Please help to take a look. Thanks!

codecov-io · 2019-11-14T20:28:19Z

Codecov Report

Merging #11362 into master will increase coverage by 0.49%.
The diff coverage is 58.68%.

@@            Coverage Diff             @@
##           master   #11362      +/-   ##
==========================================
+ Coverage   64.29%   64.79%   +0.49%     
==========================================
  Files         403      404       +1     
  Lines       38079    38751     +672     
==========================================
+ Hits        24484    25109     +625     
- Misses      11950    11988      +38     
- Partials     1645     1654       +9

Impacted Files	Coverage Δ
etcdserver/api/v3rpc/rpctypes/error.go	`90.47% <ø> (ø)`	⬆️
etcdserver/api/v3rpc/util.go	`67.74% <ø> (ø)`	⬆️
etcdserver/errors.go	`0% <ø> (ø)`	⬆️
etcdctl/ctlv3/command/printer.go	`0% <0%> (ø)`	⬆️
etcdctl/ctlv3/command/printer_simple.go	`0% <0%> (ø)`	⬆️
etcdctl/ctlv3/command/printer_fields.go	`0% <0%> (ø)`	⬆️
etcdctl/ctlv3/ctl.go	`100% <100%> (ø)`	⬆️
etcdserver/api/capability.go	`87.09% <100%> (ø)`	⬆️
clientv3/retry.go	`70.17% <100%> (+0.53%)`	⬆️
proxy/grpcproxy/maintenance.go	`51.21% <100%> (+3.85%)`	⬆️
... and 37 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb7703a...22ddb23. Read the comment docs.

gyuho · 2019-11-14T23:43:10Z

etcdserver/api/membership/cluster.go

+
+	if d.Enabled && d.TargetVersion != nil {
+		oneMinorHigher := &semver.Version{Major: d.TargetVersion.Major, Minor: d.TargetVersion.Minor + 1}
+		if !lv.Equal(*d.TargetVersion) && !lv.Equal(*oneMinorHigher) {


Can we have a separate function with unit tests?

Something like isValidDowngrade(cur, target)

Added isValidDowngrade(targetVersion, localVersion) and unit tests

gyuho · 2019-11-14T23:43:43Z

etcdserver/api/membership/cluster.go

+			}
+		}
+
+		if !d.Enabled && lv.LessThan(*cv) {


Let's separate this part as well, to a separate function.

jingyih · 2019-11-15T01:07:12Z

@gyuho
This is for etcd 3.5, do we still need to support plog in this PR?

jingyih

I did a first pass. Overall approach looks good. I think we need to explore if there is a cleaner implementation on the part where individual server quires for cluster downgrade status during startup. Thanks for implementing this feature!

etcdserver/cluster_util.go

etcdserver/v3_server.go

etcdserver/api/membership/cluster.go

etcdserver/cluster_util.go

gyuho · 2019-11-15T05:15:51Z

Yeah plog to be deprecated. No need to have them.

jingyih

Reviewed the code path during server bootstrapping. Added few comments. PTAL:)

etcdserver/server.go

etcdserver/cluster_util.go

etcdserver/v3_server.go

jingyih

Reviewed the server side implementation.

etcdserver/v3_server.go

etcdserver/apply.go

etcdserver/api/membership/store.go

etcdserver/api/membership/cluster.go

etcdserver/v3_server.go

…ation

… restore from snapshot

…ng server and modify compatibility check when new member joins in an existing cluster; add http handler to get downgrade enabled status

…nterRPC and fieldsPrinter for downgrade

…luster. It is used when adding a new member to an downgrading cluster. The new member should match the downgrade target version.

…ernalRaftRequest

…interface ServerPeerHTTP to wrap up ServerPeer.

…data

…de; remove new-added plog

YoyinZyc · 2019-12-16T20:58:22Z

Rebased the migration of cluster attributes.

jingyih

I did a quick pass on the new code changes since rebasing. Overall looks good. I'll spend more time to review later.

jingyih · 2019-12-17T04:20:35Z

etcdserver/v3_server.go

+	}
+	resp.Version = cv.String()
+
+	allowedTargetVersion := semver.Version{Major: cv.Major, Minor: cv.Minor - 1}


Does this work if current version is 4.0?

jingyih · 2019-12-17T04:24:03Z

etcdserver/v3_server.go

+		return nil, ErrIsNotDowngrading
+	}
+
+	raftRequest := membershippb.DowngradeInfoSetRequest{Enabled: false}


nit: If we set target version to 0 here, in apply stage we could simply reconstruct a new downgradeInfo from the request and overwrite the existing one.

YoyinZyc · 2020-01-02T18:43:13Z

@gyuho @jingyih @xiang90 @hexfusion Do you think this pr may be too big to review? I can divide it into smaller pr and collect more comments.

gyuho · 2020-01-02T23:05:48Z

@YoyinZyc Yes please. Maybe one for adding a new RPC in server side. And another one for clientv3 change. And another one for etcdctl command changes?

stale · 2020-04-06T18:25:37Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

codecov-commenter · 2020-05-27T08:49:48Z

Codecov Report

Merging #11362 into master will increase coverage by 0.16%.
The diff coverage is 58.68%.

@@            Coverage Diff             @@
##           master   #11362      +/-   ##
==========================================
+ Coverage   64.62%   64.79%   +0.16%     
==========================================
  Files         403      404       +1     
  Lines       38079    38751     +672     
==========================================
+ Hits        24610    25109     +499     
- Misses      11838    11988     +150     
- Partials     1631     1654      +23

Impacted Files	Coverage Δ
etcdctl/ctlv3/command/printer.go	`0.00% <0.00%> (ø)`
etcdctl/ctlv3/command/printer_fields.go	`0.00% <0.00%> (ø)`
etcdctl/ctlv3/command/printer_simple.go	`0.00% <0.00%> (ø)`
etcdserver/api/v3rpc/rpctypes/error.go	`90.47% <ø> (ø)`
etcdserver/api/v3rpc/util.go	`67.74% <ø> (ø)`
etcdserver/errors.go	`0.00% <ø> (ø)`
etcdctl/ctlv3/command/downgrade_command.go	`35.13% <35.13%> (ø)`
etcdserver/api/membership/store.go	`61.05% <46.39%> (-15.30%)`	⬇️
etcdserver/server.go	`66.58% <62.00%> (-1.40%)`	⬇️
etcdserver/api/membership/cluster.go	`75.74% <67.24%> (+2.96%)`	⬆️
... and 34 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 378b05b...22ddb23. Read the comment docs.

wenjiaswe · 2020-05-27T18:50:22Z

@YoyinZyc are you still working on this one, or is this split into smaller PRs?

YoyinZyc · 2020-05-28T00:21:37Z

@YoyinZyc are you still working on this one, or is this split into smaller PRs?

Downgrade will be implemented with other smaller PRs. It is ok to mark this one as stale.

stale · 2020-08-26T00:39:57Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

gyuho reviewed Nov 14, 2019

View reviewed changes

jingyih reviewed Nov 15, 2019

View reviewed changes

YoyinZyc force-pushed the downgradePOC branch from 6ce8fa9 to 2891693 Compare November 16, 2019 00:39

jingyih reviewed Nov 20, 2019

View reviewed changes

jingyih reviewed Nov 21, 2019

View reviewed changes

etcdserver/v3_server.go Outdated Show resolved Hide resolved

jingyih reviewed Nov 21, 2019

View reviewed changes

jingyih mentioned this pull request Nov 22, 2019

Migrate cluster attributes to use v3 backend #11380

Closed

4 tasks

YoyinZyc mentioned this pull request Nov 22, 2019

Allowed member version when cluster is downgrading #11385

Closed

YoyinZyc force-pushed the downgradePOC branch from 4aedcad to 2caeb11 Compare November 25, 2019 19:31

This was referenced Dec 6, 2019

Migrate cluster attributes to use v3 backend #11427

Merged

auth: fix NoPassWord check when add user #11418

Merged

wenjiaswe and others added 14 commits December 12, 2019 14:24

etcdserver: Add downgrade support POC

9298f84

etcdserver: add downgrade proto api

62280cc

clientV3: add downgrade api in maintenance

1aa02da

v3rpc: add downgrader api in maintenance server

336cef4

membership: add downgrade field into raftCluster

9da21e9

etcdserver: implement EtcdServer.Downgrade; implement downgrade valid…

2023979

…ation

membership: add downgrade backend support

5182d5e

membership: recover downgrade from backend when restart the server or…

a81969f

… restore from snapshot

etcdserver: add downgrade start and downgrade cancel

5df9185

etcdserver: add monitorDowngrade for monitoring the downgrade status

1d80e74

etcdctl: add downgrade commands.

4e429b7

etcdserver: update downgrade check for version when starting/restarti…

fa150ce

…ng server and modify compatibility check when new member joins in an existing cluster; add http handler to get downgrade enabled status

etcdserver: add unit&integration test for downgrade; etcdctl: add pri…

ce75238

…nterRPC and fieldsPrinter for downgrade

etcdserver: add new http handler to fetch the downgrade status of a c…

87aaab6

…luster. It is used when adding a new member to an downgrading cluster. The new member should match the downgrade target version.

YoyinZyc and others added 11 commits December 12, 2019 14:28

e2e: add etcd downgrade/upgrade test

dbc383a

test: ctl_downgrade_test

f01694a

e2e: fix ctl_v3_downgrade_test; etcdctl: update output messages

2311bc4

etcdserver: remove Downgrade Action Status in rpc.proto

fe53e35

etcdserver: change downgrade raft request type from ConfChange to Int…

5c7f7ee

…ernalRaftRequest

etcdserver: allow previous binary to restart when downgrade. add new …

61082c8

…interface ServerPeerHTTP to wrap up ServerPeer.

etcdserver: fix TestCtlV3Migrate nil pointer issue

197a8a6

etcdserver: change "/downgrade/enabled" endpoint to serve linearized …

6efeb79

…data

etcdserver: separate function isValidDowngrade from mustDetectDowngra…

f961ac9

…de; remove new-added plog

etcdserver: define error string and link grpc error code.

00a36b4

etcdserver: add new raft internal request DowngradeInfoSetRequest

22ddb23

YoyinZyc force-pushed the downgradePOC branch from 2caeb11 to 22ddb23 Compare December 16, 2019 20:51

jingyih reviewed Dec 17, 2019

View reviewed changes

YoyinZyc mentioned this pull request Jan 4, 2020

Etcd Downgrade backend #11492

Closed

YoyinZyc mentioned this pull request Mar 23, 2020

Etcd cluster downgrade #11716

Open

13 tasks

stale bot added the stale label Apr 6, 2020

wenjiaswe removed the stale label Apr 7, 2020

stale bot added the stale label Aug 26, 2020

stale bot closed this Sep 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Etcd Cluster Downgrade #11362

Etcd Cluster Downgrade #11362

YoyinZyc commented Nov 14, 2019 •

edited

Loading

wenjiaswe commented Nov 14, 2019

codecov-io commented Nov 14, 2019 •

edited

Loading

gyuho Nov 14, 2019

YoyinZyc Nov 19, 2019

gyuho Nov 14, 2019

jingyih commented Nov 15, 2019

jingyih left a comment

gyuho commented Nov 15, 2019

jingyih left a comment

jingyih left a comment

YoyinZyc commented Dec 16, 2019

jingyih left a comment

jingyih Dec 17, 2019

jingyih Dec 17, 2019

YoyinZyc commented Jan 2, 2020

gyuho commented Jan 2, 2020

stale bot commented Apr 6, 2020

codecov-commenter commented May 27, 2020

wenjiaswe commented May 27, 2020

YoyinZyc commented May 28, 2020

stale bot commented Aug 26, 2020

Etcd Cluster Downgrade #11362

Etcd Cluster Downgrade #11362

Conversation

YoyinZyc commented Nov 14, 2019 • edited Loading

wenjiaswe commented Nov 14, 2019

codecov-io commented Nov 14, 2019 • edited Loading

Codecov Report

gyuho Nov 14, 2019

Choose a reason for hiding this comment

YoyinZyc Nov 19, 2019

Choose a reason for hiding this comment

gyuho Nov 14, 2019

Choose a reason for hiding this comment

jingyih commented Nov 15, 2019

jingyih left a comment

Choose a reason for hiding this comment

gyuho commented Nov 15, 2019

jingyih left a comment

Choose a reason for hiding this comment

jingyih left a comment

Choose a reason for hiding this comment

YoyinZyc commented Dec 16, 2019

jingyih left a comment

Choose a reason for hiding this comment

jingyih Dec 17, 2019

Choose a reason for hiding this comment

jingyih Dec 17, 2019

Choose a reason for hiding this comment

YoyinZyc commented Jan 2, 2020

gyuho commented Jan 2, 2020

stale bot commented Apr 6, 2020

codecov-commenter commented May 27, 2020

Codecov Report

wenjiaswe commented May 27, 2020

YoyinZyc commented May 28, 2020

stale bot commented Aug 26, 2020

YoyinZyc commented Nov 14, 2019 •

edited

Loading

codecov-io commented Nov 14, 2019 •

edited

Loading