a data corruption bug in revoking lease when upgrading cluster from v3.2 to v3.3/v3.4+

[tangcong]

What happened:

recently, our team(TencentCloud k8s team) encountered another serious data inconsistency bug when upgrading the cluster(3.2->3.3). the number of keys every node is inconsistent. The cluster does not work when you deploy/update workload.

How to trouble-shooting it:

we add debugging log and use simple chaos monkey tool to reproduce it. we successfully reproduced it again. etcd is very hard to troubleshooting data inconsistency due to lack of log.

node A(3.2+,Leader)

637802:Mar 11 21:52:23 localhost etcd[28065]: LeaseRevoke:{LeaseRevoke 24 0  ID:2229124347589447546 } {entry-index 17 25  <nil>} {entry-term 17 2  <nil>} {rev 11 2014102077  <nil>}
637803:Mar 11 21:52:23 localhost etcd[28065]: revoke lessor,lessor id:2229124347589447546
637804:Mar 11 21:52:23 localhost etcd[28065]: revoke lessor,lessor id:2229124347589447546,key:/masterleases/A
637805:Mar 11 21:52:23 localhost etcd[28065]: apply request:header:<ID:16363671783383239993 > lease_revoke:<ID:2229124347589447546 > ,response:&{0xc000d545e0 <nil> <nil>}

node B(3.2+,Follower)

637802:Mar 11 21:52:23 localhost etcd[28065]: LeaseRevoke:{LeaseRevoke 24 0  ID:2229124347589447546 } {entry-index 17 25  <nil>} {entry-term 17 2  <nil>} {rev 11 2014102077  <nil>}
637803:Mar 11 21:52:23 localhost etcd[28065]: revoke lessor,lessor id:2229124347589447546
637804:Mar 11 21:52:23 localhost etcd[28065]: revoke lessor,lessor id:2229124347589447546,key:/masterleases/A
637805:Mar 11 21:52:23 localhost etcd[28065]: apply request:header:<ID:16363671783383239993 > lease_revoke:<ID:2229124347589447546 > ,response:&{0xc000d545e0 <nil> <nil>}

node C(3.3+,Follower)

1392010:Mar 11 21:56:42 localhost etcd[26312]: applyEntryNormal:check index, entry-index: 25, consistent-index: 24
1392011:Mar 11 21:56:42 localhost etcd[26312]: shouldApplyV3, entry-index: 25,  consistent-index: 25
1392012:Mar 11 21:56:42 localhost etcd[26312]: LeaseRevoke: ID:2229124347589447546 , entry-index: 25, entry-term: 2, rev: 2014102070
1392014:Mar 11 21:56:42 localhost etcd[26312]: request "header:<ID:16363671783383239993 > lease_revoke:<id:1eef70c8a23e4f7a>" with result "error:auth: user name is empty" took too long (1.464µs) to execute

node C(3.3+) failed to apply lease_revoke command(error:auth: user name is empty). this error will continue to amplify, causing the mvcc revision to diverge very fast, failing to execute txn command, and data corruption.

How to fix it：

In the upgrade documentation, it is better to add this bug description. Users must backup data and be careful of the cluster upgrade operation, it is a high-risk operation.
we have added a pr #11691 to release-3.2 to ensure that auth info is not nil.
if user want to upgrade cluster from 3.2 to 3.3/3.4, user can firstly upgrade the cluster to the 3.2 latest version.
do you have any other better suggestions?
@jingyih @mitake

How this bug was introduced：

pr #8031(protecting lease revoking with auth) limits the users who can revoke leases. If the user isn't granted write permission of keys which are attached the lease, the revoking request will be denied.
3.0+/3.1+/3.2+ do not limit the users who can revoke leases,so user name is empty.

Impact:

it is possible to encounter it when authentication is enabled and upgrading cluster from v3.0/v3.1/v3.2 to v3.3/v3.4.

[jpbetz]

Since only impacts cluster using auth and leases so Kubernetes is not impacted, right?

[tangcong]

@jpbetz it affects our Kubernetes cluster because our etcd cluster enables auth.

[jpbetz]

@jpbetz it affects our Kubernetes cluster because our etcd cluster enables auth.

Ah

[tangcong]

If authentication is enabled,it has a certain probability to encounter this issue in the k8s cluster scenario.

[tangcong]

Recently we encountered two data inconsistent issues(#11651,#11689) caused by authentication. Is GKE not using etcd authentication? @jpbetz @jingyih

[jingyih]

1392014:Mar 11 21:56:42 localhost etcd[26312]: request "header:<ID:16363671783383239993 > lease_revoke:id:1eef70c8a23e4f7a" with result "error:auth: user name is empty" took too long (1.464µs) to execute

Lease revoke may come from v3rpc user request [1] or internally in etcdserver [2]. Does this happen in both cases?

[1]

etcd/etcdserver/api/v3rpc/lease.go

Line 46 in 282cce7

func (ls *LeaseServer) LeaseRevoke(ctx context.Context, rr *pb.LeaseRevokeRequest) (*pb.LeaseRevokeResponse, error) {

[2]

etcd/etcdserver/server.go

Line 854 in 282cce7

_, lerr := s.LeaseRevoke(ctx, &pb.LeaseRevokeRequest{ID: int64(lid)})

[wswcfan]

@jingyih Yes, this happen in both cases. But lease revoke internally in etcdserver [2] may be easier to trigger this bug, because once the lease expires, the lease revoke call is triggered, and it doesn't carry the authentication information.

[tangcong]

pr #11691 fixed this issue.