functional-tester: revision, lease expiration mismatch

[gyuho]

2017-06-07 17:22:54.288013 I | etcd-tester: lease Stresser 10.240.0.6:2379 starting ...
2017-06-07 17:22:54.288048 I | etcd-tester: [round#0 case#0] started stressers
2017-06-07 17:22:54.289121 I | etcd-tester: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.240.0.2:2379: getsockopt: connection refused"; Reconnecting to {10.240.0.2:2379 <nil>}
@
                                                                                                                                                                    150,1         78%
2017-06-07 17:22:44.691818 I | etcd-tester: keyStresser "10.240.0.3:2379" is closed
2017-06-07 17:22:44.691848 I | etcd-tester: keyStresser "10.240.0.2:2379" is closed
2017-06-07 17:22:44.691856 I | etcd-tester: [round#0 case#0] paused stressers
2017-06-07 17:22:44.691870 I | etcd-tester: [round#0 case#0] wait until cluster is healthy
2017-06-07 17:22:44.780298 I | etcd-tester: [round#0 case#0] cluster is healthy
2017-06-07 17:22:44.780327 I | etcd-tester: [round#0 case#0] checking consistency and invariant of cluster
2017-06-07 17:22:44.945409 W | etcd-tester: retry 0. etcd cluster is not stable: [revisions: map[http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351 http://10.240.0.4:2379:62350 http://10.240.0.5:2379:62351 http://10.240.0.6:2379:62351]] and [hashes: map[http://10.240.0.2:2379:1086686394 http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260 http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394]]
2017-06-07 17:22:46.101771 W | etcd-tester: retry 1. etcd cluster is not stable: [revisions: map[http://10.240.0.6:2379:62351 http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351 http://10.240.0.4:2379:62350 http://10.240.0.5:2379:62351]] and [hashes: map[http://10.240.0.2:2379:1086686394 http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260 http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394]]
2017-06-07 17:22:47.279183 W | etcd-tester: retry 2. etcd cluster is not stable: [revisions: map[http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351 http://10.240.0.4:2379:62350 http://10.240.0.5:2379:62351 http://10.240.0.6:2379:62351]] and [hashes: map[http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260 http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394 http://10.240.0.2:2379:1086686394]]
2017-06-07 17:22:48.448153 W | etcd-tester: retry 3. etcd cluster is not stable: [revisions: map[http://10.240.0.4:2379:62350 http://10.240.0.5:2379:62351 http://10.240.0.6:2379:62351 http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351]] and [hashes: map[http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394 http://10.240.0.2:2379:1086686394 http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260]]
2017-06-07 17:22:49.609552 W | etcd-tester: retry 4. etcd cluster is not stable: [revisions: map[http://10.240.0.6:2379:62351 http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351 http://10.240.0.4:2379:62350 http://10.240.0.5:2379:62351]] and [hashes: map[http://10.240.0.2:2379:1086686394 http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260 http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394]]
2017-06-07 17:22:50.775824 W | etcd-tester: retry 5. etcd cluster is not stable: [revisions: map[http://10.240.0.6:2379:62351 http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351 http://10.240.0.4:2379:62350 http://10.240.0.5:2379:62351]] and [hashes: map[http://10.240.0.2:2379:1086686394 http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260 http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394]]
2017-06-07 17:22:51.934019 W | etcd-tester: retry 6. etcd cluster is not stable: [revisions: map[http://10.240.0.6:2379:62351 http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351 http://10.240.0.4:2379:62350 http://10.240.0.5:2379:62351]] and [hashes: map[http://10.240.0.2:2379:1086686394 http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260 http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394]]
2017-06-07 17:22:52.934377 I | etcd-tester: [round#0 case#0] lease 1971270982099336196 expiration mismatch (lease expired=true, keys expired=false), lease 4255721893083028973 expiration mismatch (lease expired=true, keys expired=false), lease 4594899240019358752 expiration mismatch (lease expired=true, keys expired=false), lease 3971995116558682132 expiration mismatch (lease expired=true, keys expired=false), etcd cluster is not stable: [revisions: map[http://10.240.0.5:2379:62351 http://10.240.0.6:2379:62351 http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351 http://10.240.0.4:2379:62350]] and [hashes: map[http://10.240.0.2:2379:1086686394 http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260 http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394]]
2017-06-07 17:22:52.934431 W | etcd-tester: [round#0 case#0] functional-tester returning with error (tt.checkConsistency error (lease 1971270982099336196 expiration mismatch (lease expired=true, keys expired=false), lease 4255721893083028973 expiration mismatch (lease expired=true, keys expired=false), lease 4594899240019358752 expiration mismatch (lease expired=true, keys expired=false), lease 3971995116558682132 expiration mismatch (lease expired=true, keys expired=false), etcd cluster is not stable: [revisions: map[http://10.240.0.5:2379:62351 http://10.240.0.6:2379:62351 http://10.240.0.2:2379:62351 http://10.240.0.3:2379:62351 http://10.240.0.4:2379:62350]] and [hashes: map[http://10.240.0.2:2379:1086686394 http://10.240.0.3:2379:1086686394 http://10.240.0.4:2379:2060741260 http://10.240.0.5:2379:1086686394 http://10.240.0.6:2379:1086686394]]))
2017-06-07 17:22:52.934448 I | etcd-tester: [round#0 case#0] closing the stressers...
2017-06-07 17:22:52.934488 I | etcd-tester: lease stresser "10.240.0.6:2379" is closed

[xiang90]

@fanminshi can you help on this one?

[fanminshi]

@xiang90 will do

[fanminshi]

can reproduce every time on local functional tester with default setting.

[fanminshi]

etcd cluster is not stable: [revisions: map[http://127.0.0.1:22379:7725 http://127.0.0.1:32379:7725 http://127.0.0.1:12379:7725]] and [hashes: map[http://127.0.0.1:32379:2954077830 http://127.0.0.1:12379:367948630 http://127.0.0.1:22379:2954077830]]

In some case, I noticed that revision for nodes are the same but hashes are different.

[fanminshi]

commit at 163fd2d causes this issue.

[xiang90]

@fanminshi this should be fixed. can you double check and close it?

[fanminshi]

verified the patch #8127 fixes the issue. closing.