Etcd cluster startup failed

[swenzi]

Hi,
My etcd cluster consists of three machines. When I started the etcd cluster, two etcds reported errors. The details are as follows:

1. http://pdqtitad6.bkt.clouddn.com/etcd-issue-1.png
2. http://pdqtitad6.bkt.clouddn.com/etcd-issue-2.png
3. http://pdqtitad6.bkt.clouddn.com/etcd-issue-3.png

Please help me to see why.
Thanks!

[hexfusion]

When I started the etcd cluster, two etcds reported errors. The details are as follows:

@swenzi can you tell us a little more information on this cluster's data? I am curious if you have updated etcd from a previous version. Specifically was etcd at anytime < 3.2.10? I assume this was an existing cluster that was previously working? Basically, we need more details to understand why we are seeing this freelist panic to understand if it is related to #8813.

Adding image for easier reference but in future txt logs are much easier for us to review.

/cc @jpbetz

[jpbetz]

+1 to @hexfusion's analysis, this does look a lot like #8813 corruption that was fixed in 3.2.10+. I'd start by examining that as cause. Was a <3.2.10 version of etcd in use recently?

[swenzi]

Yes, There was a cluster in the virtual machine before. Then my host suddenly went out of power. When I restarted the host and started the virtual machine, the above error occurred. My etcd version 3.2.18, thank you!

Detail:
http://pdqtitad6.bkt.clouddn.com/etcd-issue-4.png

[hexfusion]

can you tell us a little more information on this cluster's data? I am curious if you have updated etcd from a previous version. Specifically was etcd at anytime < 3.2.10?

Hi @swenzi can you answer my question above? We understand that you are running 3.2.18 now, thanks!

[swenzi]

I didn't have updated etcd from a previous version. I have used etcd:3.2.18 all the time. there was't an existing cluster that was previously working.

[swenzi]

Now my cluster is good because I reinstalled the etcd cluster.

[hexfusion]

@swenzi thank you for the details would you be able to share the data-dir for the corrupted nodes so that we can review? This may help us to better understand the underlying problem.

[swenzi]

Sorry, the cluster is reinstalled, and the data is not retained. If this problem arises again, I will share the data with you. Thank you!