Increase robustness for kubeadm join / add etcd

[fabriziopandini]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version: v1.17.*

What happened?

While executing Cluster API tests, in some cases it was observed kubeadm join failures when waiting for the new etcd member to report healthy state.

xref kubernetes-sigs/cluster-api#2769

What you expected to happen?

To add new etcd member more resilient by increasing the timeout/the number of retries for this operation

How to reproduce it (as minimally and precisely as possible)?

This error happens only sometimes, most probably due to slow network/slow I/O causing delays in etcd getting online or in some cases, also change fo the etcd leader.

Anything else we need to know?

Important: if possible the change should be kept as small and possible and backported

[neolit123]

@fabriziopandini note there is already a ticket for learner mode here:
#1793

[fabriziopandini]

@neolit123 thanks! nevertheless, I will keep this also one for making current implement more robust

[neolit123]

/assign

[neolit123]

To add new etcd member more resilient by increasing the timeout/the number of retries for this operation

@fabriziopandini what timeouts are we talking about?

current backoff for AddMember is:

steps: 11
duration: 50
factor: 2
jitter: 0.1
step: 0, value: 0
step: 1, value: 50.35
step: 2, value: 155.55
step: 3, value: 363.46
step: 4, value: 779.66
step: 5, value: 1611.29
step: 6, value: 3275.65
step: 7, value: 6602.45
step: 8, value: 13257.2
step: 9, value: 26565.96
step: 10, value: 53182.36

~53 sec

https://github.com/kubernetes/kubernetes/blob/8a4bf398840f1298edaca40b76bee2ace2733d12/cmd/kubeadm/app/util/etcd/etcd.go#L370-L383

also are you sure this is a AddMember issue and not an issue with the client dial?
https://github.com/kubernetes/kubernetes/blob/8a4bf398840f1298edaca40b76bee2ace2733d12/cmd/kubeadm/app/util/etcd/etcd.go#L356
https://github.com/kubernetes/kubernetes/blob/8a4bf398840f1298edaca40b76bee2ace2733d12/cmd/kubeadm/app/util/etcd/etcd.go#L215

[fabriziopandini]

@neolit123 I was thinking that we can raise this timeout up to 2 minutes (or even more)
Also, as you are pointing out, for better robustness, we can move the client dial step into the retry loop

[neolit123]

all cherry picks are about to merge, but we should keep this open to refactor the etcd client management in a similar way in mater. e.g. kubernetes/kubernetes#90645 made it so MemberAdd behaves differently than MemberRemove

[neolit123]

cherry picks merged. lowering priority as the remaining refactor is not mandatory for 1.19.

[neolit123]

actually let me log a new ticket