sig-cl/kubeadm: add provisional KEP for etcd learner mode

[neolit123]

• One-line PR description:
  use etcd's learner mode for joining etcd members to the etcd cluster managed by kubeadm

• Issue link:
  sig-cl/kubeadm: Use etcd's learner mode #3614
  RFE: use learner mode for joining etcd members kubeadm#1793

• Other comments:
  this is not targeting a release yet. the KEP is provisional and needs someone to own the work.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neolit123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-cluster-lifecycle/OWNERS [neolit123]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[neolit123]

/hold
for review
/cc @fabriziopandini @pacoxu

this is the provisional etcd learner mode KEP PR.
as discussed, we'd like to add the KEP and once we have someone who has time / is willing to work on it we can revisit and check the implementable state.

[pacoxu]

I have encountered some cases in that the ntp is not configured by joining as a control plane and ends up with errors. (Not quite sure. Can this case be covered by learner mode?)

[neolit123]

i have not experimented with learner mode.
by ntp do you mean network time protocol?
haven't played with that much either.

[harshanarayana]

+1 to what @pacoxu mentioned. We had issues where two nodes of the cluster ended up detecting different set of NTP offset (One node was ahead and other was behind the ETCD node that was already running) that lead to some issue with ETCD member join and some issue after in terms of building a consensus over raft.

[neolit123]

looks like if learner mode is added to kubeadm at the alpha stage, we have to experiment with ntp offsets.

[fabriziopandini]

IMO testing ntp offsets should be out of scope because AFAIK learner mode but it is not meant to be something that improves the resilience of the raft protocol to machine mis-configurations (or at least this is not a goal of the original design).

[neolit123]

agreed, and i will add this as out of scope. proper machine configuration is already a pre-requisite.

[pacoxu]

As there is a feature gate, we should not add an extra phase for it.

The change would happen in kubeadm join phase control-plane-join etcd for adding a new local etcd member.

I would like to work on this if you are busy with other work.

[neolit123]

As there is a feature gate, we should not add an extra phase for it.

The change would happen in kubeadm join phase control-plane-join etcd for adding a new local etcd member.

yes, it will be like a separate logic path controlled by the FG and not a new phase.

I would like to work on this if you are busy with other work.

sure if you have time. we can try getting this done for 1.27.
experimentation can start sooner of course.

[harshanarayana]

+1 to what @pacoxu mentioned. We had issues where two nodes of the cluster ended up detecting different set of NTP offset (One node was ahead and other was behind the ETCD node that was already running) that lead to some issue with ETCD member join and some issue after in terms of building a consensus over raft.

[harshanarayana]

This might not be the right place to bring up this but leaving it here just in case.

The current way of kubeadm can handle parallel joins pretty well like you have mentioned before, how would that behavior change if the leaner mode is to be put in place ? Do we promote each of them as standalone?

[neolit123]

discussing this with @fabriziopandini , we thought that learners will simply stay on hold / wait and get promoted async. what is not clear to me is if etcd only supports one learner maximum. this maximum was stated in the docs before. if that is the case what can happen instead is that we might have to retry the add member call.

[harshanarayana]

There was an option added to configure the max learner in ETCD a while ago. xref: etcd-io/etcd#13377

[neolit123]

that's good, we can cap it to an odd number > 3. but if the number is small, it feels like we still have to have the addmember retry.

alternatively we can just have it as a very large number.

[harshanarayana]

Would it be possible to deduce the number based on the parsing of initial-cluster: ${NAMES[0]}=https://${HOSTS[0]}:2380,${NAMES[1]}=https://${HOSTS[1]}:2380,${NAMES[2]}=https://${HOSTS[2]}:2380... configuration if provided ?

[neolit123]

i don't think so. initial cluster is incremented per each new member, so it goes like A; A,B; A,B,C

and we don't know the final number of CP nodes the user wants. my guess is that ~95+% of HA users have 3 CP nodes. the rest might be at 5. 7 is too much, but who knows.

[harshanarayana]

It doesn't always have to be that way though does it ? Take https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/ for example. The initial-cluster is already aware of the fact that it will have 3 nodes. But I am not sure if there is any enforcement on this kind of config. So, I get your point 😄 Thanks for the clarification

[neolit123]

external etcd is actually out of scope for this proposal, because there the user manages the etcd machines outside of the control plane machines.

[fabriziopandini]

Frankly speaking, I do not think we should claim that parallel join of CP nodes is supported until we have proper test coverage for it.

As a consequence, IMO we should optimize UX for sequential join only, eventually expose flags for people willing to explore parallel joining (or let this be done via extra args), but do not invest time in tuning this scenario.
We can also rule it a non-goal for this first iteration.

[neolit123]

will add notes that learner number will be kept as default (1).
improvements over the tentatively supported parallel join will be added as out-of-scope. basically we can retry in the logic, similarly to how it is now, but there will be no claims for improvements.

[neolit123]

updated by adding some goals / non-goals as per the discussions thus far.

[pacoxu]

https://github.com/etcd-io/etcd/blob/29c3b0f307107fd95a6eb43ddff20db952eeb2fa/server/etcdserver/api/membership/cluster_opts.go#L17

As this is a default behavior, when we join two control-plan together, one would wait for the other due to the set. Should we continue to retry if the addMemberAsLearner failed?
Or should we exit if we get the too many learner members in cluster error?

[neolit123]

i think we should set up an exponential backoff timeout of say 40 seconds / 1 minute max. potentially reuse the current timeout. then if members are joined in parallel the add member call can retry. we shouldn't exit on max members error.

[neolit123]

any further comments on this, or can we get it merged?

[pacoxu]

/lgtm

[harshanarayana]

/lgtm

Thanks @neolit123 and @pacoxu for the discussion on this.

[neolit123]

/hold cancel

[logicalhan]

/assign

[logicalhan]

There is a downside though, if you are maintaining 3/3 quorum and adding a learner, you are increasing the resource consumption of etcd by 33%.

[logicalhan]

We should probably call these things more descriptively in the KEP for those who do not have the context. I recommend "old way" to be renamed "etcd-without-learner"