Allowed member version when cluster is downgrading

[YoyinZyc]

• Only allow downgrade target version to join in cluster.
  • pros: strict enough for downgrade
  • cons: if one member crashes unexpectedly during downgrade, it cannot restart itself with old binary.(ref: Etcd Cluster Downgrade #11362 (comment))
• Allow higher than target version.
  • pros: will not break current cluster operation
  • cons: the cluster may upgrade during downgrade. For example, we have a cluster {3.4, 3.5, 3.6}. The cluster version is 3.4 and we want it to downgrade to 3.3. If we stop 3.4 and restart it with 3.5, the cluster upgrades to 3.5. It is weird that the cluster can downgrade to 3.3 now.
• Allow target version and one minor higher than target version.
  It is a trade-off between the two approaches above. For regular condition that all members in the cluster have same local version, it allows server to crash and restart itself. However, it doesn't work if the cluster has mix version members. For example, for a cluster {3.4, 3.5, 3.6}. The cluster version is 3.4 and we want it to downgrade to 3.3. If 3.5 crashes, it cannot restart itself as 3.5.
  I use this approach currently in Etcd Cluster Downgrade #11362

Which policy do you think is better? Is there any other corner case we need to take care?
@wenjiaswe @jingyih @gyuho @hexfusion

[hexfusion]

Thanks for doing this @YoyinZyc I hope to review your work soon.

[gyuho]

We should start with the simplest implementation. Just allow one version lower than the current cluster version, which is agreed in quorum. Then, we don't need worry about having 3 minor versions in the same cluster.

We can handle other edge cases later.

[gyuho]

In most cases, the downgrade would not overlap with upgrade operation. So, let's not worry about the cluster version skew greater than 1. It can be only one version difference, either upgrade or downgrade.

[jingyih]

I think the current approach (allow 1 minor version difference) is fine.

[wenjiaswe]

I agree with approach 1, only allowing 1 minor version lower. That's the policy for now, and it makes sense the code agree with the policy. If the corner case is not so corner in future, we could change the policy. But we shouldn't worry too much about it now.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.