Finding a solution for etcd

[jamiehannaford]

A few weeks a go some folks on Slack brought up the idea of defining requirements for highly available etcd on kubeadm-provisioned clusters. The idea was to define requirements before the implementation of any solution. This discussion was continued in the sig-cluster-lifecycle meeting on May 16th 2017, where we came up with some initial critieria:

1. High availability
   a. Recovers from member failure
   b. Recovers from quorum loss
   c. Recovery from full cluster failure i.e. power-off
   d. Recovers from partial / failed / interrupted upgrades
2. Handles discovery of etcd peers
3. Secure by default
   a. TLS encryption
   b. Certificate rotation
4. Support multiple form factors
   a. Non-self hosted
   b. Self-hosted (optional)
5. Ability to restore from a backup (possibly not backup itself)
6. Upgrades
   a. Rolling upgrades
   b. Downgrades (but tricky because of etcd)
7. Resize/scale cluster from 1 -> 3 -> 5 members
8. Ease of installation/teardown

Are there any I've missed?

The next stage is proposing solutions that meet the above criteria and can be verified in a fork.

cc/ @timothysc @justinsb @philips @xiang90 @aaronlevy

[jamiehannaford]

My proposed solution is the etcd-operator, since it provides nearly all criteria out of the box:

• High availability
  • Recovers from member failure (see here)
  • Recovers from quorum loss (see here)
  • Recovery from full cluster failure i.e. power-off (I think this can be accomplished using snapshotting and the operator's built in restart tolerance)
  • Recovers from partial / failed / interrupted upgrades
• Handles discovery of etcd peers
• Secure by default
  • TLS encryption (see here)
  • Certificate rotation (no, but my personal preference is to use an external tool, like a cert-rotator operator to handle TLS cert rotation across the cluster)
• Support multiple form factors
  • Non-self hosted
  • Self-hosted
• Ability to restore from a backup (see here)
• Upgrades
  • Rolling upgrades (see here)
  • Downgrades (but tricky because of etcd)
• Resize/scale cluster from 1 -> 3 -> 5 members (see here)
• Ease of installation/teardown

I've already integrated kubeadm and etcd-operator successfully in this PR, and here is the fork.

I think it's probably worthwhile to come up with a more granular disaster recovery requirement list, and also think about to what degree an etcd solution should cover all the bases. We already have several issues where this is tracked, so we should take into account all the suggested solutions there too:

• Documentation: Disaster recovery scenarios kubernetes-retired/bootkube#432
• Simple scheduler & controller-manager disaster recovery kubernetes-retired/bootkube#112
• Documentation: enumerate self-hosted etcd operator failure scenarios coreos/tectonic-installer#257

[xiang90]

Recovers from partial / failed / interrupted upgrades

see kubernetes-retired/bootkube#528

[timothysc]

@jamiehannaford if there is a branch you would like reviewed I'd be happy to go through it now.

[jamiehannaford]

@xiang90 Awesome. So if a failed upgrade occurs, the user can manual restore from a backup file. Is there a way that etcd can automatically check specific locations (like local FS, S3) for backups without the user needing to specify one manually?

For example, assume that the etcd-operator has been backing stuff up to a S3 container. When it initialises it checks the same bucket and boots from there (this assume the user hasn't changed any backup options).

[jamiehannaford]

@timothysc Thanks! The only branch I have is the one I submitted in my PR. I think you've already gone through this though. Unless you meant something else?

There are a bunch of comments on that PR which I can start to address as a next step forward. I think I'll also add TLS secrets to the PR too. Should I go ahead and do that?

[luxas]

There are a bunch of comments on that PR which I can start to address as a next step forward. I think I'll also add TLS secrets to the PR too. Should I go ahead and do that?

@jamiehannaford Feel free to. I'm gonna try to look at the TLS Secrets PR this week so it might yet change (@andrewrynhard), but I don't expect it to be part of v1.7 to give us a little more time to think about it until v1.8

[anguslees]

For example, assume that the etcd-operator has been backing stuff up to a S3 container. When it initialises it checks the same bucket and boots from there

I'd just like to highlight that doing something like this automatically is a terrible idea and will give you multiple sources of truth if etcd is internally partitioned.

I suspect recovery will need to be manually triggered, because by definition it is required when the etcd cluster is incapable of making robust automatic decisions.

[xiang90]

I'd just like to highlight that doing something like this automatically is a terrible idea and will give you multiple sources of truth if etcd is internally partitioned.

totally agree. we designed this to be a manual work, at least at etcd operator side.

[luxas]

Moving milestone to v1.9. In v1.8, we're gonna stick with a local etcd instance listening on localhost.

[luxas]

Moving milestone for this to v1.10 as we depend on changes being made to the operator before we can use it and the code freeze is coming up.

[timothysc]

Given all the history here and recent feedback, we need to go with the non-operator option.

[timothysc]

So I'm going to close this issue and open a new one to outline the doc on using the existing commands to lay down etcd, we will likely have to wait until we have some of the other phases work done as well.

/cc @chuckha @fabriziopandini @stealthybox