[feature] CA Root rotation

[cyli]

The more detailed design document is here: https://docs.google.com/document/d/1ccKB78d0vZbRnN6odZs_VocSDZQtOcU9JmJUzCV1f1I/ was

Background

Currently, the swarm root CA certificate can never be changed. This may cause issues in the following cases:

• A manager is compromised - managers have access to all the data in the cluster. While the secret data in that cluster should be rotated, it’d be helpful to maintain existing data such as service specs, networks, etc. after auditing, without having to replicate all that work. The root CA key material is the last piece of secure information that cannot be rotated.
• Related to a manager getting compromised - if the raft logs and/or snapshots which contain the root key material is ever leaked.
• Getting the root CA cert into the swarm cluster to use with an external CA requires that the CA cert be put in the correct directory before docker swarm init is run. Not being able to change the root CA means that one can never convert from an internal CA to an external CA or back.

Providing a way to rotate the root CA material would allow users to mitigate a manager or raft data compromise, and would allow users to be able to switch over to an external CA or back from an external CA.

Root rotation should not be a common occurrence.

Basic design

A common way to introduce a new root of trust is to use cross-signed certs (for example, https://letsencrypt.org/certificates/), whereby you have two certificates, each signed by a different entity, and each having the same public key and subject.

If we provide a cross-signed intermediate as part of our root rotation process, then we can distribute the new root bundle and renew the TLS certificates at the same time, so there are only 2 phases of root rotation:

• All nodes get new TLS certificates signed with the new root and also including the cross-signed root with the TLS certificates. We may also distribute root bundles containing both the old and new root cert.
• Once ALL the nodes have TLS certificates signed by the new root key, we can finish the rotation and trust only the new root cert on all nodes.

Logic

1. User updates the Root CA material, providing a new public certificate, a new private key or external CA, or both of those.
2. Swarm generates a cross-signed root that has the same public key and subject as the new root certificate, and will provide this as an intermediate certificate when generating new TLS certificates until the root rotation is complete.
3. Swarm starts propagating the old+new root CA as trusted roots, and all the nodes get new TLS certificates
4. User (or swarm) notices that all nodes have new TLS certificates, and tells all nodes to stop trusting the old certificate.

TODO items:

• When a security config is updated, all credentials that make use of it should also be updated (ca: when the Root CA is updated in the security config, update all TLS creds too #1983)
• All managers should automatically update the root CA when it changes in the raft store (ca: all managers will update their security configs when cluster root CA updated #1984)([ca][api]: Root rotation object in cluster #2037)([ca] Separate RootCA signing cert from root certs #2038)
• Add a root rotation object in the cluster object in the raft store ([ca][api]: Root rotation object in cluster #2037)
• All managers and nodes should get new TLS certificates when there is a root rotation ([ca] Root rotation reconciliation loop in the CA server #2100)
• Support intermediates in the swarm TLS logic ([ca] Add support for verifying and loading certificate chains which include intermediates #2000) (ca: Add support for appending intermediates to issued TLS certs #2027)([ca]: Add RequestAndSaveNewCertificates test with intermediates #2071)
• Add support for generating cross-signed certificates both on the local root signer and on the external CA server ([ca] Add utility to create cross-signed roots to the RootCA. #2001)
• Add control API to start a root rotation, accepting a new certificate, a new key/external URL, both, or none. Once these validate, also generate a cross-signed certificate for the root rotation. ([ca] Make the control API server take a security config instead of RootCA #2051)([controlapi] Allow starting and aborting root CA rotations during swarm update API #2052)
• Keep track of the last observed certificate used by any particular node ([agent/dispatcher] Keep track of a node's TLS info and provide root CA changes for an agent #2077)([ca] Propagate issuer info from TLS certificates to SecurityConfig #2079)
• Automatically complete a root rotation (no longer trusting the old certificate) when all nodes have new certificates ([ca] Root rotation reconciliation loop in the CA server #2100)
• CLI support for rotating to a new certificate (WIP: [cli] Interactive root rotation #2104) in docker (Add the CACert parameter to the ExternalCA object moby#32828) (Propagate TLS Info in swarm info and node info REST endpoints moby#32875) (API changes to rotate swarm root CA moby#32993)

Other bugfixes: (#2134) (#2163)(#2210) (#2212) (#2218).

[cyli]

Closing this out as the basic feature is in.