Control Plane tracking issue

[chuckha]

KubeadmControlPlane implementation

Status: work is blocked on the merging of #1613

Foundational patterns

These tasks put down patterns in order to distribute the remaining work. These will happen more or less serially

• [@detiber] Merge the new types: ✨ Add KubeadmControlPlane types / validation webhooks #1765
  • KubeadmControlPlaneSpec
  • KubeadmControlPlaneStatus
  • KubeadmControlPlane
  • KubeadmControlPlaneList
  • Documentation for new types
• Merge webhook skeleton files (including documentation) for:
  • validation
  • defaulting
• Merge skeleton of the reconciler (including documentation skeleton): ✨ Start adding KubeadmControlPlane controller #1826

Follow-up work

These will already have patterns in place and can happen in one or more PRs from one or more authors, generally at any point. There may be exceptions so please check if you are unsure.

Validations & defaulting

• Document validation and defaulting webhooks including how to set up, requirements and customization if applicable.

• If KubeadmControlPlane.Spec.KubeadmConfigSpec does not define external etcd (webhook):

  • KubeadmControlPlane.Spec.Replicas is an odd number.
  • Configuration of external etcd is determined by introspecting the provided KubeadmConfigSpec.

• KubeadmControlPlane.Spec.Replicas is >= 0 or is nil (this is covered because we have a scale subresource - Kubernetes validates this for us)

• KubeadmControlPlane.Spec.Version != "" (openapi)

• KubeadmControlPlane.Spec.KubeadmConfigSpec must be treated as immutable (via webhook)

• KubeadmControlPlane.Spec.Replicas: 1

Modifications

• Document changes to types (especially breaking changes)
• Add ControlPlaneRef and ControlPlaneReady to the ClusterSpec and ClusterStatus types respectively
• Update Cluster reconciler with new behaviors:
  • If Cluster.Spec.ControlPlaneRef is set:
    • Status.ControlPlaneInitialized is set based on the value of Status.Initialized for the referenced resource.
    • Status.ControlPlaneReady is set based on the value of Status.Ready for the referenced resource, this field is intended to eventually replace Status.ControlPlaneInitialized as a field that will be kept up to date instead of set only once.
  • Current behavior will be preserved if Cluster.Spec.ControlPlaneRef is not set.
  • CA certificate secrets that were previously generated by the Kubeadm bootstrapper will now be generated by the KubeadmControlPlane Controller, maintaining backwards compatibility with the previous behavior if the KubeadmControlPlane is not used.
  • The kubeconfig secret that was previously created by the Cluster Controller will now be generated by the KubeadmControlPlane Controller, maintaining backwards compatibility with the previous behavior if the KubeadmControlPlane is not used.
• Add FailureDomain implementation to the controller. See RFE: CAPI failure domain & control plane support #1647 for more information

Main implementation

• Document the reconciler to answer general questions such as "what is it", "how does it work", "how do i use it" (issue Add specification document for control plane provider #1727)
• Create
• Delete
• Scale up
• Scale down
• Upgrade
• Healthcheck
• Document the external contract for interoperability with alternative Control Plane implementations.
• Add support for the KubeadmControlPlane provider to the e2e test framework

[vincepri]

/milestone v0.3.0
/area api
/area control-plane
/priority important-soon

[chuckha]

Please open new issues if more detail is required on any particular task. Maintainers should be able to edit this issue if desirable. If not, please close the issue when it is sufficiently out of date.

[randomvariable]

Can't edit the thing, but

Document the reconciler to answer general questions such as "what is it", "how does it work", "how do i use it"

is being treated as #1727 unless there's objections.

[ncdc]

@randomvariable I updated it above

[detiber]

If anyone wants to volunteer to help out for something, please leave a comment before starting work to let others know that work is inflight for it already.

Current work in progress:

• @randomvariable is currently working on the control plane provider documentation
• @detiber is currently working on adding testing for the create workflow for the controller done

[dlipovetsky]

@detiber I can take up delete

[randomvariable]

Just going to note some TODO's here unless someone's got a better idea:

• conversion-webhook "msg"="failed to convert" "error"="cannot convert KubeadmConfigStatus's bootstrap data from Secret reference to inline field" - Turned out this was weave flux querying objects at all versions.
• Support minimal kubeadmConfigSpecs (for Docker provider) - lack of clusterConfiguration causes NPE

[randomvariable]

Updated WIP:

• @randomvariable - Control plane healthchecks (in concert with Docker provider for testing).

[chuckha]

I'm going to wire up the e2es with KubeadmControlPlane objects

[detiber]

I'm going to start working on scale up This is done

[detiber]

@dlipovetsky are you still planning on working on delete?

[dlipovetsky]

@detiber I was out over the holidays. Yes, working on it!

[randomvariable]

Another task:
As per https://kubernetes.slack.com/archives/C8TSNPY4T/p1578398060157200, ensure ownerreferences are set on downstream objects

[vincepri]

Seems we're already setting controller owner references:

• https://github.com/kubernetes-sigs/cluster-api/blob/master/controlplane/kubeadm/controllers/kubeadm_control_plane_controller.go#L341
• https://github.com/kubernetes-sigs/cluster-api/blob/master/controlplane/kubeadm/controllers/kubeadm_control_plane_controller.go#L412
• https://github.com/kubernetes-sigs/cluster-api/blob/master/controlplane/kubeadm/controllers/kubeadm_control_plane_controller.go#L448
• https://github.com/kubernetes-sigs/cluster-api/blob/master/controlplane/kubeadm/controllers/kubeadm_control_plane_controller.go#L499

And we rely on the Cluster controller to set an OwnerReference on KubeadmControlPlane objects, so when the Cluster is deleted, everything else should go as well.

[chuckha]

#2015 This is on hold until delete is implemented. I'll start working on scale down or investigate the issue with parallel vs serial scale up #2016

[ncdc]

/assign @randomvariable @detiber
/lifecycle active

[chuckha]

I'm going to pick up documentation here. I don't expect it to be too verbose yet as things are still rapidly changing, but at least having an example and a high level outline should help somewhat.

[chuckha]

Actually I'm going to pick up implementing the Failure Domain business in this controller

[dlipovetsky]

Now that delete is in, I'm going to pick up scale down. While etcd healthchecks are being finished, I will work around them in some way--maybe stub them out.

[chuckha]

#2241

Scale up / Scale down

[chuckha]

#2242

Upgrades

[chuckha]

#2243

Health checks

[chuckha]

#2244

Documentation

And with that, we can close this issue

/close

[k8s-ci-robot]

@chuckha: Closing this issue.

In response to this:

#2244

Documentation

And with that, we can close this issue

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.