From c904d5fbf5db878bcde3180602a3e797a88f9acb Mon Sep 17 00:00:00 2001 From: Jack Francis Date: Mon, 10 Apr 2023 12:18:12 -0700 Subject: [PATCH] CAEP: Make Cluster Infra Resource Optional --- docs/proposals/20220725-managed-kubernetes.md | 23 +- ...0230407-optional-cluster-infra-resource.md | 207 ++++++++++++++++++ 2 files changed, 221 insertions(+), 9 deletions(-) create mode 100644 docs/proposals/20230407-optional-cluster-infra-resource.md diff --git a/docs/proposals/20220725-managed-kubernetes.md b/docs/proposals/20220725-managed-kubernetes.md index ef4226f9accb..e547059175e0 100644 --- a/docs/proposals/20220725-managed-kubernetes.md +++ b/docs/proposals/20220725-managed-kubernetes.md @@ -65,13 +65,13 @@ superseded-by: - [Add-ons management](#add-ons-management) - [Upgrade Strategy](#upgrade-strategy) - [Implementation History](#implementation-history) - + ## Glossary - **Managed Kubernetes** - a Kubernetes service offered/hosted by a service provider where the control plane is run & managed by the service provider. As a cluster service consumer, you don’t have to worry about managing/operating the control plane machines. Additionally, the managed Kubernetes service may extend to cover running managed worker nodes. Examples are EKS in AWS and AKS in Azure. This is different from a traditional implementation in Cluster API, where the control plane and worker nodes are deployed and managed by the cluster admin. - **Unmanaged Kubernetes** - a Kubernetes cluster where a cluster admin is responsible for provisioning and operating the control plane and worker nodes. In Cluster API this traditionally means a Kubeadm bootstrapped cluster on infrastructure machines (virtual or physical). - **Managed Worker Node** - an individual Kubernetes worker node where the underlying compute (vm or bare-metal) is provisioned and managed by the service provider. This usually includes the joining of the newly provisioned node into a Managed Kubernetes cluster. The lifecycle is normally controlled via a higher level construct such as a Managed Node Group. -- **Managed Node Group** - is a service that a service provider offers that automates the provisioning of managed worker nodes. Depending on the service provider this group of nodes could contain a fixed number of replicas or it might contain a dynamic pool of replicas that auto-scales up and down. Examples are Node Pools in GCP and EKS managed node groups. +- **Managed Node Group** - is a service that a service provider offers that automates the provisioning of managed worker nodes. Depending on the service provider this group of nodes could contain a fixed number of replicas or it might contain a dynamic pool of replicas that auto-scales up and down. Examples are Node Pools in GCP and EKS managed node groups. - **Cluster Infrastructure Provider (Infrastructure)** - an Infrastructure provider supplies whatever prerequisites are necessary for creating & running clusters such as networking, load balancers, firewall rules, and so on. ([docs](../book/src/developer/providers/cluster-infrastructure.md)) - **ControlPlane Provider (ControlPlane)** - a control plane provider instantiates a Kubernetes control plane consisting of k8s control plane components such as kube-apiserver, etcd, kube-scheduler and kube-controller-manager. ([docs](../book/src/developer/architecture/controllers/control-plane.md#control-plane-provider)) - **MachineDeployment** - a MachineDeployment orchestrates deployments over a fleet of MachineSets, which is an immutable abstraction over Machines. ([docs](../book/src/developer/architecture/controllers/machine-deployment.md)) @@ -87,7 +87,9 @@ Cluster API was originally designed with unmanaged Kubernetes clusters in mind a Some Cluster API Providers (i.e. Azure with AKS first and then AWS with EKS) have implemented support for their managed Kubernetes services. These implementations have followed the existing documentation & contracts (that were designed for unmanaged Kubernetes) and have ended up with 2 different implementations. -While working on supporting ClusterClass for EKS in Cluster API Provider AWS (CAPA), it was discovered that the current implementation of EKS within CAPA, where a single resource kind (AWSManagedControlPlane) is used for both ControlPlane and Infrastructure, is incompatible with ClusterClass (See the [issue](https://github.com/kubernetes-sigs/cluster-api/issues/6126)). Separation of ControlPlane and Infrastructure is expected for the ClusterClass implementation to work correctly. +> _While working on supporting ClusterClass for EKS in Cluster API Provider AWS (CAPA), it was discovered that the current implementation of EKS within CAPA, where a single resource kind (AWSManagedControlPlane) is used for both ControlPlane and Infrastructure, is incompatible with ClusterClass (See the [issue](https://github.com/kubernetes-sigs/cluster-api/issues/6126)). Separation of ControlPlane and Infrastructure is expected for the ClusterClass implementation to work correctly._ + +(Note: the above quoted, italicized text matter is no longer relevant once CAEP https://github.com/kubernetes-sigs/cluster-api/pull/8500 is implemented.) The responsibilities between the CAPI control plane and infrastructure are blurred with a managed Kubernetes service like AKS or EKS. For example, when you create a EKS control plane in AWS it also creates infrastructure that CAPI would traditionally view as the responsibility of the cluster “infrastructure provider”. @@ -235,7 +237,7 @@ type GCPManagedControlPlaneSpec struct { // +optional Network NetworkSpec `json:"network"` - // AddonsConfig defines the addons to enable with the GKE cluster. + // AddonsConfig defines the addons to enable with the GKE cluster. // +optional AddonsConfig *AddonsConfig `json:"addonsConfig,omitempty"` @@ -262,7 +264,9 @@ CAPA decided to represent an EKS cluster as a CAPI control-plane. This meant tha Initially CAPA had an infrastructure cluster kind that reported back the control plane endpoint. This required less than ideal code in its controller to watch the control plane and use its value of the control plane endpoint. -As the infrastructure cluster kind only acted as a passthrough (to satisfy the contract with CAPI) it was decided that it would be removed and the control-plane kind (AWSManagedControlPlane) could be used to satisfy both the “infrastructure” and “control-plane” contracts. This worked well until ClusterClass arrived with its expectation that the “infrastructure” and “control-plane” are 2 different resource kinds. +As the infrastructure cluster kind only acted as a passthrough (to satisfy the contract with CAPI) it was decided that it would be removed and the control-plane kind (AWSManagedControlPlane) could be used to satisfy both the “infrastructure” and “control-plane” contracts. _This worked well until ClusterClass arrived with its expectation that the “infrastructure” and “control-plane” are 2 different resource kinds._ + +(Note: the above italicized text matter is no longer relevant once CAEP https://github.com/kubernetes-sigs/cluster-api/pull/8500 merges is implemented.) Note that CAPZ had a similar discussion and an [issue](https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/1396) to remove AzureManagedCluster: AzureManagedCluster is useless; let's remove it (and keep AzureManagedControlPlane) @@ -273,6 +277,7 @@ Note that CAPZ had a similar discussion and an [issue](https://github.com/kubern **Cons** - Doesn’t work with the current implementation of ClusterClass, which expects a separation of ControlPlane and Infrastructure. + - when CAEP https://github.com/kubernetes-sigs/cluster-api/pull/8500 is implemented this con will no longer be true - Doesn’t provide separation of responsibilities between creating the general cloud infrastructure for the cluster and the actual cluster control plane. - Managed Kubernetes look different from unmanaged Kubernetes where two separate kinds are used for a control plane and infrastructure. This would impact products building on top of CAPI. @@ -331,6 +336,8 @@ type GCPManagedClusterSpec struct { - Need to maintain Infra cluster kind, which is a pass-through layer and has no other functions. In addition to the CRD, controllers, webhooks and conversions webhooks need to be maintained. - Infra provider doesn’t provision infrastructure and whilst it may meet the CAPI contract, it doesn’t actually create infrastructure as this is done via the control plane. +Note: when CAEP https://github.com/kubernetes-sigs/cluster-api/pull/8500 is implemented this option will no longer be relevant, as we can simply drop the InfraCluster altogether. + #### Option 3: Two kinds with a Managed Control Plane and Managed Infra Cluster with Better Separation of Responsibilities This option more closely follows the original separation of concerns with the different CAPI provider types. With this option, 2 new resource kinds will be introduced: @@ -435,11 +442,9 @@ The reasons for this recommendation are as follows: - The infra cluster provisions and manages the general infrastructure required for the cluster but not the control plane. - By having a separate infra cluster API definition, it allows differences in the API between managed and unmanaged clusters. -Providers like CAPZ and CAPA have already implemented managed Kubernetes support and there should be no requirement on them to move to Option 3. Both Options 2 and 4 are solutions that would work with ClusterClass and so could be used if required. - -Option 1 is the only option that will not work with ClusterClass and would require a change to CAPI. Therefore this option is not recommended. +Providers like CAPZ and CAPA have already implemented managed Kubernetes support and there should be no requirement on them to move to Option 3. Both Options 4 also works well with ClusterClass and so could be used if required. Option 2 will no longer be relevant once CAEP https://github.com/kubernetes-sigs/cluster-api/pull/8500 is implemented. -*** This means that CAPA will have to make changes to move away from Option 1 if it wants to support ClusterClass. +Option 1 will be available when CAEP https://github.com/kubernetes-sigs/cluster-api/pull/8500 is implemented. Until then it is not recommended. ### Additional notes on option 3 diff --git a/docs/proposals/20230407-optional-cluster-infra-resource.md b/docs/proposals/20230407-optional-cluster-infra-resource.md new file mode 100644 index 000000000000..b81f02670e39 --- /dev/null +++ b/docs/proposals/20230407-optional-cluster-infra-resource.md @@ -0,0 +1,207 @@ +--- +title: Make Cluster Infra Resource Optional +authors: + - "@jackfrancis" +reviewers: + - "@richardcase" + - "@pydctw" + - "@mtougeron" + - “@CecileRobertMichon” + - “@fabriziopandini” + - “@sbueringer” + - "@killianmuldoon" + - "@mboersma" + - "@nojnhuh" +creation-date: 2023-04-07 +last-updated: 2023-04-07 +status: provisional +see-also: + - "/docs/proposals/20220725-managed-kubernetes.md" +--- + +# Make Cluster Infra Resource Optional + +## Table of Contents + +A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting +any additional information provided beyond the standard proposal template. +[Tools for generating](https://github.com/ekalinin/github-markdown-toc) a table of contents from markdown are available. + +- [Make Cluster Infra Resource Optional](#make-cluster-infra-resource-optional) + - [Table of Contents](#table-of-contents) + - [Glossary](#glossary) + - [Summary](#summary) + - [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) + - [Future work](#future-work) + - [Proposal](#proposal) + - [User Stories](#user-stories) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Requirements (Optional)](#requirements-optional) + - [Functional Requirements](#functional-requirements) + - [FR1](#fr1) + - [FR2](#fr2) + - [Non-Functional Requirements](#non-functional-requirements) + - [NFR1](#nfr1) + - [NFR2](#nfr2) + - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) + - [Security Model](#security-model) + - [Risks and Mitigations](#risks-and-mitigations) + - [Alternatives](#alternatives) + - [Upgrade Strategy](#upgrade-strategy) + - [Additional Details](#additional-details) + - [Test Plan [optional]](#test-plan-optional) + - [Graduation Criteria [optional]](#graduation-criteria-optional) + - [Version Skew Strategy [optional]](#version-skew-strategy-optional) + - [Implementation History](#implementation-history) + +## Glossary + +Refer to the [Cluster API Book Glossary](https://cluster-api.sigs.k8s.io/reference/glossary.html). + +The following terms will be used in this document. + +- `Cluster` + - When we say `Cluster` we refer to any provider's infra-specific implementation of the Cluster API `Cluster` resource spec. When you see ``, interpret that as a placeholder for any provider implementation. Some concrete examples of provider infra cluster implementations are Azure's CAPZ provider (e.g., `AzureCluster` and `AzureManagedCluster`), AWS's CAPA provider (e.g., `AWSCluster` and `AWSManagedCluster`), and Google Cloud's CAPG provider (e.g., `GCPCluster` and `GCPManagedCluster`). Rather than referencing any one of the preceding actual implementations of infra cluster resources, we prefer to generalize to `Cluster` so that we don't suggest any provider-specific bias informing our conclusions. +- Managed Kubernetes + - Managed Kubernetes refers to any Kubernetes Cluster provisioning and maintenance platform that is exposed by a service API. For example: [EKS](https://aws.amazon.com/eks/), [OKE](https://www.oracle.com/cloud/cloud-native/container-engine-kubernetes/), [AKS](https://azure.microsoft.com/en-us/products/kubernetes-service), [GKE](https://cloud.google.com/kubernetes-engine), [IBM Cloud Kubernetes Service](https://www.ibm.com/cloud/kubernetes-service), [DOKS](https://www.digitalocean.com/products/kubernetes), and many more throughout the Kubernetes Cloud Native ecosystem. +- _Kubernetes Cluster Infrastructure_ + - When we refer to _Kubernetes Cluster Infrastructure_ we aim to distinguish required environmental infrastructure (e.g., cloud virtual networks) in which a Kubernetes cluster resides as a "set of child resources" from the Kubernetes cluster resources themselves (e.g., virtual machines that underlie nodes, managed by Cluster API). Sometimes this is referred to as "BYO Infrastructure"; essentially, we are talking about **infrastructure that supports a Kubernetes cluster, but is not actively managed by Cluster API**. As we will see, this boundary is different when discussing Managed Kubernetes: more infrastructure resources are not managed by Cluster API when running Managed Kubernetes. +- e.g. + - This just means "For example:"! + +## Summary + +We propose to make provider `Cluster` resources optional in order to better represent Managed Kubernetes scenarios where all _Kubernetes Cluster Infrastructure_ is managed by the service provider, and not by Cluster API. + +## Motivation + +The implementation of Managed Kubernetes scenarios by Cluster API providers occurred after the architectural design of Cluster API, and thus that design process did not consider these Managed Kubernetes scenarios as a user story. In practice, Cluster API's specification has allowed Managed Kubernetes solutions to emerge that aid running fleets of clusters at scale, with CAPA's `AWSManagedCluster` and `AzureManagedCluster` being notable examples. However, because these Managed Kubernetes solutions arrived after the Cluster API contract was defined, providers have not settled on a consistent rendering of how a "Service-Managed Kubernetes" specification fits into a "Cluster API-Managed Kubernetes" surface area. + +One particular part of the existing Cluster API surface area that is inconsistent with most Managed Kubernetes user experiences is the accounting of the [Kubernetes API server](https://kubernetes.io/docs/concepts/overview/components/#kube-apiserver). In the canonical "self-managed" user story that Cluster API addresses, it is the provider implementation of Cluster API (e.g., CAPA) that is responsible for scaffolding the necessary _Kubernetes Cluster Infrastructure_ that is required in order to create the Kubernetes API server (e.g., a Load Balancer and a public IP address). This provider responsibility is declared in the `Cluster` resource, and carried out via its controllers; and then finally this reconciliation is synchronized with the parent `Cluster` Cluster API resource. + +Because there exist Managed Kubernetes scenarios that handle all _Kubernetes Cluster Infrastructure_ responsibilities themselves, Cluster API's requirement of a `Cluster` resource leads to weird implementation decisions, because in these scenarios there is no actual work for a Cluster API provider to do to scaffold _Kubernetes Cluster Infrastructure_. + +### Goals + +- Make `Cluster` resources optional. +- Enable API Server endpoint reporting from a provider's Control Plane resource rather than from its `Cluster` resource. +- Ensure any changes to the current behavioral contract are backwards-compatible. + +### Non-Goals + +- Change the Cluster API data type specification. +- Introduce new "Managed Kubernetes" data types in Cluster API. + +### Future Work + +- Detailed documentation that references the flavors of Managed Kubernetes scenarios and how they can be implemented in Cluster API, with provider examples. + +## Proposal + +### User Stories + +- Detail the things that people will be able to do if this proposal is implemented. +- Include as much detail as possible so that people can understand the "how" of the system. +- The goal here is to make this feel real for users without getting bogged down. + +#### Story 1 + +As a cluster operator, I want to use Cluster API to provision and manage the lifecycle of a control plane that utilizes my service provider's managed Kubernetes control plane (i.e. EKS, AKS, GKE), so that I don’t have to worry about the management/provisioning of control plane nodes, and so I can take advantage of any value add services offered by my cloud provider. + +#### Story 2 + +As a cluster operator, I want to be able to provision both "unmanaged" and "managed" Kubernetes clusters from the same management cluster, so that I can support different requirements and use cases as needed whilst using a single operating model. + +#### Story 3 + +As a Cluster API provider developer, I want guidance on how to incorporate a managed Kubernetes service into my provider, so that its usage is compatible with Cluster API architecture/features and its usage is consistant with other providers. + +#### Story 4 + +As a Cluster API provider developer, I want to enable the ClusterClass feature for a Managed Kubernetes service, so that users can take advantage of an improved UX with ClusterClass-based clusters. + +#### Story 5 + +As a cluster operator, I want to use Cluster API to provision and manage the lifecycle of worker nodes that utilizes my cloud providers' managed instances (if they support them), so that I don't have to worry about the management of these instances. + +#### Story 6 + +As a service provider I want to be able to offer Managed Kubernetes clusters by using CAPI referencing my own managed control plane implementation that satisfies Cluster API contracts. + +### Requirements (Optional) + +Some authors may wish to use requirements in addition to user stories. +Technical requirements should derived from user stories, and provide a trace from +use case to design, implementation and test case. Requirements can be prioritised +using the MoSCoW (MUST, SHOULD, COULD, WON'T) criteria. + +The FR and NFR notation is intended to be used as cross-references across a CAEP. + +The difference between goals and requirements is that between an executive summary +and the body of a document. Each requirement should be in support of a goal, +but narrowly scoped in a way that is verifiable or ideally - testable. + +#### Functional Requirements + +TODO + +##### FR1 + +TODO + +##### FR2 + +TODO + +#### Non-Functional Requirements + +TODO + +##### NFR1 + +TODO + +##### NFR2 + +TODO + +### Implementation Details/Notes/Constraints + +- TODO + +### Security Model + +TODO + +### Risks and Mitigations + +- TODO + +## Alternatives + +TODO + +## Upgrade Strategy + +TODO + +## Additional Details + +### Test Plan [optional] + +TODO + +### Graduation Criteria [optional] + +TODO + +### Version Skew Strategy [optional] + +TODO + +## Implementation History + +- [ ] 01/11/2023: Compile a Google Doc to organize thoughts prior to CAEP (link here)[https://docs.google.com/document/d/1rqzZfsO6k_RmOHUxx47cALSr_6SeTG89e9C44-oHHdQ/]