From 46d4cff19acd0c7f560d3a59c1d96882e03c1562 Mon Sep 17 00:00:00 2001 From: zhenggu1 Date: Thu, 2 Dec 2021 09:30:33 +0800 Subject: [PATCH] ingress: update nodepool ingress proposal Signed-off-by: zhenggu1 --- .../20210628-nodepool-ingress-support.md | 335 +++++++++++++----- 1 file changed, 246 insertions(+), 89 deletions(-) diff --git a/docs/proposals/20210628-nodepool-ingress-support.md b/docs/proposals/20210628-nodepool-ingress-support.md index 89faa1abb9b..10074388ef6 100644 --- a/docs/proposals/20210628-nodepool-ingress-support.md +++ b/docs/proposals/20210628-nodepool-ingress-support.md @@ -8,7 +8,7 @@ reviewers: - "@rambohe-ch" - "@wenjun93" creation-date: 2021-06-28 -last-updated: 2021-06-28 +last-updated: 2021-11-30 status: provisional --- @@ -60,26 +60,54 @@ To enable the ingress feature in OpenYurt NodePool is to fill the gap that curre Ingress controller service can be exposed through cloud provider SLB in Cloud Native environment, but in the Edge environment, no SLB is provided, we can adopt NodePort type service at the current stage, but in future we can evaluate whether to leverage metalLB as a feature enhancement, the reference link is shown below: + https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#a-pure-software-solution-metallb ## Proposal To implement the ingress feature support in NodePool, we need to nail down the solution design firstly. And the design can be separated into 2 parts by our understanding: -1). Ingress controller deployment - It means to deploy one ingress controller to the NodePool which needs to enable the ingress feature, - we can treat it as similar daemonset support from the NodePool level. When the ingress controller is - deployed to a NodePool, one node of the NodePool will be elected to host the deployment. The related - development is working in process by @kadisi and it is supposed to be open sourced soon. - -2). Ingress feature implementation - Once the ingress controller has been deployed to a NodePool, this NodePool has a basic ability to provide - the ingress feature, but how to implement the ingress controller and how to adapt it to NodePool? - Firstly, to ensure the ingress feature can work even in OpenYurt autonomy mode, the ingress controller - should communicate with kube-apiserver through Yurthub to cache the related resource data. - Then, the ingress controller should only monitor the resources of its own NodePool instead of whole cluster. - Here we thought about several solution alternatives and each of them has its pros and cons: - (Note that the "Edge Node" below is the elected node in a NodePool to host the ingress controller) + +### 1). Ingress controller deployment + +It means to deploy one ingress controller to the NodePool which needs to enable the ingress feature, +we can treat it as the similar daemonset support from the NodePool level. When the ingress controller is +deployed to a NodePool, one node of the NodePool will be elected to host the deployment. Here we investigated +several possible solutions to achieve the purpose: + + 1.1). Leverage YurtAppDaemon CRD to deploy ingress controller + + YurtAppDaemon seems can meet the requirement for the ingress controller deployment to specified NodePools, + but finally we found that it's not appropriate for some kind of system components deployment like ingress + controller, because ingress controller has a more complicated requirement than an independent deployment to + a NodePool, it also requires to deploy the related RBAC/ConfigMap/Service/Admission Webhook, which is beyond + the scope of YurtAppDaemon. + + 1.2). Implement the deployment in NodePool operator + + Since 1.1) above is excluded, we thought to implement the NodePool ingress controller deployment in the + NodePool operator, which aims to enhance the capability of NodePool itself, users can operate the NodePool + CR to enable/disable the ingress feature, and it seems to be an acceptable solution, but it introduces at + least 2 problems: one is it couples the NodePool and the components running on it, which is not a graceful + design, the other one is users cannot operate multi NodePools ingress simultaneously, which is not very + friendly to end users. + + 1.3). Implement the deployment in a standalone operator + + To avoid the problems of 1.2) above, after several rounds of community discussions, we decided to implement + the NodePool ingress controller deployments in a standalone operator: YurtIngress, this solution is more + graceful and we achieved an agreement in the community meeting, we are following it for the development now. + Thanks @Fei-Guo for his great suggestions about it. + +### 2). Ingress feature implementation + +Once the ingress controller has been deployed to a NodePool, this NodePool has a basic ability to provide +the ingress feature, but how to implement the ingress controller and how to adapt it to the Edge NodePool? +Firstly, to ensure ingress feature can work even in OpenYurt autonomy mode, the ingress controller should +communicate with kube-apiserver through Yurthub to cache the related resource data locally. +Then, the ingress controller should only care the resources of its own NodePool instead of whole cluster. +Here we thought about several solution alternatives and each of them has its pros and cons: +(Note that the "Edge Node" below is the elected node in a NodePool to host the ingress controller) 2.1). Solution 1: Implement a tailored ingress controller for NodePool ------------------ @@ -97,12 +125,12 @@ And the design can be separated into 2 parts by our understanding: | | controller |------- | | -------------- | --------------------------------- - Pros: Lightweight ingress controller for it is specific for NodePool + Pros: Lightweight ingress controller for it is specific for NodePool. Cons: Big effort for ingress controller needs to monitor and manage all the related resources of - ingress/service/configmap/secret/endpoint in a NodePool by itself + ingress/service/configmap/secret/endpoint/... in a NodePool by itself. 2.2). Solution 2: Leverage and modify current opensource ingress controller to adapt NodePool - Take Nginx ingress controller as an example: + Take nginx ingress controller as an example: ------------------ | kube-apiserver | ------------------ @@ -118,10 +146,10 @@ And the design can be separated into 2 parts by our understanding: | | ingress controller|---- | | --------------------- | ---------------------------------- - Pros: Little effort for it only needs to change some source codes to adapt NodePool - Cons: Intrusive to opensource ingress controller, not convenient to upgrade, it's hard to maintain in future + Pros: Little effort for it only needs to change some source codes to adapt NodePool. + Cons: Intrusive to opensource ingress controller, not convenient to upgrade, it's hard to maintain in future. - 2.3). Solution 3: Leverage current opensource ingress controller and add a sidecar to adapt NodePool + 2.3). Solution 3: Leverage current opensource ingress controller and add an ingress data filter sidecar By evaluating the functionality and maturity of the opensource ingress controllers, we recommend to adopt nginx ingress controller: ------------------ @@ -134,46 +162,71 @@ And the design can be separated into 2 parts by our understanding: |Edge Node ----------- | | | Yurthub | | | ----------- | - | ------------------- ------------- ^ | - | | nginx ingress | | ingress | | | - | | controller |--->| controller|---- | - | ------------------- | sidecar | | - | ------------- | + | ------------------- -------------- ^ | + | | nginx ingress | | ingress | | | + | | controller |--->| datafilter |--- | + | ------------------- | sidecar | | + | -------------- | -------------------------------------------------- - Pros: None-intrusive to opensource ingress controller, convenient to upgrade, easy to maintain in future - Cons: Medium effort for we need to implement an ingress controller sidecar to filter all the ingress related + Pros: None-intrusive to opensource ingress controller, convenient to upgrade, easy to maintain in future. + Cons: Medium effort for we need to implement an ingress data filter sidecar to filter all the ingress related resources of its own NodePool and communicate with nginx ingress controller for resources state update, - besides, for the sidecar will intercept all the traffic to nginx ingress controller, it improves the - development complexity and may lead to some efficiency loss + besides, for the sidecar is specific to ingress controller, it is not quite reasonable. + + 2.4). Solution 4: Leverage current opensource ingress controller and add a common NodePool data filter sidecar + By evaluating the functionality and maturity of the opensource ingress controllers, we recommend + to adopt nginx ingress controller: + ------------------ + | kube-apiserver | + ------------------ + Cloud ^ + --------------------------------------------------|------------ + Edge | + ------------------------------------------|------- + |Edge Node ----------- | + | | Yurthub | | + | ----------- | + | ------------------- -------------- ^ | + | | nginx ingress | | nodepool | | | + | | controller |--->| datafilter |--- | + | ------------------- | sidecar | | + | -------------- | + -------------------------------------------------- + Pros: None-intrusive to opensource ingress controller, convenient to upgrade, easy to maintain in future. + Cons: High effort for we need to implement a common NodePool level data filter sidecar to filter all the ingress + resources of its own NodePool and communicate with nginx ingress controller for the resources state update, + besides, as a common NodePool sidecar, it needs to take all the possible resources into account in future, + which improves the development complexity. - 2.4). Solution 4: Similar to Solution 3, except for it depends on Yurthub to filter the ingress related - resources of its own NodePool, which simplifies the implementation of sidecar + 2.5). Solution 5: Similar to Solution 3 and 4, except for it leverages Yurthub data filter framework by adding + NodePool level ingress related resources filter in the framework, which simplifies the implementation ------------------ | kube-apiserver | ------------------ Cloud ^ - ------------------------------------|-------------------------- + ------------------------------------|------------------- Edge | - ----------------------------|--------------------- - |Edge Node ----------- | - | | Yurthub | | - | ----------- | - | ------------------- ^ ^ ------------- | - | | nginx ingress | | | | ingress | | - | | controller |---- ----| controller| | - | ------------------- | sidecar | | - | ------------- | - -------------------------------------------------- + ----------------------------|---------- + |Edge Node ----------- | + | | Yurthub | | + | ----------- | + | ------------------- ^ | + | | nginx ingress | | | + | | controller |------- | + | ------------------- | + --------------------------------------- Pros: None-intrusive to opensource ingress controller, convenient to upgrade, easy to maintain in future, - high efficiency and less effort than Solution 3 - Cons: Ingress controller is not transparent to Yurthub, Yurthub needs to check requests from ingress controller - and add specific filter operation for it + high efficiency and less effort than Solution 3 and 4. + Cons: Yurthub is actually a Node level instead of NodePool level sidecar, it will lead to redundant logic check + on the Edge Nodes which don't host the ingress controller. + Ingress controller is not transparent to Yurthub, Yurthub needs to check request responses to the ingress + controller and add specific filter operation for it. Conclusion: - By evaluating all the alternatives above, we think Yurthub naturally can take the responsibility to filter the - data between Cloud and Edge, so we prefer to Solution 4 currently, thanks @wenjun93 for his great suggestions - about this proposal, any other suggestions will also be appreciated, if no different opinions, we will follow - Solution 4 to implement the feature. + By evaluating all the alternatives above, we think Yurthub naturally can take the responsibility to filter + the data between Cloud and Edge, so we prefer to Solution 5 currently, thanks @wenjun93 for his great + suggestions about this proposal, any other suggestions will also be appreciated, if no different opinions, + we will follow Solution 5 to implement the feature. ### User Stories @@ -186,45 +239,149 @@ As an end user, I want to balance the traffic into a NodePool with layer-7 load- layer-4 kube-proxy. ### Implementation Details/Notes/Constraints -1). Yurt-app-manager: - - When a NodePool is created, it can be labeled whether to enable the ingress feature, for example: - ```yaml - nodepool.openyurt.io/ingress-enable: "true" - ``` - Whether to enable ingress by default when NodePool created is TBD. - - A new CRD will be defined to provide the daemonset support from NodePool level, so that one ingress controller - can be deployed to the NodePool which needs to enable ingress feature. - - For every ingress controller deployment in different NodePools, it creates a NodePort type service to expose the - ingress controller service, and binds a corresponding configmap to differentiate the NodePools. All the ingress - controller deployments for different NodePools can use one same service account. - - Add 2 NodePool annotations for users to configure the ingress feature when NodePool is created: - ```yaml - nodepool.openyurt.io/ingress-ips: "xxx" # the ingress controller service IP scope - nodepool.openyurt.io/ingress-config: "xxx" # the ingress controller nginx config - ``` - -2). Nginx ingress controller: - - Keep the upstream version none-intrusive, so that it will be very convenient to upgrade in future. - - It specifies the ingress class to NodePool name so the ingress CR can select the corresponding ingress controller. - - It should be granted the authorities to create/update/patch the related resources in kube-system namespace. - - It connects with Yurthub to monitor the resources(ingress/service/configmap/secret/endpoint) of its own NodePool. - -3). Ingress controller sidecar: - - When sidecar is launched, it gets the NodePool id it belongs to, and connects to Yurthub to list/watch the NodePool. - - When it detects the NodePool annotation below is changed, it will update externalIPs of the corresponding service: - ```yaml - nodepool.openyurt.io/ingress-ips: "xxx" - ``` - - When it detects the NodePool annotation below is changed, it will update the corresponding configmap of the NodePool, - which will trigger the ingress controller to reload nginx config for the resource state sync. - ```yaml - nodepool.openyurt.io/ingress-config: "xxx" - ``` +#### 1). Yurt-app-manager + +Implement a standalone `YurtIngress` operator to support the NodePool ingress feature, which means that `YurtIngress` +together with `NodePool/YurtAppSet(UnitedDeployment)/YurtAppDaemon` will compose the the current OpenYurt unitization +capability, and they are all managed by Yurt-app-manager. The responsibility of `YurtIngress` is to monitor the update +of `YurtIngress` CR and trigger the corresponding operations to nginx ingress controller related components. + +`YurtIngress` CRD definition: + +```go +type IngressNotReadyType string + +const ( + IngressPending IngressNotReadyType = "Pending" + IngressFailure IngressNotReadyType = "Failure" +) + +// IngressPool defines the details of a Pool for ingress +type IngressPool struct { + // Indicates the pool name. + Name string `json:"name"` + + // Pool specific configuration will be supported in future. +} + +// IngressNotReadyPool defines the condition details of an ingress not ready Pool +type IngressNotReadyPool struct { + // Indicates the pool name. + Name string `json:"name"` + + // Type of ingress not ready condition. + Type IngressNotReadyType `json:"type,omitempty"` + + // Last time the condition transitioned from one status to another. + LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"` + + // The reason for the condition's last transition. + Reason string `json:"reason,omitempty"` + + // A human readable message indicating details about the transition. + Message string `json:"message,omitempty"` +} + +// YurtIngressSpec defines the desired state of YurtIngress +type YurtIngressSpec struct { + // Indicates the number of the ingress controllers to be deployed under all the specified nodepools. + // +optional + Replicas int32 `json:"ingress_controller_replicas_per_pool,omitempty"` + + // Indicates all the nodepools on which to enable ingress. + // +optional + Pools []IngressPool `json:"pools,omitempty"` +} + +// YurtIngressCondition describes current state of a YurtIngress +type YurtIngressCondition struct { + // Indicates the pools that ingress controller is deployed successfully. + IngressReadyPools []string `json:"readypools,omitempty"` + + // Indicates the pools that ingress controller is being deployed or deployed failed. + IngressNotReadyPools []IngressNotReadyPool `json:"unreadypools,omitempty"` +} + +// YurtIngressStatus defines the observed state of YurtIngress +type YurtIngressStatus struct { + // Indicates the number of the ingress controllers deployed under all the specified nodepools. + // +optional + Replicas int32 `json:"ingress_controller_replicas_per_pool,omitempty"` + + // Indicates all the nodepools conditions on which to enable ingress. + // +optional + Conditions YurtIngressCondition `json:"conditions,omitempty"` + + // Indicates the nginx ingress controller version deployed under all the specified nodepools. + // +optional + Version string `json:"nginx_ingress_controller_version,omitempty"` + + // Total number of ready pools on which ingress is enabled. + // +optional + ReadyNum int32 `json:"readyNum"` + + // Total number of unready pools on which ingress is enabling or enable failed. + // +optional + UnreadyNum int32 `json:"unreadyNum"` +} +``` +Some other details about the design: +- The `YurtIngress` CR is a cluster level instead of namespace level resource. +- The `YurtIngress` CR should be singleton CR which means there should be only one instance in the cluster, + otherwise, it may lead to conflict if users create different `YurtIngress` CR for same NodePools. We can implement + the related admission webhook to control it. +- In Cloud Edge usage scenarios, admission webhook should be deployed on the Cloud side for Edge network limitation, + so when `YurtIngress Controller` tries to deploy nginx ingress controller, the upstream nginx ingress controller + will be divided into 2 parts: one part is deployed on the Cloud side focusing on admission webhook, the other part + is deployed to NodePool focusing on service proxy and load balancer. These 2 parts can work collaboratively without + any conflict by setting the different startup parameters. +- A specified `ingress-nginx` namespace will be created when a NodePool ingress is enabled, then all the nginx ingress + controller related components for different NodePools will be deployed under this namespace, it must ensure all the + resources for all the NodePools in this namespace not have conflict with each other. +- For every ingress controller deployment in the specified NodePools, it exposes a NodePort type service for users + to access the nginx ingress controller service. +- The ingress class is specified to be NodePool name so users can differentiate the NodePool ingress by the class. +- Nginx ingress controller will connect to kube-apiserver through Yurthub to leverage its data cache and data filter + capability. + +#### 2). Yurthub + +We will leverage Yurthub data cache and data filter framework for the purposes below: +- Yurthub data cache to cache the ingress related resources data locally to ensure ingress can work in autonomy mode. +- Add ingress controller data filter implementation in Yurthub to filter the endpoints of its own NodePool: + +```go +if nodePoolName, ok := currentNode.Labels[nodepoolv1alpha1.LabelCurrentNodePool]; ok { + nodePool, err := fh.nodePoolLister.Get(nodePoolName) + if err != nil { + return endpoints + } + isNodePoolValidEps := false + var newEpsA []v1.EndpointAddress + for i := range endpoints.Subsets { + for j := range endpoints.Subsets[i].Addresses { + nodeName := endpoints.Subsets[i].Addresses[j].NodeName + if nodeName == nil { + continue + } + if inSameNodePool(*nodeName, nodePool.Status.Nodes) { + isNodePoolValidEps = true + newEpsA = append(newEpsA, endpoints.Subsets[i].Addresses[j]) + } + } + endpoints.Subsets[i].Addresses = newEpsA + newEpsA = nil + } + if !isNodePoolValidEps { + return nil + } +} +``` ## Implementation History -- [ ] MM/DD/YYYY: Proposed idea in an issue or [community meeting] -- [ ] MM/DD/YYYY: Compile a Google Doc following the CAEP template (link here) -- [ ] MM/DD/YYYY: First round of feedback from community -- [ ] MM/DD/YYYY: Present proposal at a [community meeting] -- [ ] MM/DD/YYYY: Open proposal PR +- [ ] 07/21/2021: Proposed idea at community meeting and collect feedbacks +- [ ] 07/21/2021: First round of feedback from community +- [ ] 09/15/2021: Second round of feedback from community +- [ ] 10/20/2021: Third round of feedback from community +- [ ] 12/08/2021: Present proposal at a community meeting