[Question] What is the DNS best practice in the OpenYurt edge cluster?

[SataQiu]

At present, we have a variety of DNS resolution implementations:

1. Deploy CoreDNS DaemonSet with hostnetwork，and then configure the Kubelet flag --cluster-dns=127.0.0.1

2. Deploy CoreDNS DaemonSet and YurtHub on all the cluster nodes, then enable CoreDNS service topology awareness by add annotation openyurt.io/topologyKeys=kubernetes.io/hostname into CoreDNS Service.

3. Deploy NodeLocalDNS DaemonSet (nodelocaldns access kube-apiserver directly)

The benefit of options 1 and 3 is that you do not need to deploy YurtHub on cloud nodes.

The benefit of option 2 is CoreDNS Pod can run in the Pod network and the topology awareness is supported on both cloud and edge.

The advantage of option 3 over option 1 is that there is no additional configuration of Kubelet and no need to modify the existing CoreDNS Deployment or DaemonSet in the system.

So what is the DNS best practice in the OpenYurt edge cluster?

[rambohe-ch]

@SataQiu Thank you for raising issue.

I think we can follow the best practice of CoreDNS community. CoreDNS is the up dns for nodeLocalDNS, and nodeLocalDNS is a daemon app for every node.
And in OpenYurt edge cluster, we can use YurtAppDaemon(plan to release in v0.6.0 @kadisi ) to deploy CoreDNS from NodePool level, and use DaemonSet to deploy nodeLocalDNS.

so, yurthub maybe need to deploy on both edge and cloud nodes.

[rambohe-ch]

@SataQiu @Fei-Guo @huangyuqi @gnunu @zzguang @yixingjia @DrmagicE @kadisi And CoreDNS is system component for OpenYurt edge cluster, so i think that end users do not need to maintain CoreDNS deployment, maybe we can add a label for NodePool to enable or disable deployment of CoreDNS like ingress controller. we can call all of these abilities(like ingress controller, CoreDNS) as NodePool governance ability.

[zzguang]

For nodepool level CoreDNS, agree with @rambohe-ch to treat it as a nodepool innate feature and add the related implementation in nodepool operator, end users only need to operate on the nodepool to enable/disable this feature while the deployment is transparent to them, so it will improve the ease of use for this feature.

[SataQiu]

@SataQiu Thank you for raising issue.

I think we can follow the best practice of CoreDNS community. CoreDNS is the up dns for nodeLocalDNS, and nodeLocalDNS is a daemon app for every node. And in OpenYurt edge cluster, we can use YurtAppDaemon(plan to release in v0.6.0 @kadisi ) to deploy CoreDNS from NodePool level, and use DaemonSet to deploy nodeLocalDNS.

so, yurthub maybe need to deploy on both edge and cloud nodes.

OK. So let's deploy YurtHub on both edge and cloud nodes to support service topology awareness.

[DrmagicE]

I think deploying apps is beyond the scope of nodepool controller, which should focus on managing node topology but not the applications on nodes. Any nodepool awareness application can be managed in the upper layer, by YurtAppDaemon, helm or other dedicated operators. To deploy CoreDNS deployment, I think YurtAppDaemon is good enough.

[zzguang]

The key point is whether we treat CoreDNS as an ordinary user app.
IMHO, CoreDNS/ingress controller are system components and their requirements are somehow different with the end user apps. Take ingress controller as an example, it requires to deploy an ingress controller deployment and a service for a nodepool, besides, in order to support validating admission webhook, it also requires to deploy webhook related deployment/service/ValidatingWebhookConfiguration... on cloud side, if we implement it through YurtAppDaemon, YurtAppDaemon needs to support multi deployments/multi services/ValidatingWebhookConfiguration/..., which seems not a generic design from the perspective of YurtAppDaemon.
Therefore, we suggest to separate system components from the end user apps, YurtAppDaemon is only responsible for the end user apps, while the nodepool related system components are treated as nodepool native features and are implemented by enhancing current nodepool operator. Besides, for the users will never care about the deployment of these system components, it improves the ease of use for these features.

[rambohe-ch]

@Fei-Guo Do you have any comments about above discussions?

[gnunu]

Yes, we should differ system components from pure application workload, for the former may involve control plane operation occuring in cloud space, such as installing webhook services. Even worse is that those system components' deployments maybe case by case. Ingress needs services and webhooks, CoreDNS has its own requirements, and so do others like load balancer, etc.

I think this pecularity is "edge native": nodepool is our edge, and not always reachable from cloud (even with yurttunnel), so when adding a nodepool level feature, we want the feature belongs to the nodepool, while being managable from cloud master at the same time, through or not yurthub. From this "edge native" viewpoint, we want most of our feature really run in nodepool and at best form a close loop, i.e., incur south-north communication as less as possible. Sounds we need a general way to deploy such "edge native" service?

As for CoreDNs and Ingress at the nodepool level, definitely they are features of nodepool (edge). So sounds it's a good idea to add a switch for nodepool to enable/disable these features. We also can add a general switch at the whole cluster level to do easy control. If the general switch is off, all nodepool have that feature switched off. While if the general switch is on, nodepool has ability to disable the feature using its own switch.

As for impelemtation, before we have a general way to deploy system level components, sounds it's good idea to reuse/enhance nodepool operator.

[kadisi]

I agree to include ingress in the node pool as a property of the node pool spec，This is an example：

apiVersion: apps.openyurt.io/v1alpha1
kind: NodePool
metadata:
  name: hangzhou
spec:
  type: Edge
  ingress: 
    version: 1.2.3
    enable: true
  annotations:
    apps.openyurt.io/example: test-hangzhou
  labels:
    apps.openyurt.io/example: test-hangzhou
  taints:
  - key: apps.openyurt.io/example
    value: test-hangzhou
    effect: NoSchedule

[rambohe-ch]

@Fei-Guo @gnunu @kadisi

1. YurtAppDaemon should be focus on NodePool level workload, in spite of system components or pure application workloads.
2. Workloads of system components(like ingress-controller) can be deployed by YurtAppDaemon, but configuration(like webhooks) can not be included in YurtAppDaemon, so we need to add an operator(include YurtAppDaemon and webhooks) to deploy some components(like ingress-controller).
3. How to use these operators by end users, through NodePool or YurtCluster need to discuss.

[rambohe-ch]

@Fei-Guo @zzguang @gnunu @SataQiu @kadisi @DrmagicE and many other community members have discussed this issue in OpenYurt community meeting(2021.10.20), and have got the following conclusion:

1. YurtAppDaemon is dedicated to workload deployment for interacting with NodePool. Patch fields are planned for the next release to be used to deploy non-Pod Objects to all NodePools to improve flexibility and extensibility of YurtAppDaemon
2. UnitedDeployment(later renamed YurtAppSet) is recommended for the workload of the NodePool level due to personalized configuration requirements (such as using different ConfigMaps)
3. To keep the Resource API simple and simplify the management of complex components, no fields will be added into NodePool or YurtCluster resource. So ingress-controller will be implemented using a standalone operator. The ability to interact with the NodePool and disclose configurations to users is defined by the operator itself
4. If a component is truly required for NodePool, consider adding field into NodePool to support this feature, but need to discuss case by case.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.