webhook: automatically insert the rdma resource to pod by DRA claim #3670

cyclinder · 2024-06-26T11:20:11Z

Thanks for contributing!

What type of PR is this?

release/feature

What this PR does / why we need it:

webhook: auto inject the rdma resource to pod.

pod.resourceClaims -> spiderclaimparameters.staticNics -> spidermultusconfig -> resourceName -> pod.resource.requests.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

codecov · 2024-06-26T11:24:43Z

Codecov Report

Attention: Patch coverage is 0% with 103 lines in your changes missing coverage. Please review.

Project coverage is 79.30%. Comparing base (5ff6c82) to head (7663fa2).

❗ Current head 7663fa2 differs from pull request most recent head 7f26f0b

Please upload reports for the commit 7f26f0b to get more accurate results.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3670      +/-   ##
==========================================
- Coverage   81.68%   79.30%   -2.39%     
==========================================
  Files          50       51       +1     
  Lines        4391     4494     +103     
==========================================
- Hits         3587     3564      -23     
- Misses        643      773     +130     
+ Partials      161      157       -4

Flag	Coverage Δ
unittests	`79.30% <0.00%> (-2.39%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pkg/podmanager/pod_webhook.go	`0.00% <0.00%> (ø)`

... and 1 file with indirect coverage changes

weizhoublue · 2024-06-27T02:49:40Z

pkg/podmanager/pod_webhook.go

+	ClientSet    *kubernetes.Clientset
+}
+
+func New(enableDra bool, clientSet *kubernetes.Clientset) *PodWebhook {


这个 pkg 下有 NewPodManager , xxxx
因此，这个 New 要调整下，随意了点
NewPodWebHook ?

然后作为一个完整的 pkg，强烈建议把新加逻辑合入 PodManager interface ，而不是新来一个 PodWebhook struct，这样后续可以最大的实现相互之间的代码复用

这个 podManager 以前是作为 get/list pod 的tools，太多地方引用，比如agent。如果合入进来会对 agent 造成干扰

webhook 的逻辑应该只在 controller 里面

这也是目前其他所有代码位置把manager 和 wehbook 分开的原因

weizhoublue · 2024-06-27T02:56:00Z

pkg/podmanager/pod_webhook.go

+var _ webhook.CustomValidator = &PodWebhook{}
+
+// Default implements admission.CustomDefaulter.
+func (pw *PodWebhook) Default(ctx context.Context, obj runtime.Object) error {


需要单位测试

weizhoublue · 2024-06-27T02:58:37Z

pkg/podmanager/pod_webhook.go

+	return ""
+}
+
+func (pw *PodWebhook) injectRdmaResourceToPod(resourceMap map[string]bool, pod *corev1.Pod) {


这个函数中的逻辑是你想的，还是从其他地方借鉴的有最佳实践的

是我自己想的，没找到可借鉴的地方

weizhoublue · 2024-06-27T03:02:11Z

pkg/podmanager/pod_webhook.go

+	for resource, found := range resourceMap {
+		if !found {
+			// inject the resource to the pod.containers[0].resources.requests
+			pod.Spec.Containers[0].Resources.Requests[corev1.ResourceName(resource)] = k8s_resource.MustParse("1")


这里缺少一行重要的 info 级别注入日志

在调用函数之前已经log过了，这里不想在函数内部再次 log，对单元测试来说比较麻烦，而且会在调用函数之后 show pod 的 yaml

pkg/podmanager/pod_webhook.go

weizhoublue · 2024-06-27T03:28:14Z

没有 E2E ，至少需要把 E2E 设计提到 doc

weizhoublue · 2024-06-27T07:40:21Z

pkg/podmanager/pod_manager.go

+			}
+
+			// try to find the resource in container resources.requests
+			if _, ok := c.Resources.Requests[corev1.ResourceName(resource)]; ok {


如果 pod 之前有 resource ，但是 values 已经不是最新的了（不正确），这里应该要纠正？如果不纠正，需要有个日志

不应该纠正，无法定义 pod.resource 是否正确，可能是用户有意为之

如果是用户改了 parameter 这边的定义呢？然后重启pod

有道理，但这里不太好区分是用户自己改了 parameter 还是手动改了 pod

这里还要思考一下

用户是不会手动改 pod，只会改 deployment ，但是判断 pod 和 deployment 的 resource 是否一致 , 又有绝大代码成本，且控制器类型很多

不管用户改 deploy 还是 pod，对于webhook 来说都是改 pod

pkg/podmanager/pod_manager.go

weizhoublue · 2024-06-28T02:31:22Z

pkg/podmanager/pod_manager.go

+	return nil
+}
+
+func (pw *podManager) getStaticNicsFromSpiderClaimParameter(ctx context.Context, pod *corev1.Pod) ([]spiderpoolv2beta1.StaticNic, error) {


这个api 应该是在 spiderDra 之类的目录下

weizhoublue · 2024-06-28T02:31:38Z

pkg/podmanager/pod_manager.go

+	return []spiderpoolv2beta1.StaticNic{}, nil
+}
+
+func (pw *podManager) getResourceMapFromStaticNics(ctx context.Context, staticNics []spiderpoolv2beta1.StaticNic) (map[string]bool, error) {


这个api 应该是在 spiderDra 之类的目录下

weizhoublue · 2024-06-28T02:31:57Z

pkg/podmanager/pod_manager.go

+}
+
+// resourceName return the resourceName for given spiderMultusConfig
+func (pw *podManager) resourceName(smc *spiderpoolv2beta1.SpiderMultusConfig) string {


这个api 应该是在 spiderDra 之类的目录下

weizhoublue · 2024-06-28T02:33:43Z

pkg/podmanager/pod_manager.go

+	return ""
+}
+
+func InjectRdmaResourceToPod(resourceMap map[string]bool, pod *corev1.Pod) {


这个放到本目录的 utils.go 比较合适

cyclinder · 2024-06-28T10:25:09Z

仍然在排查 webhook 流程问题：

Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  10s (x13 over 31s)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "pods.kubernetes.io": failed to call webhook: the server could not find the requested resource
root@10-20-1-200:~# kubectl logs  -n kube-system kube-apiserver-spider-control-plane | grep pods.kubernetes.io
W0628 10:24:25.536243       1 dispatcher.go:225] Failed calling webhook, failing closed pods.kubernetes.io: failed calling webhook "pods.kubernetes.io": failed to call webhook: the server could not find the requested resource
	logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x12\xbe\x02\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a\x8e\x01Internal error occurred: failed calling webhook \"pods.kubernetes.io\": failed to call webhook: the server could not find the requested resource\"\rInternalError*\x87\x01\n\x00\x12\x00\x1a\x00\"{\n\x00\x12ufailed calling webhook \"pods.kubernetes.io\": failed to call webhook: the server could not find the requested resource\x1a\x00(\x002\x000\xf4\x03\x1a\x00\"\x00"
W0628 10:24:25.553750       1 dispatcher.go:225] Failed calling webhook, failing closed pods.kubernetes.io: failed calling webhook "pods.kubernetes.io": failed to call webhook: the server could not find the requested resource

weizhoublue · 2024-06-28T11:53:58Z

the server could not find the requested resource

the server could not find the requested resource ---- is spiderpool parameter CRD not registerred ?

weizhoublue · 2024-06-28T12:51:44Z

just go ahead next Monday

cyclinder · 2024-07-01T02:13:03Z

the server could not find the requested resource ---- is spiderpool parameter CRD not registered ?

No, all spiderpool CRD has been registered, this webhook is just for pods, the error is extraordinary

weizhoublue · 2024-07-04T06:46:07Z

pkg/podmanager/pod_manager.go

@@ -238,3 +271,161 @@ func (pm *podManager) GetPodTopController(ctx context.Context, pod *corev1.Pod)
 		UID: podOwner.UID,
 	}, nil
 }
+
+// Default implements admission.CustomDefaulter.


suggest to add some comments above the function to indicate what the webhook will handle , it will be clear as the feature grows

// whebhook is for:
// 1. when dra is enabled, it insert RDMA resource name to the pod
// ....

pkg/podmanager/pod_manager.go

cyclinder · 2024-07-08T07:19:07Z

@weizhoublue 我建议这个webhook 只对带有某些 label 的 pod 生效(比如 resources.spidernet.io/webhook-resources: true)，不然可能会影响其他 pod 甚至是 apiserver 的正常启动, 测试过程也发现了这个问题

weizhoublue · 2024-07-15T02:33:01Z

charts/spiderpool/templates/tls.yaml

@@ -144,6 +144,40 @@ webhooks:
    - spidercoordinators
  sideEffects: None
 {{- end }}
+{{- if .Values.dra.enabled }}


这个不需要用户判断，由 controller 代码动态 patch

什么情况下patch

Signed-off-by: cyclinder <qifeng.guo@daocloud.io>

cyclinder added the release/feature-new release note for new feature label Jun 26, 2024

cyclinder requested a review from weizhoublue as a code owner June 26, 2024 11:20

cyclinder force-pushed the dra/webhook_resourceName branch from 605d5b7 to 7663fa2 Compare June 26, 2024 11:22