volcano-admission restart error, failed calling webhook "mutatepod.volcano.sh"

[nature1995]

What happened:
volcano-admission pod cannot be restarted. After the restart, the pod cannot be scheduled, and an error is reported:

Internal error occurred: failed calling webhook "mutatepod.volcano.sh": 
Post "https://volcano-admission-service.volcano-system.svc:443/pods /mutate?timeout=10s": 
no endpoints available for service "volcano-admission-service

What you expected to happen:
Pod restarts normally

How to reproduce it (as minimally and precisely as possible):
delete the volcano-admission pod or restart the volcano-admission deployments

Anything else we need to know?:

Environment:

• Volcano Version: v1.6
• Kubernetes version (use kubectl version): 1.20.8
• Cloud provider or hardware configuration: baidu
• OS (e.g. from /etc/os-release): cent7.5
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[nature1995]

This problem can be solved by delete the volcano-admission-service-pods-mutate and volcano-admission-service-pods-validate

kubectl delete mutatingwebhookconfigurations volcano-admission-service-pods-mutate
kubectl delete validatingwebhookconfigurations volcano-admission-service-pods-validate

refer the from #2346

We can find the namespaceSelector on volcano-admission-service-pods-mutate and namespaceSelector. But the selector is not work unless you add the label to the namespace. So, the promblem can be solved by add a label to the namespace's yaml.

    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values:
            - volcano-system
            - kube-system

apiVersion: v1
kind: Namespace
metadata:
  name: volcano-system
  labels:
    kubernetes.io/metadata.name: volcano-system
---
apiVersion: v1
kind: Namespace
metadata:
  name: volcano-monitoring
  labels:
    kubernetes.io/metadata.name: volcano-monitoring

[hzxuzhonghu]

@hwdef #2346 cause this, ptal

[nature1995]

@hwdef #2346 cause this, ptal

We use Baidu CCE. It not work on the old version, such as 1.20.8.

[hwdef]

@hwdef #2346 cause this, ptal

We use Baidu CCE. It not work on the old version, such as 1.20.8.

yes, Your PR looks like it could fix this.

[hwdef]

/close

[volcano-sh-bot]

@hwdef: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.