Why can't Karmada apiserver start

[Schwarao]

log：

describe：

[Schwarao]

describe：

[Schwarao]

Cluster created using kubeadm

[Schwarao]

This is my initialization command：

sudo KUBECONFIG=/home/rongqigang/.kube/config kubectl karmada init --kubeconfig=/home/rongqigang/.kube/config
--etcd-init-image=alpine:3.19.2
--karmada-aggregated-apiserver-image=karmada/karmada-aggregated-apiserver:v1.10.1
--karmada-controller-manager-image=karmada/karmada-controller-manager:v1.10.1
--karmada-scheduler-image=karmada/karmada-scheduler:v1.10.1
--karmada-webhook-image=karmada/karmada-webhook:v1.10.1
--kube-image-registry registry.cn-hangzhou.aliyuncs.com/google_containers
--crds crds.tar.gz

[RainbowMango]

cc @chaosi-zju

[chaosi-zju]

hi @Schwarao, could you please provided full karmada-apiserver logs?

if current pod logs is not complete, maybe you can use kubectl logs -p karmada-apiserver-xxx -n karmada-system to fetch previous containter logs~

[chaosi-zju]

besides, can you check the version of kubectl karmada~

[Schwarao]

This is the complete log：

[Schwarao]

此外，你能检查一下kubectl karmada~
May I ask what command to use

[chaosi-zju]

May I ask what command to use

kubectl karmada version

[chaosi-zju]

if current pod logs is not complete, maybe you can use kubectl logs -p karmada-apiserver-xxx -n karmada-system to fetch previous containter logs~

can you use -p parameter to print logs? just like: kubectl logs -p karmada-apiserver-xxx -n karmada-system

[chaosi-zju]

run.go:74] "command failed" err="context deadline exceeded"

This error is most likely because karmada-apiserver cannot connect to etcd-0, we may need more infomation:

1. use kubectl describe po karmada-apiserver-xxx -n karmada-system to get the Containers.karmada-apiserver.Command field
2. check what is etcd-server address and whether/why etcd-server is not connected.

[Schwarao]

here

[Schwarao]

kubectl describe po karmada-apiserver-8dfd9bcd7-8k5g4 -n karmada-system：

Name: karmada-apiserver-8dfd9bcd7-8k5g4
Namespace: karmada-system
Priority: 0
Service Account: default
Node: k8s-master01/10.240.10.70
Start Time: Thu, 27 Jun 2024 10:47:16 +0800
Labels: app=karmada-apiserver
pod-template-hash=8dfd9bcd7
Annotations: cni.projectcalico.org/containerID: acbea277dc4fd1c791b852e83e97d63319e3460fdefd85cad27410af86322645
cni.projectcalico.org/podIP: 10.0.32.135/32
cni.projectcalico.org/podIPs: 10.0.32.135/32
Status: Running
IP: 10.0.32.135
IPs:
IP: 10.0.32.135
Controlled By: ReplicaSet/karmada-apiserver-8dfd9bcd7
Containers:
karmada-apiserver:
Container ID: docker://8e9833a5899d34b2107e426ab17dbba78f971d894476999479bab35972b2bf29
Image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.11
Image ID: docker-pullable://registry.aliyuncs.com/google_containers/kube-apiserver@sha256:24262f37a760fdee1970f6b7f496ee5189774af38b0812a289e67518aaf32243
Port: 5443/TCP
Host Port: 0/TCP
Command:
kube-apiserver
--allow-privileged=true
--authorization-mode=Node,RBAC
--client-ca-file=/etc/karmada/pki/ca.crt
--enable-bootstrap-token-auth=true
--etcd-cafile=/etc/karmada/pki/etcd-ca.crt
--etcd-certfile=/etc/karmada/pki/etcd-client.crt
--etcd-keyfile=/etc/karmada/pki/etcd-client.key
--etcd-servers=https://etcd-0.etcd.karmada-system.svc.cluster.local:2379
--bind-address=0.0.0.0
--kubelet-client-certificate=/etc/karmada/pki/karmada.crt
--kubelet-client-key=/etc/karmada/pki/karmada.key
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
--disable-admission-plugins=StorageObjectInUseProtection,ServiceAccount
--runtime-config=
--apiserver-count=1
--secure-port=5443
--service-account-issuer=https://kubernetes.default.svc.cluster.local
--service-account-key-file=/etc/karmada/pki/karmada.key
--service-account-signing-key-file=/etc/karmada/pki/karmada.key
--service-cluster-ip-range=10.96.0.0/12
--proxy-client-cert-file=/etc/karmada/pki/front-proxy-client.crt
--proxy-client-key-file=/etc/karmada/pki/front-proxy-client.key
--requestheader-allowed-names=front-proxy-client
--requestheader-client-ca-file=/etc/karmada/pki/front-proxy-ca.crt
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
--tls-cert-file=/etc/karmada/pki/apiserver.crt
--tls-private-key-file=/etc/karmada/pki/apiserver.key
--tls-min-version=VersionTLS13
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 27 Jun 2024 18:19:50 +0800
Finished: Thu, 27 Jun 2024 18:20:10 +0800
Ready: False
Restart Count: 93
Liveness: http-get https://:5443/livez delay=15s timeout=5s period=30s #success=1 #failure=3
Readiness: http-get https://:5443/readyz delay=0s timeout=5s period=30s #success=1 #failure=3
Environment:
Mounts:
/etc/karmada/pki from karmada-cert (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
karmada-cert:
Type: Secret (a volume populated by a Secret)
SecretName: karmada-cert
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: :NoExecute op=Exists
Events:
Type Reason Age From Message

---

Warning Unhealthy 28m (x208 over 7h33m) kubelet Readiness probe failed: Get "https://10.0.32.135:5443/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning BackOff 3m4s (x1964 over 7h32m) kubelet Back-off restarting failed container karmada-apiserver in pod karmada-apiserver-8dfd9bcd7-8k5g4_karmada-system(5469299d-9754-410e-9b62-1b471b150b20)
[rongqigang@k8s-master01:~/calico_image]$

[Schwarao]

More information:

endpoints ：

etcd svc：

[Schwarao]

May I ask if Kubernetes 1.30.2 is supported
@chaosi-zju

[chaosi-zju]

May I ask if Kubernetes 1.30.2 is supported

I tested replace karmada-apiserver image to v1.30.2, it runs ok~

$ kubectl get deploy karmada-apiserver -n karmada-system -o yaml | grep -C 3 image:
        - --tls-cert-file=/etc/karmada/pki/apiserver.crt
        - --tls-private-key-file=/etc/karmada/pki/apiserver.key
        - --tls-min-version=VersionTLS13
        image: registry.k8s.io/kube-apiserver:v1.30.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 8
$ kubectl get pods -o wide -n karmada-system | grep karmada-apiserver     
karmada-apiserver-6d698678df-qfbsz                     1/1     Running   0          2m35s   172.18.0.4    karmada-host-control-plane   <none>           <none>

[Schwarao]

May I ask why I reported an error here?

[Schwarao]

Do we just need to change this to 1.30.2

[chaosi-zju]

May I ask why I reported an error here?

This should be a problem with your own kubernetes container network. There is a service named etcd in your environment, but it cannot do dns resolution on serviceName in the container.

[chaosi-zju]

Do we just need to change this to 1.30.2

You can, but it will not make any difference~

You karmada-apiserver installed failed is because it cann't connect to etcd by serviceName.

However, the reason for can not connect to etcd by serviceName is because there may be some problem with your own kubernetes container network, wihch result in dns resolution failure.

---

Maybe you can a test to check whether your kubernetes container network has problem:

• deploy a simple nginx deployment and a service
• deploy another simple ubuntu deployment and enter the container of ubuntu pod and check whether you can connect to nginx service from ubuntu container by nginx service name~

[chaosi-zju]

Same problem in #5143 (comment)

May be the comment #5143 (comment) can help this problem.