example-etcd-cluster not created

[itsbanjo]

upon running oc create -f etcd-cluster-cr.yaml the example-etcd-cluster-* pods are not created

[0xdnL]

same for kubectl create -f etcd-cluster-cr.yaml which throws

error: error validating "etcd-cluster-cr.yaml": error validating data: [ValidationError(EtcdCluster.spec): 
unknown field "size" in com.coreos.database.etcd.v1.EtcdCluster.spec, ValidationError(EtcdCluster.spec): 
unknown field "version" in com.coreos.database.etcd.v1.EtcdCluster.spec]; 
if you choose to ignore these errors, turn validation off with --validate=false

using --validate=false doesn't change anything

minikube version: v1.23.2
kubernets server: 1.22+

the versions I use also have changes for apiextensions.k8s.io/v1 which I was able to fix by changing the spec
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#customresourcedefinition-v122

but kubectl create -f etcd-cluster-cr.yaml won't bring up pods

@joshix @jdob will you update the example for later versions of the k8s-api ?

[0xdnL]

ok, I fixed the validation by using this schema (which has become required in apiextensions.k8s.io/v1 )

    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              size:
                type: integer
              version:
                type: string

now one pod is tries to start:

example-etcd-cluster-cfvbzv546g   0/1     Init:0/1   0          15m

but the init-container throws:

kubectl logs example-etcd-cluster-cfvbzv546g -c check-dns

nslookup: can't resolve 'example-etcd-cluster-cfvbzv546g.example-etcd-cluster.default.svc'
nslookup: can't resolve 'example-etcd-cluster-cfvbzv546g.example-etcd-cluster.default.svc'
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
...

[seanw2020]

@joshix @jdob Thoughts above? I spent about 2 hours researching this, and it seems that:

1. your book depends on the now stale etcd-operator repo
2. in that repo, this issue has been reported but without resolution
3. I'm using Amazon EKS on Kubernetes 1.23
4. the responsible code is around here

Is there any fix? Or is the book's demo (chapter 3) largely stale now too?

I am excited about the book, but this start is worrying.

[Ovec]

Same problem here on Kind cluster (kind v0.16.0 go1.19.1 darwin/arm64)

kubectl logs example-etcd-cluster-gdr2l9h8rv -c check-dns

nslookup: can't resolve 'example-etcd-cluster-gdr2l9h8rv.example-etcd-cluster.default.svc'
nslookup: can't resolve 'example-etcd-cluster-gdr2l9h8rv.example-etcd-cluster.default.svc'
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

[danpawlik]

The order of applying the manifests is just wrong.

kubectl apply -f etcd-operator-crd.yaml
kubectl apply -f etcd-operator-deployment.yaml
kubectl apply -f etcd-operator-rolebinding.yaml
kubectl apply -f etcd-operator-role.yaml
kubectl apply -f etcd-operator-sa.yaml
kubectl apply -f etcd-cluster-cr.yaml

After that, you will have something like:

[root@kubernetes ch03]# kubectl get pods
NAME                              READY   STATUS     RESTARTS   AGE
etcd-operator-58bc7fbd7-jwrj6     1/1     Running    0          48s
example-etcd-cluster-28h6q29vjn   0/1     Init:0/1   0          30s

But it will fail, unless you have provided Docker credentials into your Kubernetes Cluster.
Example how it looks when Docker rate limit is exceeded:

[root@kubernetes ch03]# kubectl get pods
NAME                              READY   STATUS                  RESTARTS   AGE
etcd-operator-58bc7fbd7-jwrj6     1/1     Running                 0          70s
example-etcd-cluster-28h6q29vjn   0/1     Init:ImagePullBackOff   0          52s

NOTE: I did not read the book yet, but I consider to read it and I just check that repo.

[tkrishtop]

Hi there, I tested the etcd cluster example in 2023 on k8s 1.24, and the example seems to be completely outdated. The cluster does not start; it endlessly waits for the DNS config to be created.

$ kubectl describe pod/example-etcd-cluster-fl9wp7ch4b
Name:             example-etcd-cluster-fl9wp7ch4b
-- snip --
Init Containers:
  check-dns:
    Container ID:  containerd://e16a4a902d7c7f0636c9a97eef4deaf1a3a076d24b7b633a42659acd135ccdf0
    Image:         busybox:1.28.0-glibc
    Image ID:      docker.io/library/busybox@sha256:0b55a30394294ab23b9afd58fab94e61a923f5834fba7ddbae7f8e0c11ba85e6
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      
                TIMEOUT_READY=0
                while ( ! nslookup example-etcd-cluster-fl9wp7ch4b.example-etcd-cluster.default.svc )
                do
                  # If TIMEOUT_READY is 0 we should never time out and exit 
                  TIMEOUT_READY=$(( TIMEOUT_READY-1 ))
                              if [ $TIMEOUT_READY -eq 0 ];
                                  then
                                      echo "Timed out waiting for DNS entry"
                                      exit 1
                                  fi
                              sleep 1
                            done
    State:          Running
-- snip --
Warning  DNSConfigForming  12s (x20 over 20m)  kubelet            
Search Line limits were exceeded, 
some search paths have been omitted, 
the applied search line is: 
default.svc.cluster.local svc.cluster.local cluster.local dns.podman home redhat.com

$ kubectl logs example-etcd-cluster-fl9wp7ch4b -c check-dns
nslookup: can't resolve 'example-etcd-cluster-fl9wp7ch4b.example-etcd-cluster.default.svc'
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Here is what I did (tested on both minikube and kind, with the same results):

$ kubectl create -f etcd-operator-crd.yaml
$ kubectl create -f etcd-operator-sa.yaml
$ kubectl create -f etcd-operator-role.yaml
$ kubectl create -f etcd-operator-rolebinding.yaml
$ kubectl create -f etcd-operator-deployment.yaml
$ kubectl create -f etcd-cluster-cr.yaml

The result:

$ kubectl get etcdcluster
NAME                   AGE
example-etcd-cluster   3m4s

$ kubectl get pods
NAME                              READY   STATUS     RESTARTS   AGE
etcd-operator-668c54776-vp4sz     1/1     Running    0          11m
example-etcd-cluster-fl9wp7ch4b   0/1     Init:0/1   0          102s

The reason is pointed out by @seanw2020 : the book depends on the stale etcd-operator repo and the issue is known but remains unresolved.

I'm not sure if the book maintainers have plans to fix that example. For now, I would recommend skipping the 'Running operators' chapter. Despite being somewhat outdated, the book is still highly interesting.