Error: service ... does not have any active endpoints

[ChenyuanZ]

Hi here. I observed does not have any active endpoints error in quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0. My Kubernetes version is v1.10.9-gke.5.

The same bug is recorded in #250 for version 0.9.0-beta.1.

Steps to reproduce the error:

helm repo add polyaxon https://charts.polyaxon.com
helm repo update
helm install polyaxon/polyaxon --name=polyaxon --namespace=polyaxon --set rbac.enabled=true --set ingress. enabled=true --set serviceType=LoadBalancer

Ingress controller log:

I0121 22:10:29.344764       7 main.go:240] Running in Kubernetes cluster version v1.10+ (v1.10.9-gke.5) - git (clean) commit d776b4deeb3655fa4b8f4e8e7e4651d00c5f4a98 - platform linux/amd64
I0121 22:10:29.386501       7 main.go:101] Validated polyaxon/polyaxon-polyaxon-http-backend as the default backend.
I0121 22:10:30.392505       7 nginx.go:258] Starting NGINX Ingress controller
I0121 22:10:30.457125       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"polyaxon", Name:"polyaxon-polyaxon-ingress-config", UID:"5a4a5e0d-1dc9-11e9-83c0-42010a800260", APIVersion:"v1", ResourceVersion:"13351", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap polyaxon/polyaxon-polyaxon-ingress-config
I0121 22:10:31.514043       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"polyaxon", Name:"polyaxon-polyaxon-ingress", UID:"5ad20259-1dc9-11e9-83c0-42010a800260", APIVersion:"extensions/v1beta1", ResourceVersion:"13486", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress polyaxon/polyaxon-polyaxon-ingress
I0121 22:10:31.631037       7 leaderelection.go:187] attempting to acquire leader lease  polyaxon/ingress-controller-leader-polyaxon-ingress...
I0121 22:10:31.631418       7 nginx.go:279] Starting NGINX process
W0121 22:10:31.633066       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:31.633276       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:31.633521       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
I0121 22:10:31.633775       7 controller.go:172] Configuration changes detected, backend reload required.
I0121 22:10:31.643935       7 status.go:148] new leader elected: polyaxon-polyaxon-ingress-controller-555f798499-t4ckk
I0121 22:10:31.948223       7 controller.go:190] Backend successfully reloaded.
I0121 22:10:31.948498       7 controller.go:202] Initial sync, sleeping for 1 second.
[21/Jan/2019:22:10:32 +0000]TCP200000.000
W0121 22:10:35.350817       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:35.351764       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:35.351810       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:38.704066       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:38.731051       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:38.731161       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:42.017665       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:42.017717       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:42.017748       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:45.351979       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:45.352048       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.

Ingress controller log with flag --v=5:
https://gist.github.com/ChenyuanZ/1c596c4bb91a2bf55c17d1ae7d38f6b3

Service spec:

apiVersion: v1
kind: Service
metadata:
  name: polyaxon-polyaxon-api
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: api
    nodePort: 30760
    port: 80
    protocol: TCP
    targetPort: 80
  - name: streams
    nodePort: 31901
    port: 1337
    protocol: TCP
    targetPort: 1337
  selector:
    app: polyaxon-polyaxon-api
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

Ingress spec:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: polyaxon-polyaxon-ingress
spec:
  rules:
  - host: chart-example.local
    http:
      paths:
      - backend:
          serviceName: polyaxon-polyaxon-api
          servicePort: 80
        path: /
      - backend:
          serviceName: polyaxon-polyaxon-api
          servicePort: 1337
        path: /ws
status:
  loadBalancer: {}

Ingress controller spec

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: null
  generation: 1
  labels:
    app: polyaxon-polyaxon-ingress-controller
    chart: polyaxon-0.3.7
    heritage: Tiller
    release: polyaxon
    type: polyaxon-core
  name: polyaxon-polyaxon-ingress-controller
  selfLink: /apis/extensions/v1beta1/namespaces/polyaxon/deployments/polyaxon-polyaxon-ingress-controller
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: polyaxon-polyaxon-ingress-controller
      type: polyaxon-core
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: polyaxon-polyaxon-ingress-controller
        type: polyaxon-core
    spec:
      affinity:
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: type
                  operator: In
                  values:
                  - polyaxon-core
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/polyaxon-polyaxon-http-backend
        - --configmap=$(POD_NAMESPACE)/polyaxon-polyaxon-ingress-config
        - --ingress-class=polyaxon-ingress
        - --v=5
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: nginx-ingress-controller
        ports:
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: polyaxon-polyaxon-ingress-serviceaccount
      serviceAccountName: polyaxon-polyaxon-ingress-serviceaccount
      terminationGracePeriodSeconds: 60
status: {}

[aledbf]

Closing. This is an issue with the polyaxon helm chart.

Steps to install correctly:

1. Install chart:

helm install polyaxon/polyaxon \
  --name=polyaxon \
  --namespace=polyaxon \
  --set rbac.enabled=true \
  --set ingress.enabled=false \
  --set serviceType=ClusterIP

2. Create an Ingress:

echo '
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      ingress.kubernetes.io/add-base-url: "true"
      ingress.kubernetes.io/proxy-body-size: 4G
      ingress.kubernetes.io/proxy-connect-timeout: "600"
      ingress.kubernetes.io/proxy-read-timeout: "600"
      ingress.kubernetes.io/proxy-send-timeout: "600"
      ingress.kubernetes.io/rewrite-target: /
      ingress.kubernetes.io/send-timeout: "600"
      ingress.kubernetes.io/ssl-redirect: "false"
    labels:
      chart: polyaxon-0.3.7
      heritage: Tiller
      release: polyaxon
      type: polyaxon-core
    name: polyaxon-polyaxon-ingress
    namespace: polyaxon
  spec:
    rules:
    - host: chart-example.local
      http:
        paths:
        - backend:
            serviceName: polyaxon-polyaxon-api
            servicePort: 80
          path: /
        - backend:
            serviceName: polyaxon-polyaxon-api
            servicePort: 1337
          path: /ws
' | kubectl apply -f -

Note: the polyaxon-polyaxon-api pod takes ~3 minutes to start so you will see some does not have any active Endpoint until start passing the probes.