Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

service default/... does not have any active endpoints #250

Closed
foxylion opened this issue Feb 8, 2017 · 34 comments
Closed

service default/... does not have any active endpoints #250

foxylion opened this issue Feb 8, 2017 · 34 comments

Comments

@foxylion
Copy link
Contributor

foxylion commented Feb 8, 2017

Hi,

I've upgraded gcr.io/google_containers/nginx-ingress-controller from 0.8.3 to 0.9.0-beta.1. But now I'm facing the issue that no service is available through the ingress controller. It will always result in 503 status code responses.

Log looks like this:

W0208 15:59:58.498949       6 controller.go:806] service default/webapp-master does not have any active endpoints
W0208 15:59:58.499025       6 controller.go:560] service default/ingress-nginx-default-backend does not have any active endpoints
(repeating every 10 seconds)

The generated nginx.conf contains ony upstream entries with 127.0.0.1:8181 as backend endpoint:

   upstream default-webapp-master-80 {
        least_conn;
        server 127.0.0.1:8181 max_fails=0 fail_timeout=0;
    }
    upstream upstream-default-backend {
        least_conn;
        server 127.0.0.1:8181 max_fails=0 fail_timeout=0;
    }

Any idea what is going wrong?

I only updated the ingress controller. ingress resources where not changed. One example ingress resource looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-master
  annotations:
    proxy-body-size: 100m
spec:
  rules:
  - host: hostname.domain.example
    http:
      paths:
      - path: /
        backend:
          serviceName: webapp-master
          servicePort: 80

The matching service like this:

kind: Service
apiVersion: v1
metadata:
  name: webapp-master
spec:
  ports:
  - port: 80
    targetPort: http
  selector:
    app: webapp-master
@aledbf
Copy link
Member

aledbf commented Feb 8, 2017

@foxylion there is a bug in 0.9.0-beta.1 already fixed in master (#121).
In the meantime if you change targetPort to 80 instead a string it will work.
Apologies for the inconvenience.

@foxylion
Copy link
Contributor Author

foxylion commented Feb 8, 2017

@aledbf Thanks for your quick response. I'll try that.

Is there any schedule when new releases are done?

@aledbf
Copy link
Member

aledbf commented Feb 20, 2017

Fix included in next release #303
Please reopen if you still have issues.

@aledbf aledbf closed this as completed Feb 20, 2017
@JonatanE
Copy link

This is not fixed in 0.9.0-beta.4

2017/04/27 09:19:30 [warn] 12477#12477: *39354 using uninitialized "proxy_upstream_name" variable while logging request

@0xMadao
Copy link

0xMadao commented May 24, 2017

still in in 0.9.0-beta.5

2017/05/24 08:05:38 [warn] 379#379: *7 using uninitialized "proxy_upstream_name" variable while logging request, client: myclient, server: _, request: "GET / HTTP/1.1", host: "myhost"

@paolomainardi
Copy link

I have the same problems described here and i confirm with the latest beta6 and i confirm that changing targetPort from a string to the port fixed the ingress, but i have a lot of panic like this:

 0.054 401
panic: runtime error: index out of range

goroutine 660 [running]:
k8s.io/ingress/vendor/github.com/paultag/sniff/parser.GetSNBlock(0xc4204d8070, 0xf90, 0xf90, 0xc4204d8070, 0xf90, 0xf90, 0x0, 0x0)
	/home/nicksardo/go/src/k8s.io/ingress/vendor/github.com/paultag/sniff/parser/parser.go:91 +0x1d1
k8s.io/ingress/vendor/github.com/paultag/sniff/parser.GetHostname(0xc4204d8000, 0x1000, 0x1000, 0x1000, 0x205, 0x0, 0x0)
	/home/nicksardo/go/src/k8s.io/ingress/vendor/github.com/paultag/sniff/parser/parser.go:42 +0xee
main.(*proxy).Handle(0xc4204d4f60, 0x1d6ada0, 0xc4204470e8)
	/home/nicksardo/go/src/k8s.io/ingress/controllers/nginx/pkg/cmd/controller/tcp.go:45 +0x1dc
created by main.newNGINXController.func1
	/home/nicksardo/go/src/k8s.io/ingress/controllers/nginx/pkg/cmd/controller/nginx.go:121 +0x260
nginx git:dev ❯ kubectl get pod -nnginx-ingress

@paolomainardi
Copy link

nginx git:dev ❯ kubectl get pod -nnginx-ingress                                                                                                                                                   ✹
NAME                                    READY     STATUS             RESTARTS   AGE
default-http-backend-3723061492-lbbbv   1/1       Running            0          18d
nginx-3602923031-h2582                  0/1       CrashLoopBackOff   6          15m
nginx-3602923031-hm4l8                  1/1       Running            5          15m

@paolomainardi
Copy link

I had to revert back the ingress controller to the 0.8.3.

What could i share to debug better the issue ?

@foxylion
Copy link
Contributor Author

@paolomainardi I would suggest you to create a separate issue (this shouldn't be caused by this one).
And then ideally add a minimal example to reproduce the issue (including ingress and service configurations).

@paolomainardi
Copy link

@foxylion to me it seems related as i had "service default/service does not have any active endpoints" that was causing the failure that has been fixed changing the targetPort from a string to an integer.

@foxylion
Copy link
Contributor Author

foxylion commented May 26, 2017

@paolomainardi Okay, so replacing the labeled targetPorts fixes the "index out of range" panic? Sorry, then this should maybe be reopened.

ping @aledbf

@aledbf
Copy link
Member

aledbf commented May 26, 2017

@paolomainardi can you share the service and ingress definition?

@f0
Copy link

f0 commented May 27, 2017

@aledbf i have the same problem

In gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.5 it does work, with

gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7 i got the error

W0527 15:27:27.667881       1 controller.go:842] service external-logging/kibana-svc does not have any active endpoints

Here are the deployment, service and Ingress

metadata:
  name: kibana-deployment
  namespace: external-logging
spec:
  replicas: 1
  template:
    metadata:
      labels:
        component: kibana
    spec:
      containers:
      - name: kibana
        image: example.net/kibana:5.2.2
        imagePullPolicy: IfNotPresent
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
          requests:
            cpu: 100m
        env:
        - name: "ELASTICSEARCH_URL"
          value: "http://elasticsearch-external-logging:9200"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: external-logging
spec:
  rules:
  - host: kibana.example.net
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana-svc
          servicePort: 5601
apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
  namespace: external-logging
  labels:
    component: kibana-svc
spec:
  type: NodePort
  ports:
  - port: 5601
    protocol: TCP
    targetPort: 5601
  selector:
    component: kibana

@chainlink
Copy link

I had the same error and changed my path from /* to / and it worked (I see you have / above though

@jhdehart
Copy link

jhdehart commented Jun 1, 2017

I've encountered the same issue with the fluentd-elasticsearch service and it turned out to be RBAC related.

I viewed the following in my logs

W0601 18:40:36.085507 1 controller.go:1106] error mapping service ports: error syncing service kube-system/kibana-logging:
User "system:serviceaccount:nginx-ingress:nginx-ingress-serviceaccount" cannot update services in the namespace "kube-system". (put services kibana-logging)

An easy fix was to change the targetPort parameter from ui to 5601 in the kibana-logging service definition. Proper fix was updating the permissions.

@ianmartorell
Copy link

I encountered this issue after upgrading from 0.8.3 to 0.9.0-beta.17. Changing targetPorts in services from labels to integers fixed it, so I guess this should be reopened.

@aledbf
Copy link
Member

aledbf commented Nov 12, 2017

@ianmartorell please open a new issue

@antl3x
Copy link

antl3x commented Mar 15, 2018

I was having this error in the newest version and follow the @chainlink advise.

Change the path to / and works like a charm!

It was /* before.

@0312birdzhang
Copy link

still in quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.13.0

I0509 02:52:13.119086      17 controller.go:179] ingress backend successfully reloaded...
W0509 02:52:27.241087      17 controller.go:775] service kube-system/mongoclient does not have any active endpoints
W0509 02:52:27.241269      17 controller.go:315] service go/service-ll-onegameinfo does not have any active endpoints for port 60104 and protocol TCP
W0509 02:52:27.241303      17 controller.go:315] service default/service-xxzhushou-script-detail04 does not have any active endpoints for port 12004 and protocol TCP
W0509 02:52:27.241323      17 controller.go:315] service go/service-ipa-pkg-relation does not have any active endpoints for port 20401 and protocol TCP
W0509 02:52:27.241329      17 controller.go:315] service go/service-ll-open-game-list does not have any active endpoints for port 60102 and protocol TCP
W0509 02:52:27.241353      17 controller.go:315] service go/service-gift-center does not have any active endpoints for port 30001 and protocol TCP
W0509 02:52:27.241369      17 controller.go:315] service default/service-xxzhushou-script-detail02 does not have any active endpoints for port 12002 and protocol TCP
W0509 02:52:27.241378      17 controller.go:315] service default/service-banner-center does not have any active endpoints for port 13010 and protocol TCP
W0509 02:52:27.241395      17 controller.go:315] service go/service-ipa-base-info-init does not have any active endpoints for port 20402 and protocol TCP
W0509 02:52:27.241414      17 controller.go:315] service go/service-ll-onegameinfo does not have any active endpoints for port 14000 and protocol TCP
W0509 02:52:27.241435      17 controller.go:315] service go/service-game-center-brief does not have any active endpoints for port 13000 and protocol TCP
W0509 02:52:27.241443      17 controller.go:315] service go/service-ipa-push-collect does not have any active endpoints for port 15020 and protocol TCP
.......
ame-consult-center-etcd does not have any active endpoints for port 25006 and protocol TCP
I0509 02:52:27.241807      17 controller.go:170] backend reload required
I0509 02:52:27.241816      17 util.go:67] rlimit.max=65536
I0509 02:52:27.241821      17 nginx.go:560] maximum number of open file descriptors : 7168
I0509 02:52:27.305661      17 nginx.go:661] NGINX configuration diff

@rikatz
Copy link
Contributor

rikatz commented May 9, 2018

@0312birdzhang what's your kubernetes version?

@0312birdzhang
Copy link

@rikatz sorry it's my fault, the endpoints has been deleted, so the log is right.

@ChenyuanZ
Copy link

Hi here. I observed the issue in quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0. My Kubernetes version is v1.10.9-gke.5.

Ingress controller log:

I0121 22:10:29.344764       7 main.go:240] Running in Kubernetes cluster version v1.10+ (v1.10.9-gke.5) - git (clean) commit d776b4deeb3655fa4b8f4e8e7e4651d00c5f4a98 - platform linux/amd64
I0121 22:10:29.386501       7 main.go:101] Validated polyaxon/polyaxon-polyaxon-http-backend as the default backend.
I0121 22:10:30.392505       7 nginx.go:258] Starting NGINX Ingress controller
I0121 22:10:30.457125       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"polyaxon", Name:"polyaxon-polyaxon-ingress-config", UID:"5a4a5e0d-1dc9-11e9-83c0-42010a800260", APIVersion:"v1", ResourceVersion:"13351", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap polyaxon/polyaxon-polyaxon-ingress-config
I0121 22:10:31.514043       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"polyaxon", Name:"polyaxon-polyaxon-ingress", UID:"5ad20259-1dc9-11e9-83c0-42010a800260", APIVersion:"extensions/v1beta1", ResourceVersion:"13486", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress polyaxon/polyaxon-polyaxon-ingress
I0121 22:10:31.631037       7 leaderelection.go:187] attempting to acquire leader lease  polyaxon/ingress-controller-leader-polyaxon-ingress...
I0121 22:10:31.631418       7 nginx.go:279] Starting NGINX process
W0121 22:10:31.633066       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:31.633276       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:31.633521       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
I0121 22:10:31.633775       7 controller.go:172] Configuration changes detected, backend reload required.
I0121 22:10:31.643935       7 status.go:148] new leader elected: polyaxon-polyaxon-ingress-controller-555f798499-t4ckk
I0121 22:10:31.948223       7 controller.go:190] Backend successfully reloaded.
I0121 22:10:31.948498       7 controller.go:202] Initial sync, sleeping for 1 second.
[21/Jan/2019:22:10:32 +0000]TCP200000.000
W0121 22:10:35.350817       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:35.351764       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:35.351810       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:38.704066       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:38.731051       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:38.731161       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:42.017665       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:42.017717       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:42.017748       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:45.351979       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:45.352048       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.

Service spec:

apiVersion: v1
kind: Service
metadata:
  name: polyaxon-polyaxon-api
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: api
    nodePort: 30760
    port: 80
    protocol: TCP
    targetPort: 80
  - name: streams
    nodePort: 31901
    port: 1337
    protocol: TCP
    targetPort: 1337
  selector:
    app: polyaxon-polyaxon-api
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

Ingress spec

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: polyaxon-polyaxon-ingress
spec:
  rules:
  - host: chart-example.local
    http:
      paths:
      - backend:
          serviceName: polyaxon-polyaxon-api
          servicePort: 80
        path: /
      - backend:
          serviceName: polyaxon-polyaxon-api
          servicePort: 1337
        path: /ws
status:
  loadBalancer: {}

BTW, I verified that Endpoints exists

$ kubectl get endpoints -l app=polyaxon-polyaxon-api
NAME                    ENDPOINTS                       AGE
polyaxon-polyaxon-api   x.x.x.x:1337,x.x.x.x:80         26m

Please advise how to resolve this issue. Thanks in advance!

@aledbf
Copy link
Member

aledbf commented Jan 21, 2019

@ChenyuanZ if you use type: LoadBalancer in the service, you are creating a new cloud load balancer ($$$), please change that. Also, the service and the ingress must be present in the same namespace (polyaxon/polyaxon-polyaxon-api -> namespace: polyaxon, service: polyaxon-polyaxon-api)

@ChenyuanZ
Copy link

Hi @aledbf

I have tried both NodePort and ClusterIP. But the does not have any active endpoints issue exists in all cases.

The service and ingress are in the same namespace polyaxon as shown in the example below.

In the example below, I changed service from LoadBalancer to NodePort, and restarted ingress-controller. But the same error persisted.

$ kubectl get ingress --all-namespaces
NAMESPACE   NAME                        HOSTS                 ADDRESS         PORTS     AGE
polyaxon    polyaxon-polyaxon-ingress   chart-example.local   x.x.x.x   80        38m

kubectl get svc --all-namespaces -l app=polyaxon-polyaxon-api
NAMESPACE   NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
polyaxon    polyaxon-polyaxon-api   NodePort   x.x.x.x   <none>        80:30760/TCP,1337:31901/TCP   39m

@aledbf
Copy link
Member

aledbf commented Jan 21, 2019

@ChenyuanZ please post the output kubectl get ep --all-namespaces -l app=polyaxon-polyaxon-api

@ChenyuanZ
Copy link

ChenyuanZ commented Jan 21, 2019

Hi @aledbf

$ kubectl get ep --all-namespaces -l app=polyaxon-polyaxon-api
NAMESPACE   NAME                    ENDPOINTS                       AGE
polyaxon    polyaxon-polyaxon-api   x.x.x.x:1337,x.x.x.x:80   47m

@aledbf
Copy link
Member

aledbf commented Jan 21, 2019

@ChenyuanZ please open a new issue. Before that, please change the ingress controller deployment and add the flag --v=5 and post the log in the issue (create a gist to post the log).
Also, post the deployment of the ingress controller itself.

@ChenyuanZ
Copy link

Hi @aledbf. I've created an issue at #3687

@kariae
Copy link

kariae commented May 21, 2019

Hi here, I have some Service "X" does not have any active Endpoint. also in the controller logs, I'm running nginx-ingress 0.24.1 my kubernetes version is v1.14.1, i'm running an empty nginx image to test everything, here is my deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy-main
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-main
  template:
    metadata:
      labels:
        run: nginx-main
    spec:
      containers:
      - image: nginx
        name: nginx

The service

apiVersion: v1
kind: Service
metadata:
  name: demo-example-com-nginx
spec:
  selector:
    app: nginx-deploy-main
  ports:
  - targetPort: http
    port: 80

and the ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-example-com
  annotations:
    kubernetes.io/ingress.class: "nginx"
    certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: http01
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  tls:
  - hosts:
    - demo.example.com
    secretName: letsencrypt-prod
  rules:
  - host: demo.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: demo-example-com-nginx
          servicePort: 80

The certificate is well generated, but when I try to access the url I have a 503 error with those logs in the controller

W0521 09:29:12.240234       6 controller.go:797] Service "default/demo-example-com-nginx" does not have any active Endpoint.

Any help, please? Thank you

@aledbf
Copy link
Member

aledbf commented May 21, 2019

@kariae please use the kubectl plugin to get information about the ingresses https://kubernetes.github.io/ingress-nginx/kubectl-plugin/#ingresses
Also, you can use kubectl get ep demo-example-com-nginx to get the endpoints. If you don't see any, it means you have issues with your service selectors

@kariae
Copy link

kariae commented May 21, 2019

@aledbf I have an endpoint demo-example-com-nginx <none> 40m, I'm installing the plugin to get more details

Edit:
I used the plugin to get the list of ingresses, I have it there but without endpoint:

INGRESS NAME      HOST+PATH          ADDRESSES   TLS   SERVICE                 SERVICE PORT   ENDPOINTS
demo-example-com   demo.example.com/               YES   demo-example-com-nginx   80             0

@aledbf
Copy link
Member

aledbf commented May 21, 2019

That means you don't have a pod running for that service. You should see an IP address in that field

@kariae
Copy link

kariae commented May 22, 2019

@aledbf thanks so much, I did mis-configure the selector in my service 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests