NGINX Ingress can't access service ClusterIP (502 bad gateway error)

[fabstao]

Hi! After successfully deployed application (pod looks OK, logs are ok), I created ClusterIP service, then created NGINX Ingress. When I try to access NGINX NodePort, I get 502 bad gateway error. Checking NGINX logs, it can't access the ClusterIP and port.

Details:

NGINX Ingress version

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.35.0
  Build:         54ad65e32bcab32791ab18531a838d1c0f0811ef
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.2

-------------------------------------------------------------------------------

** Kubernetes version**

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T21:52:18Z", GoVersion:"go1.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:23:04Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Service

apiVersion: v1
kind: Service
metadata:
  name: fabsgoblog-svc
  labels:
    app: fabsgoblog
spec:
  selector:
    app: fabsgoblog
  ports:
    - protocol: TCP
      port: 8019
      targetPort: 8019

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blog-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: "fabs.raketo.info"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: fabsgoblog-svc
            port:
              number: 8019

GET SERVICE

$ kubectl get svc
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
fabsgoblog-svc   ClusterIP   10.98.158.246    <none>        8019/TCP   70m
kubernetes       ClusterIP   10.96.0.1        <none>        443/TCP    3d1h
postgres         ClusterIP   10.106.174.191   <none>        5432/TCP   4h30m

$ kubectl describe svc fabsgoblog-svc
Name:              fabsgoblog-svc
Namespace:         default
Labels:            app=fabsgoblog
Annotations:       <none>
Selector:          app=fabsgoblog
Type:              ClusterIP
IP:                10.98.158.246
Port:              <unset>  8019/TCP
TargetPort:        8019/TCP
Endpoints:         172.179.7.27:8019
Session Affinity:  None
Events:            <none>

ERROR

$ kubectl describe ingress blog-ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Name:             blog-ingress
Namespace:        default
Address:          192.168.56.120
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host              Path  Backends
  ----              ----  --------
  fabs.raketo.info  
                    /   fabsgoblog-svc:8019   172.179.7.27:8019)
Annotations:        <none>
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  55m   nginx-ingress-controller  Ingress default/blog-ingress
  Normal  UPDATE  55m   nginx-ingress-controller  Ingress default/blog-ingress

LOGS

2020/09/02 00:38:35 [error] 821#821: *100266 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.56.120, server: fabs.raketo.info, request: "GET / HTTP/1.1", upstream: "http://172.179.7.27:8019/", host: "fabs.raketo.info:30618"
2020/09/02 00:38:35 [error] 821#821: *100266 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.56.120, server: fabs.raketo.info, request: "GET / HTTP/1.1", upstream: "http://172.179.7.27:8019/", host: "fabs.raketo.info:30618"
2020/09/02 00:38:35 [error] 821#821: *100266 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.56.120, server: fabs.raketo.info, request: "GET / HTTP/1.1", upstream: "http://172.179.7.27:8019/", host: "fabs.raketo.info:30618"
192.168.56.120 - - [02/Sep/2020:00:38:35 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36 OPR/70.0.3728.119" 481 0.007 [default-fabsgoblog-svc-8019] [] 172.179.7.27:8019, 172.179.7.27:8019, 172.179.7.27:8019 0, 0, 0 0.001, 0.001, 0.005 502, 502, 502 98802677fdd3c70a2a38f9749310489f
2020/09/02 00:38:35 [error] 821#821: *100266 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.56.120, server: fabs.raketo.info, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.179.7.27:8019/favicon.ico", host: "fabs.raketo.info:30618", referrer: "http://fabs.raketo.info:30618/"
2020/09/02 00:38:35 [error] 821#821: *100266 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.56.120, server: fabs.raketo.info, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.179.7.27:8019/favicon.ico", host: "fabs.raketo.info:30618", referrer: "http://fabs.raketo.info:30618/"
2020/09/02 00:38:35 [error] 821#821: *100266 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.56.120, server: fabs.raketo.info, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.179.7.27:8019/favicon.ico", host: "fabs.raketo.info:30618", referrer: "http://fabs.raketo.info:30618/"
192.168.56.120 - - [02/Sep/2020:00:38:35 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://fabs.raketo.info:30618/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36 OPR/70.0.3728.119" 435 0.001 [default-fabsgoblog-svc-8019] [] 172.179.7.27:8019, 172.179.7.27:8019, 172.179.7.27:8019 0, 0, 0 0.000, 0.001, 0.000 502, 502, 502 c8102b8fbf27f989ef62b12c13cda8ac

Please help,

Cheers!!!

[aledbf]

• remove nginx.ingress.kubernetes.io/rewrite-target: /. You only have one path
• the error
  2020/09/02 00:38:35 [error] 821#821: *100266 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.56.120, server: fabs.raketo.info, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.179.7.27:8019/favicon.ico", host: "fabs.raketo.info:30618", referrer: "http://fabs.raketo.info:30618/"

means the ingress controller pod cannot reach the node where fabsgoblog svc is running.
This usually means you have some networking issues with your CNI provider.

[cleanet]

The logs:

2021/12/28 06:17:41 [error] 3256#3256: *411627 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.*.*.207, server: _, request: "GET /demo HTTP/1.1", upstream: "http://10.42.0.16:80/demo", host: "10.*.*.207"
2021/12/28 06:17:42 [error] 3256#3256: *411627 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.*.*.207, server: _, request: "GET /demo HTTP/1.1", upstream: "http://10.42.0.14:80/demo", host: "10.*.*.207"
2021/12/28 06:17:43 [error] 3256#3256: *411627 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.*.*.207, server: _, request: "GET /demo HTTP/1.1", upstream: "http://10.42.0.15:80/demo", host: "10.*.*.207"
10.*.*.207 - - [28/Dec/2021:06:17:43 +0000] "GET /demo HTTP/1.1" 502 150 "-" "curl/7.61.1" 82 3.068 [ingress-nginx-nginx-service-8080] [] 10.42.0.16:80, 10.42.0.14:80, 10.42.0.15:80 0, 0, 0 1.020, 1.024, 1.024 502, 502, 502 93cf678d8d8710e02845a378cd59ed20

means that nginx is accessing at application since the endpoint 10.42.0.15:80.

This socket, is the endpoint of you service. You can see it, do it:

kubectl get endpoints -n nginx-service

In this case, is the endpoints of service nginx-service.
But seeing that throw a 502 Bad Gateway and the logs, this means that the ingress controller is trying access at service via endpoint (trying with all the endpoints of ingress controller). And the ingress controller's pod cannot access.

For test it, entry in the pod of ingress controller and checks the connection.

$ kubectl exec -it pod/ingress-nginx-controller-57ff8464d9-pvjpc -- bash
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ nc -zv 10.42.0.16 80
nc: 10.85.0.12 (10.85.0.12:8080): Host is unreachable
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ 

As we see exactly , this cannot access.

You look that IP has the service nginx-service and try access

$ kubectl describe service
$ kubectl exec -it pod/ingress-nginx-controller-57ff8464d9-pvjpc -- bash
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ nc -zv 10.43.89.106 8080
10.43.89.106 (10.43.89.106:8080) open

And as we see, the pod has access. With the ClusterIP and Port of the service.

So that a solution would be do the follow.

You must tell at Ingress, that uses the ClusterIP:port instead of use endpoints list of ingress controller.

For this you edit the Ingress resource and add the follow annotation.

nginx.ingress.kubernetes.io/service-upstream: "true"

FYI

Service Upstream¶

By default the Ingress-Nginx Controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration.

The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port.

This can be desirable for things like zero-downtime deployments . See issue #257.

Known Issues¶

If the service-upstream annotation is specified the following things should be taken into consideration:

• Sticky Sessions will not work as only round-robin load balancing is supported.
• The proxy_next_upstream directive will not have any effect meaning on error the request will not be dispatched to another upstream.

[JShuang7711]

add this annotation , still have this problem, same issue msg,
2024/10/16 00:22:39 [error] 30980#30980: *23522467 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.64.2, server: web2.study.com, request: "GET /bar HTTP/1.1", upstream: "http://10.109.222.227:80/", host: "web2.study.com:32275"
2024/10/16 00:22:42 [error] 30980#30980: *23522467 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.64.2, server: web2.study.com, request: "GET /bar HTTP/1.1", upstream: "http://10.109.222.227:80/", host: "web2.study.com:32275"
2024/10/16 00:22:45 [error] 30980#30980: *23522467 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.64.2, server: web2.study.com, request: "GET /bar HTTP/1.1", upstream: "http://10.109.222.227:80/", host: "web2.study.com:32275"