502 and "Connection refused while connecting to upstream"

[edtshuma]

I have deployed a service and to be accessed via ingress but it appears the ingress is not working.

When I try a GET (from Postman) to my endpoint I get 502 Bad Gateway Error. I have managed to inspect the logs for the ingress controller and this is what I am getting :

kubectl logs ingress-nginx-controller-57cb5gh78uj-abcfrt > app.log

192.198.123.115 - - [23/Jun/2022:15:14:50 +0000] "GET /bpi/weatherforecast HTTP/1.1" 308 164 "-" "Thunder Client (https://www.thunderclient.com)" 196 0.000 [labs-registrations-clusterip-80] [] - - - - fad39893c39b03d026ce9eed6d0f5b1a 2022/06/23 15:14:51 [error] 1736#1736: *37187197 connect() failed (111: Connection refused) while connecting to upstream, client: 192.198.123.193, server: dev.labs.abcservice.zw, request: "GET /bpi/weatherforecast HTTP/1.1", upstream: "http://10.244.111.111:80/bpi/weatherforecast", host: "dev.labs.abcservice.zw" 2022/06/23 15:14:51 [error] 1736#1736: *37187197 connect() failed (111: Connection refused) while connecting to upstream, client: 192.198.123.193, server: dev.labs.abcservice.zw, request: "GET /bpi/weatherforecast HTTP/1.1", upstream: "http://10.244.111.111:80/bpi/weatherforecast", host: "dev.labs.abcservice.zw" 2022/06/23 15:14:51 [error] 1736#1736: *37187197 connect() failed (111: Connection refused) while connecting to upstream, client: 192.198.123.193, server: dev.labs.abcservice.zw, request: "GET /bpi/weatherforecast HTTP/1.1", upstream: "http://10.244.111.111:80/bpi/weatherforecast", host: "dev.labs.abcservice.zw"

However when I port-forward the cluster-ip specified in the ingress resource I am able to query the GET endpoint successfully (in browser) :

kubectl port-forward svc/registrations-clusterip 80

The ingress object is defined as below :

`apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: registrations
namespace: labs
spec:
rules:

• host: dev.labs.abcservice.zw
  http:
  paths:
  • path: /bpi/weatherforecast
    pathType: Prefix
    backend:
    service:
    name: registrations-clusterip
    port:
    number: 80`

And the deployment is defined as :
`apiVersion: apps/v1
kind: Deployment
metadata:
name: registrations
namespace: labs
spec:
replicas: 1
selector:
matchLabels:
app: registrations
template:
metadata:
labels:
app: registrations
spec:
containers:
- name: registrations
image: registry.zw.abccompany.com/labs/devops/labs:dev-latest
imagePullPolicy: Always
ports:
- containerPort: 80
resources:
limits:
memory: 500Mi
requests:
cpu: 100m
memory: 128Mi
imagePullSecrets:
- name: regcred

apiVersion: v1
kind: Service
metadata:
name: registrations-clusterip
namespace: labs
labels:
app: registrations
spec:
type: ClusterIP
selector:
app: registrations
ports:

• port: 80
  targetPort: 80`

My kubeconfig for the cluster is as below. I am unsure whether I should be adding tls since my cluster is by default exposed on https or rather I should be able to invoke my service even if its running on http:

`apiVersion: v1
clusters:

• cluster:
  certificate-authority-data: Lxxxxxxxtxxxxxx
  server: https://165.175.185.77:6443
  name: labsdev-15-155
  contexts:
• context:
  cluster: labsdev-15-155`

As a check I have tried to explicitly specify the protocol in the ingress object as below:
`
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: registrations
namespace: labs
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
rules:

• host: dev.orion.foxservice.cz
  http:
  paths:
  • path: /bpi/weatherforecast
    pathType: Prefix
    backend:
    service:
    name: registrations-clusterip
    port:
    number: 80
    `
    but I am still getting same error.

NB: Ingress and the domain are working as expected for some other services deployed on the cluster. Only difference is that all the other services have TLS secret enabled in the ingress resource

What am I missing?

[k8s-ci-robot]

@ed87: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

Post info as per issue template.
Then re-open the issue.

[longwuyuan]

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[edtshuma]

Where is the issue template ?

[longwuyuan]

It's presented automatically on new issue creation. Also, better to ask support in kubernetes.slack.io . Thanks, ; Long

…

On Fri, 24 Jun, 2022, 11:22 AM Ed Tshuma, ***@***.***> wrote: Where is the issue template ? — Reply to this email directly, view it on GitHub <#8731 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWXTP46INNQEBMEZJZTVQVEJLANCNFSM5ZU3DMFQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>