Host is unreachable) while connecting to upstream ... 502 Bad Gateway

[pchar]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

root@k0s:~ # uname -srvmo; cat /etc/os-release || lsb_release -a
Linux 4.18.0-408.el8.x86_64 #1 SMP Mon Jul 18 17:42:52 UTC 2022 x86_64 GNU/Linux
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Version

v1.25.3+k0s.0

Sysinfo

`k0s sysinfo`

```text Machine ID: "28f3b6ae1bad8bb06102883b572dc716c935c5d783d8a4891e5d511b5bd716dc" (from machine) (pass) Total memory: 15.6 GiB (pass) Disk space available for /var/lib/k0s: 7.9 GiB (pass) Operating system: Linux (pass) Linux kernel release: 4.18.0-408.el8.x86_64 (pass) Max. file descriptors per process: current: 262144 / max: 262144 (pass) Executable in path: modprobe: /usr/sbin/modprobe (pass) /proc file system: mounted (0x9fa0) (pass) Control Groups: version 1 (pass) cgroup controller "cpu": available (pass) cgroup controller "cpuacct": available (pass) cgroup controller "cpuset": available (pass) cgroup controller "memory": available (pass) cgroup controller "devices": available (pass) cgroup controller "freezer": available (pass) cgroup controller "pids": available (pass) cgroup controller "hugetlb": available (pass) cgroup controller "blkio": available (pass) CONFIG_CGROUPS: Control Group support: built-in (pass) CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass) CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass) CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass) CONFIG_CPUSETS: Cpuset support: built-in (pass) CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass) CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass) CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass) CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass) CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass) CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass) CONFIG_BLK_CGROUP: Block IO controller: built-in (pass) CONFIG_NAMESPACES: Namespaces support: built-in (pass) CONFIG_UTS_NS: UTS namespace: built-in (pass) CONFIG_IPC_NS: IPC namespace: built-in (pass) CONFIG_PID_NS: PID namespace: built-in (pass) CONFIG_NET_NS: Network namespace: built-in (pass) CONFIG_NET: Networking support: built-in (pass) CONFIG_INET: TCP/IP networking: built-in (pass) CONFIG_IPV6: The IPv6 protocol: built-in (pass) CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass) CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass) CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass) CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass) CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass) CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass) CONFIG_NETFILTER_XT_SET: set target and match support: module (pass) CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: unknown (warning: also tried CONFIG_IP_NF_TARGET_MASQUERADE, CONFIG_IP6_NF_TARGET_MASQUERADE) CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass) CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass) CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass) CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass) CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass) CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass) CONFIG_NETFILTER_NETLINK: module (pass) CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass) CONFIG_NF_NAT: module (pass) CONFIG_IP_SET: IP set support: module (pass) CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass) CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass) CONFIG_IP_VS: IP virtual server support: module (pass) CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass) CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning) CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass) CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning) CONFIG_IP_NF_IPTABLES: IP tables support: module (pass) CONFIG_IP_NF_FILTER: Packet filtering: module (pass) CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass) CONFIG_IP_NF_NAT: iptables NAT support: module (pass) CONFIG_IP_NF_MANGLE: Packet mangling: module (pass) CONFIG_NF_DEFRAG_IPV4: module (pass) CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning) CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning) CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass) CONFIG_IP6_NF_FILTER: Packet filtering: module (pass) CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass) CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass) CONFIG_NF_DEFRAG_IPV6: module (pass) CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass) CONFIG_LLC: module (pass) CONFIG_STP: module (pass) CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass) CONFIG_PROC_FS: /proc file system support: built-in (pass) ```

What happened?

I follow the instructions on the web site Quick Start Guide installing a sigle node with
k0s install controller --single
and then trying to install Installing NGINX Ingress Controller but the verification fails with

kubectl get services -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.105.175.42 80:30373/TCP,443:31041/TCP 13h
ingress-nginx-controller-admission ClusterIP 10.104.121.254 443/TCP 13h

curl 10.105.175.42 -H 'Host: web.example.com'

<title>502 Bad Gateway</title>

502 Bad Gateway

---

nginx

the ingress logs shows:

2022/11/08 09:09:55 [error] 249#249: *207180 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.244.0.1, server: web.example.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.62:80/", host: "web.example.com"
10.244.0.1 - - [08/Nov/2022:09:09:55 +0000] "GET / HTTP/1.1" 502 150 "-" "curl/7.61.1" 79 3.101 [web-web-server-service-5000] [] 10.244.0.62:80, 10.244.0.62:80, 10.244.0.62:80 0, 0, 0 1.054, 1.024, 1.024 502, 502, 502 9cba7b333d382a831667fc0d2e15b960

Steps to reproduce

1. install a fresh centos 8
   2.install k0s install controller --single
   3.follow the guide to install ingress controller

Expected behavior

i should reach the pod deployed in the web namespace

Actual behavior

502 Bad Gateway

Screenshots and logs

---

NGINX Ingress controller
Release: v1.4.0
Build: 50be2bf95fd1ef480420e2aa1d6c5c7c138c95ea
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.10

---

W1107 19:27:14.413100 7 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1107 19:27:14.413197 7 main.go:209] "Creating API client" host="https://10.96.0.1:443"
I1107 19:27:14.447560 7 main.go:253] "Running in Kubernetes cluster" major="1" minor="25" git="v1.25.3+k0s" state="clean" commit="434bfd82814af038ad94d62ebe59b133fcb50506" platform="linux/amd64"
I1107 19:27:14.638930 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I1107 19:27:14.647332 7 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1107 19:27:14.659465 7 nginx.go:260] "Starting NGINX Ingress controller"
I1107 19:27:14.669529 7 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"06a10268-d844-43d2-a9f3-2e3bce73273f", APIVersion:"v1", ResourceVersion:"82445", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I1107 19:27:15.762779 7 store.go:430] "Found valid IngressClass" ingress="web/web-server-ingress" ingressclass="nginx"
I1107 19:27:15.763137 7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"web", Name:"web-server-ingress", UID:"11a9094c-6ad0-4ee9-a084-b4593b583ec9", APIVersion:"networking.k8s.io/v1", ResourceVersion:"82279", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1107 19:27:15.861684 7 nginx.go:303] "Starting NGINX process"
I1107 19:27:15.870719 7 nginx.go:323] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I1107 19:27:15.870795 7 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I1107 19:27:15.871394 7 controller.go:168] "Configuration changes detected, backend reload required"
I1107 19:27:15.916212 7 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader
I1107 19:27:15.916338 7 status.go:84] "New leader elected" identity="ingress-nginx-controller-58d76857b9-b66w6"
I1107 19:27:15.932022 7 controller.go:185] "Backend successfully reloaded"
I1107 19:27:15.932069 7 controller.go:196] "Initial sync, sleeping for 1 second"
.....
I1108 09:07:24.822132 7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-58d76857b9-b66w6", UID:"9b10fcb8-f8d2-42cd-b942-4b1fcdb6ad83", APIVersion:"v1", ResourceVersion:"82474", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1108 09:08:15.923627 7 status.go:299] "updating Ingress status" namespace="web" ingress="web-server-ingress" currentValue=[] newValue=[{IP:192.168.101.125 Hostname: Ports:[]}]
I1108 09:08:15.928525 7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"web", Name:"web-server-ingress", UID:"e1e7d012-df00-49ba-b58b-bdd38e549a18", APIVersion:"networking.k8s.io/v1", ResourceVersion:"109116", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
2022/11/08 09:09:53 [error] 249#249: *207180 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.244.0.1, server: web.example.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.62:80/", host: "web.example.com"
2022/11/08 09:09:54 [error] 249#249: *207180 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.244.0.1, server: web.example.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.62:80/", host: "web.example.com"
2022/11/08 09:09:55 [error] 249#249: *207180 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.244.0.1, server: web.example.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.62:80/", host: "web.example.com"
10.244.0.1 - - [08/Nov/2022:09:09:55 +0000] "GET / HTTP/1.1" 502 150 "-" "curl/7.61.1" 79 3.101 [web-web-server-service-5000] [] 10.244.0.62:80, 10.244.0.62:80, 10.244.0.62:80 0, 0, 0 1.054, 1.024, 1.024 502, 502, 502 9cba7b333d382a831667fc0d2e15b960

Additional context

No response

[pchar]

Same issue starting from a fresh clone of Centos 9

[jnummelin]

207180 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.244.0.1, server: web.example.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.62:80/", host: "web.example.com"

I'd assume 10.244.0.62 is the pod IP for the "upstream" service for web.example.com? So basically this would mean that the nginx-ingress pod is not able to connect to that service. Did you check that all the pods for that are running properly and the service has the pod IPs as endpoints. If you were using the example yaml from the docs, check the following:
k0s kc -n web get pod,svc,ep -o wide

From the node, can you connect to the pod IP directly? For example with curl 10.244.0.62.

Do you have firewall running on the node? Could it be blocking the pod networking?

I tried on Centos9 too:

[root@centos-4gb-hel1-1 ~]# uname -srvmo; cat /etc/os-release || lsb_release -a
Linux 5.14.0-163.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 8 13:52:29 UTC 2022 x86_64 GNU/Linux
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

With this everything worked as expected. Mine has newer kernel though...

[pchar]

Thanks!!! Definitely was the firewall i've passed two days trying to understand what's was going on but I did not think about the firewall!

May I suggest in the System Requirement or whatever other section of the guide to write down a note about the firewall status what should be or need to be in order to have a clean installation of k0s ...
Thanks again !

[pchar]

was the firewall

[jnummelin]

May I suggest in the System Requirement or whatever other section of the guide to write down a note about the firewall status what should be or need to be in order to have a clean installation of k0s

We can certainly add some notes on generic firewall stuff, but pretty impossible for us to maintain a full list of rules etc. as there's wide variety of firewall tools out there.

[cleanet]

The logs:

2021/12/28 06:17:41 [error] 3256#3256: *411627 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.*.*.207, server: _, request: "GET /demo HTTP/1.1", upstream: "http://10.42.0.16:80/demo", host: "10.*.*.207"
2021/12/28 06:17:42 [error] 3256#3256: *411627 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.*.*.207, server: _, request: "GET /demo HTTP/1.1", upstream: "http://10.42.0.14:80/demo", host: "10.*.*.207"
2021/12/28 06:17:43 [error] 3256#3256: *411627 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.*.*.207, server: _, request: "GET /demo HTTP/1.1", upstream: "http://10.42.0.15:80/demo", host: "10.*.*.207"
10.*.*.207 - - [28/Dec/2021:06:17:43 +0000] "GET /demo HTTP/1.1" 502 150 "-" "curl/7.61.1" 82 3.068 [ingress-nginx-nginx-service-8080] [] 10.42.0.16:80, 10.42.0.14:80, 10.42.0.15:80 0, 0, 0 1.020, 1.024, 1.024 502, 502, 502 93cf678d8d8710e02845a378cd59ed20

means that nginx is accessing at application since the endpoint 10.42.0.15:80.

This socket, is the endpoint of you service. You can see it, do it:

kubectl get endpoints -n nginx-service

In this case, is the endpoints of service nginx-service.
But seeing that throw a 502 Bad Gateway and the logs, this means that the ingress controller is trying access at service via endpoint (trying with all the endpoints of ingress controller). And the ingress controller's pod cannot access.

For test it, entry in the pod of ingress controller and checks the connection.

$ kubectl exec -it pod/ingress-nginx-controller-57ff8464d9-pvjpc -- bash
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ nc -zv 10.42.0.16 80
nc: 10.85.0.12 (10.85.0.12:8080): Host is unreachable
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ 

As we see exactly , this cannot access.

You look that IP has the service nginx-service and try access

$ kubectl describe service
$ kubectl exec -it pod/ingress-nginx-controller-57ff8464d9-pvjpc -- bash
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ nc -zv 10.43.89.106 8080
10.43.89.106 (10.43.89.106:8080) open

And as we see, the pod has access. With the ClusterIP and Port of the service.

So that a solution would be do the follow.

You must tell at Ingress, that uses the ClusterIP:port instead of use endpoints list of ingress controller.

For this you edit the Ingress resource and add the follow annotation.

nginx.ingress.kubernetes.io/service-upstream: "true"

FYI

Service Upstream¶

By default the Ingress-Nginx Controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration.

The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port.

This can be desirable for things like zero-downtime deployments . See issue #257.

Known Issues¶

If the service-upstream annotation is specified the following things should be taken into consideration:

• Sticky Sessions will not work as only round-robin load balancing is supported.
• The proxy_next_upstream directive will not have any effect meaning on error the request will not be dispatched to another upstream.