ingress-nginx on bare-metal with hostnetwork

[staticdev]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.25.0

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Environment: bare-metal Ubuntu 18.04.2 LTS

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): NAME="Ubuntu"
  VERSION="18.04.2 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.2 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic
• Kernel (e.g. uname -a): Linux cherokee 4.15.0-54-generic Fix "invalid port in upstream" on nginx controller #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: helm
• Others: -

What happened:
I am trying to use hostnetwork configuration with ingress-nginx on a bare-metal machine. I tried installing the controller with helm with configurations:

1. --set controller.hostNetwork=true

This (incorrectly) creates a LoadBalancer service that get pending forever since k8s does not support that on bare-metal.

2. --set controller.hostNetwork=true,controller.service.type=""

This chances the behavior of the service to be a ClusterIP, but also does not work. All services get timeout from outside the cluster.

3. --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet

This changes the deployment into a DeamonSet but has the same behavior as configuration 2).

What you expected to happen:

Get a working url on the browser.

How to reproduce it (as minimally and precisely as possible):

1. Install ingress-nginxs

   helm install stable/nginx-ingress \
     --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet

2. Create a deployment:

apiVersion: v1
kind: Service
metadata:
  labels:
    name: face
  name: face
spec:
  ports:
  - port: 80
    targetPort: 5000
  selector:
    name: face
status:
  loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: face-deployment
  labels:
    name: face
spec:
  replicas: 1
  selector:
    matchLabels:
      name: face
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: face
    spec:
      containers:
        image: myimage/face:1.0.0
        name: face
        ports:
        - containerPort: 5000
        resources: {}
      restartPolicy: Always
status: {}

3. Create an ingress object:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: staging-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"    
spec:
  rules:
  - host: my.url
    http:
      paths:
      - path: /
        backend:
          serviceName: face
          servicePort: 80

4. Try to enter a URL that the application responds to like my.url/home.

Anything else we need to know:

The configmap has no data:

NAME                              DATA   AGE
ingress-controller-leader-nginx   0      10m

The deamonset/deployment for the ingress-controller has the following logs:
kubectl logs daemonset.apps/insipid-chicken-nginx-ingress-controller

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.25.0
  Build:      git-1387f7b7e
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

I0711 21:02:07.551136       6 flags.go:192] Watching for Ingress class: nginx
W0711 21:02:07.551370       6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: openresty/1.15.8.1
W0711 21:02:07.554921       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0711 21:02:07.555173       6 main.go:183] Creating API client for https://10.96.0.1:443
I0711 21:02:07.561273       6 main.go:227] Running in Kubernetes cluster version v1.15 (v1.15.0) - git (clean) commit e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529 - platform linux/amd64
I0711 21:02:07.564658       6 main.go:91] Validated default/yellow-panda-nginx-ingress-default-backend as the default backend.
I0711 21:02:07.708843       6 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
E0711 21:02:07.709441       6 main.go:131] v1.15.0
W0711 21:02:07.755191       6 store.go:616] Unexpected error reading configuration configmap: configmaps "yellow-panda-nginx-ingress-controller" not found
I0711 21:02:07.762374       6 nginx.go:275] Starting NGINX Ingress controller
I0711 21:02:08.869585       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"a4205933-5d69-49b8-9f70-f2cbd3241fe2", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"704", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/staging-ingress
I0711 21:02:08.963055       6 nginx.go:319] Starting NGINX process
I0711 21:02:08.963082       6 leaderelection.go:235] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
I0711 21:02:08.964948       6 controller.go:133] Configuration changes detected, backend reload required.
I0711 21:02:08.986909       6 leaderelection.go:245] successfully acquired lease default/ingress-controller-leader-nginx
I0711 21:02:08.987162       6 status.go:86] new leader elected: yellow-panda-nginx-ingress-controller-nll6l
I0711 21:02:09.039137       6 status.go:296] updating Ingress default/staging-ingress status from [] to [{ }]
I0711 21:02:09.042512       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"a4205933-5d69-49b8-9f70-f2cbd3241fe2", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"774", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/staging-ingress
I0711 21:02:09.053035       6 controller.go:149] Backend successfully reloaded.
I0711 21:02:09.053078       6 controller.go:158] Initial sync, sleeping for 1 second.
[11/Jul/2019:21:02:10 +0000]TCP200000.000

[aledbf]

@staticdev this seems an issue with your apps.

If I run

kubectl  apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/http-svc.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: staging-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"    
spec:
  rules:
  - host: my.url
    http:
      paths:
      - path: /
        backend:
          serviceName: http-svc
          servicePort: 8080
      - path: /test
        backend:
          serviceName: http-svc
          servicePort: 8080

This works as expected

[staticdev]

@aledbf How did you install ingress-nginx?

[aledbf]

@staticdev I followed your instructions

[staticdev]

@aledbf anything to do with port numbers or k8s version? I am sure my apps are working.. They are already in production using docker.

[aledbf]

face-deployment-5dd988b88f-8hsjj                             0/1     ImagePullBackOff   0          17m

Also, there is no test deployment or svc in the code you initially posted (for that reason the error in the log is right)

[staticdev]

@aledbf Sorry, I updated it to not have this test service and also updated the description, logs and version of ingress-nginx to 0.25.0.

Still.. nothing works!!

[aledbf]

Still.. nothing works!!

Can you be more precise? What are you testing and how?
Even with the errors, you see in the ingress controller pod log you should, at least, get a 404 error.

[staticdev]

@aledbf I enter Firefox and try the url my.url/. It gives:

Unable to connect

Firefox can’t establish a connection to the server at my.url.

The site could be temporarily unavailable or too busy. Try again in a few moments.
If you are unable to load any pages, check your computer’s network connection.
If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.

Also, my pod is running:

NAME READY STATUS RESTARTS AGE
pod/face-deployment-76b8d77dcf-c9hmn 1/1 Running 0 25m

And no requests where routed to it from the logs.

Could it be something to do with my /etc/hosts?

I have:

127.0.0.1	localhost	cherokee
127.0.1.1	my.url

[aledbf]

Could it be something to do with my /etc/hosts?

Let's start checking the ingress controller it's working:

1. kubectl exec <ingress controller pod> -- curl localhost -H 'Host: my.url'

If that works it means the ingress controller is not the issue.

2. ssh to one of the nodes

curl localhost -H 'Host: my.url' should work. If not, it means you have some issues with the configuration of the helm chart.

3. curl <one IP of the nodes> -H 'Host: my.url' should work. If not, it means you have some connectivity or firewall issue between your workstation and the node.

[staticdev]

Note: I did all the tests putting the host domain instead of my.url (I can't disclose the domain name). But, my app does not respond to /, it responds only to /something

1. I get 404

kubectl exec pod/yellow-panda-nginx-ingress-controller-nll6l -- curl localhost -H 'Host: my.url'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    21  100    21    0     0  21000      0 --:--:-- --:--:-- --:--:-- 21000
default backend - 404

2. Same 404

curl localhost -H 'Host: my.url'
default backend - 404

3. Doesn't work either (I tried from another machine in the same network)

curl IP -H 'Host: my.url'
curl: (7) Failed to connect to IP port 80: Connection refused

If I do the test 1) with the complete my.url/something, I get the result:

kubectl exec pod/yellow-panda-nginx-ingress-controller-nll6l -- curl localhost -H 'Host: my.url/something'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   163  100   163    0     0  40750      0 --:--:-- --:--:-- --:--:-- 40750
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>

[aledbf]

This is the correct syntax: curl localhost/something -H 'Host: my.url'

[staticdev]

By calling "curl localhost/something -H 'Host: my.url'" I got the response right! The rendered HTML.

[aledbf]

3. Doesn't work either (I tried from another machine in the same network)
   curl IP -H 'Host: my.url'
   curl: (7) Failed to connect to IP port 80: Connection refused

(from my previous comment) .... should work. If not, it means you have some connectivity or firewall issue between your workstation and the node/cluster.
This is not an issue with the ingress controller.

[staticdev]

@aledbf I see, but why can't I access my.url/something from the webbrowser from the node machine then?

[aledbf]

I see, but why can't I access my.url/something from the webbrowser from the node machine then?

DNS/proxy issues?

curl --resolve my.url:80:127.0.0.1 http://my.url/something

[staticdev]

I just called people from the firewall.. the port 80 is free for the internet.

The test I did:
sudo docker run --name face -d -p 80:5000 myimage/face:1.0.0

I can access is from the node machine and from the browser in the url: my.url/something.
I also can access the same url from the internet.

[aledbf]

Closing. Clearly, this is an issue in your infrastructure, not the ingress controller.