open() "/etc/ingress-controller/auth/monitoring-dashboard.passwd" failed (13: Permission denied)

[webwurst]

I am testing kubeadm/nginx-ingress-controller.yaml with gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1.

Using annotations to configure basic-auth seems to be broken:

  annotations:
    ingress.kubernetes.io/auth-secret: dashboard-basic-auth
    ingress.kubernetes.io/auth-type: basic

The following error shows up:

2017/02/02 15:06:10 [crit] 4082#4082: *57727 open() "/etc/ingress-controller/auth/monitoring-dashboard.passwd" failed (13: Permission denied), client: 127.0.0.1, server: dashboard.kube.codeformuenster.org, request: "GET / HTTP/2.0", host: "dashboard.kube.codeformuenster.org"
127.0.0.1 - [127.0.0.1] - admin [02/Feb/2017:15:06:10 +0000] "GET / HTTP/2.0" 500 706 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36" 254 0.000 [monitoring-dashboard-9090] - - - -

Here the access rights for that specific file:

root@5f40511e-8ca2-4968-95b5-aabcb56b15cd:/# ls -ld /etc
drwxr-x--- 1 root root 4096 Feb  1 23:58 /etc
root@5f40511e-8ca2-4968-95b5-aabcb56b15cd:/# ls -ld /etc/ingress-controller 
drw-r-xr-x 3 root root 4096 Feb  1 23:58 /etc/ingress-controller
root@5f40511e-8ca2-4968-95b5-aabcb56b15cd:/# ls -ld /etc/ingress-controller/auth 
drw-r-xr-x 2 root root 4096 Feb  2 00:09 /etc/ingress-controller/auth
root@5f40511e-8ca2-4968-95b5-aabcb56b15cd:/# ls -l /etc/ingress-controller/auth/monitoring-dashboard.passwd  
-rwxr-xr-x 1 root root 44 Feb  2 15:07 /etc/ingress-controller/auth/monitoring-dashboard.passwd

When setting chmod o+rx /etc this works fine. But not sure if this is a proper fix..

[bprashanth]

Slightly OT, but please add an example for this in: https://github.com/kubernetes/ingress/tree/master/examples if you have the time.

I'll need to debug to give more meaningful help, maybe @aledbf knows off the top of his head

[aledbf]

@webwurst are you mounting the /etc directory? (I cannot reproduce this error)

[webwurst]

I am using this configuration https://github.com/codeformuenster/kubernetes-deployment/blob/master/manifests/ingress/daemonset.yaml which is the same as the kubeadm example from this repo, except of the labeling. No mounting. I just restarted the Pods and verified that there is the permission error. chmod fixed it again. I just checked the docker images and there is something different: 🎉

$ docker run -ti gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 bash -c "ls -ld /etc"
drwxr-x--- 1 root root 4096 Feb  3 11:01 /etc

$ docker run -ti gcr.io/google_containers/nginx-ingress-controller:0.8.3 bash -c "ls -ld /etc"
drwxr-xr-x 1 root root 4096 Feb  3 11:01 /etc

[pstadler]

I can confirm that 0.9.0-beta.1 is currently broken, whereas basic auth with 0.8.3 works well.

In both cases I used the following template to create the ingress controller: https://github.com/kubernetes/ingress/blob/3a37607138e3bbdb83dd393241d1fdd3a6232176/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml

[andrewstuart]

https://github.com/kubernetes/ingress/blob/master/core/pkg/ingress/annotations/auth/main.go#L62

Pretty sure this TODO needs to be completed. The /etc/ingress-controller has owner root:root and permissions 0750. The nginx user/worker user (nobody) cannot traverse to the auth directory.

[Hronom]

Hi guys, when this fix will be available in new images ?

[aledbf]

@Hronom not yet. After this PR #303

[Hronom]

Thanks, am I understand correctly that I can use 0.8.3 without problems?

[evgf]

@Hronom Yes, 0.8.3 works with the basic authentication without any issues.

[Hronom]

@evgf Yeah, already switched, thanks

[pstadler]

This seems to work well with the newly released 0.9.0-beta.2