Leader election fails when RBAC is enabled

[wallrj]

/kind bug

When I deploy Navigator using helm install contrib/charts/navigator --name navigator --namespace navigator --wait as described in the quick start guide:

• https://github.com/jetstack/navigator/tree/master/docs/quick-start

The navigator controller fails during leader election

kubectl logs deployments/nav-e2e-navigator-controller
...
ERROR: logging before flag.Parse: I1030 17:40:31.035544       1 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: navigator-controller/v1.8.2 (linux/amd64) kubernetes/$Format/leader-election" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im5hdi1lMmUtbmF2aWdhdG9yLWNvbnRyb2xsZXItdG9rZW4tenRsNDUiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibmF2LWUyZS1uYXZpZ2F0b3ItY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImI5MzM3ZmI3LWJkOTYtMTFlNy04ZWQ4LTUyNTQwMDg2NzBlYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0Om5hdi1lMmUtbmF2aWdhdG9yLWNvbnRyb2xsZXIifQ.rq5OBsceqNzikBLbjIViy-yk1A22nn-dfRuRF_MWZiFjbRwBmMe4ZACk2O06mSPb-GaDvHS6ryeAaEwXNQZB_cyIKKgxPUabcRprgTh0-Ghl6K2w4d77s2gdERb-yBgRjffaa1QGAj_n8M0MbAGOVfHPvs4x8M83QnjIDfwmkIDw0u_-GboOWS1qKyb42sU3tFu7ByoMPqvlV7VX5gXmdJWcSKBoyv7GkoNFP_1bNp_2NieCC_XgmcmGAHOMxUgawFD4idtrMr3I3ReCIoC_p_mKJyqMaQKyNbuINRQ72lqgna3ZgCg7Nlo2h8eqoFvxOvKFWnsPA5HzAeRE7jxJRA" https://10.0.0.1:443/api/v1/namespaces/kube-system/endpoints/navigator-controller
ERROR: logging before flag.Parse: I1030 17:40:31.158865       1 round_trippers.go:436] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/endpoints/navigator-controller 403 Forbidden in 123 milliseconds
ERROR: logging before flag.Parse: I1030 17:40:31.158889       1 round_trippers.go:442] Response Headers:
ERROR: logging before flag.Parse: I1030 17:40:31.158894       1 round_trippers.go:445]     Date: Mon, 30 Oct 2017 17:40:31 GMT
ERROR: logging before flag.Parse: I1030 17:40:31.158898       1 round_trippers.go:445]     Content-Type: text/plain
ERROR: logging before flag.Parse: I1030 17:40:31.158901       1 round_trippers.go:445]     X-Content-Type-Options: nosniff
ERROR: logging before flag.Parse: I1030 17:40:31.158905       1 round_trippers.go:445]     Content-Length: 118
ERROR: logging before flag.Parse: I1030 17:40:31.158928       1 request.go:836] Response Body: User "system:serviceaccount:default:nav-e2e-navigator-controller" cannot get endpoints in the namespace "kube-system".
ERROR: logging before flag.Parse: E1030 17:40:31.158961       1 leaderelection.go:224] error retrieving resource lock kube-system/navigator-controller: User "system:serviceaccount:default:nav-e2e-navigator-controller" cannot get endpoints in the namespace "kube-system". (get endpoints navigator-controller)
ERROR: logging before flag.Parse: I1030 17:40:31.158970       1 leaderelection.go:180] failed to acquire lease kube-system/navigator-controller

Environment:

• Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-10-06T20:53:14Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration**:

minikube version: v0.22.3

[munnerz]

Yep, we'll need to set --leader-election-namespace appropriately too (or grant navigator access to the appropriate configmap in the kube-system namespace, which is the default) if you want to heavily restrict the permissions that Navigator runs with.

[wallrj]

I found the following PRs in service-catalog, which might be a good example to follow:

• Add leader election namespace configuration kubernetes-retired/service-catalog#920
• Adding rbac definition for v1 api endpoint. kubernetes-retired/service-catalog#1284
• RBAC setup behind the aggregator. kubernetes-retired/service-catalog#936

[wallrj]

Fixed by: #70