Fix kube-dns RBAC issues

[wallrj]

Allow kube-dns and other kube-system services full access to the API.
See:

• RBAC is broken kubernetes/minikube#1734
• Enable RBAC by default kubernetes/minikube#1722

Fixes: #107

Release note:

NONE

[munnerz]

Hmm - are you sure that this change actually fixes anything?

By the looks of what the rules previously were, I don't think it will. We previously granted kube-system:default the system:kube-dns role, which (I'd imagine) is sufficient for kube-dns to run properly.

The kube-dns service account that's been created above is actually not used from what I can see.

[munnerz]

The only difference between before and after, is that we've removed a service account that's unused ('kube-dns'), and granted kube-dns full access to the API instead of using the kube-dns ClusterRole (which in a default k8s cluster looks like the following):

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2017-10-23T15:15:15Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-dns
  resourceVersion: "57"
  selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterroles/system%3Akube-dns
  uid: f9194879-b804-11e7-a76b-42010a8401a1
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  verbs:
  - list
  - watch

[wallrj]

Here's an extract from the logs before applying any RBAC fixes.

kubectl logs deployment/kube-dns --namespace kube-system kubedns
I1108 13:34:41.182340       1 dns.go:48] version: 1.14.4-2-g5584e04
I1108 13:34:41.183027       1 server.go:66] Using configuration read from ConfigMap: kube-system:kube-dns
I1108 13:34:41.183070       1 server.go:113] FLAG: --alsologtostderr="false"
I1108 13:34:41.183085       1 server.go:113] FLAG: --config-dir=""
I1108 13:34:41.183091       1 server.go:113] FLAG: --config-map="kube-dns"
I1108 13:34:41.183095       1 server.go:113] FLAG: --config-map-namespace="kube-system"
I1108 13:34:41.183098       1 server.go:113] FLAG: --config-period="10s"
I1108 13:34:41.183103       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I1108 13:34:41.183107       1 server.go:113] FLAG: --dns-port="10053"
I1108 13:34:41.183113       1 server.go:113] FLAG: --domain="cluster.local."
I1108 13:34:41.183124       1 server.go:113] FLAG: --federations=""
I1108 13:34:41.183129       1 server.go:113] FLAG: --healthz-port="8081"
I1108 13:34:41.183133       1 server.go:113] FLAG: --initial-sync-timeout="1m0s"
I1108 13:34:41.183136       1 server.go:113] FLAG: --kube-master-url=""
I1108 13:34:41.183147       1 server.go:113] FLAG: --kubecfg-file=""
I1108 13:34:41.183150       1 server.go:113] FLAG: --log-backtrace-at=":0"
I1108 13:34:41.183157       1 server.go:113] FLAG: --log-dir=""
I1108 13:34:41.183161       1 server.go:113] FLAG: --log-flush-frequency="5s"
I1108 13:34:41.183164       1 server.go:113] FLAG: --logtostderr="true"
I1108 13:34:41.183168       1 server.go:113] FLAG: --nameservers=""
I1108 13:34:41.183171       1 server.go:113] FLAG: --stderrthreshold="2"
I1108 13:34:41.183174       1 server.go:113] FLAG: --v="2"
I1108 13:34:41.183177       1 server.go:113] FLAG: --version="false"
I1108 13:34:41.183183       1 server.go:113] FLAG: --vmodule=""
I1108 13:34:41.183257       1 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
I1108 13:34:41.183483       1 server.go:198] Skydns metrics enabled (/metrics:10055)
I1108 13:34:41.183491       1 dns.go:147] Starting endpointsController
I1108 13:34:41.183494       1 dns.go:150] Starting serviceController
I1108 13:34:41.183577       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I1108 13:34:41.183588       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
E1108 13:34:41.195392       1 sync_configmap.go:86] Error getting ConfigMap kube-system:kube-dns err: configmaps "kube-dns" is forbidden: User "system:serviceaccount:kube-system:default" cannot get configmaps in the namespace "kube-system"
E1108 13:34:41.195420       1 dns.go:183] Error getting initial ConfigMap: configmaps "kube-dns" is forbidden: User "system:serviceaccount:kube-system:default" cannot get configmaps in the namespace "kube-system", starting with default values
E1108 13:34:41.196072       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list services at the cluster scope
E1108 13:34:41.196147       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:default" cannot list endpoints at the cluster scope
E1108 13:34:41.196762       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"

It seemed to me that cannot list configmaps might be the problem and was missing from the original binding.

[munnerz]

👍 cool, sounds about right then! I wonder why that wasn't in the Role already 🤔

[munnerz]

/lgtm
/approve

[jetstack-bot]

/lgtm cancel //PR changed after LGTM, removing LGTM. @munnerz @wallrj

[wallrj]

There was an unexpected error in the logs which I have now added to the list:

• log.txt

E1108 14:18:37.610718       1 reflector.go:205] github.com/jetstack/navigator/pkg/client/informers/externalversions/factory.go:68: Failed to list *v1alpha1.Pilot: an error on the server ("Error: 'dial tcp 10.0.0.233:443: getsockopt: connection refused'\nTrying to reach: 'https://10.0.0.233:443/apis/navigator.jetstack.io/v1alpha1/pilots?resourceVersion=0'") has prevented the request from succeeding (get pilots.navigator.jetstack.io)

[munnerz]

/lgtm

[jetstack-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: munnerz

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• OWNERS [munnerz]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[jetstack-bot]

Automatic merge from submit-queue.