Replace kube-dns with CoreDNS

[dghubble]

• Add system:coredns ClusterRole and binding
• Annotate CoreDNS for Prometheus metrics scraping
• Remove kube-dns deployment, service, & service account
• Switch from kube-dns to CoreDNS terraform-render-bootstrap#71
• https://kubernetes.io/blog/2018/06/27/kubernetes-1.11-release-announcement/

Note: CoreDNS does not use a pod nanny. For large clusters (100+ nodes), either increase the replica count by hand or use a horizontal autoscaling addon https://kubernetes.io/docs/tasks/administer-cluster/dns-horizontal-autoscaling/

Testing

Switched AWs, GCP, bare-metal, and DO clusters to v1.11.0 with this PR to use CoreDNS.

Workloads have been unaffected across environments. I did discover an incorrect dnsPolicy: Default in one of my manifests for an application that expects to interact with headless service cluster.local pods. Most applications (including the one I found) should leave dnsPolicy unset to get the default ClusterFirst. CoreDNS seems to be in the right.

You can verify basic behaviors by running a debug pod (I used fedora).

$ cat /etc/resolv.conf
nameserver 10.3.0.10
search default.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
options ndots:5

$ dig kubernetes.default.svc.cluster.local +noall +answer

; <<>> DiG 9.11.3-RedHat-9.11.3-12.fc28 <<>> kubernetes.default.svc.cluster.local +noall +answer
;; global options: +cmd
kubernetes.default.svc.cluster.local. 5 IN A    10.3.0.1

$ dig ptr 1.0.3.10.in-addr.arpa. +noall +answer

; <<>> DiG 9.11.3-RedHat-9.11.3-12.fc28curl <<>> ptr 1.0.3.10.in-addr.arpa. +noall +answer
;; global options: +cmd
1.0.3.10.in-addr.arpa.  5       IN      PTR     kubernetes.default.svc.cluster.local.

curl -k https://kubernetes
curl -k https://kubernetes.default
curl -k https://kubernetes.default.svc.cluster.local

On all platforms, the upstream DNS resolver is inherited from the host /etc/resolv.conf. That means on AWS, you'll use AWS DNS servers. On Google, you'll use Google Cloud's DNS servers.