Getting custom metrics on GKE

[kerneljack]

Hello, I've tried the custom-metrics-metrics-apiserver and the sample-metrics-app on a 1.8.6 GKE cluster and I think there is an error I'm making with regards to authentication or RBAC perhaps, but I'm not sure what it is, hopefully someone can point me in the right direction? I tried to run the server in the custom-metrics namespace and the kube-system namespace, both of which don't seem to work.

I've got the following in the logs from the custom-metrics-apiserver

I0128 12:37:20.988281       1 logs.go:41] http: TLS handshake error from 10.x.1.4:37000: tls: first record does not look like a TLS handshake

I0128 12:37:38.700217       1 request.go:836] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:generic-garbage-collector","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true}}
I0128 12:37:38.700654       1 handler.go:150] prometheus-custom-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0128 12:37:38.701803       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (26.724675ms) 200 [[kube-controller-manager/v1.8.6 (linux/amd64) kubernetes/ee9a976/system:serviceaccount:kube-system:generic-garbage-collector] 10.191.0.14:34362]

I0128 12:43:28.775871       1 request.go:836] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/swagger.json","verb":"get"},"user":"system:aggregator","group":["system:authenticated"]},"status":{"allowed":false}}