I can't see cpu and memory graph in kubernetes dashboard

[yinhua-dai]

Issue details

I can't see cpu and memory graph in kubernetes dashboard

Environment

Dashboard version: 1.5.0
Kubernetes version: 1.5.0
Operating system: centos

Steps to reproduce

start the dashboard, and there are no cpu and memory graph in the dashboard

[maciaszczykm]

Quoting README.md:

Note that for the metrics and graphs to be available you need to have Heapster running in your cluster.

[floreks]

Additionally heapster has to be installed in kube-system namespace and have heapster service that is accessible through api proxy.

[maciaszczykm]

Please reopen if it will not work after you set it up as explained.

[matjam]

This was one of many threads I came across searching for instructions on how to configure Kubernetes with the dashboard, if your instance didn't get built with it. In my case, I used Kops, so I needed to do the following:

kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.6.3.yaml

I hope this is useful for other people googling who are lost.

[arno01]

And for the RBAC-enabled K8s deployments don't forget to apply the heapster-rbac, otherwise you won't see the graphs:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml

[oz123]

I tried applying what @matjam wrote, but as of today I am getting the following errors:

Failed to pull image "gcr.io/google_containers/heapster-grafana-amd64:v4.4.3": rpc error: code = 2 desc = Error: Status 405 trying to pull repository google_containers/heapster-grafana-amd64: "v1 Registry API is disabled. If you are not explicitly using the v1 Registry API, it is possible your v2 image could not be found. Verify that your image is available, or retry with `dockerd --disable-legacy-registry`. See https://cloud.google.com/container-registry/docs/support/deprecation-notices" Error syncing pod

It seems to me that not all the components are properly synced. I cloned the heapster repository, and did git checkout c674a16f74782b326f02345486b5f9520891f395. From this commit I installed:

kubectl create -f deploy/kube-config/influxdb/influxdb.yaml
kubectl create -f deploy/kube-config/influxdb/grafana.yaml
kubectl create -f deploy/kube-config/influxdb/heapster.yaml

[floreks]

This is not related to dashboard. Adapt heapster yaml according to error message.

[arno01]

@oz123 the gcr.io/google_containers/heapster-grafana-amd64:v4.4.3 has not been released yet.
Just change it to the latest available v4.4.1
kubernetes-retired/heapster#1783 (comment)

[oz123]

@arno01 I did the steps in kubernetes-retired/heapster#1783 (comment), I am still no seeing the promised metrics. I also don't see errors ...

 kubectl get pods --all-namespaces
NAMESPACE     NAME                                                   READY     STATUS    RESTARTS   AGE
default       jenkins-1747232041-b709r                               1/1       Running   0          10m
kube-system   dns-controller-1283262501-6ccdc                        1/1       Running   0          44m
kube-system   etcd-server-events-ip-172-21-62-27.ec2.internal        1/1       Running   0          43m
kube-system   etcd-server-ip-172-21-62-27.ec2.internal               1/1       Running   0          43m
kube-system   heapster-84017538-kk39v                                1/1       Running   0          17m
kube-system   kube-apiserver-ip-172-21-62-27.ec2.internal            1/1       Running   0          43m
kube-system   kube-controller-manager-ip-172-21-62-27.ec2.internal   1/1       Running   0          43m
kube-system   kube-dns-479524115-h8tdf                               3/3       Running   0          42m
kube-system   kube-dns-479524115-ndgkr                               3/3       Running   0          44m
kube-system   kube-dns-autoscaler-1818915203-cdg7v                   1/1       Running   0          44m
kube-system   kube-proxy-ip-172-21-54-177.ec2.internal               1/1       Running   0          41m
kube-system   kube-proxy-ip-172-21-60-224.ec2.internal               1/1       Running   0          41m
kube-system   kube-proxy-ip-172-21-62-27.ec2.internal                1/1       Running   0          43m
kube-system   kube-scheduler-ip-172-21-62-27.ec2.internal            1/1       Running   0          43m
kube-system   kubernetes-dashboard-417166742-nbglz                   1/1       Running   0          35m
kube-system   monitoring-grafana-1500490092-vlkz6                    1/1       Running   0          17m
kube-system   monitoring-influxdb-1870447071-zj02j                   1/1       Running   0          17m

[floreks]

@oz123 #2306 #2304 #2180

[oz123]

The only thing that might help, is this: in the dashboard logs I see this:

2017/08/31 10:48:49 log.go:26: No metric client provided. Skipping metrics.

Is this the reason?

[floreks]

It is not. Check beginning of the logs. If heapster health check fails then it will be disabled.

[oz123]

I don't see any indication for that:

 I0831 10:30:16.670557       1 heapster.go:72] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
I0831 10:30:16.670600       1 heapster.go:73] Heapster version v1.3.0
I0831 10:30:16.670809       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0831 10:30:16.670831       1 configs.go:62] Using kubelet port 10255
I0831 10:30:16.704808       1 influxdb.go:252] created influxdb sink with options: host:monitoring-influxdb.kube-system.svc:8086 user:root db:k8s
I0831 10:30:16.704833       1 heapster.go:196] Starting with InfluxDB Sink
I0831 10:30:16.704836       1 heapster.go:196] Starting with Metric Sink
I0831 10:30:16.714705       1 heapster.go:106] Starting heapster on port 8082
I0831 10:31:05.034239       1 influxdb.go:215] Created database "k8s" on influxDB server at "monitoring-influxdb.kube-system.svc:8086" ```

[floreks]

Dashboard logs.

[oz123]

OK, seems you are right,

 Creating in-cluster Heapster client
Could not enable metric client: Health check failed: an error on the server ("Error: 'dial tcp 100.96.1.4:8082: getsockopt: connection refused'\nTrying to reach: 'http://100.96.1.4:8082/healthz'") has prevented the request from succeeding (get services heapster). Continuing. 

But how do I recover from this? Should I simply restart the dashboard?

[floreks]

You can try restarting it and check again but if it can't reach heapster at all then you have networking issue in your cluster.

[oz123]

got the metrics working!

kubectl get pod kubernetes-dashboard-417166742-nbglz -n kube-system -o yaml | kubectl replace --force -f -

[floreks]

As stated in #2306, in case dashboard and heapster are deployed together then dashboard has to be restarted after heapster scrapes some initial metrics, otherwise heapster health check will fail.