Heapster does not work any more out of the box

[saschagrunert]

If I am running 1.6.2 the logs give me the output:

[restful] 2017/07/25 15:49:13 log.go:26: No metric client provided. Skipping metrics.
[restful] 2017/07/25 15:49:13 log.go:26: No metric client provided. Skipping metrics.

And no metrics will be shown on the dashboard pages. I am using K8S 1.7.1 with the following heapster image config:

gcr.io/google_containers/heapster-grafana-amd64:v4.4.1
gcr.io/google_containers/heapster-amd64:v1.4.0
gcr.io/google_containers/heapster-influxdb-amd64:v1.1.1

Is there any configuration missing within the yml file here?

[floreks]

Something had to change on either kubernetes or heapster side because we haven't touched it for quite some time and it just stopped working. The actual reason is listed on backend start: Could not enable metric client: Health check failed: the server could not find the requested resource (get services heapster). Continuing.. We'll investigate that and fix ASAP. Probably roll out new release because this functionality is very important.

[floreks]

As a workaround for now you can pass heapster-host argument to dashboard container in the yaml file. Direct connection still works. There are only problems with using in-cluster configuration.

[floreks]

Take a look at #2181. You can use these changes to make it work for now. It will be pretty much the same as in-cluster config.

Sent from my Google Nexus 5X using FastHub

[saschagrunert]

Alright, thank you!

[miramar-labs]

what is the solution exactly?

[rbjorklin]

@miramar-labs this is the workaround they mention: https://github.com/coreos/coreos-kubernetes/blob/3e0986915c1e6123c409aab7418656cb83e89ab5/multi-node/generic/controller-install.sh#L822-L830

[floreks]

It is not required anymore in 1.6.3 version. In-cluster config works fine again.

[rbjorklin]

@floreks I tried this 15 minutes ago with Kubernetes 1.7.4 and Dashboard 1.6.3 and it didn't work without that flag.

[floreks]

It usually means networking/cluster issue. You can check dashbord logs. It prints initial heapster health check result.

[rbjorklin]

@floreks I'm not sure this is the relevant snippet but here goes:

[restful] 2017/08/24 12:19:51 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
[restful] 2017/08/24 12:19:51 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
[restful] 2017/08/24 12:19:51 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
[2017-08-24T12:19:51Z] Outcoming response to 127.0.0.1:45428 with 200 status code                                                                                                                                
[2017-08-24T12:19:55Z] Incoming HTTP/1.1 GET /api/v1/replicaset/kube-system/heapster-v1.4.1-2524717099?filterBy=&itemsPerPage=15&page=1&sortBy=d,creationTimestamp request from 127.0.0.1:45428: {}              
Getting details of heapster-v1.4.1-2524717099 service in kube-system namespace                                                                                                                                   
Getting replication controller heapster-v1.4.1-2524717099 pods in namespace kube-system                                                                                                                          
Getting pod metrics                                                                                                                                                                                              
[restful] 2017/08/24 12:19:55 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
[restful] 2017/08/24 12:19:55 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
[2017-08-24T12:19:55Z] Outcoming response to 127.0.0.1:45428 with 200 status code                                                                                                                                
[2017-08-24T12:19:58Z] Incoming HTTP/1.1 GET /api/v1/pod/kube-system/heapster-v1.4.1-2524717099-qv53l request from 127.0.0.1:45428: {}                                                                           
Getting details of heapster-v1.4.1-2524717099-qv53l pod in kube-system namespace                                                                                                                                 
[restful] 2017/08/24 12:19:58 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
Found 10 events related to heapster-v1.4.1-2524717099-qv53l pod in kube-system namespace                                                                                                                         
[2017-08-24T12:19:58Z] Outcoming response to 127.0.0.1:45428 with 200 status code                                                                                                                                
[2017-08-24T12:20:01Z] Incoming HTTP/1.1 GET /api/v1/replicaset/kube-system/kubernetes-dashboard-921590658?filterBy=&itemsPerPage=15&page=1&sortBy=d,creationTimestamp request from 127.0.0.1:45428: {}          
Getting details of kubernetes-dashboard-921590658 service in kube-system namespace                                                                                                                               
Getting replication controller kubernetes-dashboard-921590658 pods in namespace kube-system                                                                                                                      
Getting pod metrics                                                                                                                                                                                              
[restful] 2017/08/24 12:20:01 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
[restful] 2017/08/24 12:20:01 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
[2017-08-24T12:20:01Z] Outcoming response to 127.0.0.1:45428 with 200 status code                                                                                                                                
[2017-08-24T12:20:03Z] Incoming HTTP/1.1 GET /api/v1/pod/kube-system/kubernetes-dashboard-921590658-0wsrt request from 127.0.0.1:45428: {}                                                                       
Getting details of kubernetes-dashboard-921590658-0wsrt pod in kube-system namespace                                                                                                                             
[restful] 2017/08/24 12:20:04 log.go:26: No metric client provided. Skipping metrics.                                                                                                                            
Found 7 events related to kubernetes-dashboard-921590658-0wsrt pod in kube-system namespace                                                                                                                      
[2017-08-24T12:20:04Z] Outcoming response to 127.0.0.1:45428 with 200 status code                                                                                                                                
Getting application global configuration                                                                                                                                                                         
Application configuration {"serverTime":1503577208806}

To easily recreate this environment clone this pull request and run vagrant up.

[floreks]

I don't see any issues here.

No metric client provided. Skipping metrics.

This is just an information. Not every page/resource contains metrics thus it's skipped for some.

[rbjorklin]

@floreks Pasted all of the log here: https://pastebin.com/KU8RbWpM

EDIT: Heapster log below if it matters:

I0824 11:57:18.581236       5 heapster.go:72] /heapster --source=kubernetes.summary_api:''
I0824 11:57:18.581391       5 heapster.go:73] Heapster version v1.4.1
I0824 11:57:18.581632       5 configs.go:61] Using Kubernetes client with master "https://10.3.0.1:443" and version v1
I0824 11:57:18.581665       5 configs.go:62] Using kubelet port 10255
I0824 11:57:18.582338       5 heapster.go:196] Starting with Metric Sink
I0824 11:57:18.589987       5 heapster.go:106] Starting heapster on port 8082

[floreks]

Everything looks fine. No errors.

[rbjorklin]

So why do I still need that argument to the dashboard for it to pick up metrics?

[floreks]

Remove argument restart dashboard and show logs from dashboard.

[rbjorklin]

That's what you're looking at.

[floreks]

It is not full log. At the beginning when dashboard is starting there are more information. Use kubectl logs -n kube-system <dashboard_pod_name>

[rbjorklin]

For anyone who finds this discussion in the future this is what came out of it: #2306

[jordanjennings]

In my case I had to grant a new RBAC permission to the ClusterRole on the dashboard. Looks like between 1.6.1 and 1.6.3 it now requires the get verb on the services resource. In other words:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "proxy"]
    resourceNames: ["heapster"]

[floreks]

Looks like between 1.6.1 and 1.6.3 it now requires the get verb on the services resource.

It has always required it for in-cluster heapster detection. Also installation command from our README makes sure that dashboard has admin privileges in the cluster.

[jordanjennings]

@floreks Interesting, metrics definitely worked before without needing that permission. We don't give dashboard cluster-admin, we use the token header authentication and use an oath proxy in front of it.

[floreks]

Well it was always required if cluster had authorization mode RBAC enabled. Method to detect heapster using in-cluster config did not change, ever. We have only added additional health check.

[miramar-labs]

surely this is all to do with https://github.com/kubernetes/dashboard/issues/2306
so due to this change in heapster, right now, if you create your dashboard pod right after you create your heapster pod, the dashboard will not load the metrics client because not enough time has passed in order for heapster to gather some data ... wait 5 minutes or so and restart the dashboard, it will then get the right value back from the heapster health check and metrics/graphs will suddenly start working in the dashboard.... but this is far from optimal.
how about removing that health check until you figure out a scheme that works reliably?

[mcwienczek]

restarting dashboard after 5 mins worked in my case:
minikube 0.22
dashboard 1.6.3
heapster 1.3.0

[floreks]

I'm currently working on adding periodic check so dashboard will try to discover heapster for around 10 min after start. In case they are deployed together it will be more than enough to automatically enable metrics once heapster scrapes some data.

[jitendra-saal]

We are seeing the similar problem in v1.10 -
2018/04/08 11:09:06 Starting overwatch
2018/04/08 11:09:06 Using in-cluster config to connect to apiserver
2018/04/08 11:09:06 Using service account token for csrf signing
2018/04/08 11:09:06 No request provided. Skipping authorization
2018/04/08 11:09:06 Successful initial request to the apiserver, version: v1.10.0
2018/04/08 11:09:06 Generating JWE encryption key
2018/04/08 11:09:06 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2018/04/08 11:09:06 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/04/08 11:09:06 Initializing JWE encryption key from synchronized object
2018/04/08 11:09:06 Creating in-cluster Heapster client
2018/04/08 11:09:06 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/04/08 11:09:06 Serving insecurely on HTTP port: 9090
2018/04/08 11:09:36 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.

Version details:-
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Created the dashboard using -
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

[klynch]

I'm seeing this as well.

[klynch]

@jitendra-saal Upgrading to 1.8.3 works for me now.

[shashankpai]

@jitendra-saal were you able to solve it . Even i am getting this same message .
2018/04/15 06:47:11 Using in-cluster config to connect to apiserver
2018/04/15 06:47:11 Using service account token for csrf signing
2018/04/15 06:47:11 No request provided. Skipping authorization
2018/04/15 06:47:11 Starting overwatch
2018/04/15 06:47:11 Successful initial request to the apiserver, version: v1.10.1
2018/04/15 06:47:11 Generating JWE encryption key
2018/04/15 06:47:11 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2018/04/15 06:47:11 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/04/15 06:47:11 Initializing JWE encryption key from synchronized object
2018/04/15 06:47:11 Creating in-cluster Heapster client
2018/04/15 06:47:11 Auto-generating certificates
2018/04/15 06:47:11 Successfully created certificates
2018/04/15 06:47:11 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/04/15 06:47:11 Serving securely on HTTPS port: 8443
2018/04/15 06:47:41 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/04/15 06:48:11 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.

[komaldhiman112]

@shashankpai I am facing same issue in v1.14.1