Prediction analytics dashboard not capturing prediction API calls

[aritrachatterjee15]

I have been successful in deploying transformers and models in Seldon by following the Minikube with RBAC notebook. I am also able to successfully call the prediction API to get back scores. However, the API calls are not being logged by the Grafana dashboard.

I checked the prometheus logs and it is throwing out the following errors repeatedly:

time="2018-06-15T23:33:43Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:seldon:seldon" cannot list endpoints at the cluster scope" component="kube_client_runtime" source="kubernetes.go:75" 
time="2018-06-15T23:33:43Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:seldon:seldon" cannot list services at the cluster scope" component="kube_client_runtime" source="kubernetes.go:75" 
time="2018-06-15T23:33:43Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:seldon:seldon" cannot list pods at the cluster scope" component="kube_client_runtime" source="kubernetes.go:75" 

I wonder if prometheus deployment should be done using the prometheus service account available under the default namespace instead of using the seldon service account.

[ukclivecox]

Can you tell us which commands/notebook you have followed to setup seldon-core? Also, how you created your kubernetes cluster?

This certainly seems like a missing/incorrect rolebinding in the RBAC setup.

[aritrachatterjee15]

I followed the kubectl_demo_minikube_rbac notebook.

[ukclivecox]

if you run:

kubectl create clusterrolebinding kube-system-cluster-admin-seldon --clusterrole=cluster-admin --serviceaccount=seldon:seldon

Does it fix it for you? If so we can update the notebook.

[aritrachatterjee15]

Yes, that fixes the issue. Thanks!

[ukclivecox]

There was an issue in the helm chart for the analytics setting up the serviceaccount. This should be fixed by #169.