[grafana] grafana-sc-dashboard sidecard works only at initializatrion phase, subsequent connection failed by ProtocolError when calling kubernetes

[tringuyen-yw]

Using Grafana helm chart version 6.13.2, which is an upstream dependency from kube-prometheus-stack.

The Grafana pod is created and works correctly. The grafana-sc-dashboard sidecar container picked up and parsed correctly the ConfigMaps containing dashboards. However, subsequent modification (add/delete/update) of configmap is ignored by grafana-sc-dashboard. Probably because the container can no longer connect to kubernetes. Below is the logs:

[2021-06-25 18:54:14] Starting collector
[2021-06-25 18:54:14] No folder annotation was provided, defaulting to k8s-sidecar-target-directory
[2021-06-25 18:54:14] Selected resource type: ('secret', 'configmap')
[2021-06-25 18:54:14] Loading incluster config ...
[2021-06-25 18:54:14] Config for cluster api at 'https://172.24.0.1:443' loaded...
[2021-06-25 18:54:14] Unique filenames will not be enforced.
[2021-06-25 18:54:14] 5xx response content will not be enabled.
[2021-06-25 18:54:19] Working on ADDED configmap monitoring/my-kubpromstack-controller-manager
[2021-06-25 18:54:19] Working on ADDED configmap monitoring/my-kubpromstack-k8s-resources-namespace
[2021-06-25 18:54:19] Working on ADDED configmap monitoring/my-kubpromstack-etcd
[2021-06-25 18:54:19] Working on ADDED configmap monitoring/my-kubpromstack-statefulset
... etc ...
[2021-06-25 18:54:19] Working on ADDED configmap monitoring/my-kubpromstack-kubelet
[2021-06-25 18:54:19] Working on ADDED configmap monitoring/my-kubpromstack-namespace-by-pod
[2021-06-25 18:54:19] Working on ADDED configmap monitoring/my-kubpromstack-k8s-resources-pod

[2021-06-25 18:58:37] ProtocolError when calling kubernetes: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

[2021-06-25 19:03:02] ProtocolError when calling kubernetes: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

[2021-06-25 19:07:29] ProtocolError when calling kubernetes: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

... same error repeated every 4 minutes ....

[Ottovsky]

I have the same issue on AKS, does anyone knows what can be the possible issue?

[transkonduktor]

I have the same issue on AKS, does anyone knows what can be the possible issue?

Probably the only workaround for now is to use watchMethod: SLEEP.

kiwigrid/k8s-sidecar#85

[bergerx]

I believe this one is a duplicate of the long-standing #18 and will eventually be closed once the kiwigrid/k8s-sidecar#85 drives the original issue to a conclusion.

Currently, there seem to be 3 workarounds:

• watchMethod: SLEEP: works fine if you don't care about auto-deletion of dashboards when you remove ConfigMaps: Watching stops working after 10 minutes kiwigrid/k8s-sidecar#85 (comment)
• restart grafana sidecar regularly (e.g. every 10 minutes): a very ugly hack mentioned here by me Dashboard sidecar stops working after 10 minutes #18 (comment)
• use an alternative sidecar implementation: Seems to be working fine, but by doing so you are not using the upstream's default one anymore, new versions/changes are more likely to be found earlier for the default one. Mentioned in this comment, you can read the rest of that ticket Watching stops working after 10 minutes kiwigrid/k8s-sidecar#85 (comment)