inputs.prometheus require cluster level permissions even when scoped to a namespace

[redbaron]

Relevant telegraf.conf

[[inputs.prometheus]]
monitor_kubernetes_pods = true
monitor_kubernetes_pods_namespace = "ns1"
monitor_kubernetes_pods_port = 7354

Logs from Telegraf

E0302 16:25:26.890248       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:ns1:telegraf-prometheus" cannot list resource "pods" in API group "" at the cluster scope

System info

1.25.1

Docker

No response

Steps to reproduce

Grant kubernetes service account used to run telegraf followig role (important, NOT cluster role)

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: telegraf-prometheus
  namespace: n1
  rules:
  - verbs:
      - list
      - watch
      - get
    apiGroups:
      - ''
    resources:
      - namespaces
      - pods

Run telegraf with config from the above in the namespace ns1

Expected behavior

Scrapes metrics successfully

Actual behavior

Unable to list pods

Additional info

No response

[powersj]

@redbaron have you looked into what might be required to reduce the permissions? It does look like the readme example expected cluster level permissions as you point out.

[skrech]

Hello, we're facing the same issue where we need to scrape pods but don't have permissions in our cluster to create ClusterRoles.
In another related issue (#10928), a limitation of informer (on kubernetes side, kubernetes-sigs/controller-runtime#124) is referred to as the cause of the problem, but it seems it's been resolved in kubernetes-sigs/controller-runtime#136 so maybe telegraf could alleviate the requirement for ClusterRole.

[powersj]

@redbaron and @skrech,

Can you try the artifacts in #13063 to see if they resolve this issue?

[skrech]

Uh, It was very hard for me to setup the environment, but I think I manage to do it in the end.
I used this Dockerfile (https://github.com/influxdata/influxdata-docker/blob/master/telegraf/nightly/Dockerfile), modified it to download the CI build linked in the PR, built it locally, and used a clever trick in Skaffold to test with local cluster.

The good news is the fix is working! With only Role and RoleBinding (as opposed to ClusterRole and ClusterRoleBining) I was able to scrape Pods from the namespace specified in monitor_kubernetes_pods_namespace.

Thank you @Ivaylogi98 for the fix!

[powersj]

@skrech thank you for taking the time to confirm!