systemd collector blocked by AppArmor when non root under kubernetes

[glennpratt]

I realize there is a desire to deprecate this for systemd_exporter. I wasn't able to test systemd_exporter yet because of limitations with it's helm chart. I will try to replicate this there and report my findings.

Host operating system: output of uname -a

Linux dev-master-0 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Ubuntu 22.04.4

node_exporter version: output of node_exporter --version

node_exporter, version 1.8.2 (branch: HEAD, revision: f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)
  build user:       root@03d440803209
  build date:       20240714-11:53:45
  go version:       go1.22.5
  platform:         linux/amd64
  tags:             unknown

node_exporter command line flags

/bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root --path.udev.data=/host/root/run/udev/data --web.listen-address=[0.0.0.0]:9100 --collector.systemd --collector.systemd.unit-include=(rke2-.*\.service) --collector.systemd.enable-restarts-metrics --collector.systemd.enable-start-time-metrics --collector.systemd.enable-task-metrics

node_exporter log output

Non-root

root

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Enabled systemd collector

What did you expect to see?

No AppArmor error, as seen when run as root.

root

ts=2024-09-10T15:31:49.504Z caller=node_exporter.go:196 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
ts=2024-09-10T15:31:49.505Z caller=systemd_linux.go:153 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-include" flag=(rke2-.*\.service)                             
ts=2024-09-10T15:31:49.506Z caller=systemd_linux.go:155 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-exclude" flag=.+\.(automount|device|mount|scope|slice)       
ts=2024-09-10T15:31:49.506Z caller=node_exporter.go:118 level=info collector=systemd 

What did you see instead?

non-root

ts=2024-09-10T16:20:08.238Z caller=systemd_linux.go:153 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-include" flag=(rke2-.*\.service)                             
ts=2024-09-10T16:20:08.238Z caller=systemd_linux.go:155 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-exclude" flag=.+\.(automount|device|mount|scope|slice)       
ts=2024-09-10T16:20:08.240Z caller=node_exporter.go:118 level=info collector=systemd                                                                                                        
ts=2024-09-10T16:20:28.809Z caller=collector.go:169 level=error msg="collector failed" name=systemd duration_seconds=0.0484503 err="couldn't get dbus connection: An AppArmor policy prevent
s this sender from sending this message to this recipient; type=\"method_call\", sender=\"(null)\" (inactive) interface=\"org.freedesktop.DBus\" member=\"Hello\" error name=\"(unset)\" req
uested_reply=\"0\" destination=\"org.freedesktop.DBus\" (bus)"  

[glennpratt]

systemd_exporter exhibits the same behavior:

prometheus-systemd-exporter-b55jn ts=2024-09-10T23:22:30.626Z caller=systemd.go:225 level=error msg="error collecting metrics" err="couldn't get dbus connection: An AppArmor policy prevent
s this sender from sending this message to this recipient; type=\"method_call\", sender=\"(null)\" (inactive) interface=\"org.freedesktop.DBus\" member=\"Hello\" error name=\"(unset)\" req
uested_reply=\"0\" destination=\"org.freedesktop.DBus\" (bus)"

[discordianfish]

seems like a apparmor configuration issue, not an issue in the node-exporter

[glennpratt]

@discordianfish thanks, I thought it was unconfined, but I must have messed up my check earlier. It appears to be an AppArmor profile from containerd.

/ # cat /proc/self/attr/apparmor/current 
cri-containerd.apparmor.d (enforce)

This seems to have resolved the issue prometheus-community/helm-charts#2304 (comment)

/ $ cat /proc/self/attr/apparmor/current
unconfined

This is vanilla Ubuntu 22.04 and rke2. I've found other reports out there, e.g. m-lab/k8s-support#708

It would seem reasonable to handle this by default or at least document it.

[discordianfish]

The node-exporter itself has nothing to do with confining or not confining a container, this needs to be handled by whatever is deploying the node-exporter