Prometheus memory usage for containers is always 0

[airbone42]

I just set up the default system using docker-compose.
Unfortunately, in Grafana is see all metric except of the Container Memory Usage and Cached Container Memory Usage.

Unfortunately already when I search in Prometheus for memory, all values are 0.

I'm using a PI4, probably the issue is related to that. Did anyone have similar issues and has an idea how to fix it or what would be the next step to look into, when Prometheus has the wrong/no values?

[nightah]

Is your cadvisor container running?
As far as I know the official cadvisor container doesn't support ARM so you would have had to modify the docker-compose.yml to utilise an image that does.

[airbone42]

Is your cadvisor container running?
As far as I know the official cadvisor container doesn't support ARM so you would have had to modify the docker-compose.yml to utilise an image that does.

Indeed that's what I did, easily replaced it with: image: budry/cadvisor-arm

I don't know all these tools enough, is cadvisor actually the tool that gets the raw container memory data? If yes I would look deeper into that specific arm-image.

[nightah]

Yes it is I would try run this query in Prometheus directly to see what shows up:
container_memory_usage_bytes{image!="",container_label_org_label_schema_group=""}

[airbone42]

This is the result:

<!--StartFragment-->
ontainer_memory_usage_bytes{container_label_com_docker_compose_config_hash="2554b9322524180575582f3546abe6114846a48c2ba54071ec3731dde4cda855",&nbsp;container_label_com_docker_compose_container_number="1",&nbsp;container_label_com_docker_compose_oneoff="False",&nbsp;container_label_com_docker_compose_project="openhab",&nbsp;container_label_com_docker_compose_service="openhab",&nbsp;container_label_com_docker_compose_version="1.21.0",&nbsp;container_label_maintainer="openHAB &lt;info@openhabfoundation.org&gt;",&nbsp;container_label_org_label_schema_build_date="2021-01-27T21:40:00Z",&nbsp;container_label_org_label_schema_description="An open source, technology agnostic home automation platform",&nbsp;container_label_org_label_schema_docker_dockerfile="/Dockerfile",&nbsp;container_label_org_label_schema_license="EPL-2.0",&nbsp;container_label_org_label_schema_name="openHAB",&nbsp;container_label_org_label_schema_url="https://www.openhab.com/",&nbsp;container_label_org_label_schema_vcs_ref="75d0fffe8526866c2830d83612fbd7ed64a866bb",&nbsp;container_label_org_label_schema_vcs_type="Git",&nbsp;container_label_org_label_schema_vcs_url="https://github.com/openhab/openhab-docker.git",&nbsp;container_label_org_label_schema_vendor="openHAB Foundation e.V.",&nbsp;container_label_org_label_schema_version="2.5.12",&nbsp;id="/docker/f1b366c9b8cef49fe90bcdeb3476045ae0acbddde557fd622bb1875a79ebcd94",&nbsp;image="openhab_openhab",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="openhab"} | 0
-- | --
container_memory_usage_bytes{container_label_com_docker_compose_config_hash="e2cdee5c548ccaece03d188ba0ac441d6ae9f38178f329cd361079a54d15dae5",&nbsp;container_label_com_docker_compose_container_number="1",&nbsp;container_label_com_docker_compose_oneoff="False",&nbsp;container_label_com_docker_compose_project="openhab3",&nbsp;container_label_com_docker_compose_service="openhab",&nbsp;container_label_com_docker_compose_version="1.21.0",&nbsp;container_label_maintainer="openHAB &lt;info@openhabfoundation.org&gt;",&nbsp;container_label_org_label_schema_build_date="2021-04-21T12:56:20Z",&nbsp;container_label_org_label_schema_description="An open source, technology agnostic home automation platform",&nbsp;container_label_org_label_schema_docker_dockerfile="/Dockerfile",&nbsp;container_label_org_label_schema_license="EPL-2.0",&nbsp;container_label_org_label_schema_name="openHAB",&nbsp;container_label_org_label_schema_url="https://www.openhab.com/",&nbsp;container_label_org_label_schema_vcs_ref="d4bc2cfb570ee70d74369fccd168ccbd078a4c79",&nbsp;container_label_org_label_schema_vcs_type="Git",&nbsp;container_label_org_label_schema_vcs_url="https://github.com/openhab/openhab-docker.git",&nbsp;container_label_org_label_schema_vendor="openHAB Foundation e.V.",&nbsp;container_label_org_label_schema_version="3.0.2",&nbsp;id="/docker/63adaa6b9219ba98ae1e58774adaba79d25a0beba57e49135cc887c40c0e6f62",&nbsp;image="openhab3_openhab",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="openhab3"} | 0
container_memory_usage_bytes{container_label_restartcount="1",&nbsp;id="/docker/a0a96fa86ff26b06b022e281cde454882b93c43c5db822092648a2387882de7e",&nbsp;image="marthoc/deconz",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="deconz"} | 0
container_memory_usage_bytes{container_label_restartcount="9",&nbsp;id="/docker/cd860d38fad0d7bd999c36ae043e6a6bac2756fd1f689e5090412d7085a03f5a",&nbsp;image="pi/heos-api",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="heos-api"} | 0
container_memory_usage_bytes{id="/docker/550fa5b733dd182cbf17c273ff5f687c592bd158cd8393020377d38c41e8ed7c",&nbsp;image="pi/shelly-proxy",&nbsp;instance="cadvisor:8080",&nbsp;job="cadvisor",&nbsp;name="shelly-proxy"}

<!--EndFragment-->

[nightah]

That's the same query that's being run so you should definitely be seeing results in Grafana.

[chintanp]

@airbone42 Not sure if you have the same problem, but the new default on Raspbian is to not report out the memory. The docker stats command does not work as expected too, as reported here: docker/for-linux#1112 and the fix there was to append the following to the \boot\cmdline.txt: cgroup_memory=1 cgroup_enable=memory. Once I did that and restarted my Pi 4, the memory section in Grafana displays a non-zero value. Still need to verify if it is correct.

[nightah]

Other thing to make sure is that you're not running the SNAP version of Docker, this seems to also cause issues.

[nightah]

Did you manage to resolve this @airbone42?

[nightah]

Closing due to inactivity if you're still having this issue please let me know and I'll reopen the issue.