cAdvisor reports incorrect container_memory_cache metrics

[vboginskey]

Environment

OS: CoreOS 1745.7.0
Kernel: 4.14.48-coreos-r2
systemd: 238
cAdvisor: google/cadvisor:latest

Summary

cAdvisor seems to report incorrect metrics for container_memory_cache with nested cgroups. container_memory_rss is rolled up from child cgroups, whereas container_memory_cache is not. This succinctly illustrates the problem:

container_memory_cache{container_name="",id="/podruntime.slice",image="",name="",namespace="",pod_name=""} 0
container_memory_cache{container_name="",id="/podruntime.slice/podruntime-kubelet.slice",image="",name="",namespace="",pod_name=""} 0
container_memory_cache{container_name="",id="/podruntime.slice/podruntime-kubelet.slice/kubelet.service",image="",name="",namespace="",pod_name=""} 1.3787136e+09
container_memory_cache{container_name="",id="/podruntime.slice/podruntime-runtime.slice",image="",name="",namespace="",pod_name=""} 0
container_memory_cache{container_name="",id="/podruntime.slice/podruntime-runtime.slice/containerd.service",image="",name="",namespace="",pod_name=""} 3.231744e+06
container_memory_cache{container_name="",id="/podruntime.slice/podruntime-runtime.slice/docker.service",image="",name="",namespace="",pod_name=""} 1.7538719744e+10

container_memory_rss{container_name="",id="/podruntime.slice",image="",name="",namespace="",pod_name=""} 2.53140992e+08
container_memory_rss{container_name="",id="/podruntime.slice/podruntime-kubelet.slice",image="",name="",namespace="",pod_name=""} 9.3331456e+07
container_memory_rss{container_name="",id="/podruntime.slice/podruntime-kubelet.slice/kubelet.service",image="",name="",namespace="",pod_name=""} 9.3331456e+07
container_memory_rss{container_name="",id="/podruntime.slice/podruntime-runtime.slice",image="",name="",namespace="",pod_name=""} 1.59809536e+08
container_memory_rss{container_name="",id="/podruntime.slice/podruntime-runtime.slice/containerd.service",image="",name="",namespace="",pod_name=""} 6.6609152e+07
container_memory_rss{container_name="",id="/podruntime.slice/podruntime-runtime.slice/docker.service",image="",name="",namespace="",pod_name=""} 9.3134848e+07

I would have expected consistent behavior here, with either cache/rss or total_cache/total_rss reported from /sys/fs/cgroup/memory/podruntime.slice/memory.stat.

The cgroup layout used is as follows:

/sys/fs/cgroup/memory/podruntime.slice $ systemd-cgls | grep -e slice -e service
Working directory /sys/fs/cgroup/memory/podruntime.slice:
├─podruntime-kubelet.slice
│ └─kubelet.service
└─podruntime-runtime.slice
  ├─containerd.service
  └─docker.service

[dashpole]

Interesting. It looks like when memory.use_hierarchy is set, cadvisor returns total_rss, but regardless of hierarchy, runc returns cache.

We made this change here: #1728, but runc still has the old behavior. Can you open an issue with opencontainers/runc, and see if they are willing to change the behavior to match?

[vboginskey]

It's not clear to me why the runc changes are necessary, since cadvisor uses MemoryStats.Stats["cache"] directly rather than MemoryStats.Cache. Wouldn't something like this address the discrepancy?

[dashpole]

You are right. Feel free to open the PR, and Ill review it

[vboginskey]

Sure, here it is: #2014