Metrics always showing zero

[achille-roussel]

Hi all, I've just deployed cAdvisor to an existing infra, simply running the default docker image as explained in the README.

Things work just fine but some stats are always showing zero values at the container level (host-level stats have meaningful values), so far:

• all memory related stats are zero
• all io related stats are zero
• all task related stats are zero

Is there something I may have to configure? I don't know if those stats should be shown "out-of-the-box".

Thanks for your help!

[mboussaa]

These stats are related to disk metrics. The are is no issue I think

[achille-roussel]

I'm looking at the Prometheus interface and I'm seeing these stats:

# HELP container_memory_usage_bytes Current memory usage in bytes.
# TYPE container_memory_usage_bytes gauge
container_memory_usage_bytes{id="/"} 0
...

Are these disk metrics? How can I figure out what they are actually reporting?

[vishh]

Sounds like cAdvisor is having trouble collecting stats. 0 doesn't seem like a valid value for memory usage.

[achille-roussel]

Do you guys have an idea what may be causing this? I really just used the default setup and there's nothing fancy about the docker install either (using Docker 1.8 on a Debian on AWS).

[vishh]

Check the logs.

[achille-roussel]

Sorry I forgot to mention that, there's about nothing in the logs:

docker logs --tail=10 -f container-exporter
I0108 21:21:58.873729 00001 manager.go:158] Machine: {NumCores:4 CpuFrequency:2400072 MemoryCapacity:16863678464 MachineID: SystemUUID:*** BootID:6a8d7a36-e177-44ec-a42a-7562c3677d15 Filesystems:[{Device:/dev/xvda1 Capacity:8316706816} {Device:/dev/xvdb Capacity:270565117952}] DiskMap:map[202:0:{Name:xvda Major:202 Minor:0 Size:8589934592 Scheduler:cfq} 202:16:{Name:xvdb Major:202 Minor:16 Size:274877906944 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:*** Speed:10000 Mtu:9001}] Topology:[{Id:0 Memory:16863678464 Cores:[{Id:0 Threads:[0 2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:1 Threads:[1 3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:31457280 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown}
I0108 21:21:58.874528 00001 manager.go:164] Version: {KernelVersion:3.16.0-4-amd64 ContainerOsVersion:Buildroot 2014.02 DockerVersion:1.8.3 CadvisorVersion:0.18.0}
I0108 21:21:58.968268 00001 factory.go:227] System is using systemd
I0108 21:21:58.971804 00001 factory.go:235] Registering Docker factory
I0108 21:21:58.975366 00001 factory.go:93] Registering Raw factory
I0108 21:21:59.122014 00001 manager.go:1001] Started watching for new ooms in manager
W0108 21:21:59.122174 00001 manager.go:232] Could not configure a source for OOM detection, disabling OOM events: exec: "journalctl": executable file not found in $PATH
I0108 21:21:59.141765 00001 manager.go:245] Starting recovery of all containers
I0108 21:21:59.215566 00001 manager.go:250] Recovery completed
I0108 21:21:59.249463 00001 cadvisor.go:94] Starting cAdvisor version: "0.18.0" on port 8080

Nothing more than this is shown, Prometheus is pulling every 15 seconds and I forced generating a few HTTP requests as well but the logs stay silent.
I'm also checking the command line options, the -stderrthreshold and -v are at 0 which as I understand means all logs will be printed... I'm not sure how to get more data on what's going on.

[vishh]

-v=0 means minimal logging is enabled. You can try setting --v=4 to get
more logs.

On Fri, Jan 8, 2016 at 1:31 PM, Achille notifications@github.com wrote:

Sorry I forgot to mention that, there's about nothing in the logs:

docker logs --tail=10 -f container-exporter
I0108 21:21:58.873729 00001 manager.go:158] Machine: {NumCores:4 CpuFrequency:2400072 MemoryCapacity:16863678464 MachineID: SystemUUID:* BootID:6a8d7a36-e177-44ec-a42a-7562c3677d15 Filesystems:[{Device:/dev/xvda1 Capacity:8316706816} {Device:/dev/xvdb Capacity:270565117952}] DiskMap:map[202:0:{Name:xvda Major:202 Minor:0 Size:8589934592 Scheduler:cfq} 202:16:{Name:xvdb Major:202 Minor:16 Size:274877906944 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:* Speed:10000 Mtu:9001}] Topology:[{Id:0 Memory:16863678464 Cores:[{Id:0 Threads:[0 2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:1 Threads:[1 3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:31457280 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown}
I0108 21:21:58.874528 00001 manager.go:164] Version: {KernelVersion:3.16.0-4-amd64 ContainerOsVersion:Buildroot 2014.02 DockerVersion:1.8.3 CadvisorVersion:0.18.0}
I0108 21:21:58.968268 00001 factory.go:227] System is using systemd
I0108 21:21:58.971804 00001 factory.go:235] Registering Docker factory
I0108 21:21:58.975366 00001 factory.go:93] Registering Raw factory
I0108 21:21:59.122014 00001 manager.go:1001] Started watching for new ooms in manager
W0108 21:21:59.122174 00001 manager.go:232] Could not configure a source for OOM detection, disabling OOM events: exec: "journalctl": executable file not found in $PATH
I0108 21:21:59.141765 00001 manager.go:245] Starting recovery of all containers
I0108 21:21:59.215566 00001 manager.go:250] Recovery completed
I0108 21:21:59.249463 00001 cadvisor.go:94] Starting cAdvisor version: "0.18.0" on port 8080

Nothing more than this is shown, Prometheus is pulling every 15 seconds
and I forced generating a few HTTP requests as well but the logs stay
silent.
I'm also checking the command line options, the -stderrthreshold and -v
are at 0 which as I understand means all logs will be printed... I'm not
sure how to get more data on what's going on.

—
Reply to this email directly or view it on GitHub
#1039 (comment).

[achille-roussel]

Alright, that was confusing, -stderrthreshold=0 sending all stats to stderr but -v=0 is actually turning off stats, thanks for your advise.

I've got more data now, but there's nothing explicit about memory stats not being read, a bunch of stuff that look like warnings show up tho but I'm too inexperienced with cadvisor to tell what would be giving hints on what's going on.

Things that look weird to me, I'm gonna investigate more:

I0111 20:20:05.495421 00001 machine.go:49] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"

I0111 20:20:05.749432 00001 factory.go:71] Error trying to work out if we can handle /: error inspecting container: unexpected end of JSON input

Here are all the logs if someone has some spare cycles to take a look:

I0111 20:20:05.487631 00001 manager.go:127] cAdvisor running in container: "/system.slice/docker-1833c88c617a80f2182b7dd588f119db49175ec75e303f42ba9b715703ddd75f.scope"
I0111 20:20:05.491407 00001 fs.go:93] Filesystem partitions: map[/dev/xvda1:{mountpoint:/rootfs major:202 minor:1} /dev/xvdb:{mountpoint:/rootfs/mnt/prometheus major:202 minor:16}]
I0111 20:20:05.495421 00001 machine.go:49] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
I0111 20:20:05.495466 00001 manager.go:158] Machine: {NumCores:4 CpuFrequency:2400072 MemoryCapacity:16863678464 MachineID: SystemUUID:EC2A5423-30E0-02E9-E596-2F0EA2365FBB BootID:6a8d7a36-e177-44ec-a42a-7562c3677d15 Filesystems:[{Device:/dev/xvda1 Capacity:8316706816} {Device:/dev/xvdb Capacity:270565117952}] DiskMap:map[202:0:{Name:xvda Major:202 Minor:0 Size:8589934592 Scheduler:cfq} 202:16:{Name:xvdb Major:202 Minor:16 Size:274877906944 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:12:38:96:44:90:13 Speed:10000 Mtu:9001}] Topology:[{Id:0 Memory:16863678464 Cores:[{Id:0 Threads:[0 2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:1 Threads:[1 3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:31457280 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown}
I0111 20:20:05.496289 00001 manager.go:164] Version: {KernelVersion:3.16.0-4-amd64 ContainerOsVersion:Buildroot 2014.02 DockerVersion:1.8.3 CadvisorVersion:0.18.0}
I0111 20:20:05.583212 00001 factory.go:227] System is using systemd
I0111 20:20:05.586936 00001 factory.go:235] Registering Docker factory
I0111 20:20:05.590411 00001 factory.go:93] Registering Raw factory
I0111 20:20:05.746294 00001 manager.go:1001] Started watching for new ooms in manager
W0111 20:20:05.746392 00001 manager.go:232] Could not configure a source for OOM detection, disabling OOM events: exec: "journalctl": executable file not found in $PATH
I0111 20:20:05.749432 00001 factory.go:71] Error trying to work out if we can handle /: error inspecting container: unexpected end of JSON input
I0111 20:20:05.749449 00001 factory.go:82] Factory "docker" was unable to handle container "/"
I0111 20:20:05.749462 00001 factory.go:78] Using factory "raw" for container "/"
I0111 20:20:05.750068 00001 manager.go:798] Added container: "/" (aliases: [], namespace: "")
I0111 20:20:05.750363 00001 handler.go:322] Added event &{/ 2016-01-05 01:11:41.221597572 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.750445 00001 manager.go:245] Starting recovery of all containers
I0111 20:20:05.750466 00001 container.go:368] Start housekeeping for container "/"
I0111 20:20:05.766835 00001 factory.go:71] Error trying to work out if we can handle /system.slice: error inspecting container: No such container: system.slice
I0111 20:20:05.766902 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice"
I0111 20:20:05.766914 00001 factory.go:78] Using factory "raw" for container "/system.slice"
I0111 20:20:05.767039 00001 manager.go:798] Added container: "/system.slice" (aliases: [], namespace: "")
I0111 20:20:05.767139 00001 handler.go:322] Added event &{/system.slice 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.767209 00001 container.go:368] Start housekeeping for container "/system.slice"
I0111 20:20:05.767818 00001 factory.go:71] Error trying to work out if we can handle /system.slice/mnt-prometheus.mount: error inspecting container: No such container: mnt-prometheus.mount
I0111 20:20:05.767850 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/mnt-prometheus.mount"
I0111 20:20:05.767864 00001 factory.go:78] Using factory "raw" for container "/system.slice/mnt-prometheus.mount"
I0111 20:20:05.767997 00001 manager.go:798] Added container: "/system.slice/mnt-prometheus.mount" (aliases: [], namespace: "")
I0111 20:20:05.768127 00001 handler.go:322] Added event &{/system.slice/mnt-prometheus.mount 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.768163 00001 container.go:368] Start housekeeping for container "/system.slice/mnt-prometheus.mount"
I0111 20:20:05.768832 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-fsck-root.service: error inspecting container: No such container: systemd-fsck-root.service
I0111 20:20:05.768851 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-fsck-root.service"
I0111 20:20:05.768863 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-fsck-root.service"
I0111 20:20:05.768988 00001 manager.go:798] Added container: "/system.slice/systemd-fsck-root.service" (aliases: [], namespace: "")
I0111 20:20:05.769087 00001 handler.go:322] Added event &{/system.slice/systemd-fsck-root.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.769143 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-fsck-root.service"
I0111 20:20:05.769887 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-update-utmp.service: error inspecting container: No such container: systemd-update-utmp.service
I0111 20:20:05.769916 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-update-utmp.service"
I0111 20:20:05.769932 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-update-utmp.service"
I0111 20:20:05.770071 00001 manager.go:798] Added container: "/system.slice/systemd-update-utmp.service" (aliases: [], namespace: "")
I0111 20:20:05.770163 00001 handler.go:322] Added event &{/system.slice/systemd-update-utmp.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.770244 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-update-utmp.service"
I0111 20:20:05.770971 00001 factory.go:71] Error trying to work out if we can handle /system.slice/udev-finish.service: error inspecting container: No such container: udev-finish.service
I0111 20:20:05.770986 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/udev-finish.service"
I0111 20:20:05.770997 00001 factory.go:78] Using factory "raw" for container "/system.slice/udev-finish.service"
I0111 20:20:05.771148 00001 manager.go:798] Added container: "/system.slice/udev-finish.service" (aliases: [], namespace: "")
I0111 20:20:05.771249 00001 handler.go:322] Added event &{/system.slice/udev-finish.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.771295 00001 container.go:368] Start housekeeping for container "/system.slice/udev-finish.service"
I0111 20:20:05.771736 00001 factory.go:71] Error trying to work out if we can handle /system.slice/dev-hugepages.mount: error inspecting container: No such container: dev-hugepages.mount
I0111 20:20:05.771756 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/dev-hugepages.mount"
I0111 20:20:05.771768 00001 factory.go:78] Using factory "raw" for container "/system.slice/dev-hugepages.mount"
I0111 20:20:05.771894 00001 manager.go:798] Added container: "/system.slice/dev-hugepages.mount" (aliases: [], namespace: "")
I0111 20:20:05.772006 00001 handler.go:322] Added event &{/system.slice/dev-hugepages.mount 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.772054 00001 container.go:368] Start housekeeping for container "/system.slice/dev-hugepages.mount"
I0111 20:20:05.773055 00001 factory.go:71] Error trying to work out if we can handle /system.slice/rsyslog.service: error inspecting container: No such container: rsyslog.service
I0111 20:20:05.773076 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/rsyslog.service"
I0111 20:20:05.773088 00001 factory.go:78] Using factory "raw" for container "/system.slice/rsyslog.service"
I0111 20:20:05.773233 00001 manager.go:798] Added container: "/system.slice/rsyslog.service" (aliases: [], namespace: "")
I0111 20:20:05.773346 00001 handler.go:322] Added event &{/system.slice/rsyslog.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.773387 00001 container.go:368] Start housekeeping for container "/system.slice/rsyslog.service"
I0111 20:20:05.774404 00001 factory.go:71] Error trying to work out if we can handle /system.slice/-.mount: error inspecting container: No such container: -.mount
I0111 20:20:05.774427 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/-.mount"
I0111 20:20:05.774439 00001 factory.go:78] Using factory "raw" for container "/system.slice/-.mount"
I0111 20:20:05.774543 00001 manager.go:798] Added container: "/system.slice/-.mount" (aliases: [], namespace: "")
I0111 20:20:05.774642 00001 handler.go:322] Added event &{/system.slice/-.mount 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.774691 00001 container.go:368] Start housekeeping for container "/system.slice/-.mount"
I0111 20:20:05.775412 00001 factory.go:71] Error trying to work out if we can handle /system.slice/cgroupfs-mount.service: error inspecting container: No such container: cgroupfs-mount.service
I0111 20:20:05.775425 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/cgroupfs-mount.service"
I0111 20:20:05.775436 00001 factory.go:78] Using factory "raw" for container "/system.slice/cgroupfs-mount.service"
I0111 20:20:05.775580 00001 manager.go:798] Added container: "/system.slice/cgroupfs-mount.service" (aliases: [], namespace: "")
I0111 20:20:05.775684 00001 handler.go:322] Added event &{/system.slice/cgroupfs-mount.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.775769 00001 container.go:368] Start housekeeping for container "/system.slice/cgroupfs-mount.service"
I0111 20:20:05.777341 00001 factory.go:71] Error trying to work out if we can handle /system.slice/system-getty.slice: error inspecting container: No such container: system-getty.slice
I0111 20:20:05.777353 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/system-getty.slice"
I0111 20:20:05.777360 00001 factory.go:78] Using factory "raw" for container "/system.slice/system-getty.slice"
I0111 20:20:05.777462 00001 manager.go:798] Added container: "/system.slice/system-getty.slice" (aliases: [], namespace: "")
I0111 20:20:05.777533 00001 handler.go:322] Added event &{/system.slice/system-getty.slice 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.777583 00001 container.go:368] Start housekeeping for container "/system.slice/system-getty.slice"
I0111 20:20:05.778414 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-modules-load.service: error inspecting container: No such container: systemd-modules-load.service
I0111 20:20:05.778429 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-modules-load.service"
I0111 20:20:05.778441 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-modules-load.service"
I0111 20:20:05.778548 00001 manager.go:798] Added container: "/system.slice/systemd-modules-load.service" (aliases: [], namespace: "")
I0111 20:20:05.778641 00001 handler.go:322] Added event &{/system.slice/systemd-modules-load.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.778686 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-modules-load.service"
I0111 20:20:05.779238 00001 factory.go:71] Error trying to work out if we can handle /user.slice: error inspecting container: No such container: user.slice
I0111 20:20:05.779250 00001 factory.go:82] Factory "docker" was unable to handle container "/user.slice"
I0111 20:20:05.779260 00001 factory.go:78] Using factory "raw" for container "/user.slice"
I0111 20:20:05.779431 00001 manager.go:798] Added container: "/user.slice" (aliases: [], namespace: "")
I0111 20:20:05.779570 00001 handler.go:322] Added event &{/user.slice 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.779640 00001 container.go:368] Start housekeeping for container "/user.slice"
I0111 20:20:05.780430 00001 factory.go:71] Error trying to work out if we can handle /system.slice/dev-mqueue.mount: error inspecting container: No such container: dev-mqueue.mount
I0111 20:20:05.780445 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/dev-mqueue.mount"
I0111 20:20:05.780456 00001 factory.go:78] Using factory "raw" for container "/system.slice/dev-mqueue.mount"
I0111 20:20:05.780589 00001 manager.go:798] Added container: "/system.slice/dev-mqueue.mount" (aliases: [], namespace: "")
I0111 20:20:05.780695 00001 handler.go:322] Added event &{/system.slice/dev-mqueue.mount 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.780801 00001 container.go:368] Start housekeeping for container "/system.slice/dev-mqueue.mount"
I0111 20:20:05.782080 00001 factory.go:78] Using factory "docker" for container "/system.slice/docker-5422c86bb0116f43be90b82b76942bc3cb2a0edcb1779641751ab7af35e81950.scope"
I0111 20:20:05.783713 00001 manager.go:798] Added container: "/system.slice/docker-5422c86bb0116f43be90b82b76942bc3cb2a0edcb1779641751ab7af35e81950.scope" (aliases: [prometheus 5422c86bb0116f43be90b82b76942bc3cb2a0edcb1779641751ab7af35e81950], namespace: "docker")
I0111 20:20:05.784149 00001 handler.go:322] Added event &{/system.slice/docker-5422c86bb0116f43be90b82b76942bc3cb2a0edcb1779641751ab7af35e81950.scope 2016-01-05 02:47:36.063077392 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.784194 00001 container.go:368] Start housekeeping for container "/system.slice/docker-5422c86bb0116f43be90b82b76942bc3cb2a0edcb1779641751ab7af35e81950.scope"
I0111 20:20:05.784879 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-logind.service: error inspecting container: No such container: systemd-logind.service
I0111 20:20:05.784893 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-logind.service"
I0111 20:20:05.784904 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-logind.service"
I0111 20:20:05.785424 00001 manager.go:798] Added container: "/system.slice/systemd-logind.service" (aliases: [], namespace: "")
I0111 20:20:05.785525 00001 handler.go:322] Added event &{/system.slice/systemd-logind.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.785647 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-logind.service"
I0111 20:20:05.786148 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-random-seed.service: error inspecting container: No such container: systemd-random-seed.service
I0111 20:20:05.786171 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-random-seed.service"
I0111 20:20:05.786182 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-random-seed.service"
I0111 20:20:05.786333 00001 manager.go:798] Added container: "/system.slice/systemd-random-seed.service" (aliases: [], namespace: "")
I0111 20:20:05.786452 00001 handler.go:322] Added event &{/system.slice/systemd-random-seed.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.786525 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-random-seed.service"
I0111 20:20:05.787321 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-remount-fs.service: error inspecting container: No such container: systemd-remount-fs.service
I0111 20:20:05.787335 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-remount-fs.service"
I0111 20:20:05.787346 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-remount-fs.service"
I0111 20:20:05.787475 00001 manager.go:798] Added container: "/system.slice/systemd-remount-fs.service" (aliases: [], namespace: "")
I0111 20:20:05.787580 00001 handler.go:322] Added event &{/system.slice/systemd-remount-fs.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.787631 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-remount-fs.service"
I0111 20:20:05.788730 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-udevd.service: error inspecting container: No such container: systemd-udevd.service
I0111 20:20:05.788752 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-udevd.service"
I0111 20:20:05.788765 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-udevd.service"
I0111 20:20:05.788904 00001 manager.go:798] Added container: "/system.slice/systemd-udevd.service" (aliases: [], namespace: "")
I0111 20:20:05.789010 00001 handler.go:322] Added event &{/system.slice/systemd-udevd.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.789115 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-udevd.service"
I0111 20:20:05.789820 00001 factory.go:71] Error trying to work out if we can handle /system.slice/cloud-config.service: error inspecting container: No such container: cloud-config.service
I0111 20:20:05.789836 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/cloud-config.service"
I0111 20:20:05.789847 00001 factory.go:78] Using factory "raw" for container "/system.slice/cloud-config.service"
I0111 20:20:05.789969 00001 manager.go:798] Added container: "/system.slice/cloud-config.service" (aliases: [], namespace: "")
I0111 20:20:05.790103 00001 handler.go:322] Added event &{/system.slice/cloud-config.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.790204 00001 container.go:368] Start housekeeping for container "/system.slice/cloud-config.service"
I0111 20:20:05.790925 00001 factory.go:71] Error trying to work out if we can handle /system.slice/cloud-final.service: error inspecting container: No such container: cloud-final.service
I0111 20:20:05.790940 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/cloud-final.service"
I0111 20:20:05.790951 00001 factory.go:78] Using factory "raw" for container "/system.slice/cloud-final.service"
I0111 20:20:05.791041 00001 manager.go:798] Added container: "/system.slice/cloud-final.service" (aliases: [], namespace: "")
I0111 20:20:05.791110 00001 handler.go:322] Added event &{/system.slice/cloud-final.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.791184 00001 container.go:368] Start housekeeping for container "/system.slice/cloud-final.service"
I0111 20:20:05.791977 00001 factory.go:71] Error trying to work out if we can handle /system.slice/cloud-init.service: error inspecting container: No such container: cloud-init.service
I0111 20:20:05.792034 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/cloud-init.service"
I0111 20:20:05.792059 00001 factory.go:78] Using factory "raw" for container "/system.slice/cloud-init.service"
I0111 20:20:05.792222 00001 manager.go:798] Added container: "/system.slice/cloud-init.service" (aliases: [], namespace: "")
I0111 20:20:05.792358 00001 handler.go:322] Added event &{/system.slice/cloud-init.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.792471 00001 container.go:368] Start housekeeping for container "/system.slice/cloud-init.service"
I0111 20:20:05.793570 00001 factory.go:78] Using factory "docker" for container "/system.slice/docker-0be3a0479f7265d299a28e607d0deb01195116a21f611260deb8f27ff57f42f9.scope"
I0111 20:20:05.795002 00001 manager.go:798] Added container: "/system.slice/docker-0be3a0479f7265d299a28e607d0deb01195116a21f611260deb8f27ff57f42f9.scope" (aliases: [consul 0be3a0479f7265d299a28e607d0deb01195116a21f611260deb8f27ff57f42f9], namespace: "docker")
I0111 20:20:05.795395 00001 handler.go:322] Added event &{/system.slice/docker-0be3a0479f7265d299a28e607d0deb01195116a21f611260deb8f27ff57f42f9.scope 2016-01-05 21:55:04.770951619 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.795490 00001 container.go:368] Start housekeeping for container "/system.slice/docker-0be3a0479f7265d299a28e607d0deb01195116a21f611260deb8f27ff57f42f9.scope"
I0111 20:20:05.796702 00001 factory.go:78] Using factory "docker" for container "/system.slice/docker-76bbd245a77ce85d0697235348a556828160d134900987a8d3c4b417b602d0cb.scope"
I0111 20:20:05.798091 00001 manager.go:798] Added container: "/system.slice/docker-76bbd245a77ce85d0697235348a556828160d134900987a8d3c4b417b602d0cb.scope" (aliases: [alertmanager 76bbd245a77ce85d0697235348a556828160d134900987a8d3c4b417b602d0cb], namespace: "docker")
I0111 20:20:05.798469 00001 handler.go:322] Added event &{/system.slice/docker-76bbd245a77ce85d0697235348a556828160d134900987a8d3c4b417b602d0cb.scope 2016-01-05 02:46:32.70334153 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.798518 00001 container.go:368] Start housekeeping for container "/system.slice/docker-76bbd245a77ce85d0697235348a556828160d134900987a8d3c4b417b602d0cb.scope"
I0111 20:20:05.799279 00001 factory.go:71] Error trying to work out if we can handle /system.slice/ssh.service: error inspecting container: No such container: ssh.service
I0111 20:20:05.799351 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/ssh.service"
I0111 20:20:05.799375 00001 factory.go:78] Using factory "raw" for container "/system.slice/ssh.service"
I0111 20:20:05.799799 00001 manager.go:798] Added container: "/system.slice/ssh.service" (aliases: [], namespace: "")
I0111 20:20:05.799907 00001 handler.go:322] Added event &{/system.slice/ssh.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.800021 00001 container.go:368] Start housekeeping for container "/system.slice/ssh.service"
I0111 20:20:05.801985 00001 factory.go:71] Error trying to work out if we can handle /system.slice/sys-kernel-debug.mount: error inspecting container: No such container: sys-kernel-debug.mount
I0111 20:20:05.802035 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/sys-kernel-debug.mount"
I0111 20:20:05.802083 00001 factory.go:78] Using factory "raw" for container "/system.slice/sys-kernel-debug.mount"
I0111 20:20:05.802247 00001 manager.go:798] Added container: "/system.slice/sys-kernel-debug.mount" (aliases: [], namespace: "")
I0111 20:20:05.802387 00001 handler.go:322] Added event &{/system.slice/sys-kernel-debug.mount 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.802451 00001 container.go:368] Start housekeeping for container "/system.slice/sys-kernel-debug.mount"
I0111 20:20:05.803421 00001 factory.go:71] Error trying to work out if we can handle /system.slice/sysinfo.service: error inspecting container: No such container: sysinfo.service
I0111 20:20:05.803466 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/sysinfo.service"
I0111 20:20:05.803489 00001 factory.go:78] Using factory "raw" for container "/system.slice/sysinfo.service"
I0111 20:20:05.803648 00001 manager.go:798] Added container: "/system.slice/sysinfo.service" (aliases: [], namespace: "")
I0111 20:20:05.803807 00001 handler.go:322] Added event &{/system.slice/sysinfo.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.803865 00001 container.go:368] Start housekeeping for container "/system.slice/sysinfo.service"
I0111 20:20:05.804508 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-journald.service: error inspecting container: No such container: systemd-journald.service
I0111 20:20:05.804540 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-journald.service"
I0111 20:20:05.804593 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-journald.service"
I0111 20:20:05.804737 00001 manager.go:798] Added container: "/system.slice/systemd-journald.service" (aliases: [], namespace: "")
I0111 20:20:05.804850 00001 handler.go:322] Added event &{/system.slice/systemd-journald.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.804954 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-journald.service"
I0111 20:20:05.805704 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-setup-dgram-qlen.service: error inspecting container: No such container: systemd-setup-dgram-qlen.service
I0111 20:20:05.805740 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-setup-dgram-qlen.service"
I0111 20:20:05.805753 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-setup-dgram-qlen.service"
I0111 20:20:05.805930 00001 manager.go:798] Added container: "/system.slice/systemd-setup-dgram-qlen.service" (aliases: [], namespace: "")
I0111 20:20:05.806094 00001 handler.go:322] Added event &{/system.slice/systemd-setup-dgram-qlen.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.806143 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-setup-dgram-qlen.service"
I0111 20:20:05.806936 00001 factory.go:71] Error trying to work out if we can handle /system.slice/dbus.service: error inspecting container: No such container: dbus.service
I0111 20:20:05.806968 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/dbus.service"
I0111 20:20:05.806990 00001 factory.go:78] Using factory "raw" for container "/system.slice/dbus.service"
I0111 20:20:05.807153 00001 manager.go:798] Added container: "/system.slice/dbus.service" (aliases: [], namespace: "")
I0111 20:20:05.807285 00001 handler.go:322] Added event &{/system.slice/dbus.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.807341 00001 container.go:368] Start housekeeping for container "/system.slice/dbus.service"
I0111 20:20:05.808130 00001 factory.go:71] Error trying to work out if we can handle /system.slice/docker.service: error inspecting container: No such container: docker.service
I0111 20:20:05.808144 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/docker.service"
I0111 20:20:05.808155 00001 factory.go:78] Using factory "raw" for container "/system.slice/docker.service"
I0111 20:20:05.808322 00001 manager.go:798] Added container: "/system.slice/docker.service" (aliases: [], namespace: "")
I0111 20:20:05.808417 00001 handler.go:322] Added event &{/system.slice/docker.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.808469 00001 container.go:368] Start housekeeping for container "/system.slice/docker.service"
I0111 20:20:05.809226 00001 factory.go:71] Error trying to work out if we can handle /system.slice/networking.service: error inspecting container: No such container: networking.service
I0111 20:20:05.809242 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/networking.service"
I0111 20:20:05.809253 00001 factory.go:78] Using factory "raw" for container "/system.slice/networking.service"
I0111 20:20:05.809354 00001 manager.go:798] Added container: "/system.slice/networking.service" (aliases: [], namespace: "")
I0111 20:20:05.809417 00001 handler.go:322] Added event &{/system.slice/networking.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.809468 00001 container.go:368] Start housekeeping for container "/system.slice/networking.service"
I0111 20:20:05.809951 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-tmpfiles-setup-dev.service: error inspecting container: No such container: systemd-tmpfiles-setup-dev.service
I0111 20:20:05.810075 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-tmpfiles-setup-dev.service"
I0111 20:20:05.810120 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-tmpfiles-setup-dev.service"
I0111 20:20:05.810277 00001 manager.go:798] Added container: "/system.slice/systemd-tmpfiles-setup-dev.service" (aliases: [], namespace: "")
I0111 20:20:05.810383 00001 handler.go:322] Added event &{/system.slice/systemd-tmpfiles-setup-dev.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.810431 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-tmpfiles-setup-dev.service"
I0111 20:20:05.811092 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-tmpfiles-setup.service: error inspecting container: No such container: systemd-tmpfiles-setup.service
I0111 20:20:05.811107 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-tmpfiles-setup.service"
I0111 20:20:05.811120 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-tmpfiles-setup.service"
I0111 20:20:05.811210 00001 manager.go:798] Added container: "/system.slice/systemd-tmpfiles-setup.service" (aliases: [], namespace: "")
I0111 20:20:05.811290 00001 handler.go:322] Added event &{/system.slice/systemd-tmpfiles-setup.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.811365 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-tmpfiles-setup.service"
I0111 20:20:05.811992 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-user-sessions.service: error inspecting container: No such container: systemd-user-sessions.service
I0111 20:20:05.812009 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-user-sessions.service"
I0111 20:20:05.812020 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-user-sessions.service"
I0111 20:20:05.812126 00001 manager.go:798] Added container: "/system.slice/systemd-user-sessions.service" (aliases: [], namespace: "")
I0111 20:20:05.812189 00001 handler.go:322] Added event &{/system.slice/systemd-user-sessions.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.812266 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-user-sessions.service"
I0111 20:20:05.813371 00001 factory.go:71] Error trying to work out if we can handle /system.slice/cloud-init-local.service: error inspecting container: No such container: cloud-init-local.service
I0111 20:20:05.813398 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/cloud-init-local.service"
I0111 20:20:05.813414 00001 factory.go:78] Using factory "raw" for container "/system.slice/cloud-init-local.service"
I0111 20:20:05.813512 00001 manager.go:798] Added container: "/system.slice/cloud-init-local.service" (aliases: [], namespace: "")
I0111 20:20:05.813591 00001 handler.go:322] Added event &{/system.slice/cloud-init-local.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.813642 00001 container.go:368] Start housekeeping for container "/system.slice/cloud-init-local.service"
I0111 20:20:05.814450 00001 factory.go:71] Error trying to work out if we can handle /system.slice/cron.service: error inspecting container: No such container: cron.service
I0111 20:20:05.814474 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/cron.service"
I0111 20:20:05.814494 00001 factory.go:78] Using factory "raw" for container "/system.slice/cron.service"
I0111 20:20:05.814604 00001 manager.go:798] Added container: "/system.slice/cron.service" (aliases: [], namespace: "")
I0111 20:20:05.814703 00001 handler.go:322] Added event &{/system.slice/cron.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.814769 00001 container.go:368] Start housekeeping for container "/system.slice/cron.service"
I0111 20:20:05.816147 00001 factory.go:78] Using factory "docker" for container "/system.slice/docker-1833c88c617a80f2182b7dd588f119db49175ec75e303f42ba9b715703ddd75f.scope"
I0111 20:20:05.817461 00001 manager.go:798] Added container: "/system.slice/docker-1833c88c617a80f2182b7dd588f119db49175ec75e303f42ba9b715703ddd75f.scope" (aliases: [container-exporter 1833c88c617a80f2182b7dd588f119db49175ec75e303f42ba9b715703ddd75f], namespace: "docker")
I0111 20:20:05.817893 00001 handler.go:322] Added event &{/system.slice/docker-1833c88c617a80f2182b7dd588f119db49175ec75e303f42ba9b715703ddd75f.scope 2016-01-11 20:16:15.794547335 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.817942 00001 container.go:368] Start housekeeping for container "/system.slice/docker-1833c88c617a80f2182b7dd588f119db49175ec75e303f42ba9b715703ddd75f.scope"
I0111 20:20:05.819276 00001 factory.go:78] Using factory "docker" for container "/system.slice/docker-4dfcb5a1439c7c93fb3ce1ad9a0b35b27ef6b0bf8c916dbd95374d9ae21f4ded.scope"
I0111 20:20:05.820637 00001 manager.go:798] Added container: "/system.slice/docker-4dfcb5a1439c7c93fb3ce1ad9a0b35b27ef6b0bf8c916dbd95374d9ae21f4ded.scope" (aliases: [grafana 4dfcb5a1439c7c93fb3ce1ad9a0b35b27ef6b0bf8c916dbd95374d9ae21f4ded], namespace: "docker")
I0111 20:20:05.821048 00001 handler.go:322] Added event &{/system.slice/docker-4dfcb5a1439c7c93fb3ce1ad9a0b35b27ef6b0bf8c916dbd95374d9ae21f4ded.scope 2016-01-05 21:55:47.398292817 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.821096 00001 container.go:368] Start housekeeping for container "/system.slice/docker-4dfcb5a1439c7c93fb3ce1ad9a0b35b27ef6b0bf8c916dbd95374d9ae21f4ded.scope"
I0111 20:20:05.822090 00001 factory.go:78] Using factory "docker" for container "/system.slice/docker-ac25b22d15d4097f93b3f7ca2349cb1c0dfe90537d336193f08aeb1d42f71a68.scope"
I0111 20:20:05.823574 00001 manager.go:798] Added container: "/system.slice/docker-ac25b22d15d4097f93b3f7ca2349cb1c0dfe90537d336193f08aeb1d42f71a68.scope" (aliases: [consul-exporter ac25b22d15d4097f93b3f7ca2349cb1c0dfe90537d336193f08aeb1d42f71a68], namespace: "docker")
I0111 20:20:05.823967 00001 handler.go:322] Added event &{/system.slice/docker-ac25b22d15d4097f93b3f7ca2349cb1c0dfe90537d336193f08aeb1d42f71a68.scope 2015-12-04 08:29:34.425481508 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.824023 00001 container.go:368] Start housekeeping for container "/system.slice/docker-ac25b22d15d4097f93b3f7ca2349cb1c0dfe90537d336193f08aeb1d42f71a68.scope"
I0111 20:20:05.824944 00001 factory.go:71] Error trying to work out if we can handle /system.slice/kmod-static-nodes.service: error inspecting container: No such container: kmod-static-nodes.service
I0111 20:20:05.824983 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/kmod-static-nodes.service"
I0111 20:20:05.824996 00001 factory.go:78] Using factory "raw" for container "/system.slice/kmod-static-nodes.service"
I0111 20:20:05.825143 00001 manager.go:798] Added container: "/system.slice/kmod-static-nodes.service" (aliases: [], namespace: "")
I0111 20:20:05.825228 00001 handler.go:322] Added event &{/system.slice/kmod-static-nodes.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.825341 00001 container.go:368] Start housekeeping for container "/system.slice/kmod-static-nodes.service"
I0111 20:20:05.826009 00001 factory.go:71] Error trying to work out if we can handle /system.slice/ntp.service: error inspecting container: No such container: ntp.service
I0111 20:20:05.826029 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/ntp.service"
I0111 20:20:05.826039 00001 factory.go:78] Using factory "raw" for container "/system.slice/ntp.service"
I0111 20:20:05.826185 00001 manager.go:798] Added container: "/system.slice/ntp.service" (aliases: [], namespace: "")
I0111 20:20:05.826292 00001 handler.go:322] Added event &{/system.slice/ntp.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.826389 00001 container.go:368] Start housekeeping for container "/system.slice/ntp.service"
I0111 20:20:05.827069 00001 factory.go:71] Error trying to work out if we can handle /system.slice/rc-local.service: error inspecting container: No such container: rc-local.service
I0111 20:20:05.827104 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/rc-local.service"
I0111 20:20:05.827204 00001 factory.go:78] Using factory "raw" for container "/system.slice/rc-local.service"
I0111 20:20:05.827427 00001 manager.go:798] Added container: "/system.slice/rc-local.service" (aliases: [], namespace: "")
I0111 20:20:05.827503 00001 handler.go:322] Added event &{/system.slice/rc-local.service 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.827567 00001 container.go:368] Start housekeeping for container "/system.slice/rc-local.service"
I0111 20:20:05.828282 00001 factory.go:71] Error trying to work out if we can handle /system.slice/system-serial\x2dgetty.slice: error inspecting container: No such container: system-serial\x2dgetty.slice
I0111 20:20:05.828296 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/system-serial\\x2dgetty.slice"
I0111 20:20:05.828303 00001 factory.go:78] Using factory "raw" for container "/system.slice/system-serial\\x2dgetty.slice"
I0111 20:20:05.828425 00001 manager.go:798] Added container: "/system.slice/system-serial\\x2dgetty.slice" (aliases: [], namespace: "")
I0111 20:20:05.828502 00001 handler.go:322] Added event &{/system.slice/system-serial\x2dgetty.slice 2016-01-05 01:35:21.033221591 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.828593 00001 container.go:368] Start housekeeping for container "/system.slice/system-serial\\x2dgetty.slice"
I0111 20:20:05.829680 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-sysctl.service: error inspecting container: No such container: systemd-sysctl.service
I0111 20:20:05.829723 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-sysctl.service"
I0111 20:20:05.829771 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-sysctl.service"
I0111 20:20:05.829999 00001 manager.go:798] Added container: "/system.slice/systemd-sysctl.service" (aliases: [], namespace: "")
I0111 20:20:05.830141 00001 handler.go:322] Added event &{/system.slice/systemd-sysctl.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.830191 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-sysctl.service"
I0111 20:20:05.831006 00001 factory.go:71] Error trying to work out if we can handle /system.slice/systemd-udev-trigger.service: error inspecting container: No such container: systemd-udev-trigger.service
I0111 20:20:05.831024 00001 factory.go:82] Factory "docker" was unable to handle container "/system.slice/systemd-udev-trigger.service"
I0111 20:20:05.831035 00001 factory.go:78] Using factory "raw" for container "/system.slice/systemd-udev-trigger.service"
I0111 20:20:05.831212 00001 manager.go:798] Added container: "/system.slice/systemd-udev-trigger.service" (aliases: [], namespace: "")
I0111 20:20:05.831328 00001 handler.go:322] Added event &{/system.slice/systemd-udev-trigger.service 2016-01-05 01:35:21.037221544 +0000 UTC containerCreation {<nil>}}
I0111 20:20:05.831403 00001 manager.go:250] Recovery completed
I0111 20:20:05.831426 00001 container.go:368] Start housekeeping for container "/system.slice/systemd-udev-trigger.service"
I0111 20:20:05.859808 00001 cadvisor.go:94] Starting cAdvisor version: "0.18.0" on port 8080

[vishh]

I0111 20:20:05.495421 00001 machine.go:49] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"

This is not an issue. This file is distro specific.

I0111 20:20:05.749432 00001 factory.go:71] Error trying to work out if we can handle /: error inspecting container: unexpected end of JSON input

This log line sounds like a bug in the code. It should be harmless in any case.

The logs you posted is for the setup phase and does not seem to highlight any errors. Are there any more logs that refer to metrics collection after cAdvisor is up logically?

Is cAdvisor running in privileged mode?

[achille-roussel]

Nothing is being logged after starting (which I find strange), but metrics are definitely getting updated, it's being fetched into prometheus and graphed via grafana, all working well except for memory, tasks and IO which are all at zero.

And yes, cAdvisor is running as root (in the container):

$ docker exec container-exporter ps ax | grep cadvisor
    1 root     /usr/bin/cadvisor -enable_load_reader=true -logtostderr=true -log_cadvisor_usage=true -v=4

[achille-roussel]

So as far as I understand the memory stats are pulled from this https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/fs/memory.go#L101

I've scanned the host and container file systems and couldn't find any trace of a memory.stat file, do you know if that could be the cause of the problem?

It's weird that no error is reported by cAdvisor tho.

[vishh]

Ah! You need to enable memory cgroup. This is a kernel configuration.
Add a kernel boot option "cgroup_enable=memory" to your bootloader setting
(e.g. /etc/default/grub) to enable it.

On Mon, Jan 11, 2016 at 3:01 PM, Achille notifications@github.com wrote:

So as far as I understand the memory stats are pulled from this
https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/fs/memory.go#L101

I've scanned the host file system and couldn't find any trace of a
memory.stat file, do you know if that could be the cause of the problem?

It's weird that no error is reported by cAdvisor tho.

—
Reply to this email directly or view it on GitHub
#1039 (comment).

[vishh]

I'm closing this issue. Feel free to re-open if there are any more concerns.

[achille-roussel]

I'm wondering if it may be related to this:

• https://docs.docker.com/engine/articles/runmetrics/
• "WARNING: No {swap,memory} limit support" Ubuntu LTS (Precise) moby/moby#4250
• Debian stock kernel doesn't support limiting memory moby/moby#396

I'm gonna set these config options and see if that fixed the issue with memory stats.

[achille-roussel]

Yep that worked!

Adding GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1" then running update-grub && reboot, then I've got this in the logs:

I0111 23:21:15.384420 00001 container.go:410] [/system.slice/docker-574001368f978a8e11caa8b471cebda7d741b010eb0b3543b34f3c17837563fd.scope] 0.046 cores (average: 0.068 cores), 33.37 MB of memory

And the memory metrics are being reported properly now.

Thanks a lot for your help, hopefully this will be useful to someone else as well.

[jamshid]

Thanks @achille-roussel and looks like it's documented officially below. I'm surprised docker-machine doesn't do this grub change automatically, or at least warn that you might want to.
https://docs.docker.com/engine/installation/linux/ubuntulinux/#adjust-memory-and-swap-accounting

[ajmathews]

Thanks @achille-roussel and looks like it's documented officially below. I'm surprised docker-machine doesn't do this grub change automatically, or at least warn that you might want to. https://docs.docker.com/engine/installation/linux/ubuntulinux/#adjust-memory-and-swap-accounting

Updated link to the docker doc above,
https://docs.docker.com/config/containers/runmetrics/

[sarpuser]

I am having trouble following the links and the comments from above. I am on a Raspberry Pi and running into the issue of my memory metrics not reporting correctly. Would anyone be able to point me in the correct direction (Pi does not use GRUB). Thanks!

[Kleinendorst]

I am having trouble following the links and the comments from above. I am on a Raspberry Pi and running into the issue of my memory metrics not reporting correctly. Would anyone be able to point me in the correct direction (Pi does not use GRUB). Thanks!

I was also facing this issue using a RaspberryPi, this suggestion fixed it for me (also a reboot was required).