when docker use --cpu-quota and systemd daemon-reload, old container's cpu.cfs_quota_us become -1

[suxiaobei1024]

hello ,I got a problem , But I am not sure it is systemd's problem or docker's problem, It is easy to reproduce.

When I use docker run --cpu-quota=50000 --cpu-period=100000 -d docker.io/centos /sbin/initto start the first container A, and then execute systemctl daemon-reload ; then , I start the second docker container , the first docker container's value cpu.cfs_quota_us will become -1

Is systemd's logical or is docker's bug?

docker version

[root@iZrj9cft0g87g90carlmnsZ ~]# docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-253:1-401976-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 255.8 MB
 Data Space Total: 107.4 GB
 Data Space Available: 40.5 GB
 Metadata Space Used: 897 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.135-RHEL7 (2016-09-28)
Execution Driver: native-0.2
Logging Driver: journald
Plugins:
 Volume: local
 Network: host bridge null
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 4
Total Memory: 15.51 GiB
Name: iZrj9cft0g87g90carlmnsZ
ID: 2P26:F7ZL:RTPF:IYEM:RUK7:MGTF:J5IR:O66A:KF3I:RX66:USZB:4QUC
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Registries: docker.io (secure)

[root@iZrj9cft0g87g90carlmnsZ ~]# docker version
Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-59.el7.centos.x86_64
 Go version:      go1.6.3
 Git commit:      3999ccb-unsupported
 Built:           Thu Dec 15 17:24:43 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-59.el7.centos.x86_64
 Go version:      go1.6.3
 Git commit:      3999ccb-unsupported
 Built:           Thu Dec 15 17:24:43 2016
 OS/Arch:         linux/amd64

[root@iZrj9cft0g87g90carlmnsZ ~]# rpm -qa | grep systemd
systemd-libs-219-30.el7_3.6.x86_64
systemd-219-30.el7_3.6.x86_64
systemd-sysv-219-30.el7_3.6.x86_64
oci-systemd-hook-0.1.4-6.git337078c.el7.x86_64

[root@iZrj9cft0g87g90carlmnsZ ~]# uname -r
3.10.0-327.22.2.el7.x86_64

How to reproduce

1. docker use --cpu-quota start a container

[root@iZrj9cft0g87g90carlmnsZ ~]# docker run --cpu-quota=50000 --cpu-period=100000 -d docker.io/centos /sbin/init
19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20
[root@iZrj9cft0g87g90carlmnsZ ~]#

[root@iZrj9cft0g87g90carlmnsZ ~]# systemd-cgls  | grep 19d5b47bbedd
│     └─2959 grep --color=auto 19d5b47bbedd
  ├─docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope
  

[root@iZrj9cft0g87g90carlmnsZ ~]# systemctl cat docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope

# /run/systemd/system/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope.d/50-BlockIOAccounting.conf
[Scope]
BlockIOAccounting=yes
# /run/systemd/system/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope.d/50-CPUAccounting.conf
[Scope]
CPUAccounting=yes
# /run/systemd/system/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope.d/50-DefaultDependencies.conf
[Unit]
DefaultDependencies=no
# /run/systemd/system/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope.d/50-Delegate.conf
[Scope]
Delegate=yes
# /run/systemd/system/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope.d/50-Description.conf
[Unit]
Description=docker container 19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20
# /run/systemd/system/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope.d/50-MemoryAccounting.conf
[Scope]
MemoryAccounting=yes
# /run/systemd/system/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope.d/50-Slice.conf
[Scope]
Slice=system.slice
[root@iZrj9cft0g87g90carlmnsZ ~]#

2. systemctl daemon-reload

we can see cpu.cfs_quota_us

[root@iZrj9cft0g87g90carlmnsZ ~]# cat /sys/fs/cgroup/cpu/system.slice/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope/cpu.cfs_quota_us
50000
[root@iZrj9cft0g87g90carlmnsZ ~]# systemctl daemon-reload
[root@iZrj9cft0g87g90carlmnsZ ~]#

3. start another new container

[root@iZrj9cft0g87g90carlmnsZ ~]# docker run --cpu-quota=50000 --cpu-period=100000 -d docker.io/centos /sbin/init
bdf76c111116d60252cb9075bc69f22da9ac970551fc6a55a49d0f8cee163441

4. old docker container's cpu.cfs_quota_us value become -1

[root@iZrj9cft0g87g90carlmnsZ ~]# cat /sys/fs/cgroup/cpu/system.slice/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope/cpu.cfs_quota_us
-1
[root@iZrj9cft0g87g90carlmnsZ ~]#

5. My Analysis

BUt , If I do systemctl set-property xxx CPUQuota=10% , and systemctl daemon-reload , I find , this value will not be changed , So I think if it is systemd's logic and it will likely to be a docker's bug ?

[root@iZrj9cft0g87g90carlmnsZ ~]# systemctl set-property docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope CPUQuota=10%
[root@iZrj9cft0g87g90carlmnsZ ~]#
[root@iZrj9cft0g87g90carlmnsZ ~]# cat /sys/fs/cgroup/cpu/system.slice/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope/cpu.cfs_quota_us
10000
[root@iZrj9cft0g87g90carlmnsZ ~]#
[root@iZrj9cft0g87g90carlmnsZ ~]# systemctl daemon-reload
[root@iZrj9cft0g87g90carlmnsZ ~]#
[root@iZrj9cft0g87g90carlmnsZ ~]# cat /sys/fs/cgroup/cpu/system.slice/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope/cpu.cfs_quota_us
10000
[root@iZrj9cft0g87g90carlmnsZ ~]#
[root@iZrj9cft0g87g90carlmnsZ ~]# docker run --cpu-quota=50000 --cpu-period=100000 -d docker.io/centos /sbin/init
4c77256624b929b62494766062526f506da445585923374628e4a5c30d34159b
[root@iZrj9cft0g87g90carlmnsZ ~]#
[root@iZrj9cft0g87g90carlmnsZ ~]# cat /sys/fs/cgroup/cpu/system.slice/docker-19d5b47bbedd802ade53eee6a0a2d53a9639d47f67be5247c3a9f6b55b5cde20.scope/cpu.cfs_quota_us
10000

[thaJeztah]

It looks like you're running the Red Hat fork of Docker, which containers various modifications to run "systemd" inside a container (e.g. the oci-systemd-hook). I highly suspect this is specific to their fork, and we can't really help with that. I suggest to report this issue in the Red Hat issue tracker, or try installing the official builds https://docs.docker.com/engine/installation/linux/centos/

Be aware though, that running systemd inside a container on the official builds requires you to give additional privileges to the container, because those privileges are blocked by default (for security reasons).

I'm closing this issue because of the above, but happy to answer questions after that

[xuxinkun]

@thaJeztah The daemon process in my container is just a java program. But I also face the same problem as @muahao say.
So, I think it may be a systemd or docker bug. I think it is better to reopen this issue and call more members to solve it. It really affect the old containers.

[thaJeztah]

Looks like I may have closed prematurely, and this may be related to systemd cgroups;

Using the default options (native.cgroupdriver=cgroupfs);

docker run --cpu-quota=50000 --cpu-period=100000 -d centos /sbin/init
d6833321af721240058e8b7668a6d67fd4c7857d672a9c15e9c976e6dd91dc3e

systemctl daemon-reload

cat /sys/fs/cgroup/cpu/docker/d6833321af721240058e8b7668a6d67fd4c7857d672a9c15e9c976e6dd91dc3e/cpu.cfs_quota_us
50000

docker run --cpu-quota=50000 --cpu-period=100000 -d centos /sbin/init
9c533c2fdf11641ba89d36d650012e19fbafaa67fee58877557f1ae518ba5f10

cat /sys/fs/cgroup/cpu/docker/d6833321af721240058e8b7668a6d67fd4c7857d672a9c15e9c976e6dd91dc3e/cpu.cfs_quota_us
50000

cat /sys/fs/cgroup/cpu/docker/9c533c2fdf11641ba89d36d650012e19fbafaa67fee58877557f1ae518ba5f10/cpu.cfs_quota_us
50000

Switching to native.cgroupdriver=systemd;

mkdir -p /etc/docker/
echo '{"exec-opts":["native.cgroupdriver=systemd"]}' > /etc/docker/daemon.json

systemctl restart docker

docker run --cpu-quota=50000 --cpu-period=100000 -d centos /sbin/init
fe0ea0071a597f654131856d031ade48fd63bb28e381492a61b45a807af6a53e

systemctl daemon-reload

cat /sys/fs/cgroup/cpu/system.slice/docker-fe0ea0071a597f654131856d031ade48fd63bb28e381492a61b45a807af6a53e.scope/cpu.cfs_quota_us
50000

docker run --cpu-quota=50000 --cpu-period=100000 -d centos /sbin/init
680203484ecdfc2b23d57218c0b01c4a3882e63b6cbed0583b7fe9a6e3111959

cat /sys/fs/cgroup/cpu/system.slice/docker-fe0ea0071a597f654131856d031ade48fd63bb28e381492a61b45a807af6a53e.scope/cpu.cfs_quota_us
-1

I think those cgroups are handled by RunC though @mlaventure ?

[xuxinkun]

@thaJeztah I have found the cause of the problem, and I have a solution to solve it. Later I will post a pull request of docker and make a link to this issue.

[thaJeztah]

Thanks @xuxinkun !

[xuxinkun]

cpu.cfs_quota_us and cpu.cfs_period_us should be controlled by systemd.
The percentage cpuQuota refered specifies how much CPU time the unit shall get at maximum, relative to the total CPU time available on one CPU. The percentage should be greater than 1%.
I fix this bug for runc.
Also I checkout a branch to fix it for docker v1.10.3. Here it is:https://github.com/xuxinkun/docker/commits/fixcpuQuota20170223.
I don't know how to make a PR from my branch to tag v1.10.3 of docker/docker. Is there anyway to do it? @thaJeztah

[thaJeztah]

Docker 1.10.3 is EOL and no longer maintained, so you'll have to open the change against "master". I think that part of the code is no longer in docker/docker though, and now in the RunC repository;

https://github.com/opencontainers/runc/blob/2940d2e2e9365b4b3fec8ae24822435195f56b61/libcontainer/cgroups/fs/cpu.go#L70-L91

[thaJeztah]

oh, actually, that link is for cgroupfs; guess this part is for systemd cgroups; https://github.com/opencontainers/runc/blob/2940d2e2e9365b4b3fec8ae24822435195f56b61/libcontainer/cgroups/systemd/apply_systemd.go#L168-L265

[xuxinkun]

Yes, I also make a PR for runc opencontainers/runc#1344

[thaJeztah]

Thanks!

[thaJeztah]

Let me reopen, because opencontainers/runc#1344 has not been vendored yet in this repository

[xuxinkun]

@thaJeztah Should I update the runc vendor or just wait some one to update it?

[thaJeztah]

@xuxinkun due to RunC not being "stable" yet (no 1.0 release), docker currently uses a fork, so we need that fork to be updated, and the patch cherry-picked into the 17.03.x branch; https://github.com/docker/runc/tree/17.03.x. Once that's done, the version can be bumped by updating the commit here; https://github.com/docker/docker/blob/9bf36cb4431b9f06f290658668415d9150618739/hack/dockerfile/binaries-commits#L6

ping @mlaventure @crosbymichael are you able to assist with this?