Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove btrfs quota groups after containers destroyed #29427

Merged
merged 2 commits into from
Jun 6, 2017

Conversation

yongtang
Copy link
Member

@yongtang yongtang commented Dec 14, 2016

- What I did

This fix tries to address the issue raised in #29325 where btrfs quota groups are not clean up even after containers have been destroyed.

- How I did it

The reason for the issue is that btrfs quota groups have to be explicitly destroyed. This fix fixes this issue.

- How to verify it

This fix is tested manually in Ubuntu 16.04:

pre-fix behavior with qgroup leftover:

ubuntu@ubuntu:~$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for busybox:latest
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257       112.00KiB    112.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker run --name top -d busybox top
b59e6e3da6ed3e601f081c11c44d3f4dd44aefee5584d34ba913e68207c9694d
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257         1.17MiB     64.00KiB
0/258         1.17MiB     16.00KiB
0/259         1.17MiB     16.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker rm -f top
top
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         144.00KiB    144.00KiB
0/257         1.17MiB      1.17MiB
0/258           0.00B        0.00B
0/259           0.00B        0.00B

PR with qgroup cleaned (non --live-restore case):

ubuntu@ubuntu:~$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for busybox:latest
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257       112.00KiB    112.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker run --name top -d busybox top
201d444f74943247c7794fae2215f763cad0dd726d03285004c54ffa04882d6b
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257         1.17MiB     64.00KiB
0/258         1.17MiB     16.00KiB
0/259         1.17MiB     16.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker rm -f top
top
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         144.00KiB    144.00KiB
0/257         1.17MiB      1.17MiB

PR with qgroup cleaned (--live-restore case):

# [sudo dockerd --debug --live-restore -s btrfs --storage-opt btrfs.min_space=1G]

ubuntu@ubuntu:~$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for busybox:latest
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257       112.00KiB    112.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker run --name top -d busybox top
73c2184a4414d47b5f5dc27cd2ee28116aa761f947196618ca66bf05c4705456
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257         1.17MiB     64.00KiB
0/258         1.17MiB     16.00KiB
0/259         1.17MiB     16.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$

# [Ctrl-C ]
# [sudo dockerd --debug --live-restore -s btrfs]

ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
73c2184a4414        busybox             "top"               About a minute ago   Up About a minute                       top
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         152.00KiB    152.00KiB
0/257         1.17MiB     64.00KiB
0/258         1.17MiB     48.00KiB
0/259         1.17MiB     48.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker rm -f top
top
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         168.00KiB    168.00KiB
0/257         1.17MiB     64.00KiB

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

c03904d6028381a6bd0d1b1bfd1523bb

This fix fixes #29325.

Signed-off-by: Yong Tang yong.tang.github@outlook.com

@yongtang yongtang force-pushed the 29325-cleanup-btrfs-quota-groups branch 2 times, most recently from 421140b to 57c74de Compare December 15, 2016 16:45
@yongtang
Copy link
Member Author

/cc @justincormack @mlaventure

@zhuguihua
Copy link
Contributor

Sorry for late reply. LGTM.

@yongtang yongtang force-pushed the 29325-cleanup-btrfs-quota-groups branch from 57c74de to 44dee9c Compare January 3, 2017 01:47
@yongtang
Copy link
Member Author

yongtang commented Jan 3, 2017

Previously the PR didn't take into consideration that, if quotaEnabled is false, there is no need to clean up the qgroup (See #29810 for similar issue)

Now the PR has been updated to add the check for quotaEnabled.

@thaJeztah
Copy link
Member

Also wondering the same here as on #29835 (comment)

@mlaventure
Copy link
Contributor

I had a quick look at the code, it looks like @thaJeztah may be correct.

Would wait on @cpuguy83 for confirmation

@thaJeztah
Copy link
Member

if I'm correct, we may need some mechanism to detect if a quota is set (perhaps querying btrfs), otherwise it's possible we don't know when to cleanup / manage quotas

@cpuguy83
Copy link
Member

cpuguy83 commented Jan 3, 2017

I think you are right. On daemon restore it could really be a problem.

@LK4D4
Copy link
Contributor

LK4D4 commented Jan 26, 2017

@cpuguy83 @tonistiigi @yongtang so, what we're going to do with this and #29385
If we have no proper solution for now, I propose to close PRs.
Thanks!

@yongtang
Copy link
Member Author

@cpuguy83 @tonistiigi @LK4D4 This PR (#29427) and the other PR (#29835) are related.

I updated the other PR #29835 first to try to address the issue by looking up the qgroup and see if it returns something.

Please take a look at the (#29835) first and see if it address the concerns.

@yongtang yongtang force-pushed the 29325-cleanup-btrfs-quota-groups branch from 44dee9c to ba10de8 Compare January 27, 2017 07:39
@yongtang
Copy link
Member Author

Here is the complete steps to verify that the issue is fixed (Including --live-restore case).

Would be good to setup integration test for btrfs in Jenkins

Note: this PR has been updated to incorporate #30497. If this one is merged then #30497 could be closed

pre-fix behavior with qgroup leftover:

ubuntu@ubuntu:~$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for busybox:latest
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257       112.00KiB    112.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker run --name top -d busybox top
b59e6e3da6ed3e601f081c11c44d3f4dd44aefee5584d34ba913e68207c9694d
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257         1.17MiB     64.00KiB
0/258         1.17MiB     16.00KiB
0/259         1.17MiB     16.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker rm -f top
top
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         144.00KiB    144.00KiB
0/257         1.17MiB      1.17MiB
0/258           0.00B        0.00B
0/259           0.00B        0.00B

PR with qgroup cleaned (non --live-restore case):

ubuntu@ubuntu:~$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for busybox:latest
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257       112.00KiB    112.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker run --name top -d busybox top
201d444f74943247c7794fae2215f763cad0dd726d03285004c54ffa04882d6b
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257         1.17MiB     64.00KiB
0/258         1.17MiB     16.00KiB
0/259         1.17MiB     16.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker rm -f top
top
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         144.00KiB    144.00KiB
0/257         1.17MiB      1.17MiB

PR with qgroup cleaned (--live-restore case):

# [sudo dockerd --debug --live-restore -s btrfs --storage-opt btrfs.min_space=1G]

ubuntu@ubuntu:~$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for busybox:latest
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257       112.00KiB    112.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker run --name top -d busybox top
73c2184a4414d47b5f5dc27cd2ee28116aa761f947196618ca66bf05c4705456
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         128.00KiB    128.00KiB
0/257         1.17MiB     64.00KiB
0/258         1.17MiB     16.00KiB
0/259         1.17MiB     16.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$

# [Ctrl-C ]
# [sudo dockerd --debug --live-restore -s btrfs]

ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
73c2184a4414        busybox             "top"               About a minute ago   Up About a minute                       top
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         152.00KiB    152.00KiB
0/257         1.17MiB     64.00KiB
0/258         1.17MiB     48.00KiB
0/259         1.17MiB     48.00KiB
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker rm -f top
top
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ sudo btrfs qgroup show /var/lib/docker/btrfs
qgroupid         rfer         excl
--------         ----         ----
0/5         168.00KiB    168.00KiB
0/257         1.17MiB     64.00KiB

@@ -320,7 +337,7 @@ func (d *Driver) subvolEnableQuota() error {
func (d *Driver) subvolDisableQuota() error {
if !d.quotaEnabled {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be turned into a isQuotaEnabled() routine to avoid this code to be duplicated?

Also, quotaEnabled is just a boolean, but don't we need a lock around it?

@mlaventure
Copy link
Contributor

ping @cpuguy83 @tonistiigi

@yongtang yongtang force-pushed the 29325-cleanup-btrfs-quota-groups branch 2 times, most recently from d7f3c09 to 4d312d6 Compare February 13, 2017 18:41
@yongtang
Copy link
Member Author

Thanks @mlaventure for the review. The PR has been updated with isQuotaEnabled() and lock added. Please take a look and let me know if there are any issues.

@vdemeester vdemeester added this to the 1.14.0 milestone Feb 13, 2017
}

func (d *Driver) subvolEnableQuota() error {
d.mu.Lock()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move the lock in isQuotaEnabled?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mlaventure.

Inside the subvolEnableQuota the d.quotaEnabled will be changed again to true at the end of subvolEnableQuota (located in Line 343). For that I placed the Lock() outside of isQuotaEnabled().

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, my bad. Thanks for clarifying.

@mlaventure
Copy link
Contributor

ping @cpuguy83 @tonistiigi this is waiting for reviews 👼

@thaJeztah
Copy link
Member

ping @cpuguy83 @tonistiigi PTAL

@cpuguy83
Copy link
Member

cpuguy83 commented Apr 5, 2017

This is not re-applying qutoas on daemon restart.

$ docker run -d --name test --storage-opt size=100m busybox top
$ docker exec test sh -c 'dd if=/dev/urandom of=/data bs=1 count=200' # errors around 100M
$ # restart docker
$ # note that `btrfs qgroup show` does not find the qgroup
$ docker start test
$ # note still no qgroup
$ docker exec test sh -c 'dd if=/dev/urandom of=/data bs=1 count=200'  # write successful, no qutoa

@cpuguy83
Copy link
Member

cpuguy83 commented Apr 5, 2017

It looks like we need to rescan for quota in Get()... or not disable quota on Cleanup()

@cpuguy83
Copy link
Member

cpuguy83 commented Apr 5, 2017

Or I guess write out a file to trigger an auto-enable of the quota on restart.

@mlaventure
Copy link
Contributor

ping @yongtang 👼

@thaJeztah
Copy link
Member

ping @yongtang do you still have time to work on this?

@thaJeztah thaJeztah modified the milestones: 17.06.0, 17.05.0 May 12, 2017
@yongtang
Copy link
Member Author

@thaJeztah Sorry for the late response. Let me spend the next several days to get this one fixed.

@yongtang yongtang force-pushed the 29325-cleanup-btrfs-quota-groups branch 2 times, most recently from 8106d15 to 9792f99 Compare May 25, 2017 01:02
@yongtang
Copy link
Member Author

@cpuguy83 The PR has been updated. Now the quota is saved in /lib/var/docker/btrfs/quotas and recovered in case of a daemon restart. Not sure this is the best place to persist so any suggestions would be greatly appreciated. Please take a look.

@thaJeztah
Copy link
Member

ping @cpuguy83

also @cyphar, who may be interested in this 😃

@sargun
Copy link
Contributor

sargun commented May 30, 2017

Alternatively, you can just wait for this patchset to land: http://www.spinics.net/lists/linux-btrfs/msg65928.html

@thaJeztah
Copy link
Member

@sargun nice! Unfortunately we don't control the host, so cannot guarantee that patch is present on each distro / version, so not sure we can rely on that

if quota, err := ioutil.ReadFile(d.quotasDirID(id)); err == nil {
if size, err := strconv.ParseUint(string(quota), 10, 64); err == nil && size >= d.options.minSpace {
d.subvolEnableQuota()
subvolLimitQgroup(dir, size)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error check?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -557,6 +635,13 @@ func (d *Driver) Get(id, mountLabel string) (string, error) {
return "", fmt.Errorf("%s: not a directory", dir)
}

if quota, err := ioutil.ReadFile(d.quotasDirID(id)); err == nil {
if size, err := strconv.ParseUint(string(quota), 10, 64); err == nil && size >= d.options.minSpace {
d.subvolEnableQuota()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error check?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -533,7 +606,12 @@ func (d *Driver) Remove(id string) error {
if _, err := os.Stat(dir); err != nil {
return err
}
if err := subvolDelete(d.subvolumesDir(), id); err != nil {
os.Remove(d.quotasDirID(id))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error check?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cpuguy83. Done.

if err := idtools.MkdirAllAs(quotas, 0700, rootUID, rootGID); err != nil {
return err
}
if err := ioutil.WriteFile(path.Join(quotas, id), []byte(fmt.Sprint(driver.options.size)), 0666); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

644?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cpuguy83. The PR has been updated.

This fix tries to address the issue raised in 29325 where
btrfs quota groups are not clean up even after containers
have been destroyed.

The reason for the issue is that btrfs quota groups have
to be explicitly destroyed. This fix fixes this issue.

This fix is tested manually in Ubuntu 16.04,
with steps specified in 29325.

This fix fixes 29325.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This commit is an extension of fix for 29325 based on the review comment.
In this commit, the quota size for btrfs is kept in `/var/lib/docker/btrfs/quotas`
so that a daemon restart keeps quota.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
@yongtang yongtang force-pushed the 29325-cleanup-btrfs-quota-groups branch from 9792f99 to 16328cc Compare June 2, 2017 04:16
@yongtang
Copy link
Member Author

yongtang commented Jun 2, 2017

Thanks @cpuguy83 for the review. The PR has been updated. Please take a look.

Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yongtang
Copy link
Member Author

yongtang commented Jun 6, 2017

All tests passed now, merging...

@yongtang yongtang merged commit 54dacaf into moby:master Jun 6, 2017
@yongtang yongtang deleted the 29325-cleanup-btrfs-quota-groups branch June 6, 2017 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Docker does not remove Btrfs quota groups when destroying container
10 participants