docker cp cause /dev/pts to be unmounted.

[baopx]

docker version:
[root@ip-172-31-25-218 porx]# docker version
Client:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765
Built: Thu Feb 11 20:39:58 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765
Built: Thu Feb 11 20:39:58 2016
OS/Arch: linux/amd64

docker info:
[root@ip-172-31-25-218 porx]# docker info
Containers: 3
Running: 2
Paused: 0
Stopped: 1
Images: 3
Server Version: 1.10.1
Storage Driver: btrfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local pxd
Network: null host bridge
Kernel Version: 3.19.3-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.304 GiB
Name: ip-172-31-25-218.us-west-1.compute.internal
ID: XRC7:XGJG:5VVJ:MO5H:LS4N:KTXO:HEKL:W57E:DNHE:STUN:QQ57:D6Q7
Username: XYZ
Registry: https://index.docker.io/v1/

Provide additional environment details (AWS, VirtualBox, physical, etc.):
AWS, EC2.

List the steps to reproduce the issue:

1. [root@ip-172-31-25-218 porx]# mount |grep pts
   devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
2. [root@ip-172-31-25-218 porx]# docker run -it --name test_container -v /dev/:/dev/ busybox
3. [root@ip-172-31-25-218 porx]# docker cp test_container:/bin/sh /tmp
4. [root@ip-172-31-25-218 porx]# mount |grep pts

Describe the results you received:
/dev/pts is unmounted.

Describe the results you expected:
I ran -v /dev:/dev to map it all the host's devices into the container. After docker cp, I expect the device to still be mounted on the host.

/dev/pts should not be unmount.

[AkihiroSuda]

I can also deterministically reproduce the bug, with any storage drivers (aufs/overlay/btrfs) on Ubuntu 15.10 amd64 on AWS.
(dfebb60, kernel 4.2.0-30-generic)

However, I cannot reproduce the bug on these environments:

• Boot2Docker 1.10.2 on VirtualBox
• Ubuntu 15.10 amd64 on baremetal box (docker 1.10.2-rc1, kernel 4.2.0-27-generic)
• Ubuntu 15.10 amd64 on baremetal box (docker 1.10.2-rc1, kernel 4.2.0-30-generic)

Oddly I can only reproduce the bug on AWS, but I don't think this is related to AWS..

[AkihiroSuda]

/dev/pts is unmounted within detachMounted("/var/lib/docker/aufs/mnt/foobar/dev") .

func detachMounted(path string) error {
    return syscall.Unmount(path, syscall.MNT_DETACH)
}

Call stack:

• https://github.com/docker/docker/blob/v1.10.2/container/container_unix.go#L655
• https://github.com/docker/docker/blob/v1.10.2/container/container_unix.go#L682
• https://github.com/docker/docker/blob/v1.10.2/daemon/archive.go#L157
• https://github.com/docker/docker/blob/v1.10.2/daemon/archive.go#L57
• ..

Printf-debug patch:

diff --git a/container/container_unix.go b/container/container_unix.go
index b8bae23..140e783 100644
--- a/container/container_unix.go
+++ b/container/container_unix.go
@@ -7,7 +7,9 @@ import (
        "io/ioutil"
        "net"
        "os"
+       "os/exec"
        "path/filepath"
+       "runtime/debug"
        "strconv"
        "strings"
        "syscall"
@@ -653,7 +655,19 @@ func (container *Container) UpdateContainer(hostConfig *containertypes.HostConfi
 }

 func detachMounted(path string) error {
-       return syscall.Unmount(path, syscall.MNT_DETACH)
+       grepDEV(fmt.Sprintf("detachMounted(%s) ENTER", path))
+       logrus.Debugf("STACK: %s", debug.Stack())
+       err := syscall.Unmount(path, syscall.MNT_DETACH)
+       grepDEV(fmt.Sprintf("detachMounted(%s) LEAVE", path))
+       return err
+}
+
+func grepDEV(id string) {
+       out, err := exec.Command("sh","-c","mount | grep \"on /dev\"").Output()
+       if err != nil {
+               panic(err)
+       }
+       logrus.Debugf("grepDEV(%s):\n%s\n", id, out)
 }

 // UnmountVolumes unmounts all volumes

[AkihiroSuda]

Identified the cause: systemd option MountFlags=slave is required to avoid the bug.
https://www.freedesktop.org/software/systemd/man/systemd.exec.html
Without this option, I can always hit the bug.

/lib/systemd/system/docker.service (default configuration, 1.10.2-0~wily)

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

[thaJeztah]

@AkihiroSuda so, this is a packaging issue? The 1.10.2 service file uses "slave" https://github.com/docker/docker/blob/v1.10.2/contrib/init/systemd/docker.service - where does the service file on AWS come from?

[AkihiroSuda]

@thaJeztah
I was manually running the daemon.
So this is not a packaging issue.

I think docker should not unmount the host pts even when running without systemd,
because user cannot login to the host after pts is unmounted.
This can be critical for operation.

Can we move MountFlags=slave logic from the systemd service into the docker daemon itself?
(As in POC AkihiroSuda/docker@f2158c1?)

[thaJeztah]

@AkihiroSuda can you open a PR with that change? We can then discuss the change on that PR

[baopx]

@AkihiroSuda sounds good..removing Mountflags=slave is not an option because mount propagation doesn’t work with that flag: #14630'

[AkihiroSuda]

@thaJeztah Sure, #20712

[sgallagher]

Just to note, this is also affecting users of Docker 1.10.3 on Fedora 24 systems. I have filed https://bugzilla.redhat.com/show_bug.cgi?id=1326532 downstream.

[AkihiroSuda]

The issue is still reproducible with 1.11-dev (348d902 + opencontainers/runc@445d184).

[AkihiroSuda]

non-docker repro instruction

$ mkdir /tmp/t1
$ mount -o rbind,rw /dev /tmp/t1/dev  # this mimics `daemon/container_operations_unix.go:mountVolumes()`
$ mount | grep pts
none on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
none on /tmp/t1/dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
$ umount -l /tmp/t1/dev # this mimics `daemon/container_operations_unix.go:detachMounted()`
$ mount | grep pts # /dev/pts should be kept, but we get an empty output here

I'm looking into this issue again.

Perhaps we can solve the issue by use bind instead of rbind, and do recursion manually

[rhvgoyal]

I have proposed a PR #22009 to solve this issue. See if it makes sense.