error mounting image volumes: error relabeling directory

[mbaldessari]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

Description
Trying to play with podman with the idea of using it in openstack. Run a podman run command which fails with:
root@foobar-0 ~]# podman run -d --name=test_podman-docker-2 -e PCMK_stderr=1
--net=host -e PCMK_remote_port=3121 -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro
--user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
docker.io/sdelrio/docker-minimal-nginx
error mounting image volumes: error relabeling directory "/var/lib/containers/storage/overlay-containers/cfc94bc4a68034da8bd0731189afd7425f24272c05bf8a1ea7129e1cbe51ba60/userdata/volumes/usr/share/nginx/html" for volume "/usr/share/nginx/html" in container %!q(func() string=0xe77020): no such file or directory

Also after the above failure I see the following:

[root@foobar-0 ~]# podman ps
[root@foobar-0 ~]# 

Interestingly though there seems to be an awkward state I am in:

[root@foobar-0 ~]# podman ps -a
CONTAINER ID   IMAGE                                           COMMAND                CREATED          STATUS    PORTS   NAMES
cfc94bc4a680   docker.io/sdelrio/docker-minimal-nginx:latest   nginx -g daemon off;   21 minutes ago   Created           test_podman-docker-2
  
[root@foobar-0 ~]# podman stop cfc94bc4a680
failed to stop container cfc94bc4a68034da8bd0731189afd7425f24272c05bf8a1ea7129e1cbe51ba60: can only stop created, running, or stopped containers: container state improper
[root@foobar-0 ~]# podman kill cfc94bc4a680
unable to find container cfc94bc4a680: can only kill running containers: container state improper

Note that Selinux is in permissive mode:

[root@foobar-0 ~]# getenforce
Permissive
[root@foobar-0 ~]# grep denied /var/log/audit/audit.log 
[root@foobar-0 ~]# 

Describe the results you received:

root@foobar-0 ~]# podman run -d --name=test_podman-docker-2 -e PCMK_stderr=1 --net=host -e PCMK_remote_port=3121 -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro --user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS docker.io/sdelrio/docker-minimal-nginx
error mounting image volumes: error relabeling directory "/var/lib/containers/storage/overlay-containers/cfc94bc4a68034da8bd0731189afd7425f24272c05bf8a1ea7129e1cbe51ba60/userdata/volumes/usr/share/nginx/html" for volume "/usr/share/nginx/html" in container %!q(func() string=0xe77020): no such file or directory

Describe the results you expected:
Docker seems to just work:

[root@foobar-0 ~]# docker run -d --name=test_podman-docker-2 -e PCMK_stderr=1 --net=host -e PCMK_remote_port=3121 -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro --user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS docker.io/sdelrio/docker-minimal-nginx
cbfa08a22b46c73b242f1e2ff8924375c7b7550a2f7f2b16b26b600294f2fc31

Additional information you deem important (e.g. issue happens only occasionally):
Reproduced this on centos7 using the following repos:
https://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/
https://cbs.centos.org/repos/paas7-crio-311-candidate/x86_64/os/

Output of podman version:

[root@foobar-0 ~]# podman version
Version:       0.6.4
Go Version:    go1.10.2
OS/Arch:       linux/amd64

Output of podman info:

[root@foobar-0 ~]# podman info
host:
  MemFree: 6562877440
  MemTotal: 8112017408
  SwapFree: 2147479552
  SwapTotal: 2147479552
  arch: amd64
  cpus: 4
  hostname: foobar-0.int.rhx
  kernel: 3.10.0-862.9.1.el7.x86_64
  os: linux
  uptime: 3h 40m 3.58s (Approximately 0.12 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.centos.org
  - registry.access.redhat.com
store:
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

[root@foobar-0 ~]# rpm -q podman docker runc
podman-0.6.4-1.gitd5beb2f.el7.x86_64
docker-1.13.1-72.git6f36bd4.el7.x86_64
runc-1.0.0-27.rc5.dev.git4bb1fe4.el7.x86_64

Physical box (Intel Nuc) running: 3.10.0-862.9.1.el7.x86_64

[vrothberg]

I tried reproducing locally (podman 0.7.3 on openSUSE Leap 15) where it works. I'll setup a VM and report back. There were a bunch of mount-related fixes since 0.6.4.

I wonder why CentOS 7 is still on 0.6.4 (Cc: @rhatdan).

[mbaldessari]

Ah, good point, I blindly used the two repos I was pointed at (see bottom of the initial description), but if there is something more recent to play with, I'll be happy to try that.

[vrothberg]

@mbaldessari that seems to be a regression in 0.6.4. I ran it in a VM with the podman from the main repos (i.e., 0.6.1) where it works. Using 0.6.4 breaks exactly as you describe. I suggest closing the issue as it seems to be related to 0.6.4 in this specific repository.

[mbaldessari]

Ack, thanks for your help. Worksforme. I will hunt down some newer repos for this then.