cp -afl --preserve=xattr failing on directories

[xedrac]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When executing cp -afl --preserve=xattr <dir0> <dir1) in a podman container, it appears that podman is attempting to create hard-links for the directories themselves (which is not permitted, even as root), rather than copy the directory names and create hard-links for the files. If I remove the --preserve=xattr, it works as expected. Note that this works correctly in Docker.

Steps to reproduce the issue:

1. podman run -it <hash> bash

2. mkdir /tmp/systemd

3. cp -afl --preserve=xattr /etc/systemd/* /tmp/systemd

Describe the results you received:

cp -afl --preserve=xattr /etc/systemd/* /tmp/systemd
cp: setting attributes for '/tmp/systemd/network': Operation not supported
cp: setting attributes for '/tmp/systemd/system/timers.target.wants': Operation not supported
cp: setting attributes for '/tmp/systemd/system/getty.target.wants': Operation not supported
cp: setting attributes for '/tmp/systemd/system/multi-user.target.wants': Operation not supported
cp: setting attributes for '/tmp/systemd/system/sockets.target.wants': Operation not supported
cp: setting attributes for '/tmp/systemd/system/sysinit.target.wants': Operation not supported
cp: setting attributes for '/tmp/systemd/system': Operation not supported
cp: setting attributes for '/tmp/systemd/user': Operation not supported
root@4ac3d766f5d3:/# echo $?
1

Describe the results you expected:

cp -afl --preserve=xattr /etc/systemd/* /tmp/systemd
root@4ac3d766f5d3:/# echo $?
0

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.14-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.14, commit: 083a0be12178013d44ff51ceda3090ea741b6516'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 14982782976
  MemTotal: 33680461824
  OCIRuntime:
    name: crun
    package: crun-0.13-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 5318897664
  SwapTotal: 5368705024
  arch: amd64
  cpus: 16
  eventlogger: journald
  hostname: localhost.localdomain
  kernel: 5.5.13-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 70h 53m 17.5s (Approximately 2.92 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/sbadger/.config/containers/storage.conf
  ContainerStore:
    number: 91
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.8-1.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.8
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/sbadger/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 129
  RunRoot: /run/user/1000
  VolumePath: /home/sbadger/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.2-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
physical

[rhatdan]

This is probably an SELinux issue. I would figure it is getting permission denied when it attempts to reset the SELinux labels.

[rhatdan]

Are you seeing AVC messages?

[xedrac]

I am not seeing any AVC messages.
I've also tried running this as:
--privileged
and
--security-opt label=disable

Although I'm not entirely sure it's related.

[rhatdan]

The problem I because you are attempting to set labels on fuse-overlayfs and that is not allowed. As root is it also not allowed, since the container uses a context mount on overlay.

I have no idea why this is allowed on Docker, other then perhaps you aren't using overlay or SELinux is not enabled inside the container.

lsetxattr("/tmp/systemd/network", "security.selinux", "system_u:object_r:container_file"..., 48, 0) = -1 EOPNOTSUPP (Operation not supported)
write(2, ": Operation not supported", 25) = 25
lsetxattr("/tmp/systemd/system/multi-user.target.wants", "security.selinux", "system_u:object_r:container_file"..., 48, 0) = -1 EOPNOTSUPP (Operation not supported)
write(2, ": Operation not supported", 25) = 25
lsetxattr("/tmp/systemd/system/getty.target.wants", "security.selinux", "system_u:object_r:container_file"..., 48, 0) = -1 EOPNOTSUPP (Operation not supported)
write(2, ": Operation not supported", 25) = 25
lsetxattr("/tmp/systemd/system/sockets.target.wants", "security.selinux", "system_u:object_r:container_file"..., 48, 0) = -1 EOPNOTSUPP (Operation not supported)
write(2, ": Operation not supported", 25) = 25
lsetxattr("/tmp/systemd/system", "security.selinux", "system_u:object_r:container_file"..., 48, 0) = -1 EOPNOTSUPP (Operation not supported)
write(2, ": Operation not supported", 25) = 25
lsetxattr("/tmp/systemd/user/sockets.target.wants", "security.selinux", "system_u:object_r:container_file"..., 48, 0) = -1 EOPNOTSUPP (Operation not supported)
write(2, ": Operation not supported", 25) = 25
lsetxattr("/tmp/systemd/user", "security.selinux", "system_u:object_r:container_file"..., 48, 0) = -1 EOPNOTSUPP (Operation not supported)
write(2, ": Operation not supported", 25) = 25

[rhatdan]

Definitely SELinux and not much we can do about it.
What Xattrs are you trying to preserve?

[xedrac]

Hmm... This is from building quilt-native in a yocto build running inside a podman container. I'm not really sure what Xattrs they are trying to preserve. Is there a way to disable SELinux for now?

[rhatdan]

No the issue is cp is seeing the SELinux labels and trying to maintain them.
Unless you disable the SELinux for the system.

It is too bad the kernel is not a little smarter or cp, in that it is attempting to set the same Xattr that is already on disk.

[xedrac]

Thanks for the info. I guess I'm left wondering how Docker mitigates this.

[rhatdan]

The docker version you are using is not using SELinux for separation.
Is this a Dockerfile that you are trying to build?

[xedrac]

It is an image that I built from a Dockerfile, yes. But the issue manifests while I'm running the cp command inside the image.

[rhatdan]

And you can not change the Dockerfile?
Not sure if buildah bud or buildah bud --isolation chroot would work any better.

We might want to look at not setting up the mount label when running with SELinux disabled. But this still would not allow this to work on a rootless podman.

[xedrac]

Yes I can change the Dockerfile. I'm currently using podman build ... to perform the build. I'm not sure how to translate that to buildah bud.

One more piece of information. If I disable SELinux on my host machine before building the image by doing sudo setenforce 0, I still encounter the problem.

[TomSweeneyRedHat]

buildah bud --layers ... is equivalent to podman build ...

[rhatdan]

xedrac, I am talking about disabling SELinux on the host, not about putting it in permissive mode. But this is not something I am recommending. I think it is best to eliminate the preservation of the xattrs. I am not sure what you are worried about copying from there.

[xedrac]

I would eliminate it, but that command is buried deep in what is considered a read-only yocto (poky) repository. I generate these images so infrequently that I would rather disable SELinux temporarily just to produce a functional image. However, I'm a little confused. Doesn't setting SELinux to permissive mode mean that it will not enforce any policies, just logs warnings? If this is true, I would have expected that change to get me beyond this issue. Either way, I've battled this a little too long and had to temporarily revert back to cgroupsv1 and docker in order to meet a deadline. I'll resume my dance with podman at another time. Thank you both for your help!

[giuseppe]

since you are using fuse-overlayfs, there is an option that tells fuse-overlayfs to disable xattrs support, you can try by creating a container like:

podman --storage-opt overlay.mountopt=noxattrs=1,overlay.mount_program=/usr/bin/fuse-overlayfs run ....

I think this could solve the issue you've

[giuseppe]

@xedrac does the proposed solution work for you?

[xedrac]

I'm not currently in a position to test it unfortunately. It looks like a promising workaround for sure. Once I've finished what I'm working on, I'll revert back to cgroupsv2 and try again. But that might be a few weeks.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I am going to close this since we have a work around.

[Rahix]

I just ran into this issue, also with a yocto/open-embedded build. For reference, the following command works for me:

podman --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs --storage-opt overlay.mountopt=nodev,metacopy=on,noxattrs=1 run ...

[sivaece06]

I tried to run the command. but it throws error.

$ podman --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs --storage-opt overlay.mountopt=nodev,metacopy=on,noxattrs=1 run ...
Error: invalid reference format
$

[rhatdan]

Is storage driver overlay?
Please open new issues, do not attach to old closed issues.