rootless: can't install httpd into a Fedora container because of capabilities #5364

praiskup · 2020-03-02T07:29:58Z

With this dockerfile:

FROM fedora:31
RUN dnf install -y httpd

I run buildah bud ., and I see:

  Installing       : httpd-2.4.41-12.fc31.x86_64                                                                                                                                                                                       10/10 
Error unpacking rpm package httpd-2.4.41-12.fc31.x86_64
  Running scriptlet: httpd-2.4.41-12.fc31.x86_64                                                                                                                                                                                       10/10 
error: unpacking of archive failed on file /usr/sbin/suexec;5e5cb536: cpio: cap_set_file
error: httpd-2.4.41-12.fc31.x86_64: install failed

  Verifying        : httpd-2.4.41-12.fc31.x86_64
...
Error: Transaction failed
error building at STEP "RUN dnf install -y httpd": error while running runtime: exit status 1

Output of podman version:

$ podman version
ERRO[0000] Error refreshing volume 0a2798ccfb48ec87b701c6424fe2ef70bcd69beddbc2014b97bac5237ea29bbd: error acquiring lock 0 for volume 0a2798ccfb48ec87b701c6424fe2ef70bcd69beddbc2014b97bac5237ea29bbd: file exists 
ERRO[0000] Error refreshing volume 3224d753c9abf610930bb237eb097ffcff5cc9235520a1de26c5513626de53e2: error acquiring lock 0 for volume 3224d753c9abf610930bb237eb097ffcff5cc9235520a1de26c5513626de53e2: file exists 
ERRO[0000] Error refreshing volume 48beddd03d9df833495956b005176c347c38cab7bd365a50e313adcd4ab17d13: error acquiring lock 0 for volume 48beddd03d9df833495956b005176c347c38cab7bd365a50e313adcd4ab17d13: file exists 
ERRO[0000] Error refreshing volume c2c9404d41ad15657c9eb52bfc9da5898d876de92c85f13a314bafa98d173780: error acquiring lock 0 for volume c2c9404d41ad15657c9eb52bfc9da5898d876de92c85f13a314bafa98d173780: file exists 
ERRO[0000] Error refreshing volume cee8d22a0f9c27b23621f56baf4fedbed77486dc7977a39cfd711e7c1490d10e: error acquiring lock 0 for volume cee8d22a0f9c27b23621f56baf4fedbed77486dc7977a39cfd711e7c1490d10e: file exists 
ERRO[0000] Error refreshing volume f8cef33ef853227f5d7bbd63eb1012725fda6ae171ca1312ac68d8d7624a7165: error acquiring lock 0 for volume f8cef33ef853227f5d7bbd63eb1012725fda6ae171ca1312ac68d8d7624a7165: file exists 
Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 17122
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 17122
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 18502434816
  MemTotal: 33271529472
  OCIRuntime:
    name: crun
    package: crun-0.12.2.1-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.2.1
      commit: cd7cea7114db5f6aa35fbb69fa307c19c2728a31
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: raiskup
  kernel: 5.5.6-201.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 13h 37m 11.95s (Approximately 0.54 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/praiskup/.config/containers/storage.conf
  ContainerStore:
    number: 29
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/praiskup/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 48
  RunRoot: /run/user/17122
  VolumePath: /home/praiskup/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-2.fc31.x86_64

The text was updated successfully, but these errors were encountered:

laped83 · 2020-03-02T09:54:57Z

Adding the needed capability seems to work. I was fighting with the same for iputils. Not sure if this is the correct way to do it.

buildah bud --cap-add="CAP_SETFCAP" .

github-actions · 2020-04-02T00:04:08Z

A friendly reminder that this issue had no activity for 30 days.

praiskup · 2020-04-02T05:03:17Z

A friendly reminder that this issue had no activity for 30 days.

So ping then. This is pretty serious, we should be able to install packages
rootless. The question is whether we have to allow capabilities by default
in podman - or whether we have to fix the packages.

mheon · 2020-04-02T05:09:09Z

@rhatdan Does this sound like expected behavior? I don't think this is fuse-overlay related, since adding the capability fixes this.

praiskup · 2020-04-02T05:29:47Z

Well, I tried again today, and it works again. With the following list of packages

$ rpm -q podman buildah runc crun container-selinux
podman-1.8.2-2.fc31.x86_64
buildah-1.14.3-1.fc31.x86_64
runc-1.0.0-102.dev.gitdc9208a.fc31.x86_64
crun-0.13-1.fc31.x86_64
container-selinux-2.124.0-3.fc31.noarch

I don't know, perhaps it was fixed. Or it is not always reproducible.

rhatdan · 2020-04-02T13:14:58Z

The default list of capabilities to run containers is:

AUDIT_WRITE,CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,MKNOD,NET_BIND_SERVICE,NET_RAW,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT

So the buildah container should havv SETFCAP by default. No idea why this is working and then not working.

rhatdan · 2020-06-09T18:45:33Z

I don't believe this issue still exists. Closing, reopen if I am mistaken.

praiskup · 2020-09-25T14:44:22Z

I can not reopen this, but the problem regressed back, freshly after upgrade to Fedora 33:

Version:      2.1.0
API Version:  2.0.0
Go Version:   go1.15
Built:        Wed Sep 23 22:47:06 2020
OS/Arch:      linux/amd64

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: raiskup
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 17122
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 17122
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.11-300.fc33.x86_64
  linkmode: dynamic
  memFree: 19985932288
  memTotal: 33269727232
  ociRuntime:
    name: crun
    package: crun-0.15-2.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/17122/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.4+dev
      commit: eecccdb96f587b11d7764556ffacfeaffe4b6e11
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 1h 23m 59.15s (Approximately 0.04 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/praiskup/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/praiskup/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 42
  runRoot: /run/user/17122/containers
  volumePath: /home/praiskup/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1600894026
  BuiltTime: Wed Sep 23 22:47:06 2020
  GitCommit: ""
  GoVersion: go1.15
  OsArch: linux/amd64
  Version: 2.1.0

$ rpm -q podman buildah runc crun container-selinux
podman-2.1.0-2.fc33.x86_64
buildah-1.16.2-1.fc33.x86_64
runc-1.0.0-279.dev.gitdedadbf.fc33.x86_64
crun-0.15-2.fc33.x86_64
container-selinux-2.145.0-1.fc33.noarch

praiskup · 2020-09-25T14:51:28Z

I can not reopen this

I mean, there's no "reopen" button in this issue. Perhaps I should fill a new one?

vrothberg · 2020-09-25T14:53:46Z

Thanks! @rhatdan is on it :)

praiskup · 2020-09-25T15:04:56Z

FTR, I also removed all containers/images, and then did
mv $HOME/.local/share/containers{,~`date -I`}
and this problem is still reproducible.

praiskup · 2020-09-25T15:13:29Z

I'm not sure it is related, but when I built the image with the mentioned work-around
above (--cap-add=CAP_SETFCAP), I can not run the container:

podman run --rm -ti                  \
    -p 55555:55555               \
    -p 54321                         \
    -v `pwd`/../copr/frontend/coprs_frontend:/copr/coprs_frontend:Z,rw        \
    -v `pwd`/../copr/common:/copr/common:Z,rw \
    -e TEST_REMOTE_USER=praiskup     \
    --cap-add="CAP_SETFCAP"          \
    --tmpfs=/tmp                     \
    -v /home/praiskup/rh/projects/copr/docker-testing/initdir:/copr.init.d:ro,Z                 \
    copr-frontend 
Error: capset: Operation not permitted: OCI runtime permission denied error

mheon · 2020-09-25T16:03:03Z

I'm seeing that in several issues right now - I believe it all traces back to some attempts to prune default capabilities by @rhatdan

rhatdan · 2020-09-25T20:47:59Z

Yes if you remove /usr/share/containers/containers.conf it should be fixed. We are attempting to run containers tighter security in Fedora 33.

praiskup · 2020-09-25T21:10:37Z

Hm, from package installation POV, I don't think we can live without setcap. And it would be a pity
to by-design discourage 'dnf' operations in rootless container. So is this going to be fixed, or
what is the plan? (I can confirm that downgrade to podman-2:2.1.0-2.fc33.x86_64 restored the
expected behavior for me).

remove /usr/share/containers/containers.conf

Removing the default config helps.

dustymabe · 2020-10-02T16:14:22Z

This is a problem for us when trying to move Fedora CoreOS to Fedora 33. This is a simple reproducer:

cd $(mktemp -d)
cat <<EOF > Containerfile
FROM registry.fedoraproject.org/fedora:33
RUN dnf -y install systemd httpd \
&& dnf clean all \
&& systemctl enable httpd
ENTRYPOINT [ "/sbin/init" ]
EOF
podman build -t httpd .

Ends up with:

  Running scriptlet: httpd-2.4.46-1.fc33.x86_64
error: unpacking of archive failed on file /usr/sbin/suexec;5f77513d: cpio: cap_set_file
error: httpd-2.4.46-1.fc33.x86_64: install failed

$ rpm -q podman containers-common skopeo crun
podman-2.1.1-7.fc33.x86_64
containers-common-1.2.0-1.fc33.x86_64
skopeo-1.2.0-1.fc33.x86_64
crun-0.15-5.fc33.x86_64

Those packages are a part of https://bodhi.fedoraproject.org/updates/FEDORA-2020-7b6058fec9

cgwalters · 2020-10-02T16:24:45Z

Probably rpm (and other systems) should be patched to not try to set file caps if the process doesn't have the capability.

cgwalters · 2020-10-02T16:30:21Z

That said I also kind of question the value of dropping this. File caps are still bounded by user namespaces. I could imagine there is some kernel bug that this could mitigate but it seems unlikely to me.

rhatdan · 2020-10-02T17:10:43Z

Building skopeo 1.2.1-2 which adds in CAP_SETFCAP.

dustymabe · 2020-10-02T17:32:19Z

Thanks @rhatdan - I just tested with skopeo-1.2.0-2.fc33 and that seemed to work great. Can you add that build to the update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7b6058fec9

rhatdan · 2020-10-02T21:16:49Z

I think @lsm5 is taking care of this.

rhatdan · 2020-10-07T17:17:25Z

THis is now fixed.

praiskup · 2021-03-08T09:51:17Z

This happens again on Fedora 34:

FROM fedora:33
RUN dnf install -y httpd

Build fails on:

error: unpacking of archive failed on file /usr/sbin/suexec;6045f32f: cpio: cap_set_file failed - Inappropriate ioctl for device
error: httpd-2.4.46-9.fc33.x86_64: install failed

I'm curious if we could install some test case for this somewhere, e.g. to Fedoa CI?

praiskup · 2021-03-08T09:52:36Z

Now with:

$ rpm -q podman containers-common skopeo crun
podman-3.0.1-1.fc34.x86_64
containers-common-1.2.1-28.dev.git1b813f8.fc34.x86_64
skopeo-1.2.1-28.dev.git1b813f8.fc34.x86_64
crun-0.18-1.fc34.x86_64

praiskup · 2021-03-08T10:05:15Z

buildah bud --layers --cap-add=CAP_SETFCAP ... doesn't help now (it helped before). Also removing the default containers.conf doesn't help.

rhatdan · 2021-03-08T20:41:51Z

Please open a new issue on this. On Buildah. I have no idea if this is the same issue as before.

ensc · 2021-03-23T14:38:55Z

I do not think that the new

| ... cpio: cap_set_file failed - Inappropriate ioctl for device

error is capability related. It sounds like a problem with setting these xattr on the overlay filesystem. Does fuse-overlayfs supports them?

Perhaps it worked before because rpm ignored these errors when capabilities are missing. Now, capabilities are given and the operation itself fails.

rhatdan · 2021-03-23T21:04:13Z

grep SETFCAP /usr/share/containers/containers.conf
"SETFCAP",

dustymabe · 2021-03-24T18:50:50Z

FYI: containers/buildah#3071 (comment) - try updating your kernel and see if that helps.

praiskup changed the title ~~Can not install httpd into fedora:31 container because of capabilities~~ rootless: can't install httpd into fedora:31 container because of capabilities Mar 2, 2020

praiskup mentioned this issue Mar 29, 2020

mock install transactions in toolbox blocked by setcap() problems containers/toolbox#397

Closed

github-actions bot added the stale-issue label Apr 2, 2020

rhatdan closed this as completed Jun 9, 2020

praiskup changed the title ~~rootless: can't install httpd into fedora:31 container because of capabilities~~ rootless: can't install httpd into a Fedora container because of capabilities Sep 25, 2020

vrothberg reopened this Sep 25, 2020

dustymabe mentioned this issue Oct 2, 2020

[next-devel] rebase to Fedora 33 coreos/fedora-coreos-config#640

Merged

rhatdan closed this as completed Oct 7, 2020

praiskup mentioned this issue Mar 9, 2021

Missing FCAP I fail to install httpd into Fedora 33 container in build-using-dockerfile containers/buildah#3071

Closed

BarDweller mentioned this issue Sep 5, 2023

nodejs extension: Failed to extend build image redhat-buildpacks/testing#42

Open

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rootless: can't install httpd into a Fedora container because of capabilities #5364

rootless: can't install httpd into a Fedora container because of capabilities #5364

praiskup commented Mar 2, 2020

laped83 commented Mar 2, 2020

github-actions bot commented Apr 2, 2020

praiskup commented Apr 2, 2020

mheon commented Apr 2, 2020

praiskup commented Apr 2, 2020

rhatdan commented Apr 2, 2020

rhatdan commented Jun 9, 2020

praiskup commented Sep 25, 2020 •

edited

Loading

praiskup commented Sep 25, 2020

vrothberg commented Sep 25, 2020

praiskup commented Sep 25, 2020 •

edited

Loading

praiskup commented Sep 25, 2020

mheon commented Sep 25, 2020

rhatdan commented Sep 25, 2020

praiskup commented Sep 25, 2020 •

edited

Loading

dustymabe commented Oct 2, 2020 •

edited

Loading

cgwalters commented Oct 2, 2020

cgwalters commented Oct 2, 2020

rhatdan commented Oct 2, 2020

dustymabe commented Oct 2, 2020

rhatdan commented Oct 2, 2020

rhatdan commented Oct 7, 2020

praiskup commented Mar 8, 2021 •

edited

Loading

praiskup commented Mar 8, 2021

praiskup commented Mar 8, 2021

rhatdan commented Mar 8, 2021

ensc commented Mar 23, 2021

rhatdan commented Mar 23, 2021

dustymabe commented Mar 24, 2021

rootless: can't install httpd into a Fedora container because of capabilities #5364

rootless: can't install httpd into a Fedora container because of capabilities #5364

Comments

praiskup commented Mar 2, 2020

laped83 commented Mar 2, 2020

github-actions bot commented Apr 2, 2020

praiskup commented Apr 2, 2020

mheon commented Apr 2, 2020

praiskup commented Apr 2, 2020

rhatdan commented Apr 2, 2020

rhatdan commented Jun 9, 2020

praiskup commented Sep 25, 2020 • edited Loading

praiskup commented Sep 25, 2020

vrothberg commented Sep 25, 2020

praiskup commented Sep 25, 2020 • edited Loading

praiskup commented Sep 25, 2020

mheon commented Sep 25, 2020

rhatdan commented Sep 25, 2020

praiskup commented Sep 25, 2020 • edited Loading

dustymabe commented Oct 2, 2020 • edited Loading

cgwalters commented Oct 2, 2020

cgwalters commented Oct 2, 2020

rhatdan commented Oct 2, 2020

dustymabe commented Oct 2, 2020

rhatdan commented Oct 2, 2020

rhatdan commented Oct 7, 2020

praiskup commented Mar 8, 2021 • edited Loading

praiskup commented Mar 8, 2021

praiskup commented Mar 8, 2021

rhatdan commented Mar 8, 2021

ensc commented Mar 23, 2021

rhatdan commented Mar 23, 2021

dustymabe commented Mar 24, 2021

praiskup commented Sep 25, 2020 •

edited

Loading

praiskup commented Sep 25, 2020 •

edited

Loading

praiskup commented Sep 25, 2020 •

edited

Loading

dustymabe commented Oct 2, 2020 •

edited

Loading

praiskup commented Mar 8, 2021 •

edited

Loading