Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Security IMA: Fail to set xattr on catatonit #18064

Closed
paul-grozav opened this issue Apr 5, 2023 · 9 comments
Closed

Security IMA: Fail to set xattr on catatonit #18064

paul-grozav opened this issue Apr 5, 2023 · 9 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@paul-grozav
Copy link

Issue Description

I can not create a pod (infra) on an Oracle Linux 9.1 UEK with IMA(Integrity Measurement Architecture).

This seems to be a known issue, and described by Oracle in their podman manual: https://docs.oracle.com/en/operating-systems/oracle-linux/podman/podman-KnownIssues.html#topic_efh_1n3_55b

From their manual:

Podman Pod Create Fails on Oracle Linux 9 For An Unprivileged User With IMA Enabled
On systems, such as Oracle Linux 9, where Integrity Measurement Architecture (IMA) is enabled and enforcing, running podman pod create as an unprivileged user can fail with an error similar to:

...
RemoveOptions:copier.RemoveOptions{All:false}}: copier: put: error setting
extended attributes on "/catatonit": error setting value of extended
attribute "security.ima" on "/catatonit": operation not permitted

This issue occurs because the catatonit binary, used by Podman to provide init services to containers, is an IMA signed file and unprivileged users do not have permissions to set security extended attributes (xattrs) on the file system.

(Bug 34578553)

I'm not sure about the bug ID 34578553 - I guess it points to a bug in their internal(not open) issue tracking system.

I am pretty sure that this is an OS configuration issue, probably not podman's fault, but I couldn't find much info about this online, and I thought I should open a ticket on a public system.

Do you have any idea what I could change in the OS config to allow podman to work?

Also, I'm not sure if podman could skip applying xattrs on the copied binary (or maybe make it a non-fatal error?) - or if it is always important to keep the xattrs.

(0)dbadmin@qboro2:~ $ podman version
Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.9
Built:        Tue Feb 28 13:07:06 2023
OS/Arch:      linux/amd64

# See podman info below, in it's own section

(0)dbadmin@qboro2:~ $ rpm -qa | grep podman
podman-catatonit-4.2.0-11.0.1.el9_1.x86_64
podman-4.2.0-11.0.1.el9_1.x86_64

Steps to reproduce the issue

I have a similar VM and I can't reproduce the issue there 😞 - I'm not sure what's different. I will keep trying and come back with details as I have them.

Describe the results you received

(0)dbadmin@qboro2:~ $ podman pod create --infra=true --publish=0.0.0.0:3306:3306 --publish=0.0.0.0:9104:9104 --name pod_fcm_db --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called create.PersistentPreRunE(podman pod create --infra=true --publish=0.0.0.0:3306:3306 --publish=0.0.0.0:9104:9104 --name pod_fcm_db --log-level=debug)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /qst/podman_root/dbadmin/libpod/bolt_state.db
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /qst/podman_root/dbadmin
DEBU[0000] Using run root /run/user/2115/containers
DEBU[0000] Using static dir /qst/podman_root/dbadmin/libpod
DEBU[0000] Using tmp dir /run/user/2115/libpod/tmp
DEBU[0000] Using volume path /qst/podman_root/dbadmin/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 73
DEBU[0000] Adding port mapping from 3306 to 3306 length 1 protocol ""
DEBU[0000] Adding port mapping from 9104 to 9104 length 1 protocol ""
DEBU[0000] Looking up image "localhost/podman-pause:4.2.0-1677582426" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ...
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ...
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ...
DEBU[0000] FROM "scratch"
DEBU[0000] Check for idmapped mounts support create mapped mount: operation not permitted
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] overlay: test mount indicated that volatile is being used
DEBU[0000] overlay: mount_data=lowerdir=/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/empty,upperdir=/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/diff,workdir=/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/work,,userxattr,volatile
DEBU[0000] Container ID: b5f97f921a29b36b7ebff0225435c3d165909ad8171b438efca336848ab0ed3a
DEBU[0000] Parsed Step: {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:copy Args:[/usr/libexec/podman/catatonit /catatonit] Flags:[] Attrs:map[] Message:COPY /usr/libexec/podman/catatonit /catatonit Original:COPY /usr/libexec/podman/catatonit /catatonit}                            
DEBU[0000] COPY []string(nil), imagebuilder.Copy{FromFS:false, From:"", Src:[]string{"/usr/libexec/podman/catatonit"}, Dest:"/catatonit", Download:false, Chown:"", Chmod:""}



DEBU[0000] Error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:copy Args:[/usr/libexec/podman/catatonit /catatonit] Flags:[] Attrs:map[] Message:COPY /usr/libexec/podman/catatonit /catatonit Original:COPY /usr/libexec/podman/catatonit /catatonit}: error storing "/usr/libexec/podman/catatonit": error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/merged", rootPrefix:"/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/merged", Directory:"/", preservedDirectory:"/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/merged", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap{}, GIDMap:[]idtools.IDMap{}, DefaultDirOwner:(*idtools.IDPair)(0xc000d48520), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:true, NoOverwriteDirNonDir:false, NoOverwriteNonDirDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: put: error setting extended attributes on "/catatonit": error setting value of extended attribute "security.ima" on "/catatonit": operation not permitted
Error: building local pause image: error building at STEP "COPY /usr/libexec/podman/catatonit /catatonit": error storing "/usr/libexec/podman/catatonit": error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/merged", rootPrefix:"/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/merged", Directory:"/", preservedDirectory:"/qst/podman_root/dbadmin/overlay/0eba375e398d40be3feac26e28fbc5faa9eea9cc1802d927da929fc25d7fcdba/merged", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap{}, GIDMap:[]idtools.IDMap{}, DefaultDirOwner:(*idtools.IDPair)(0xc000d48520), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:true, NoOverwriteDirNonDir:false, NoOverwriteNonDirDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: put: error setting extended attributes on "/catatonit": error setting value of extended attribute "security.ima" on "/catatonit": operation not permitted

While creating the podman-pause container image (from scratch) it tries to COPY /usr/libexec/podman/catatonit /catatonit and I guess it tries to keep the extended attributes by setting them on the destination binary(same as on the source binary).

Although I can't see any xattrs on the source file either:

(0)dbadmin@qboro2:~ $ getfattr -d /usr/libexec/catatonit/catatonit
(0)dbadmin@qboro2:~ $ lsattr /usr/libexec/catatonit/catatonit
--------------e------- /usr/libexec/catatonit/catatonit
(0)dbadmin@qboro2:~ $ lsattr -l /usr/libexec/catatonit/catatonit
/usr/libexec/catatonit/catatonit Extents

Describe the results you expected

I was hoping that it would create the pod - something similar to:

DEBU[0000] Container ID: b32eb4599c8501b2b0528b9494de44d9467f07135d512d946502a09f7927afa8 
DEBU[0000] Parsed Step: {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:copy Args:[/usr/libexec/podman/catatonit /catatonit] Flags:[] Attrs:map[] Message:COPY /usr/libexec/podman/catatonit /catatonit Original:COPY /usr/libexec/podman/catatonit /catatonit} 
DEBU[0000] COPY []string(nil), imagebuilder.Copy{FromFS:false, From:"", Src:[]string{"/usr/libexec/podman/catatonit"}, Dest:"/catatonit", Download:false, Chown:"", Chmod:""} 



DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] added content file:bbf8e8210d48f6deb44924dcd084f1c623294a644ac4f42e429517c5a1dae773 
DEBU[0000] Parsed Step: {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:entrypoint Args:[/catatonit -P] Flags:[] Attrs:map[json:true] Message:ENTRYPOINT /catatonit -P Original:ENTRYPOINT ["/catatonit", "-P"]} 
DEBU[0000] COMMIT localhost/podman-pause:4.2.0-1677582426 
DEBU[0000] Looking up image "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ... 
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ... 
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ... 
DEBU[0000] parsed reference into "[overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]localhost/podman-pause:4.2.0-1677582426" 
DEBU[0000] COMMIT "containers-storage:[overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]localhost/podman-pause:4.2.0-1677582426" 
DEBU[0000] committing image with reference "containers-storage:[overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]localhost/podman-pause:4.2.0-1677582426" is allowed by policy 
DEBU[0000] layer list: ["d75a538fd5a9f9b153d0cf0c608ab2774f8784128ab86e976553bb530db2cab0"] 
DEBU[0000] using "/var/tmp/buildah1328397854" to hold temporary data 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Tar with options on /home/qstadmin/.local/share/containers/storage/overlay/d75a538fd5a9f9b153d0cf0c608ab2774f8784128ab86e976553bb530db2cab0/diff 
DEBU[0000] layer "d75a538fd5a9f9b153d0cf0c608ab2774f8784128ab86e976553bb530db2cab0" size is 807936 bytes, uncompressed digest sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2, possibly-compressed digest sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2 
DEBU[0000] OCIv1 config = {"created":"2023-04-05T11:08:35.453868336Z","architecture":"amd64","os":"linux","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["/catatonit","-P"],"Labels":{"io.buildah.version":"1.27.3"}},"rootfs":{"type":"layers","diff_ids":["sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2"]},"history":[{"created":"2023-04-05T11:08:35.45282091Z","created_by":"/bin/sh -c #(nop) COPY file:bbf8e8210d48f6deb44924dcd084f1c623294a644ac4f42e429517c5a1dae773 in /catatonit ","empty_layer":true},{"created":"2023-04-05T11:08:35.459927564Z","created_by":"/bin/sh -c #(nop) ENTRYPOINT [\"/catatonit\", \"-P\"]"}]} 
DEBU[0000] OCIv1 manifest = {"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","digest":"sha256:d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae","size":667},"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2","size":807936}],"annotations":{"org.opencontainers.image.base.digest":"","org.opencontainers.image.base.name":""}} 
DEBU[0000] Docker v2s2 config = {"created":"2023-04-05T11:08:35.453868336Z","container":"b32eb4599c8501b2b0528b9494de44d9467f07135d512d946502a09f7927afa8","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":[],"Image":"","Volumes":{},"WorkingDir":"","Entrypoint":["/catatonit","-P"],"OnBuild":[],"Labels":{"io.buildah.version":"1.27.3"}},"config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":[],"Image":"","Volumes":{},"WorkingDir":"","Entrypoint":["/catatonit","-P"],"OnBuild":[],"Labels":{"io.buildah.version":"1.27.3"}},"architecture":"amd64","os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2"]},"history":[{"created":"2023-04-05T11:08:35.45282091Z","created_by":"/bin/sh -c #(nop) COPY file:bbf8e8210d48f6deb44924dcd084f1c623294a644ac4f42e429517c5a1dae773 in /catatonit ","empty_layer":true},{"created":"2023-04-05T11:08:35.459927564Z","created_by":"/bin/sh -c #(nop) ENTRYPOINT [\"/catatonit\", \"-P\"]"}]} 
DEBU[0000] Docker v2s2 manifest = {"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":1341,"digest":"sha256:fd80e5dbca8986db5ed5c4aface4522dd12ec626ebaed150174692ad777e0dfc"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar","size":807936,"digest":"sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2"}]} 
DEBU[0000] Using blob info cache at /home/qstadmin/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage: 
DEBU[0000]  Using transport "containers-storage" policy section  
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0000] start reading config                         
DEBU[0000] finished reading config                      
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking if we can reuse blob sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true 
DEBU[0000] reading layer "sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2" 
DEBU[0000] No compression detected                      
DEBU[0000] Using original blob without modification     
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported 
DEBU[0000] Check for idmapped mounts support            
DEBU[0000] Applying tar in /home/qstadmin/.local/share/containers/storage/overlay/38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2/diff 
DEBU[0000] finished reading layer "sha256:38af3032cde21560c771e7dda1a788d89f8ea3ba06d43cd403a8d7cd3bfe6fe2" 
DEBU[0000] No compression detected                      
DEBU[0000] Compression change for blob sha256:d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae ("application/vnd.oci.image.config.v1+json") not supported 
DEBU[0000] Using original blob without modification     
DEBU[0000] setting image creation date to 2023-04-05 11:08:35.453868336 +0000 UTC 
DEBU[0000] created new image ID "d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] saved image metadata "{\"signatures-sizes\":{\"sha256:38c419a19c9116ce51a3d1bbfc711ea9fc4cf445f37306b1fc5270adb72fe29a\":[]}}" 
DEBU[0000] added name "localhost/podman-pause:4.2.0-1677582426" to image "d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] Looking up image "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ... 
DEBU[0000] parsed reference into "[overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]@d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] Found image "localhost/podman-pause:4.2.0-1677582426" as "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Found image "localhost/podman-pause:4.2.0-1677582426" as "localhost/podman-pause:4.2.0-1677582426" in local containers storage ([overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]@d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae) 
DEBU[0000] exporting opaque data as blob "sha256:d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] parsed reference into "[overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]localhost/podman-pause:4.2.0-1677582426" 
DEBU[0000] printing final image id "d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] Created cgroup path user.slice/user-libpod_pod_6f92970ddaa898a2664ace9741d175925ef6f90543c884dc589330d95e31b43f.slice for parent user.slice and name libpod_pod_6f92970ddaa898a2664ace9741d175925ef6f90543c884dc589330d95e31b43f 
DEBU[0000] Created cgroup user.slice/user-libpod_pod_6f92970ddaa898a2664ace9741d175925ef6f90543c884dc589330d95e31b43f.slice 
DEBU[0000] Got pod cgroup as user.slice/user-libpod_pod_6f92970ddaa898a2664ace9741d175925ef6f90543c884dc589330d95e31b43f.slice 
DEBU[0000] Looking up image "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ... 
DEBU[0000] parsed reference into "[overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]@d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] Found image "localhost/podman-pause:4.2.0-1677582426" as "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Found image "localhost/podman-pause:4.2.0-1677582426" as "localhost/podman-pause:4.2.0-1677582426" in local containers storage ([overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]@d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae) 
DEBU[0000] exporting opaque data as blob "sha256:d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] Inspecting image d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae 
DEBU[0000] exporting opaque data as blob "sha256:d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] Inspecting image d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae 
DEBU[0000] Inspecting image d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae 
DEBU[0000] Inspecting image d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae 
DEBU[0000] using systemd mode: false                    
DEBU[0000] setting container name 6f92970ddaa8-infra    
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] Allocated lock 3 for container 4c59e53452317cea44d440299571499cd4d2bf9a24be8d4bfc27263eca421407 
DEBU[0000] parsed reference into "[overlay@/home/qstadmin/.local/share/containers/storage+/run/user/1000/containers]@d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] exporting opaque data as blob "sha256:d60db05bf4e4f37384855215302109c7c669c0d114246775f7e0daad1448a1ae" 
DEBU[0000] Created container "4c59e53452317cea44d440299571499cd4d2bf9a24be8d4bfc27263eca421407" 
DEBU[0000] Container "4c59e53452317cea44d440299571499cd4d2bf9a24be8d4bfc27263eca421407" has work directory "/home/qstadmin/.local/share/containers/storage/overlay-containers/4c59e53452317cea44d440299571499cd4d2bf9a24be8d4bfc27263eca421407/userdata" 
DEBU[0000] Container "4c59e53452317cea44d440299571499cd4d2bf9a24be8d4bfc27263eca421407" has run directory "/run/user/1000/containers/overlay-containers/4c59e53452317cea44d440299571499cd4d2bf9a24be8d4bfc27263eca421407/userdata" 
6f92970ddaa898a2664ace9741d175925ef6f90543c884dc589330d95e31b43f
DEBU[0000] Called create.PersistentPostRunE(podman pod create --infra=true --publish=0.0.0.0:3306:3306 --publish=0.0.0.0:9104:9104 --name pod_fcm_db --log-level=debug) 


[dbadmin@oracle-linux ~]$ podman pod ls
POD ID        NAME        STATUS      CREATED         INFRA ID      # OF CONTAINERS
0f0caf4384c8  pod_fcm_db  Created     41 minutes ago  396ef3fb9112  1
[dbadmin@oracle-linux ~]$ podman ps -a
CONTAINER ID  IMAGE                                    COMMAND     CREATED         STATUS      PORTS                                           NAMES
396ef3fb9112  localhost/podman-pause:4.2.0-1677582426              41 minutes ago  Created     0.0.0.0:3306->3306/tcp, 0.0.0.0:9104->9104/tcp  0f0caf4384c8-infra

podman info output

host:
  arch: amd64
  buildahVersion: 1.27.3
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: 72c22139fbca75534d8c39fd72457900f499ce2b'
  cpuUtilization:
    idlePercent: 96.63
    systemPercent: 0.27
    userPercent: 3.1
  cpus: 24
  distribution:
    distribution: '"ol"'
    variant: server
    version: "9.1"
  eventLogger: file
  hostname: qboro2
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 588
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2115
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 5.15.0-8.91.4.1.el9uek.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3709337600
  memTotal: 133620297728
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.5-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/2115/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.el9_0.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 334h 56m 16.00s (Approximately 13.92 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - container-registry.oracle.com
  - docker.io
store:
  configFile: /home/dbadmin/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /qst/podman_root/dbadmin
  graphRootAllocated: 918260506624
  graphRootUsed: 223702593536
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/2115/containers
  volumePath: /qst/podman_root/dbadmin/volumes
version:
  APIVersion: 4.2.0
  Built: 1677582426
  BuiltTime: Tue Feb 28 13:07:06 2023
  GitCommit: ""
  GoVersion: go1.18.9
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

(0)dbadmin@server:~ $ cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="9.1"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Oracle Linux Server 9.1"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:9:1:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 9"
ORACLE_BUGZILLA_PRODUCT_VERSION=9.1
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=9.1

(0)dbadmin@server:~ $ uname -a
Linux qboro2 5.15.0-8.91.4.1.el9uek.x86_64 #2 SMP Tue Mar 7 21:03:42 PST 2023 x86_64 x86_64 x86_64 GNU/Linux

Additional information

The podman pod create command works on root, on the same system.

I was also thinking about using a custom IMA policy to, somehow allow setting xattrs on a certain executable, or to a certain user - but I'm not sure if that's configurable through the IMA policy or would require mounting with user_xattr.

I saw that podman is mounting with userxattr and both my partitions are missing that mount flag:

(0)dbadmin@qboro2:~ $ mount | grep " on / "
/dev/nvme0n1p4 on / type ext4 (rw,relatime)
(0)dbadmin@qboro2:~ $ mount | grep " on /qst "
/dev/nvme0n1p3 on /qst type ext4 (rw,relatime)
@paul-grozav paul-grozav added the kind/bug Categorizes issue or PR as related to a bug. label Apr 5, 2023
@rhatdan
Copy link
Member

rhatdan commented Apr 5, 2023

@giuseppe PTAL

@paul-grozav
Copy link
Author

paul-grozav commented Apr 5, 2023 •
edited
Loading

The interesting thing is that podman cp fails while cp does not. And the manual doesn't mention anything related to this.

Also, it seems that the binary is copied into the container, it's just that the xattr is not set. Started from alpine here instead of scratch:

(125)dbadmin@qboro2:~/pgrozav_container_test $ podman stop -t0 pgrozav_test_ima ; podman rm pgrozav_test_ima
pgrozav_test_ima
d2f61db59d6698ab621fd44110108b7f2e7d8d4eaa1e4c5e9b244084718074f2
(0)dbadmin@qboro2:~/pgrozav_container_test $ podman run -d --name=pgrozav_test_ima docker.io/alpine:3.16.2 sleep infinity
479a1328e9b47276a16e1f8fbeabff6bad2646ff995a268dce01804a53326ac6
(0)dbadmin@qboro2:~/pgrozav_container_test $ podman cp /usr/libexec/podman/catatonit pgrozav_test_ima:/root/catatonit --log-level=debug
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called cp.PersistentPreRunE(podman cp /usr/libexec/podman/catatonit pgrozav_test_ima:/root/catatonit --log-level=debug) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /qst/podman_root/dbadmin/libpod/bolt_state.db 
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /qst/podman_root/dbadmin    
DEBU[0000] Using run root /run/user/2115/containers     
DEBU[0000] Using static dir /qst/podman_root/dbadmin/libpod 
DEBU[0000] Using tmp dir /run/user/2115/libpod/tmp      
DEBU[0000] Using volume path /qst/podman_root/dbadmin/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 73             
DEBU[0000] Container copy *to* "/root" (resolved: "/root") on container "pgrozav_test_ima" (ID: 479a1328e9b47276a16e1f8fbeabff6bad2646ff995a268dce01804a53326ac6) 
Error: 1 error occurred:
        * error copying to container: copier: put: error setting extended attributes on "/root/catatonit": error setting value of extended attribute "security.ima" on "/root/catatonit": operation not permitted
(125)dbadmin@qboro2:~/pgrozav_container_test $ podman inspect pgrozav_test_ima | grep UpperDir
                    "UpperDir": "/qst/podman_root/dbadmin/overlay/19f136657d516599e334b4af899c080760e438550ffa28063fae3d2fa4b93169/diff",
(0)dbadmin@qboro2:~/pgrozav_container_test $ ls -la /qst/podman_root/dbadmin/overlay/19f136657d516599e334b4af899c080760e438550ffa28063fae3d2fa4b93169/diff/root/
total 796
drwx------ 2 dbadmin dba   4096 Apr  5 17:54 .
dr-xr-xr-x 5 dbadmin dba   4096 Apr  5 17:54 ..
-rwxr-xr-x 1 dbadmin dba 806144 Apr  5 17:54 catatonit
(0)dbadmin@qboro2:~/pgrozav_container_test $ cp /usr/libexec/podman/catatonit /qst/podman_root/dbadmin/overlay/19f136657d516599e334b4af899c080760e438550ffa28063fae3d2fa4b93169/diff/root/catatonit 
(0)dbadmin@qboro2:~/pgrozav_container_test $ ls -la /qst/podman_root/dbadmin/overlay/19f136657d516599e334b4af899c080760e438550ffa28063fae3d2fa4b93169/diff/root/
total 796
drwx------ 2 dbadmin dba   4096 Apr  5 17:54 .
dr-xr-xr-x 5 dbadmin dba   4096 Apr  5 17:54 ..
-rwxr-xr-x 1 dbadmin dba 806144 Apr  5 17:55 catatonit
(0)dbadmin@qboro2:~/pgrozav_container_test $

I also saw these flags in the code context: StripXattrs:false, IgnoreXattrErrors:false - but I'm not sure how I could change/set them.

However, this command:

# Fails on qboro2
(0)dbadmin@qboro2:~/pgrozav_container_test $ cp --preserve=xattr /usr/libexec/podman/catatonit /qst/podman_root/dbadmin/overlay/19f136657d516599e334b4af899c080760e438550ffa28063fae3d2fa4b93169/diff/root/catatonit 
cp: setting attribute 'security.ima' for 'security.ima': Operation not permitted

# works in VM
[dbadmin@oracle-linux container]$ cp --preserve=xattr /usr/libexec/podman/catatonit ./catatonit 
[dbadmin@oracle-linux container]$

And setfattr fails on both:

(0)dbadmin@qboro2:~/pgrozav_container_test $ setfattr -n 'security.ima' -v 'FOO' ./test
setfattr: ./test: Operation not permitted

[dbadmin@oracle-linux container]$ setfattr -n 'security.ima' -v 'FOO' ./catatonit 
setfattr: ./catatonit: Operation not permitted

Seems to be related to #5781 .

@paul-grozav
Copy link
Author

paul-grozav commented Apr 6, 2023 •
edited
Loading

Finally, I figured it out, why it's not reproduced on my VM(even though it seems to have IMA enabled):

(0)dbadmin@qboro2:~ $ getfattr -dm- /usr/libexec/podman/catatonit
getfattr: Removing leading '/' from absolute path names
# file: usr/libexec/podman/catatonit
security.ima=0sAwIEmiKuyQIAmRIpCLtWAbiLXRd5nrJLWgpR9YFxvQAdhudVUSOi/+QRTw0T4b/mOgXT4Mz0CjdoApKgBkY0e9A3gfUHlC+n9+Do65pnONIofb6/ba60esak7gixy1dkArPLYj+CYenLZXk5bSoQkfKaur76ulpC3iAzzULiIiHAatsPW9hKkkfc1QVEyp+Suqd06oMoe7+aAGIDdx+RSi3Mq9wYXXncmTgyihvFAyWniUSGfvm0Kko+tlVjjqGVtRD2CTn2HXtqm4N2grDLT1UytDvP/byZngn7HVqgTuPLo9Gz3rvEDs7A87oI7Qr8LrG4jFZSn78zeBuK//wfseNMJIQ9e2gip0kSwLWjzuaGjMI5pUp5lEPhAkyK/Y/wn4bw2TbDFmGk6uCBbmIytaF54CZBvRcNgaS7QFI92iX+4/fN5YhZZcKO0+6Q5qmmFlJgIl6bKD77uyAnH8hc5WaakKXtOwT82lOvHBSq7NWS4/5cESTcr+omDjEXWeCAfJw77ALZfrOmiR3cCV7yZhyqFW/52un0W9BgOKnTguuXsWRqAi0/BYvx00NFqc/7jp068vFz/fokjQuO1U+90+Q4EyNmT5i7Gqkm6QGZcfoDkY3iIsRQBDDE17CpBOrGfSSAoXWUh45kNb1ARHdihluUy5Rsgi8Wxhfht94laXoWDiohAHyBKe4=

[dbadmin@oracle-linux container]$ getfattr -dm- /usr/libexec/podman/catatonit
[dbadmin@oracle-linux container]$

I was trying with getfattr -d not -d -m- (thanks).

So, I was not able to reproduce it because the file in my VM was not IMA signed.

I guess the question that persists is: how can we configure podman to ignore xattr while copying(or continue if setting xattr fails).

As a workaround, I managed to avoid the problem by saving the pause image from another machine(without an IMA signed catatonit), and loading it locally(on the machine with IMA signatures)

[dbadmin@oracle-linux container]$ podman save -o pause.tgz localhost/podman-pause:4.2.0-1677582426
Copying blob 38af3032cde2 done  
Copying config 4a0170bd2f done  
Writing manifest to image destination
Storing signatures

# Then copied pause.tgz to qboro2 and loaded it
(0)dbadmin@qboro2:~ $ podman load -i pause.tgz 
Getting image source signatures
Copying blob 38af3032cde2 done  
Copying config 4a0170bd2f done  
Writing manifest to image destination
Storing signatures
Loaded image: localhost/podman-pause:4.2.0-1677582426

(0)dbadmin@qboro2:~ $ podman images
REPOSITORY                TAG               IMAGE ID      CREATED       SIZE
localhost/podman-pause    4.2.0-1677582426  4a0170bd2fa6  18 hours ago  809 kB
docker.io/library/alpine  3.16.2            9c6f07244728  7 months ago  5.83 MB

# Then it finally worked to create a pod:
(0)dbadmin@qboro2:~ $ podman pod create --infra=true --publish=0.0.0.0:13306:3306 --publish=0.0.0.0:19104:9104 --name pod_fcm_db --log-level=debug
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called create.PersistentPreRunE(podman pod create --infra=true --publish=0.0.0.0:13306:3306 --publish=0.0.0.0:19104:9104 --name pod_fcm_db --log-level=debug) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /qst/podman_root/dbadmin/libpod/bolt_state.db 
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /qst/podman_root/dbadmin    
DEBU[0000] Using run root /run/user/2115/containers     
DEBU[0000] Using static dir /qst/podman_root/dbadmin/libpod 
DEBU[0000] Using tmp dir /run/user/2115/libpod/tmp      
DEBU[0000] Using volume path /qst/podman_root/dbadmin/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: storage already configured with a mount-program 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 73             
DEBU[0000] Adding port mapping from 13306 to 3306 length 1 protocol "" 
DEBU[0000] Adding port mapping from 19104 to 9104 length 1 protocol "" 
DEBU[0000] Looking up image "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ... 
DEBU[0000] parsed reference into "[overlay@/qst/podman_root/dbadmin+/run/user/2115/containers]@4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933" 
DEBU[0000] Found image "localhost/podman-pause:4.2.0-1677582426" as "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Found image "localhost/podman-pause:4.2.0-1677582426" as "localhost/podman-pause:4.2.0-1677582426" in local containers storage ([overlay@/qst/podman_root/dbadmin+/run/user/2115/containers]@4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933) 
DEBU[0000] exporting opaque data as blob "sha256:4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933" 
DEBU[0000] Created cgroup path user.slice/user-libpod_pod_e355f6f2756a840cf859c54e9979748e6910049125263ed6fca4b18604488f52.slice for parent user.slice and name libpod_pod_e355f6f2756a840cf859c54e9979748e6910049125263ed6fca4b18604488f52 
DEBU[0000] Created cgroup user.slice/user-libpod_pod_e355f6f2756a840cf859c54e9979748e6910049125263ed6fca4b18604488f52.slice 
DEBU[0000] Got pod cgroup as user.slice/user-libpod_pod_e355f6f2756a840cf859c54e9979748e6910049125263ed6fca4b18604488f52.slice 
DEBU[0000] Looking up image "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "localhost/podman-pause:4.2.0-1677582426" ... 
DEBU[0000] parsed reference into "[overlay@/qst/podman_root/dbadmin+/run/user/2115/containers]@4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933" 
DEBU[0000] Found image "localhost/podman-pause:4.2.0-1677582426" as "localhost/podman-pause:4.2.0-1677582426" in local containers storage 
DEBU[0000] Found image "localhost/podman-pause:4.2.0-1677582426" as "localhost/podman-pause:4.2.0-1677582426" in local containers storage ([overlay@/qst/podman_root/dbadmin+/run/user/2115/containers]@4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933) 
DEBU[0000] exporting opaque data as blob "sha256:4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933" 
DEBU[0000] Inspecting image 4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933 
DEBU[0000] exporting opaque data as blob "sha256:4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933" 
DEBU[0000] exporting opaque data as blob "sha256:4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933" 
DEBU[0000] Inspecting image 4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933 
DEBU[0000] Inspecting image 4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933 
DEBU[0000] Inspecting image 4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933 
DEBU[0000] using systemd mode: false                    
DEBU[0000] setting container name e355f6f2756a-infra    
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] Allocated lock 2 for container 8ad1e2b7246ed2cfea57cc917f47d1b2077424b469e2211778e3e80be7fd2201 
DEBU[0000] parsed reference into "[overlay@/qst/podman_root/dbadmin+/run/user/2115/containers]@4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933" 
DEBU[0000] exporting opaque data as blob "sha256:4a0170bd2fa6dbb0bd094e3dd7ec7c211b99fff127def0c6141f00c1db37e933" 
DEBU[0000] Created container "8ad1e2b7246ed2cfea57cc917f47d1b2077424b469e2211778e3e80be7fd2201" 
DEBU[0000] Container "8ad1e2b7246ed2cfea57cc917f47d1b2077424b469e2211778e3e80be7fd2201" has work directory "/qst/podman_root/dbadmin/overlay-containers/8ad1e2b7246ed2cfea57cc917f47d1b2077424b469e2211778e3e80be7fd2201/userdata" 
DEBU[0000] Container "8ad1e2b7246ed2cfea57cc917f47d1b2077424b469e2211778e3e80be7fd2201" has run directory "/run/user/2115/containers/overlay-containers/8ad1e2b7246ed2cfea57cc917f47d1b2077424b469e2211778e3e80be7fd2201/userdata" 
e355f6f2756a840cf859c54e9979748e6910049125263ed6fca4b18604488f52
DEBU[0000] Called create.PersistentPostRunE(podman pod create --infra=true --publish=0.0.0.0:13306:3306 --publish=0.0.0.0:19104:9104 --name pod_fcm_db --log-level=debug)
(0)dbadmin@qboro2:~ $ echo ${?}
0

by the way, can we pull this image from a registry ? Seems related to: #11256 .

@giuseppe
Copy link
Member

giuseppe commented Apr 6, 2023

couldn't you copy the catatonit file somewhere else and use that (with --init-path)?

@paul-grozav
Copy link
Author

couldn't you copy the catatonit file somewhere else and use that (with --init-path)?

My context (as in the oracle documentation) was podman pod create --infra=true . I see that the --init-path is an option for podman run - creating a container, not a pod(the pod infrastructure container).

Is there a way I could provide the "Path to the container-init binary" to podman pod create ?

Or, the alternative(with lower chances of success) would be to create the "infra" container using regular podman run --init-path ... and then somehow, ... could I create a new pod and specify an existing infrastructure container that it should use?

Maybe a possible alternative would be to somehow use podman pod create with --infra-image and --infra-command to avoid creating that pause image, and use something else instead, another pre-built image(with catatonit), or just a a busybox sleep.

I've tested the latter idea, and it seems to be working 😃

(0)dbadmin@qboro2:~ $ podman pod create --infra=true --infra-image=busybox:1.36.0 --infra-command="sleep infinity" --publish=0.0.0.0:8153:80 --name pgrozav_ima_test
c1d03e1539f733d4b782d3c0f72aa87b78bd5bc84d43865a0b763319b22d66dc
(0)dbadmin@qboro2:~ $ podman run --pod pgrozav_ima_test --name some-nginx -d docker.io/nginx:1.23.4
9666a89ceeadcd4de9c6cf5461743b8ffcd0d0e1b45f796abd3d15532014db9d
(0)dbadmin@qboro2:~ $ curl -vvv http://qboro2:8153/
*   Trying 192.168.200.95:8153...
* Connected to qboro2 (192.168.200.95) port 8153 (#0)
> GET / HTTP/1.1
> Host: qboro2:8153
> User-Agent: curl/7.76.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.23.4
...
</html>
* Connection #0 to host qboro2 left intact
(0)dbadmin@qboro2:~ $ podman pod stop -t0 pgrozav_ima_test && podman pod rm pgrozav_ima_test
c1d03e1539f733d4b782d3c0f72aa87b78bd5bc84d43865a0b763319b22d66dc
c1d03e1539f733d4b782d3c0f72aa87b78bd5bc84d43865a0b763319b22d66dc

So, at least we have a working solution even on Oracle :) - I'm not sure how important it is to run catatonit or that exact podman-pause:4.2.0-1677582426 image for the infra container. Are there any disadvantages(from a security perspective maybe) if we run busybox sleep instead?