Podman takes several minutes to load a compressed .tar.xz file

[RevNight]

Issue Description

The issue we are facing is that when we load podman images that are compressed .tar.xz. These .tar.xz files have multiple podman images in them. When we load this file, it takes several minutes to complete the load. The first 4 minutes are it just sitting there, the last 30 seconds is work actually being done as we see the images being loaded.

The reason for this bug is that docker (via docker images, not podman) loads its version of the compressed .tar.xz file in like 30 seconds.

4m30s vs 30s.

Why is Podman taking so long? And yes, both files are roughly the same size.

I have tried this on my workstation (I9 9900K) and home VM (Ryzen 7 7700X). Both are using Fedora33 VM with 4 CPUs. So yeah I could increase VM specs, but that wouldn't be realistic to what I have at work. Also wouldn't solve the issue as Docker tar.xz loads just fine.

Tried on Podman 2.1.1 and 3.3.1.

Steps to reproduce the issue

Steps to reproduce the issue

1. Load Fedora 33 VM
2. Install 3.3.1 Podman

• sudo dnf install podman-docker-3:3.3.1-1.fc33.noarch

3. Load public images to simulate number and size of work images

• sudo podman pull cockroachdb/cockroach:latest
• sudo podman pull minio/minio:RELEASE.2023-08-23T10-07-06Z.fips
• sudo podman pull minio/mc:latest
• sudo podman pull alpine:latest
• sudo podman pull busybox:latest
• sudo podman pull docker:latest
• sudo podman pull rabbitmq:latest

4. Save multiple podman images to singular compressed .tar.xz. This will take 5-10 minutes.
   sudo podman save -m \
   docker.io/library/rabbitmq:latest \
   docker.io/minio/mc:latest \
   docker.io/minio/minio:RELEASE.2023-08-23T10-07-06Z.fips \
   docker.io/library/alpine:latest \
   docker.io/cockroachdb/cockroach:latest \
   docker.io/library/docker:latest \
   docker.io/library/busybox:latest | xz > ImagesPodmanCompressed.tar.xz
5. Clear out podman images so you can then load this file again.

• sudo podman rmi $(sudo podman images -q)

6. Load this ImagesPodmanCompressed.tar.xz file. Notice how it just sits there for several minutes (BAD) before it begins work.

• time sudo podman load -i ImagesPodmanCompressed.tar.xz

Describe the results you received

Notice how it just sits there for several minutes (BAD) before it begins work. It takes 4m30s for Podman to load this type of file but it only takes Docker 30s.

Describe the results you expected

Images should start to load immediately. This is the behavior of Docker for this same type of singular compressed .tar.zx.

podman info output

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.15-301.fc33.x86_64
  linkmode: dynamic
  memFree: 3793604608
  memTotal: 12334272512
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: AP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAPSETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-4.dev.giteecccdb.fc33.x86_64
    version: |-
      slirp4netns version 1.1.4+dev
      commit: eecccdb96f587b11d7764556ffacfeaffe4b6e11
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 2h 45m 30.77s (Approximately 0.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356406
  BuiltTime: Mon Aug 30 16:46:46 2021
  GitCommit: ""
  GoVersion: go1.15.14
  OsArch: linux/amd64
  Version: 3.3.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

Workstation:
I9 9900K
16 GB ram
Windows 10
Oracle VM VirtualBox, Fedora 33, 4cpu/8gb ram
Podman version 2.1.1
File takes about 4m30s to load

Home PC:
Ryzen 7 7700X
32GB ram
Windows 10
Oracle VM VirtualBox, Fedora 33, 4cpu/16gb ram
Podman version 3.3.1
File takes about 3min to load

Additional information

None

[flouthoc]

Hi @RevNight , Thanks for creating the issue. I'll check this in a recent version of podman but could you confirm one thing. The versions of podman which you are using are significantly old, could you consider using a more recent version and see if issue persists ?

[baude]

Fedora 33 is EOL ... agree with trying a more recent podman.

[RevNight]

Hi all. Thanks for your replies! We are given these workstations so that's what I have to emulate at my house. I assume that the newest podman version won't work on Fedora 33? I tried installing 4.6.1 and I got red text errors. Fedora natively came with version 2.1.1 and typing "sudo dnf install TAB" allowed me to download 'podman-docker-3:3.3.1' or something similar. So when I type 'podman --version' it now says 3.3.1. I know this is a far cry from 4.6.1 (newest at time of this writing) but it's better than 2.1.1.

Just realized that the most important OS is the one of the VM we are loading these on (not my dev station). That is CentOS 7 (linux 5.4) kernel. What version of Podman can support this? They currently have 1.6.4 and it doesnt even work haha. Instead of taking 4:30 it will sit there for like 20-30 minutes and then timeout. It does give an error but I can't remember what

On a related note, how do I install podman in offline mode? I'm aware of the Podman Installation page and Podman Release page. The Installation page just says point to the repo, which won't work in my case. The release page, I see several links for each version, but none that are install packages (aka sudo dnf install podman-package.tar.gz). If my research is correct, podman-remote is the client software, so I don't need that.

Thanks all!

[afbjorklund]

I would check if it is any significant difference when using xzcat | sudo podman load, as opposed to .tar.xz

unxz < ImagesPodmanCompressed.tar.xz | time sudo podman load

Since https://github.com/ulikunitz/xz says "At this time the package cannot compete with the xz tool regarding compression speed and size", it might be slower for large images. You could also use a different compression...

Docker just forks the external xz binary, instead of trying to do the decompression within the image library itself.

[afbjorklund]

Apparently the reason why it is so incredibly slow, is that it doesn't do any buffering but just calls ReadByte...

• Unzipping is too slow ulikunitz/xz#23

Should be a simple fix in containers/image:

r, err := xz.NewReader(bufio.NewReader(f))

[RevNight]

I would check if it is any significant difference when using xzcat | sudo podman load, as opposed to .tar.xz

unxz < ImagesPodmanCompressed.tar.xz | time sudo podman load

Since https://github.com/ulikunitz/xz says "At this time the package cannot compete with the xz tool regarding compression speed and size", it might be slower for large images. You could also use a different compression...

Docker just forks the external xz binary, instead of trying to do the decompression within the image library itself.

OH SHIT. Give this man a beer

By running this command, my times are:

33.96 user
7.48 system
1.02.37 elapsed

That's on my Fedora 33 (2.1.1). On the Centos OS 7 box (1.6.4), it still fails for the same error, but it fails quickly! No more waiting for 20-30 minutes to fail.

Error:
Unexpected tar manifest.json: expected 1 item, got 5
open /var/tmp/podman49180063/manifest.json: not a directory
Error: error pulling "": unable to pull dir: /var/tmp/podman49180063: unable to pull image: Error determining manifest MIME type for dir /var/tmp/podman49180063: open /var/tmp/podman49180063/manifest.json: not a directory

The above error is fine, I can work with the team to get to 2.1.1, Ideally a more recent one.

Now all I need to do is this:

How can I update Podman offline? I need the package I can point to locally because the work stuff is air gapped.

[afbjorklund]

I think the old podman could only load one image per tarball, so that might be a limitation of the 1.x series?

EDIT: Apparently it was 2.1.0: 7fea467

Some background on using the decompression library, instead of forking a program: containers/image#427

[RevNight]

I think the old podman could only load one image per tarball, so that might be a limitation of the 1.x series?

Some background on using the decompression library, instead of forking a program: containers/image#427

Yep! I figured the age of the version would be the reason since the newer one is working fine.

Still, that begs the question: How do I update podman offline via a local package? Where can I get these packages from?

[afbjorklund]

How do I update podman offline via a local package? Where can I get these packages from?

I think it comes with the OS, so it might be the end-of-the-line for Fedora 33 and CentOS 7 (based on Fedora 18)

If my research is correct, podman-remote is the client software, so I don't need that.

That is correct, the program is renamed to "podman" on the macOS and Windows platforms

[RevNight]

Yeah but even if it comes with the OS there has got to be a way to update it manually. Is there a chart somewhere that shows what each version of Podman supports which OS number?

[afbjorklund]

I could only find the old blog post (currently all offline, due to the website reorganization):
https://github.com/containers/podman.io_old/blob/main/old/_posts/2021-03-02-podman-support-for-older-distros.md

Podman 3.0 will be the last major build on CentOS 7, Debian 10 and Ubuntu 18.04.

But I think 2.2.1 el7 never made it out of testing, so 1.6.4 is where it ended up for CentOS 7.
For Ubuntu, it is currently at 3.4.2 for 20.04 and 3.4.4 for 22.04 - Debian 12 is at podman 4.3.1

You can probably get a better answer for supported distribution versions, Fedora / Stream?
https://repology.org/project/podman/badges indicates they are now running podman 4.6.1

https://docs.fedoraproject.org/en-US/releases/eol/ says that Fedora 36 and earlier are EOL.

[RevNight]

Ok this page was perfect (https://repology.org/project/podman/versions)

It shows what version came native to the OS, and what is the latest available that will work. For Fedora 33, it correctly shows that 2.1.1 came native, and the most recent it can do is 3.3.1. For CentOS 7, 1.4.4 came native, 1.6.4 is most recent possible.

So if we want a newer version of Podman, we have to go to Cent OS 8 (3.3.1) or 9 (4.6.0).

At this point, I don't think knowing how to manually update it is an issue. Upgrade the OS and chances are it will have a much newer version, and all we need is anything 2.1.1+ which so far every OS is.

[afbjorklund]

If you really need to support it, you can do a for loop and save one image per tarball. That is what we did in minikube, for cri-o (which uses podman to load and build images). The sane approach would be to use a local registry server...

If you don't need the disk space savings, you could also just use gzip for the tarball? Unless that is also slow for some reason, then you would have to gunzip the same way you did the unxz. We used lz4 for performance reasons.

[RevNight]

Oh its a client demand. A single file that holds multiple images that is compressed.

[afbjorklund]

Sounds like a zip of the image tarballs (the "cache") would meet the requirement :-)