"kind load" tries to call docker though podman is used as runtime

[Unveraenderbar]

What happened:
kind load fails (with a misleading error message about the image not being available locally) if podman is used as runtime,
because it tries to do image management by calling a binary called "docker"

What you expected to happen:
If the cluster is provisioned by using podman as runtime management tool, kind should also call the podman bnary for image
management on load operations

How to reproduce it (as minimally and precisely as possible):

• export KIND_EXPERIMENTAL_PROVIDER=podman as environment variable
• set up kind cluster
• pull an additional image into the local podman image store
• try to load said image into kind cluster

This provokes an error

$ podman images
REPOSITORY                                 TAG         IMAGE ID      CREATED       SIZE
docker.io/kindest/node                     v1.24.0     f607f70d3a1a  2 days ago    938 MB
docker.io/linuxserver/openssh-server       latest      6ecbeb69716b  4 days ago    35.5 MB
$ kind load docker-image --name $(kind get clusters) docker.io/linuxserver/openssh-server:latest
enabling experimental podman provider
enabling experimental podman provider
ERROR: image: "docker.io/linuxserver/openssh-server:latest" not present locally
$ ln -s /usr/bin/podman $HOME/.local/bin
$ kind load docker-image --name $(kind get clusters) docker.io/linuxserver/openssh-server:latest
enabling experimental podman provider
enabling experimental podman provider
Image: "docker.io/linuxserver/openssh-server:latest" with ID "6ecbeb69716b9cd757187df0ea154dd56d1bdc1093ff115559e70e25d094662a" not yet present on node "k8s-v1.24.0-control-plane", loading...

Environment:

• kind version: (use kind version): v0.13
• Kubernetes version: (use kubectl version): v1.24.0
• Docker version: (use docker info): none -- but podman v3.4.7 and v4.1.1 work fine
• OS (e.g. from /etc/os-release): Fedora 35 Silverblue

[BenTheElder]

This is only true for "kind load docker-image" which yes that uses docker. See discussion in #2729

[Unveraenderbar]

I see.. thx for the quick answer! I'm fine with a symlink or image file pipe until #2729 is realized.

[BenTheElder]

I would love to see #2038 to add a new command dropping docker from the name and supporting all the inputs, I don't think it's super challenging, but there's just so many critical things (bug fixes...) to do between this project and elsewhere in kubernetes that nobody seems to have gotten to it yet.

[Unveraenderbar]

Hm now this is strange

$ podman save library/busybox:stable | kind load image-archive --name k8s-v1.24.0 -
enabling experimental podman provider
ERROR: failed to load image: command "podman exec --privileged -i k8s-v1.24.0-control-plane ctr --namespace=k8s.io images import --snapshotter overlayfs -" failed with error: exit status 1
Command Output: ctr: unrecognized image format
$ podman save library/busybox:stable | podman exec --privileged -i k8s-v1.24.0-worker2 sh -c 'cat > /tmp/busybox-stable.tar'
$ podman exec --privileged -i k8s-v1.24.0-worker2 ctr --namespace=k8s.io images import --snapshotter overlayfs /tmp/busybox-stable.tar
unpacking docker.io/library/busybox:stable (sha256:5f93b79941db8d790bc551bb66623d5c3a931385de4b35af01d097afc9fe275d)...done

Don't know how the image-archive transfer (which works fine if fed by a real file instead of a pipe) is realized internally... seems ctr behaves differently when writing to / reading from a pipe.

[BenTheElder]

Maybe it's defaulting --format differently when piping? https://docs.podman.io/en/latest/markdown/podman-save.1.html

[Unveraenderbar]

Don't know... the same error occurs regardless whether I call podman save --format docker-archive or ... --format oci-format
Weird is that directly piping from podman to ctr also works

$ podman  save --format docker-archive library/busybox:stable \
   | podman exec --privileged -i k8s-v1.24.0-worker2 ctr -namespace=k8s.io images import --snapshotter overlayfs -
unpacking docker.io/library/busybox:stable (sha256:5f93b79941db8d790bc551bb66623d5c3a931385de4b35af01d097afc9fe275d)...done

[BenTheElder]

We save to a tempdir and then pipe in that file with go

kind/pkg/cmd/kind/load/docker-image/docker-image.go

Lines 149 to 158 in c8b759f

	// Setup the tar path where the images will be saved
	dir, err := fs.TempDir("", "images-tar")
	if err != nil {
	return errors.Wrap(err, "failed to create tempdir")
	}
	defer os.RemoveAll(dir)
	imagesTarPath := filepath.Join(dir, "images.tar")
	// Save the images into a tar
	err = save(imageNames, imagesTarPath)
	if err != nil {

I don't know why podman would be different here. It is equivalent to podman save -o $tmpfile $image on the save side. On the load side it is the same as above, but piping in $tmpfile in go.

[aojea]

@mheon @vrothberg does this ring any bell? why should podman have a different behavior piping the images?

[vrothberg]

Thanks for the ping, @aojea! Can you share a reproducer? I see a number of issues in this thread and am not sure which one to pick.

In theory, there should be no difference when piping.