Using Dockerfile to build devcontainer with Podman causes VSCode to hang

[awesomekyle]

When creating a devcontainer from a Dockerfile using Podman, VSCode hangs because Podman doesn't pull from localhost/ when using the --platform build arg. Because the terminal Dev Container terminal window can't be interacted with at all, this means VSCode hangs forever and the only way to stop it is to kill VSCode.

The Dockerfile builds correctly ("Successfully tagged localhost/vsc-podman....") and then VSCode does some additional setup using the /tmp/devcontainercli-kyle/updateUID.Dockerfile-0.58.0 file, which has the following first two lines:

ARG BASE_IMAGE
FROM $BASE_IMAGE
...

BASE_IMAGE=vsc-podman-devcontainer-41....53dbb is being set in the podman build command. The issue is that podman build prompts for the registry as shown in the image below:

.devcontainer/devcontainer.json

If I use the image option, the container is created correctly. It only fails when BASE_IMAGE is a local image.

{
  "name": "Alpine",

  // "image": "mcr.microsoft.com/devcontainers/base:alpine-3.19",
  "build": { "dockerfile": "Dockerfile" },

  "workspaceMount": "",
  "runArgs": [
    "--userns=keep-id",
    "--volume=${localWorkspaceFolder}:/workspaces/${localWorkspaceFolderBasename}:z"
  ],
  "containerUser": "vscode"
}

.devcontainer/Dockerfile

FROM mcr.microsoft.com/devcontainers/base:alpine-3.19

Workaround:

• If /etc/containers/registries.conf only has one unqualified-search-registries, podman won't prompt for a selection so the container creation will work correctly.
• After it fails, manually run the command that hung, choosing any prompted registry (it doesn't matter, Podman pulls locally regardless). Edit: After further testing this isn't true. VSCode tries to rebuild it regardless.
• Push the base container to a registry instead of using the build option. Edit: Using features also causes new containers to be built, so this workaround only works if you don't use devcontainer features.

[chrmarti]

Is this a bug in Podman? You mention --platform, does it not ask for the registry without --platform?

[awesomekyle]

I'm not sure if this is a bug, I haven't yet filed a bug with Podman. But that's correct, if --platform is not passed in, Podman doesn't prompt for anything. This could be intentional behavior in Podman. If you don't specify a platform, it will automatically use localhost because it's the correct platform by definition.

Testing a bit more locally this also means that features can't be used either because they also build a new container dynamically. The following devcontainer.json will cause a hang when running with Podman on Fedora with "dev.containers.dockerPath": "podman",.

{
  "name": "Alpine",

  "image": "mcr.microsoft.com/devcontainers/base:alpine-3.19",
  // "build": { "dockerfile": "Dockerfile" },

  "workspaceMount": "",
  "runArgs": [
    "--userns=keep-id",
    "--volume=${localWorkspaceFolder}:/workspaces/${localWorkspaceFolderBasename}:z"
  ],
  "containerUser": "vscode",
  "features": {
    "ghcr.io/wxw-matt/devcontainer-features/command_runner:0": {}
  }
}

After testing more I updated my original post; manually running the failed Podman command doesn't work, VSCode continues trying to build the container.

[awesomekyle]

In case any body else happens to run into this, you can use the CONTAINERS_REGISTRIES_CONF environment variable to override the registries file:

~/.local/bin/podman-devcontainers:

#!/bin/bash
CONTAINERS_REGISTRIES_CONF="$HOME/.config/devcontainers-registries.conf" podman "$@"

~/.config/devcontainers-registries.conf:

unqualified-search-registries = ["docker.io"]

And then use "dev.containers.dockerPath": "/home/username/.local/bin/podman-devcontainers", in VSCode's settings. This will then ensure Podman only has one registry to pull from and doesn't hang.

@chrmarti, at the very least there should be a way to cancel the container creation process. If the process fails due to an error, VSCode prompts the user what to do. In this case there's no option other than killing VSCode entirely.

[lgonzalezsa]

I was having same issue described here and reversed to v0.348.0 of the extension and now works fine with podman
Issue with v0.354.0:

[chrmarti]

I think devcontainers/cli#746 might have changed it, so we now always pass --platform, also when the platform matches the local hardware.

[pjlammertyn]

@awesomekyle correct, restricting the number of registries is a workaround

mkdir $HOME/.config/containers
cat <<EOF > $HOME/.config/containers/registries.conf
unqualified-search-registries = ['docker.io']
EOF

[soiamsoNG]

@chrmarti
encounter same issue and after some search get below

podman tag behavior is not as docker
podman will alwasy add localhost/ before tag of image without specify registry part
even you are not hosting a local registry server.

but BASE_IMAGE miss localhost/ before it.

[chrmarti]

Not sure what the correct fix is here. It's unfortunate that a change on our side broke things, but I also think adding --platform shouldn't suddenly require user input. I'm leaning towards considering this a Podman issue.

[pgourinchas]

Is it possible to prefix the BASE_IMAGE argument with localhost/ when the docker path setting contains podman ?

[chrmarti]

That appears to work. We already use docker/podman -v to check if it is Podman and can reuse that.

Adding the suggested fix, could someone file an issue with Podman to check if the behavior is expected? It seems the addition of --platform triggered this, correct?

[chrmarti]

This fix is available with Dev Containers 0.360.0-pre-release. Let us know if this works, thanks!

[lgonzalezsa]

Testing Dev Containers 0.360.0-pre-release when the local image exist and works.
But if I do a rebuild without cache, I do have an image in our own docker hub (hub.docker.mycorp.net) but seems the extension now adds localhost/ in front of that one as well, causing certainly the connection refused because I do not host that repository on my local machine.

Let me know if you need other logs from my side of if I should post on other GitHub issue.

Thanks!

[2024-04-19T14:07:43.813Z] STEP 1/9: FROM localhost/hub.docker.mycorp.net/my_repo/base:latest
[2024-04-19T14:07:43.816Z] Trying to pull localhost/hub.docker.mycorp.net/my_repo/base:latest...
[2024-04-19T14:07:43.818Z] WARN[0000] Failed, retrying in 2s ... (1/3). Error: initializing source docker://localhost/hub.docker.mycorp.net/my_repo/base:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
[2024-04-19T14:07:45.820Z] WARN[0002] Failed, retrying in 2s ... (2/3). Error: initializing source docker://localhost/hub.docker.mycorp.net/my_repo/base:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
[2024-04-19T14:07:47.822Z] WARN[0004] Failed, retrying in 2s ... (3/3). Error: initializing source docker://localhost/hub.docker.mycorp.net/my_repo/base:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
[2024-04-19T14:07:49.825Z] Error: creating build container: initializing source docker://localhost/hub.docker.mycorp.net/my_repo/base:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

[chrmarti]

Thanks @lgonzalezsa!

I need to add a check if there is already a registry name since we go through the same code path with that. PR is out: devcontainers/cli#810

[damaestro]

Using the pre-release channel resolved this issue for me.

[chrmarti]

@lgonzalezsa The updated fix is available with Dev Containers 0.361.0-pre-release. Please give that a try, thanks!

[lgonzalezsa]

Awesome! works for me!