docker buildx bake handles environment variables as arguments differently in v0.6.0

[andrewsmith]

Hello! We noticed a breakage in our build today and I started investigating this afternoon. We're using buildx within GitHub Actions. I was able to isolate the behavior down to a fairly simple set of steps.

Suppose we have a Dockerfile that looks like:

$ cat Dockerfile 
FROM busybox:latest
ARG FOO
RUN echo $FOO > foo

and a docker-compose.yaml that looks like:

$ cat docker-compose.yaml 
version: "3.8"
services:
  example:
    image: example
    build:
      context: .
      dockerfile: Dockerfile
      args:
        FOO: $FOO

Essentially, we want to take the environment variable $FOO in the build environment and store that in a file 'foo' as part of the image.

When we run the following with buildx v0.5.1, this is what we get:

$ docker buildx version
github.com/docker/buildx v0.5.1 11057da37336192bfc57d81e02359ba7ba848e4a

$ FOO="example-value" docker buildx bake --no-cache --load --progress=plain
#1 [internal] load build definition from Dockerfile
#1 sha256:4ba49b5593c48775beaffc27706380eef457bb090c0e7695146e045b468a409f
#1 transferring dockerfile: 144B done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 sha256:93e26e32c334e3577ccb8bef529124bfe1ffeebce8758941cb89e9402ae4fb2e
#2 transferring context: 2B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/busybox:latest
#3 sha256:da853382a7535e068feae4d80bdd0ad2567df3d5cd484fd68f919294d091b053
#3 DONE 0.5s

#4 [1/2] FROM docker.io/library/busybox:latest@sha256:0f354ec1728d9ff32edcd7d1b8bbdfc798277ad36120dc3dc683be44524c8b60
#4 sha256:be47039383562d6f7c9560dd8b9c4e805f9f039bb1d09801bb3b3e30232e16c5
#4 resolve docker.io/library/busybox:latest@sha256:0f354ec1728d9ff32edcd7d1b8bbdfc798277ad36120dc3dc683be44524c8b60 0.0s done
#4 DONE 0.0s

#4 [1/2] FROM docker.io/library/busybox:latest@sha256:0f354ec1728d9ff32edcd7d1b8bbdfc798277ad36120dc3dc683be44524c8b60
#4 sha256:be47039383562d6f7c9560dd8b9c4e805f9f039bb1d09801bb3b3e30232e16c5
#4 CACHED

#5 [2/2] RUN echo example-value > foo
#5 sha256:4b2c046e1a82380556925c1d12da255a5c57d52dbd08f9b2fe55139a341d4094
#5 DONE 0.2s

#6 exporting to oci image format
#6 sha256:69a2560eef4d3ece902c3b5149d142e9bd132f25db0bc9e35b94201534c415d2
#6 exporting layers
#6 exporting layers 0.2s done
#6 exporting manifest sha256:2e66b84a17468e5ea239f97f694a82ccf523048c4fa61193d3ff91aa393b49a2 0.0s done
#6 exporting config sha256:78449c845abda8b7674ca9b889929671b80df35f6e206569d33592ccb2de8944 0.0s done
#6 sending tarball
#6 sending tarball 0.1s done
#6 DONE 0.4s

#7 importing to docker
#7 sha256:bec083dd7470da0c1b8d7e879e251c6ce8df021ca0f43d380ecb6adee668c579
#7 DONE 0.1s

$ docker run --rm 78449c845abd cat foo
example-value

With the recently released v0.6.0, we get:

$ docker buildx version
github.com/docker/buildx v0.6.0 d9ee3b134cbc2d09513fa7fee4176a3919e05887

$ FOO="example-value" docker buildx bake --no-cache --load --progress=plain
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.1s

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 144B done
#1 DONE 0.1s

#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 0.5s

#4 [1/2] FROM docker.io/library/busybox:latest@sha256:0f354ec1728d9ff32edcd7d1b8bbdfc798277ad36120dc3dc683be44524c8b60
#4 resolve docker.io/library/busybox:latest@sha256:0f354ec1728d9ff32edcd7d1b8bbdfc798277ad36120dc3dc683be44524c8b60 0.0s done
#4 CACHED

#5 [2/2] RUN echo $FOO > foo
#5 DONE 0.1s

#6 exporting to oci image format
#6 exporting layers
#6 exporting layers 0.2s done
#6 exporting manifest sha256:eb828ab01dbf41ef0fc27f058ef41eab8837e537e68b2463e69359302dbca7b9 0.0s done
#6 exporting config sha256:847ddfc7cf05d2c8ea64722a808e96df505adce4f74a0848ac78f296ff3b9b05
#6 exporting config sha256:847ddfc7cf05d2c8ea64722a808e96df505adce4f74a0848ac78f296ff3b9b05 0.0s done
#6 sending tarball
#6 sending tarball 0.1s done
#6 DONE 0.4s

#7 importing to docker
#7 DONE 0.1s

$ docker run --rm 847ddfc7cf05 cat foo
$FOO

Notice that in this case, "$FOO" is written to the file in the image (step #5), rather than the value from the environment variable. It doesn't seem to be expanded properly.

For comparison, docker-compose build (not using buildx), gives us:

$ FOO="example-value" docker-compose build --no-cache
Building example
Sending build context to Docker daemon  3.072kB

Step 1/3 : FROM busybox:latest
 ---> 69593048aa3a
Step 2/3 : ARG FOO
 ---> Running in 800296a089dc
Removing intermediate container 800296a089dc
 ---> 9e808cd18bbc
Step 3/3 : RUN echo $FOO > foo
 ---> Running in 757f4f6c83b8
Removing intermediate container 757f4f6c83b8
 ---> 560f0aa9dd30
Successfully built 560f0aa9dd30
Successfully tagged example:latest

$ docker run --rm 560f0aa9dd30 cat foo
example-value

Which is the same behavior as buildx bake in v0.5.1.

Not sure if this is helpful too, but here is the version of Docker we are seeing this in:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:57:20 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:55:04 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[crazy-max]

@andrewsmith Thanks for your feedback and repro, what is the output of FOO="example-value" docker buildx bake --print?

[sanderegg]

seeing the same problem with buildx v0.6.0. The bake application seems to not resolve env variables as before.
Our github CI builds are failing since yesterday.

[crazy-max]

Was able to reproduce. Keep you in touch with a fix.