DOCKER_BUILDKIT=0 is not applied by docker-compose build

[carlosdoordash]

Description of the issue

Context information (for bug reports)

• Using Compose V2 docker compose ...
• Using Compose V1 docker-compose ...

Output of docker(-)compose version

Docker Compose version v2.0.0-rc.3

Output of docker version

Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

services:
  test:
    build:
      context: /Users/carlosdoordash/Projects/test
      dockerfile: /Users/carlosdoordash/Projects/test/Dockerfile
    networks:
      default: null
networks:
  default:
    name: test_default

Steps to reproduce the issue

1. Create a Dockerfile

FROM ubuntu
RUN echo "127.0.0.1 test.localhost" >> /etc/hosts

2. Create a docker-compose.yml

version: '3'
services:
  test:
    build:
      context: .

3. Run docker-compose build with and without DOCKER_BUILDKIT.

$ COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build
[+] Building 0.4s (5/5) FINISHED                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 31B                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                  0.0s
 => CACHED [1/2] FROM docker.io/library/ubuntu                                                                                                                    0.0s
 => ERROR [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts                                                                                                 0.2s
------
 > [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts:
#5 0.217 /bin/sh: 1: cannot create /etc/hosts: Read-only file system
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c echo "127.0.0.1 test.localhost" >> /etc/hosts]: exit code: 2


$ COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=0 docker-compose build
[+] Building 0.4s (5/5) FINISHED                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 31B                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                  0.0s
 => CACHED [1/2] FROM docker.io/library/ubuntu                                                                                                                    0.0s
 => ERROR [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts                                                                                                 0.3s
------
 > [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts:
#5 0.242 /bin/sh: 1: cannot create /etc/hosts: Read-only file system
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c echo "127.0.0.1 test.localhost" >> /etc/hosts]: exit code: 2


$ COMPOSE_DOCKER_CLI_BUILD=0 DOCKER_BUILDKIT=0 docker-compose build
[+] Building 0.3s (5/5) FINISHED                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 31B                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                  0.0s
 => CACHED [1/2] FROM docker.io/library/ubuntu                                                                                                                    0.0s
 => ERROR [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts                                                                                                 0.2s
------
 > [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts:
#5 0.208 /bin/sh: 1: cannot create /etc/hosts: Read-only file system
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c echo "127.0.0.1 test.localhost" >> /etc/hosts]: exit code: 2
$

4. Compare to running docker build with and without DOCKER_BUILDKIT.

$ DOCKER_BUILDKIT=1 docker build .                                 
[+] Building 0.4s (5/5) FINISHED                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 36B                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                  0.0s
 => CACHED [1/2] FROM docker.io/library/ubuntu                                                                                                                    0.0s
 => ERROR [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts                                                                                                 0.2s
------
 > [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts:
#5 0.233 /bin/sh: 1: cannot create /etc/hosts: Read-only file system
------
executor failed running [/bin/sh -c echo "127.0.0.1 test.localhost" >> /etc/hosts]: exit code: 2


$ DOCKER_BUILDKIT=0 docker build .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM ubuntu
 ---> fb52e22af1b0
Step 2/2 : RUN echo "127.0.0.1 test.localhost" >> /etc/hosts
 ---> Using cache
 ---> ea73e60b789f
Successfully built ea73e60b789f

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
$ 

Observed result

In step 3, note how running DOCKER_BUILDKIT=0 docker-compose build still results in a failed build because /etc/hosts is on a read-only filesystem.
In step 4, note how running DOCKER_BUILDKIT=0 docker build, the docker build succeeds because /etc/hosts isn't on a read-only filesystem.

Expected result

In step 3, running DOCKER_BUILDKIT=0 docker-compose build should result in a successful build.

Stacktrace / full error message

 => ERROR [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts                                                                                                 0.2s
------
 > [2/2] RUN echo "127.0.0.1 test.localhost" >> /etc/hosts:
#5 0.208 /bin/sh: 1: cannot create /etc/hosts: Read-only file system
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c echo "127.0.0.1 test.localhost" >> /etc/hosts]: exit code: 2

Additional information

OS version / distribution: macOS Big Sur version 11.6
docker-compose install method: Docker Desktop version 4.0.1 (68347)

[ndeloof]

Compose V2 always uses BuildKit by design.
If Buildkit can't build some Dockerfile, either there's good reasons, otherwise it has to be fixed.

IIRC there's plans to make Buildkit the default builder everywhere, and maybe to drop support for classic builder at some point

[carlosdoordash]

I found out that when DOCKER_BUILDKIT=0, docker builds succeed by silently ignoring the change to the /etc/hosts file. I verified that the /etc/hosts file in the docker images do not have the test.localhost entries.

More detail about how the classic builder silently ignores the changes to /etc/hosts in moby/buildkit#1267 (comment)