Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Creating a build context with a specific platform #151

Closed
ohnotnow opened this issue Sep 25, 2019 · 7 comments
Closed

Creating a build context with a specific platform #151

ohnotnow opened this issue Sep 25, 2019 · 7 comments

Comments

@ohnotnow
Copy link

Hi,

I'm trying to set up a build context that uses my local machine (mac) for the amd64 builds and a raspberry pi for the arm/v7 ones. So I do :

$ docker buildx create --name mybuilder --platform linux/amd64
$ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS   PLATFORMS
mybuilder *  docker-container
  mybuilder0 unix:///var/run/docker.sock inactive linux/amd64
rpi          docker
  rpi        rpi                         running  linux/arm/v7, linux/arm/v6
default      docker
  default    default                     running  linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

$ docker buildx create --append --name mybuilder rpi
$ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS   PLATFORMS
mybuilder *  docker-container
  mybuilder0 unix:///var/run/docker.sock inactive linux/amd64
  mybuilder1 rpi                         inactive
rpi          docker
  rpi        rpi                         running  linux/arm/v7, linux/arm/v6
default      docker
  default    default                     running  linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

$ docker buildx build --load .
...
$ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS   PLATFORMS
mybuilder *  docker-container
  mybuilder0 unix:///var/run/docker.sock running  linux/amd64, linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
  mybuilder1 rpi                         inactive
rpi          docker
  rpi        rpi                         running  linux/arm/v7, linux/arm/v6
default      docker
  default    default                     running  linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

So the mybuilder0 has picked up all the other platforms - so if I specify a --platform=linux/arm/v7 on the build it seems to use the local mybuilder0 rather than the raspberry pi.

I'm not sure if there's a work-around for that, or if I'm doing something wrong. Probably the latter ;-)

Docker version is 19.03.2, build 6a30dfc.
@ohnotnow
Copy link
Author

:: sigh :: Ok - ignore me. I'd not noticed the docker version on the pi was still 18.09 and didn't have buildx at all. Updated to 19.03 and put the buildx binary in place and it seems to do The Right Thing(tm). Sorry!

@ohnotnow ohnotnow reopened this Sep 25, 2019
@ohnotnow
Copy link
Author

Actually - I'm still seeing the same thing and it doesn't seem to be using the pi to build the arm image. I have :

$ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS  PLATFORMS                      mybuilder *  docker-container
  mybuilder0 unix:///var/run/docker.sock running linux/amd64, linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
  mybuilder1 rpi                         running linux/arm/v7, linux/arm/v6
rpi          docker
  rpi        rpi                         running linux/arm/v7, linux/arm/v6
default      docker
  default    default                     running linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

If I kick off a build using docker buildx build --pull --push --platform linux/amd64,linux/arm/v7 . then I see in docker events on the pi :

2019-09-25T13:35:51.683420995+01:00 container exec_create: buildctl dial-stdio 487bf565fa69a7b04b25d0eae7cfe729f27b63c5dcbbc2db43341246da299c27 (execID=bf5e1bf408c24bce640135a47c5ec0b2b944c3ac8ba650b003aa178387912b51, image=moby/buildkit:buildx-stable-1, name=buildx_buildkit_mybuilder1)
2019-09-25T13:35:52.046550529+01:00 container exec_start: buildctl dial-stdio 487bf565fa69a7b04b25d0eae7cfe729f27b63c5dcbbc2db43341246da299c27 (execID=bf5e1bf408c24bce640135a47c5ec0b2b944c3ac8ba650b003aa178387912b51, image=moby/buildkit:buildx-stable-1, name=buildx_buildkit_mybuilder1)

Which made me think all was well. But the image still seems to be building on the local mac. The system load on the pi stays down at about 0, while the mac's disk thrashes away even though the amd64 part is done and it's still building the arm/v7 one. If only build with --platform=linux/arm/v7 I see the same thing. Zero load on the pi, while the mac disk+cpu spike.

@tonistiigi
Copy link
Member

@ohnotnow As you see your local mybuilder0 node also supports building for linux/arm/v7 (probably through Docker desktop) so there is no need for buildx to switch to it. If you want it to prefer rpi you can set the --platform linux/arm/v7 manually when appending/modifying the node with buildx create as manual platform setting should take precedence over the automatically detected ones.

Btw, you don't need to install buildx/19.03 on the rpi for this to work.

@ohnotnow
Copy link
Author

ohnotnow commented Sep 26, 2019 •
edited
Loading

@tonistiigi ahh! Yep - specifying the platform as linux/arm/v7 when appending the rpi node worked perfectly! Thank you!

Just for anyone else who ends up here and is trying the same thing, my process was (shortened!) :

# create a context for the raspberry pi
docker context create --docker "host=ssh://me@192.168.1.77" rpi
# create a new builder with the local mac docker install
docker buildx create --name mybuilder
# add the rpi context to it
docker buildx create --append --name mybuilder --platform linux/arm/v7 rpi
# and now buildx to your hearts content

@nvtkaszpir
Copy link

nvtkaszpir commented Mar 25, 2020 •
edited
Loading

I believe this should be opened?
Trying to do the same and looks like --platform is not really applied when adding/modifying a node - docker still detects all platforms on the given host.
Notice that linux/amd64 is presented twice in some of the outputs, so the platforms are appended and not overwritten?
Or is that the fact that the first plarform in the list shows the preference?

Check current status:

$ docker buildx create --append --name farm --platform linux/amd64 ssh://kaszpir@nyx-m6v
farm

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT       STATUS  PLATFORMS
farm *    docker-container              
  farm0   hormes                running linux/arm/v7, linux/arm/v6
  farm1   ssh://kaszpir@nyx-m6v running linux/amd64, linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
hormes    docker                        
  hormes  hormes                running linux/arm/v7, linux/arm/v6
nyx       docker                        
  nyx     nyx                   running linux/amd64, linux/386
default   docker                        
  default default               running linux/amd64, linux/386

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT       STATUS  PLATFORMS
farm      docker-container              
  farm0   hormes                running linux/arm/v7, linux/arm/v6
  farm1   ssh://kaszpir@nyx-m6v running linux/amd64, linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
hormes    docker                        
  hormes  hormes                running linux/arm/v7, linux/arm/v6
nyx       docker                        
  nyx     nyx                   Cannot connect to the Docker daemon at http://docker. Is the docker daemon running?: driver not connecting
default * docker                        
  default default               running linux/amd64, linux/386

Let's add node or let's try to alter its platform:

$ docker buildx create --name farm --node farm1 --platform linux/amd64 ssh://kaszpir@nyx-m6v
farm

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT       STATUS  PLATFORMS
farm      docker-container              
  farm0   hormes                running linux/arm/v7, linux/arm/v6
  farm1   ssh://kaszpir@nyx-m6v running linux/amd64, linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
hormes    docker                        
  hormes  hormes                Cannot connect to the Docker daemon at http://docker. Is the docker daemon running?: driver not connecting
nyx       docker                        
  nyx     nyx                   running linux/amd64, linux/386
default * docker                        
  default default               running linux/amd64, linux/386

$ docker buildx create --name farm --node farm1 --platform linux/amd64
farm

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT       STATUS  PLATFORMS
farm      docker-container              
  farm0   hormes                running linux/arm/v7, linux/arm/v6
  farm1   ssh://kaszpir@nyx-m6v running linux/amd64, linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
hormes    docker                        
  hormes  hormes                Cannot connect to the Docker daemon at http://docker. Is the docker daemon running?: driver not connecting
nyx       docker                 
  nyx     nyx                   Cannot connect to the Docker daemon at http://docker. Is the docker daemon running?: driver not connecting
default * docker                        
  default default               running linux/amd64, linux/386

Lets add it as context
(first remove it, list docker contexts, add new node with specific platform)

$ docker context ls
NAME                DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT                ORCHESTRATOR
default *           Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   https://127.0.0.1:6443 (default)   swarm
hormes              rpi direct                                ssh://pi@hormes                                                  
nyx                 nyx                                       ssh://kaszpir@nyx-m6v                                            

$ docker buildx create --name farm --node farm1 --leave
farm

$ docker buildx create --append --name farm --node farm1 --platform linux/amd64 nyx
farm

$ docker context ls
NAME                DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT                ORCHESTRATOR
default *           Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   https://127.0.0.1:6443 (default)   swarm
hormes              rpi direct                                ssh://pi@hormes                                                  
nyx                 nyx                                       ssh://kaszpir@nyx-m6v                                            
22:14:41 kaszpir@misiek ~/src/docker-multiarch  $ docker buildx ls
WARN[0007] commandConn.CloseWrite: commandconn: failed to wait: signal: killed 
WARN[0007] commandConn.CloseWrite: commandconn: failed to wait: signal: killed 
WARN[0007] commandConn.CloseWrite: commandconn: failed to wait: signal: killed 
WARN[0007] commandConn.CloseRead: commandconn: failed to wait: signal: killed 
WARN[0007] commandConn.CloseWrite: commandconn: failed to wait: signal: killed 
WARN[0017] commandConn.CloseWrite: commandconn: failed to wait: exec: Wait was already called 
NAME/NODE DRIVER/ENDPOINT  STATUS  PLATFORMS
farm      docker-container         
  farm0   hormes           running linux/arm/v7, linux/arm/v6
  farm1   nyx              running linux/amd64, linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
hormes    docker                   
  hormes  hormes           running linux/arm/v7, linux/arm/v6
nyx       docker                   
  nyx     nyx              Cannot connect to the Docker daemon at http://docker. Is the docker daemon running?: driver not connecting
default * docker                   
  default default          running linux/amd64, linux/386

Notice that the auto-added builders from contexts also have issues with the status command (hormes/hormes or nyx/nyx), while when they are added as contexts are displayed okay (farm/farm0 and farm/farm1).

@tonistiigi
Copy link
Member

Manual and detected platforms are merged together and have a preference priority. Showing duplicate values is a bug and should be fixed.

@nvtkaszpir
Copy link

nvtkaszpir commented Mar 25, 2020 •
edited
Loading

After some tests the platform preference indeed works, although the docker buildx ls output is a bit misleading, because it is not properly displayed (thus the duplicates). Moreover docker buildx ls shows the preference, while docker buildx inspect does not:

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT  STATUS  PLATFORMS
farm *    docker-container         
  farm0   hormes           running linux/arm/v7, linux/arm/v6, linux/arm/v7, linux/arm/v7, linux/arm/v6
  farm1   nyx              running linux/amd64, linux/arm64, linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
hormes    docker                   
  hormes  hormes           running linux/arm/v7, linux/arm/v6
nyx       docker                   
  nyx     nyx              running linux/amd64, linux/386
default   docker                   
  default default          running linux/amd64, linux/386
$ docker buildx inspect
Name:   farm
Driver: docker-container

Nodes:
Name:      farm0
Endpoint:  hormes
Status:    running
Platforms: linux/arm/v7, linux/arm/v6

Name:      farm1
Endpoint:  nyx
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tonistiigi @nvtkaszpir @ohnotnow