-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-31778][K8S][BUILD] Support cross-building docker images #28615
Conversation
Kubernetes integration test starting |
Kubernetes integration test status success |
Kubernetes integration test starting |
Kubernetes integration test status success |
Test build #123013 has finished for PR 28615 at commit
|
Test build #123014 has finished for PR 28615 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't work on Docker Desktop
. I tested on the latest one, Docker Desktop Edge 2020-05-20 2.3.1.0
. In this case, we had better describe the limitation of this new feature. Could you describe the used and expected environment in your PR description?
$ bin/docker-image-tool.sh -r docker.io/myrepo -t v3.0.0 -X -b java_image_tag=11-jre-slim build
...
+ docker buildx build --platform linux/amd64,linux/arm64 --build-arg java_image_tag=11-jre-slim -t docker.io/myrepo/spark:v3.0.0 -f kubernetes/dockerfiles/spark/Dockerfile .
multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
It does work on docker desktop but you need to be setup to cross build (currently experimental). I'll add a link in the script for those instructions. |
…bly don't have it enabled already
Kubernetes integration test starting |
Kubernetes integration test status success |
Test build #123068 has finished for PR 28615 at commit
|
Thank you for updating. Given the new URL at the last commit, it seems that this feature is pretty immature. Can we wait until it become stable? BTW, I've also been in
|
Cross building is relatively new, but a lot of docker images are cross built already. It’s optional and disabled by default (eg to have this build you’d have to explicitly configure it). For example the JDK11 base image is already cross built on amd64 and aarm64. You need to follow the buildx setup instructions not just turn on the expiremental channel. |
Okay. I removed my previous change request. |
### What changes were proposed in this pull request? Add cross build support to our docker image script using the new dockerx extension. ### Why are the changes needed? We have a CI for Spark on ARM, we should support building images for ARM and AMD64. ### Does this PR introduce _any_ user-facing change? Yes, a new flag is added to the docker image build script to cross-build ### How was this patch tested? Manually ran build script & pushed to https://hub.docker.com/repository/registry-1.docker.io/holdenk/spark/tags?page=1 verified amd64 & arm64 listed. Closes #28615 from holdenk/cross-build. Lead-authored-by: Holden Karau <hkarau@apple.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca> Signed-off-by: Holden Karau <hkarau@apple.com> (cherry picked from commit 2570228) Signed-off-by: Holden Karau <hkarau@apple.com>
What changes were proposed in this pull request?
Add cross build support to our docker image script using the new dockerx extension.
Why are the changes needed?
We have a CI for Spark on ARM, we should support building images for ARM and AMD64.
Does this PR introduce any user-facing change?
Yes, a new flag is added to the docker image build script to cross-build
How was this patch tested?
Manually ran build script & pushed to https://hub.docker.com/repository/registry-1.docker.io/holdenk/spark/tags?page=1 verified amd64 & arm64 listed.