Add s390x support to docker images #1749

kun-lu20 · 2021-09-23T14:59:14Z

Description

The aim of this PR is to add s390x support to docker images executor, executor(slim), executor(debug) and warmer. Related PR: #1684.

This PR also addresses the building issue of dependency docker-credential-gcr in Kaniko's Dockerfiles. Project docker-credential-gcr removed its Makefile in recent commits, so the make deps command used in kaniko's Dockerfile fails. This PR uses command go get -u -t ./... instead which will achieve the same goal. Related PR: #1741 and #1744.

Signed-off-by: Kun-Lu kun.lu@ibm.com

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes unit tests
Adds integration tests if needed.

This PR could be verified locally using docker buildx and qemu, or via CI workflow in GitHub Actions.

See the contribution guide for more details.

Reviewer Notes

The code flow looks good.
Unit tests and or integration tests added.

Release Notes

Describe any changes here so maintainer can include it in the release notes, or delete this block.

- kaniko adds s390x support to docker images

1. add s390x support to docker images `executor`, `executor(slim)`, `executor(debug)` and `warmer`. Fixes GoogleContainerTools#1462 and GoogleContainerTools#1665. 2. Address the building issue of dependency `docker-credential-gcr` in Dockerfiles. This issue was introduced when recent commits in `docker-credential-gcr` removed the Makefile. Signed-off-by: Kun-Lu <kun.lu@ibm.com>

kun-lu20 · 2021-10-01T14:03:48Z

Hi @tejal29 , could you please take a look at this PR? Thank you!

tejal29 · 2021-10-18T18:46:42Z

@kun-lu20 Sorry for the delay. Can you please rebase?
Also can you add some testing notes e.g. the image produced is correct platform and works?

Signed-off-by: Kun-Lu <kun.lu@ibm.com>

kun-lu20 · 2021-10-18T22:42:03Z

@tejal29 Sure, I've merged the code changes from master branch. Thanks very much for reviewing this PR.

Below is the output of docker manifest inspect command on the built multi-arch image, it indicates that the platforms are correct.

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:764adac47a34da7bd82cec0bb2f2bdc26a4994b1bd42630939260a2813b22a7d",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:54567721e174f9935e007536d2c448fbe77eaa07e24c2721a362d6a25b7b94d2",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:d2f789f116cb8f6936620ee37e34959505b214591974319764dfd2d3bfd1f597",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:b68a19acea6c083f51ecfb99869d9e43f524869f85c2f8cf8c73773730828020",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

kun-lu20 · 2021-10-19T03:31:40Z

The built image was also verified on s390x system:

$ docker run --rm -it kaniko-project/executor:v1.6.0 version
Kaniko version :  v1.6.0

$ echo -e 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | docker run \
  --interactive -v $(pwd):/workspace kaniko-project/executor:v1.6.0 \
  --context tar://stdin \
  --no-push
INFO[0000] To simulate EOF and exit, press 'Ctrl+D'     
INFO[0000] Retrieving image manifest alpine             
INFO[0000] Retrieving image alpine from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest alpine             
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Unpacking rootfs as cmd RUN echo "created from standard input" requires it. 
INFO[0002] RUN echo "created from standard input"       
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] cmd: /bin/sh                                 
INFO[0002] args: [-c echo "created from standard input"] 
INFO[0002] Running: [/bin/sh -c echo "created from standard input"] 
created from standard input
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0002] Skipping push to container registry due to --no-push flag

tejal29 · 2021-10-19T06:01:06Z

Awesome! thanks.

tejal29 · 2021-10-19T06:42:13Z

if you are still around, please rebase again on latest master or else, I have a PR with your commits here #1769

tejal29 · 2021-10-19T07:02:22Z

closing this, since I chery-picked your commit in #1769 on master with tests enabled to hit the target release for Oct 20th

kun-lu20 · 2021-10-19T14:09:34Z

@tejal29 Thank you so much! Sorry I was not around due to timezone difference.

tejal29 · 2021-10-19T20:39:50Z

@kun-lu20 I am having trouble releasing with linux/s390x.
The issue is with building a docker-credentials-gcr binary of this platform and linux/ppc64le.

What do you think about just adding these platforms for kaniko:slim images?

tejal29 · 2021-10-19T20:41:41Z

For error logs, See https://github.com/GoogleContainerTools/kaniko/runs/3943786520?check_suite_focus=true

kun-lu20 · 2021-10-19T20:51:16Z

Hi @tejal29 , thanks, I observed these issues as well.

For building a docker-credentials-gcr binary, I found that using the following lines in Dockerfiles works on all platforms:

go get -u -t ./...                                                         && \
go build -ldflags "-linkmode external -extldflags -static" -i -o /usr/local/bin/docker-credential-gcr main.go

kun-lu20 · 2021-10-19T20:56:36Z

@tejal29 Could you please give it a try? We'd be very glad to see all the kaniko images work on s390x. Thanks!

diff --git a/deploy/Dockerfile b/deploy/Dockerfile
index 331b1062..cddd2fd9 100644
--- a/deploy/Dockerfile
+++ b/deploy/Dockerfile
@@ -32,8 +32,8 @@ RUN GOARCH=$(cat /goarch) && CGO_ENABLED=0 && \
   cd /go/src/github.com/GoogleCloudPlatform                                  && \
   git clone https://github.com/GoogleCloudPlatform/docker-credential-gcr.git && \
   cd /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr            && \
-  git checkout 4cdd60d0f2d8a69bc70933f4d7718f9c4e956ff8                      && \
-  go build -o /usr/local/bin/docker-credential-gcr main.go
+  go get -u -t ./...                                                         && \
+  go build -ldflags "-linkmode external -extldflags -static" -i -o /usr/local/bin/docker-credential-gcr main.go
 
 # Get Amazon ECR credential helper
 RUN GOARCH=$(cat /goarch) && go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login && \

tejal29 · 2021-10-20T00:12:29Z

Trying now.

tejal29 · 2021-10-20T00:19:10Z

Here is the build for your suggestion b5257e8

tejal29 · 2021-10-20T00:29:23Z

It still now working

> [linux/ppc64le stage-0  7/12] RUN GOARCH=$(cat /goarch) && CGO_ENABLED=0 &&   (mkdir -p /go/src/github.com/GoogleCloudPlatform || true)                  &&   cd /go/src/github.com/GoogleCloudPlatform                                  &&   git clone https://github.com/GoogleCloudPlatform/docker-credential-gcr.git &&   cd /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr            &&   git checkout 4cdd60d0f2d8a69bc70933f4d7718f9c4e956ff8                      &&   go get -u -t ./...                                                         &&   go build -ldflags "-linkmode external -extldflags -static" -i -o /usr/local/bin/docker-credential-gcr main.go:
#55 8.670 changes and commit them, and you can discard any commits you make in this
#55 8.670 state without impacting any branches by performing another checkout.
#55 8.670 
#55 8.670 If you want to create a new branch to retain commits you create, you may
#55 8.670 do so (now or later) by using -b with the checkout command again. Example:
#55 8.670 
#55 8.670   git checkout -b <new-branch-name>
#55 8.670 
#55 8.684 HEAD is now at 4cdd60d Merge pull request #102 from rafibarash/automate-release
#55 25.17 go: github.com/docker/cli@v20.10.8+incompatible: Get "https://proxy.golang.org/github.com/docker/cli/@v/v20.10.8+incompatible.mod": net/http: TLS handshake timeout
------
Dockerfile_debug:33
--------------------
  32 |     # Get GCR credential helper
  33 | >>> RUN GOARCH=$(cat /goarch) && CGO_ENABLED=0 && \
  34 | >>>   (mkdir -p /go/src/github.com/GoogleCloudPlatform || true)                  && \
  35 | >>>   cd /go/src/github.com/GoogleCloudPlatform                                  && \
  36 | >>>   git clone https://github.com/GoogleCloudPlatform/docker-credential-gcr.git && \
  37 | >>>   cd /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr            && \
  38 | >>>   git checkout 4cdd60d0f2d8a69bc70933f4d7718f9c4e956ff8                      && \
  39 | >>>   go get -u -t ./...                                                         && \
  40 | >>>   go build -ldflags "-linkmode external -extldflags -static" -i -o /usr/local/bin/docker-credential-gcr main.go
  41 |     
--------------------
Error: failed to solve: process "/bin/sh -c GOARCH=$(cat /goarch) && CGO_ENABLED=0 &&   (mkdir -p /go/src/github.com/GoogleCloudPlatform || true)                  &&   cd /go/src/github.com/GoogleCloudPlatform                                  &&   git clone https://github.com/GoogleCloudPlatform/docker-credential-gcr.git &&   cd /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr            &&   git checkout 4cdd60d0f2d8a69bc70933f4d7718f9c4e956ff8                      &&   go get -u -t ./...                                                         &&   go build -ldflags \"-linkmode external -extldflags -static\" -i -o /usr/local/bin/docker-credential-gcr main.go" did not complete successfully: exit code: 1
Error: buildx failed with: error: failed to solve: process "/bin/sh -c GOARCH=$(cat /goarch) && CGO_ENABLED=0 &&   (mkdir -p /go/src/github.com/GoogleCloudPlatform || true)                  &&   cd /go/src/github.com/GoogleCloudPlatform                                  &&   git clone https://github.com/GoogleCloudPlatform/docker-credential-gcr.git &&   cd /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr            &&   git checkout 4cdd60d0f2d8a69bc70933f4d7718f9c4e956ff8                      &&   go get -u -t ./...                                                         &&   go build -ldflags \"-linkmode external -extldflags -static\" -i -o /usr/local/bin/docker-credential-gcr main.go" did not complete successfully: exit code: 1

kun-lu20 · 2021-10-20T03:51:56Z

@tejal29 Thank you very much! It looks like at least building kaniko-project/executor image works on s390x. Could we add s390x support to both executor and executor:slim images on this release?

I'll further look into this issue and try to fix it.

Signed-off-by: Kun-Lu <kun.lu@ibm.com>

kun-lu20 · 2021-10-20T22:59:02Z

Hi @tejal29 , I've added s390x support to executor and warmer images. The image building CI workflow worked fine. I've also verified the built image on s390x system. s390x support is not added to executor:debug image since dep busybox does not work properly on it at this moment.

Please take a look when you have some time. Thank you very much!

deploy/Dockerfile_debug

imjasonh · 2022-01-04T19:15:34Z

The executor-debug image based on busybox seems to have failed building due to:

#66 [stage-2 11/12] RUN ["/busybox/mkdir", "-p", "/bin"]
#66 0.090 qemu-s390x: Could not open '/lib/ld64.so.1': No such file or directory
#66 ERROR: process "/busybox/mkdir -p /bin" did not complete successfully: exit code: 255
------
 > [stage-2 11/12] RUN ["/busybox/mkdir", "-p", "/bin"]:
#66 0.090 qemu-s390x: Could not open '/lib/ld64.so.1': No such file or directory

(https://github.com/GoogleContainerTools/kaniko/runs/4703060523?check_suite_focus=true#step:7:2095)

Do you happen to know why this would happen? I'm not familiar enough with busybox on s390x to understand why this might fail.

kun-lu20 · 2022-01-04T19:26:18Z

Hi @imjasonh , I haven't encountered this issue before. I am looking into it to try to find a workaround.

kun-lu20 · 2022-01-04T21:32:06Z

Hi @imjasonh , I made some tests on s390x VM. It looks like busybox needs some lib files which lie in /lib directory to run the executables on s390x.

A COPY command is added to Dockerfile_debug to address this issue. This extra copy action will not happen on amd64 or arm64 platforms since /lib does not exist in amd64 or arm64 version of busybox container.

imjasonh · 2022-01-05T14:17:32Z

Hi @imjasonh , I made some tests on s390x VM. It looks like busybox needs some lib files which lie in /lib directory to run the executables on s390x.

A COPY command is added to Dockerfile_debug to address this issue. This extra copy action will not happen on amd64 or arm64 platforms since /lib does not exist in amd64 or arm64 version of busybox container.

Thanks for looking into that!

I'm a bit worried that, since we don't have automated tests for s390x, we might unintentionally break this. I guess just building the image was sufficient to discover the bug... 🤔

At the very least, can you add a comment in the Dockerfile describing why we copy /lib, and why this is only needed for s390x, with some link to more information?

kun-lu20 · 2022-01-05T16:05:41Z

Yes, I think so. I will keep an eye on it as well and we'll be very happy to solve any issues reg s390x.

I've added a comment in the Dockerfile, could you please take a look? Thanks @imjasonh

kun-lu20 · 2022-01-06T19:18:17Z

Thanks @imjasonh !

google-cla bot added the cla: yes CLA signed by all commit authors label Sep 23, 2021

Merge branch 'master' into add_s390x_docker_images

bb6b683

Signed-off-by: Kun-Lu <kun.lu@ibm.com>

tejal29 approved these changes Oct 19, 2021

View reviewed changes

tejal29 mentioned this pull request Oct 19, 2021

added linux/s390x under platforms, for people that work with IBM Linu… #1684

Closed

tejal29 closed this Oct 19, 2021

tejal29 reopened this Oct 19, 2021

kun-lu20 added 2 commits October 20, 2021 16:56

Merge branch 'master' into add_s390x_docker_images

c42996e

Signed-off-by: Kun-Lu <kun.lu@ibm.com>

Add s390x support to all images except debug image

b1046b9

tejal29 reviewed Oct 21, 2021

View reviewed changes

deploy/Dockerfile_debug Outdated Show resolved Hide resolved

imjasonh added the needs-rebase label Dec 28, 2021

kun-lu20 added 2 commits January 4, 2022 09:16

Merge branch 'master' into add_s390x_docker_images

c246f70

Add s390x support to images.yaml

fa581ae

kun-lu20 requested review from imjasonh and tejal29 January 4, 2022 14:23

Fix ld64.so.1 not found issue on s390x

524a281

Add a comment for copying /lib from busybox container on s390x

ed6e2e5

imjasonh approved these changes Jan 6, 2022

View reviewed changes

imjasonh merged commit ccaa38d into GoogleContainerTools:master Jan 6, 2022

kun-lu20 mentioned this pull request Feb 11, 2022

Add Jenkins scripts for running system tests on s390x nodes strimzi/strimzi-kafka-operator#6354

Merged

8 tasks

imjasonh mentioned this pull request Jun 28, 2022

fix(Dockerfile): use temporary busybox mount to create /kaniko directory #2155

Merged

4 tasks

aaron-prindle mentioned this pull request May 30, 2023

Image for s390x #1665

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add s390x support to docker images #1749

Add s390x support to docker images #1749

kun-lu20 commented Sep 23, 2021

kun-lu20 commented Oct 1, 2021

tejal29 commented Oct 18, 2021 •

edited

Loading

kun-lu20 commented Oct 18, 2021 •

edited

Loading

kun-lu20 commented Oct 19, 2021

tejal29 commented Oct 19, 2021

tejal29 commented Oct 19, 2021

tejal29 commented Oct 19, 2021 •

edited

Loading

kun-lu20 commented Oct 19, 2021

tejal29 commented Oct 19, 2021

tejal29 commented Oct 19, 2021

kun-lu20 commented Oct 19, 2021

kun-lu20 commented Oct 19, 2021 •

edited

Loading

tejal29 commented Oct 20, 2021

tejal29 commented Oct 20, 2021

tejal29 commented Oct 20, 2021

kun-lu20 commented Oct 20, 2021 •

edited

Loading

kun-lu20 commented Oct 20, 2021

imjasonh commented Jan 4, 2022

kun-lu20 commented Jan 4, 2022

kun-lu20 commented Jan 4, 2022

imjasonh commented Jan 5, 2022

kun-lu20 commented Jan 5, 2022

kun-lu20 commented Jan 6, 2022

Add s390x support to docker images #1749

Add s390x support to docker images #1749

Conversation

kun-lu20 commented Sep 23, 2021

kun-lu20 commented Oct 1, 2021

tejal29 commented Oct 18, 2021 • edited Loading

kun-lu20 commented Oct 18, 2021 • edited Loading

kun-lu20 commented Oct 19, 2021

tejal29 commented Oct 19, 2021

tejal29 commented Oct 19, 2021

tejal29 commented Oct 19, 2021 • edited Loading

kun-lu20 commented Oct 19, 2021

tejal29 commented Oct 19, 2021

tejal29 commented Oct 19, 2021

kun-lu20 commented Oct 19, 2021

kun-lu20 commented Oct 19, 2021 • edited Loading

tejal29 commented Oct 20, 2021

tejal29 commented Oct 20, 2021

tejal29 commented Oct 20, 2021

kun-lu20 commented Oct 20, 2021 • edited Loading

kun-lu20 commented Oct 20, 2021

imjasonh commented Jan 4, 2022

kun-lu20 commented Jan 4, 2022

kun-lu20 commented Jan 4, 2022

imjasonh commented Jan 5, 2022

kun-lu20 commented Jan 5, 2022

kun-lu20 commented Jan 6, 2022

tejal29 commented Oct 18, 2021 •

edited

Loading

kun-lu20 commented Oct 18, 2021 •

edited

Loading

tejal29 commented Oct 19, 2021 •

edited

Loading

kun-lu20 commented Oct 19, 2021 •

edited

Loading

kun-lu20 commented Oct 20, 2021 •

edited

Loading