Slow process for updating gem libraries with ARM

[hayashier]

When we run the commands for updating gem libraries　(e.g. gem update or gem pristine),
The process for x86 is completed quickly(153.3s). On the other hand, it is extremely slower in arm (8606.2s) than x86 to execute the same process.

We would like to resolve slowness about arm. Does anyone have some ideas about that?

Background

When I try to use our image artifacts in our environment, it takes a few minutes to update gem libraries with x86. On the other hand, it takes more than 3 hours to do the same process with arm. We don't see this problem when we build our image artifacts on ARM node directly without tonistiigi/binfmt.
So we would like to resolve them.

Reproduction Steps

I can reproduce this easily in my development environment with the minimum step.

• Environment (Amazon Linux 2)

$ docker -v
Docker version 20.10.25, build b82b9f3

$ docker run --privileged --rm tonistiigi/binfmt --version
binfmt/a161c41 qemu/v7.0.0 go/1.18.5

$ uname -a
Linux xxx.xxx.xxx.xxx 5.10.209-175.812.amzn2int.x86_64 #1 SMP Tue Jan 30 21:29:45 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

1. Create Dockerfile like this.

FROM public.ecr.aws/amazonlinux/amazonlinux:2023

RUN dnf install -y ruby-devel gcc make libyaml-devel
RUN gem list --local | awk '{print $1}' | xargs gem update -V

3. Set up environment for multi-platform

# Installing emulators
$ docker run --privileged --rm tonistiigi/binfmt --install amd64,arm64
Unable to find image 'tonistiigi/binfmt:latest' locally
latest: Pulling from tonistiigi/binfmt
8d4d64c318a5: Pull complete 
e9c608ddc3cb: Pull complete 
Digest: sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55
Status: Downloaded newer image for tonistiigi/binfmt:latest
installing: amd64 cannot register "/usr/bin/qemu-x86_64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: no such file or directory
installing: arm64 OK
{
  "supported": [
    "linux/amd64",
    "linux/arm64",
    "linux/386"
  ],
  "emulators": [
    "kshcomp",
    "qemu-aarch64"
  ]
}

# Create builder instance for multi-platform
$ docker buildx create --use --name builder-multiarch
builder-multiarch

# Check the list of builder instances
$ docker buildx ls
NAME/NODE            DRIVER/ENDPOINT             STATUS   BUILDKIT     PLATFORMS
builder-multiarch *  docker-container                                  
  builder-multiarch0 unix:///var/run/docker.sock inactive              
default              docker                                            
  default            default                     running  v0.8+unknown linux/amd64, linux/386, linux/arm64

4. Build images with buildx.

• gem update

$ docker buildx build --no-cache --platform=linux/amd64,linux/arm64 -t qemu-gem-update ./
[+] Building 9004.4s (10/10) FINISHED                                                           docker-container:builder-multiarch
 => [internal] load build definition from Dockerfile                                                                          0.0s
 => => transferring dockerfile: 204B                                                                                          0.0s
 => [linux/arm64 internal] load metadata for public.ecr.aws/amazonlinux/amazonlinux:2023                                      3.0s
 => [linux/amd64 internal] load metadata for public.ecr.aws/amazonlinux/amazonlinux:2023                                      3.0s
 => [internal] load .dockerignore                                                                                             0.0s
 => => transferring context: 2B                                                                                               0.0s
 => [linux/arm64 1/3] FROM public.ecr.aws/amazonlinux/amazonlinux:2023@sha256:38701a173dc0dea352df1bb934c3269053cf4137a9325  20.9s
 => => resolve public.ecr.aws/amazonlinux/amazonlinux:2023@sha256:38701a173dc0dea352df1bb934c3269053cf4137a9325e68ebef971a3d  0.0s
 => => sha256:f534013dbea5ef16c757c5298f993b98988a6e0833221735408a89b0a475dd63 51.30MB / 51.30MB                              6.0s
 => => extracting sha256:f534013dbea5ef16c757c5298f993b98988a6e0833221735408a89b0a475dd63                                    14.9s
 => [linux/amd64 1/3] FROM public.ecr.aws/amazonlinux/amazonlinux:2023@sha256:38701a173dc0dea352df1bb934c3269053cf4137a9325  15.5s
 => => resolve public.ecr.aws/amazonlinux/amazonlinux:2023@sha256:38701a173dc0dea352df1bb934c3269053cf4137a9325e68ebef971a3d  0.0s
 => => sha256:8784573bb84d178812057375084b2df4e8a0ffb22734f522709063f9581c296f 52.21MB / 52.21MB                              5.4s
 => => extracting sha256:8784573bb84d178812057375084b2df4e8a0ffb22734f522709063f9581c296f                                    10.0s
 => [linux/amd64 2/3] RUN dnf install -y ruby-devel gcc make libyaml-devel                                                   53.2s
 => [linux/arm64 2/3] RUN dnf install -y ruby-devel gcc make libyaml-devel                                                  374.0s
 => [linux/amd64 3/3] RUN gem list --local | awk '{print $1}' | xargs gem update -V                                         153.3s
 => [linux/arm64 3/3] RUN gem list --local | awk '{print $1}' | xargs gem update -V                                        8606.2s
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 

The point is

• we can see RUN gem pristine --all -V for both of linux/amd64 and linux/arm64.
• For x86, it takes 153.3s. For arm, it takes 8606.2s.

We also see this problem with gem pristine as well.

FROM public.ecr.aws/amazonlinux/amazonlinux:2023

RUN dnf install -y ruby-devel gcc make libyaml-devel
RUN gem pristine --all -V

 => [linux/arm64 2/3] RUN dnf install -y ruby-devel gcc make libyaml-devel                                                    294.1s
 => [linux/amd64 2/3] RUN dnf install -y ruby-devel gcc make libyaml-devel                                                     12.9s
 => [linux/amd64 3/3] RUN gem pristine --all -V                                                                                23.8s 
 => [linux/arm64 3/3] RUN gem pristine --all -V                                                                              1839.2s 
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

We would like to resolve the slowness about arm.