Failure due to data exceeding the limit of 491994 byte #1257

erez-levi · 2024-11-13T10:46:32Z

Contributing guidelines

I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

The documentation does not mention anything about my problem
There are no open or closed issues that are related to my problem

Description

Starting to encounter this issue when running the action:

Error: buildx failed with: ERROR: failed to solve: Internal: rpc error: code = Internal desc = received 4294967293-bytes data exceeding the limit 491994 bytes

It started after I added the load: true parameter to my action but I didn't manage to reproduce it to know if that is indeed what triggered it, when it fails a rerun is always successful afterwards which isn't expected.

Expected behaviour

Build should run smoothly without this error

Actual behaviour

Getting this error in the beginning of the docker build:

Error: buildx failed with: ERROR: failed to solve: Internal: rpc error: code = Internal desc = received 4294967293-bytes data exceeding the limit 491994 bytes

When rerunning it the build is successful

Repository URL

No response

Workflow run URL

No response

YAML workflow

- name: Build image
    uses: docker/build-push-action@v6
    with:
      push: ${{ inputs.push_image }}
      context: ./${{ inputs.image_path }}
      load: ${{ inputs.load }}
      provenance: false
      tags: |
        ${{ inputs.docker_repo }}/${{ env.NAME }}:rc-${{ env.TAG }}
      build-args: |
        GIT_BRANCH=${{ inputs.branch }}
        GIT_REPO=${{ github.repository }}
        GIT_TAG=${{ env.TAG }}
        ${{ inputs.args }}
      target: ${{ inputs.target }}
      shm-size: 4g
      file: ./${{ inputs.image_path }}/${{ inputs.dockerfile_name }}
      secret-files: |
        "pip-conf=${{ env.PYPI_HOME }}/.pip/pip.conf"
        "netrc=${{ env.PYPI_HOME }}/.netrc"
      cache-from: ${{ env.CACHE_FROM }}
      cache-to: ${{ env.CACHE_TO }}

Workflow logs

2024-11-13T09:42:10.6274606Z 
2024-11-13T09:42:10.6275552Z #9 [ 1/17] FROM gcr.io/<repo>/cuda:cuda12.2.0-base-ubuntu22.04-py310@sha256:91f19aa3fcb87600d933a7b721ab74add6f2861f5a388e76917bdbf9532d033f
2024-11-13T09:42:10.6277190Z #9 resolve gcr.io/<repo>/cuda:cuda12.2.0-base-ubuntu22.04-py310@sha256:91f19aa3fcb87600d933a7b721ab74add6f2861f5a388e76917bdbf9532d033f done
2024-11-13T09:42:10.6278009Z #9 DONE 0.0s
2024-11-13T09:42:10.6278155Z 
2024-11-13T09:42:10.6278361Z #10 importing cache manifest from local:11625040072298951821
2024-11-13T09:42:10.6279312Z #10 inferred cache manifest type: application/vnd.oci.image.index.v1+json done
2024-11-13T09:42:10.7453819Z #10 DONE 0.1s
2024-11-13T09:42:10.7454639Z 
2024-11-13T09:42:10.7455017Z #8 [internal] load build context
2024-11-13T09:42:15.0605981Z #8 transferring context: 627.95MB 4.4s
2024-11-13T09:42:15.4751102Z #8 transferring context: 675.39MB 4.8s done
2024-11-13T09:42:15.4752605Z #8 ERROR: rpc error: code = Internal desc = received 4294967272-bytes data exceeding the limit 426398 bytes
2024-11-13T09:42:15.4753410Z ------
2024-11-13T09:42:15.4753815Z  > [internal] load build context:
2024-11-13T09:42:15.4754406Z ------
2024-11-13T09:42:15.4761878Z ERROR: failed to solve: Internal: rpc error: code = Internal desc = received 4294967272-bytes data exceeding the limit 426398 bytes
2024-11-13T09:42:15.4818235Z ##[group]Reference
2024-11-13T09:42:15.5800856Z builder-828661dc-b680-4e3d-84ab-8e499aa3a5ed/builder-828661dc-b680-4e3d-84ab-8e499aa3a5ed0/m9gqlgajeeqeq8odh49r0rf90
2024-11-13T09:42:15.5802806Z ##[endgroup]
2024-11-13T09:42:15.5803507Z ##[group]Check build summary support
2024-11-13T09:42:15.5805817Z Build summary supported!
2024-11-13T09:42:15.5807143Z ##[endgroup]
2024-11-13T09:42:15.5836255Z ##[error]buildx failed with: ERROR: failed to solve: Internal: rpc error: code = Internal desc = received 4294967272-bytes data exceeding the limit 426398 bytes
2024-11-13T09:42:15.5947391Z ##[group]Run dev-drprasad/delete-tag-and-release@v0.2.1
2024-11-13T09:42:15.5947798Z with:
2024-11-13T09:42:15.5948029Z   delete_release: false
2024-11-13T09:42:15.5948314Z   tag_name: v0.76.0
2024-11-13T09:42:15.5948552Z env:
2024-11-13T09:42:15.5948771Z   NAME: <repo>
2024-11-13T09:42:15.5949033Z   TAG: v0.76.0
2024-11-13T09:42:15.5949286Z   PYPI_HOME: /home/runner
2024-11-13T09:42:15.5949938Z   CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/_work/<repo>/<repo>/<creds.json>
2024-11-13T09:42:15.5951300Z   GOOGLE_APPLICATION_CREDENTIALS: /home/runner/_work/<repo>/<repo>/<creds.json>
2024-11-13T09:42:15.5952278Z   GOOGLE_GHA_CREDS_PATH: /home/runner/_work/<repo>/<repo>/<creds.json>
2024-11-13T09:42:15.5952908Z   CLOUDSDK_CORE_PROJECT: <gcp-project>
2024-11-13T09:42:15.5953256Z   CLOUDSDK_PROJECT: <gcp-project>
2024-11-13T09:42:15.5953578Z   GCLOUD_PROJECT: <gcp-project>
2024-11-13T09:42:15.5953892Z   GCP_PROJECT: <gcp-project>
2024-11-13T09:42:15.5954220Z   GOOGLE_CLOUD_PROJECT: l<gcp-project>
2024-11-13T09:42:15.5954698Z   CACHE_FROM: type=local,src=/var/lib/docker/<repo>/.buildx-cache
2024-11-13T09:42:15.5955371Z   CACHE_TO: type=local,dest=/var/lib/docker/<repo>/.buildx-cache-new,mode=max
2024-11-13T09:42:15.5956025Z   GITHUB_TOKEN: ***
2024-11-13T09:42:15.5956288Z ##[endgroup]

BuildKit logs

Added them as attachments

Additional info

buildkit-4.txt
buildkit-3.txt
buildkit-5.txt
buildkit-2txt.txt
buildkit-1.txt

The text was updated successfully, but these errors were encountered:

crazy-max · 2024-11-13T11:11:34Z

@erez-levi Thanks! Do you have a summary generated for this build?: https://docs.docker.com/build/ci/github-actions/build-summary/

If so can you attach the .dockerbuild record like this one:

erez-levi · 2024-11-13T11:27:30Z

@crazy-max Hi, I attached all of the files from this as attachments

crazy-max · 2024-11-13T11:29:16Z

@crazy-max Hi, I attached all of the files from this as attachments

Ok so no .dockerbuild file from summary iiuc?

erez-levi · 2024-11-13T12:00:13Z

those files are exported from the .dockerbuild after I went over them to make sure I am not exposing anything private from my org, so those are the contents of the .dockerbuild as you requested

crazy-max · 2024-11-13T12:00:20Z

@erez-levi For "Workflow logs" can you show the full logs of name: Build image step please? Want to check builder information like: https://github.com/docker/build-push-action/actions/runs/11814963949/job/32915142064#step:6:121

And

As well.

erez-levi · 2024-11-13T12:07:30Z

Builder info

  {
    "nodes": [
      {
        "name": "builder-828661dc-b680-4e3d-84ab-8e499aa3a5ed0",
        "endpoint": "unix:///var/run/docker.sock",
        "status": "running",
        "buildkitd-flags": "--debug --allow-insecure-entitlement=network.host",
        "buildkit": "v0.17.1",
        "platforms": "linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/amd64/v4,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6",
        "features": {
          "Automatically load images to the Docker Engine image store": true,
          "Cache export": true,
          "Docker exporter": true,
          "Multi-platform build": true,
          "OCI exporter": true
        },
        "labels": {
          "org.mobyproject.buildkit.worker.executor": "oci",
          "org.mobyproject.buildkit.worker.hostname": "c9737dba4b68",
          "org.mobyproject.buildkit.worker.network": "host",
          "org.mobyproject.buildkit.worker.oci.process-mode": "sandbox",
          "org.mobyproject.buildkit.worker.selinux.enabled": "false",
          "org.mobyproject.buildkit.worker.snapshotter": "overlayfs"
        },
        "gcPolicy": [
          {
            "all": false,
            "filter": [
              "type==source.local",
              "type==exec.cachemount",
              "type==source.git.checkout"
            ],
            "keepDuration": "48h0m0s"
          },
          {
            "all": false,
            "keepDuration": "1440h0m0s",
            "keepBytes": "9.313GiB"
          },
          {
            "all": false,
            "keepBytes": "9.313GiB"
          },
          {
            "all": true,
            "keepBytes": "9.313GiB"
          }
        ]
      }
    ],
    "name": "builder-828661dc-b680-4e3d-84ab-8e499aa3a5ed",
    "driver": "docker-container",
    "lastActivity": "2024-11-13T09:42:01.000Z"
  }

buildx-version:

  /usr/bin/docker buildx version
  github.com/docker/buildx v0.16.2 99dea6dacacc3d604788953088560b9880550570

docker info:

 /usr/bin/docker version
  Client:
   Version:           27.1.1
   API version:       1.46
   Go version:        go1.21.12
   Git commit:        6312585
   Built:             Tue Jul 23 19:55:52 2024
   OS/Arch:           linux/amd64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          27.3.1
    API version:      1.47 (minimum version 1.24)
    Go version:       go1.22.7
    Git commit:       41ca978
    Built:            Fri Sep 20 11:41:02 2024
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          v1.7.22
    GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
   runc:
    Version:          1.1.14
    GitCommit:        v1.1.14-0-g2c9f560
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/bin/docker info
  Client:
   Version:    27.1.1
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.16.2
      Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  
  Server:
   Containers: 1
    Running: 1
    Paused: 0
    Stopped: 0
   Images: 2
   Server Version: 27.3.1
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: true
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
   runc version: v1.1.14-0-g2c9f560
   init version: de40ad0
   Security Options:
    apparmor
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.1.100+
   Operating System: Alpine Linux v3.20 (containerized)
   OSType: linux
   Architecture: x86_64
   CPUs: 8
   Total Memory: 31.35GiB
   Name: cache-jtzdl-runner-z46rc
   ID: 16b5115b-2001-4016-b67d-2cee2703ee86
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false
   Product License: Community Engine

lmk if I missed any info

crazy-max · 2024-11-13T12:14:25Z

buildx-version:

  /usr/bin/docker buildx version
  github.com/docker/buildx v0.16.2 99dea6dacacc3d604788953088560b9880550570

Looking at builder info, I guess you're using https://github.com/docker/setup-buildx-action in your workflow. Can you add version: latest to it so it downloads latest Buildx stable? (current v0.18.0):

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          version: latest

   Kernel Version: 6.1.100+
   Operating System: Alpine Linux v3.20 (containerized)

Looking at the os in docker info, I guess you're on a self-hosted runner?

Can you show the full YAML workflow please?

erez-levi · 2024-11-13T12:27:46Z

name: 'build_docker_push'
description: 'build a docker image from code and push it to registry'
inputs:

  image_path:
    description: 'path of the dockerfile in git'
    required: false
    default: '.'
  local_runner:
    description: 'use local runner'
    required: false
    default: 'false'
  image_name:
    description: 'Image name'
    required: false
    default: ''
  use_hash:
    description: 'use commit hash as the image tag'
    required: false
    default: 'false'
  use_release:
    description: 'use the tag of the latest release'
    required: false
    default: 'false'
  image_tag:
    description: 'Image tag'
    required: false
    default: ''
  buildx:
    description: 'if you want to setup the buildx'
    required: false
    default: 'true'
  bot_token:
    description: 'bot token'
    required: false
    default: ''
  pypi_reader:
    description: 'package creds'
    required: false
    default: ''
  target:
    description: 'target name'
    required: false
    default: ''
  load:
    description: 'if you want to build the action and load it without pushing'
    required: false
    default: 'true'
  branch:
    description: 'branch name'
    required: false
    default: 'master'
  args:
    description: 'args'
    required: false
    default: ''
  checkout:
    description: 'Do you want to checkout'
    required: false
    default: 'true'
  dockerfile_name:
    description: 'dockerfile name is different'
    required: false
    default: 'Dockerfile'
  ref_to_checkout:
    description: 'git ref to checkout, defaults to main'
    required: false
    default: ''
  checkout_submodule:
    description: 'use in case you want to also checkout submodules (true, false, recursive)'
    required: false
    default: 'false'
  push_image:
    description: 'whether to push the image to the registry (true or false, default is true)'
    required: false
    default: 'true'
  image_tag_latest:
    description: 'use latest tag in conjunction with the image_tag'
    required: false
    default: 'false'

runs:
  using: "composite"
  steps:

  - uses: 'actions/checkout@v3'
    if: ${{ inputs.checkout == 'true' }}
    with:
      ref: ${{ inputs.ref_to_checkout }}
      fetch-depth: 0
      lfs: true
      submodules: ${{ inputs.checkout_submodule }}
      token: ${{ env.GITHUB_TOKEN }}

  - name: Set up QEMU
    uses: docker/setup-qemu-action@v2

  - name: Set up Docker Buildx
    continue-on-error: true
    uses: docker/setup-buildx-action@v3.0.0

  - name: Set up docker cache
    if: ${{ inputs.local_runner == 'true' }}
    shell: bash
    run: |
      sudo mkdir -p /var/lib/docker/${{ env.NAME }}/.buildx-cache
      sudo mkdir -p /var/lib/docker/${{ env.NAME }}/.buildx-cache-new
      sudo chmod 777 /var/lib/docker/${{ env.NAME }}/.buildx-cache-new

  - id: 'auth'
    name: 'Authenticate to Google Cloud'
    uses: 'google-github-actions/auth@v2'
    with:
      token_format: 'access_token'
      workload_identity_provider: ...
      service_account: '...

  - name: Login to GCR
    uses: docker/login-action@v2
    with:
      registry: gcr.io
      username: oauth2accesstoken
      password: ${{ steps.auth.outputs.access_token }}

  - name: Set cache settings
    shell: bash
    run: |
      if [[ "${{ inputs.local_runner }}" == "true" ]]; then
        echo "CACHE_FROM=type=local,src=/var/lib/docker/${{ env.NAME }}/.buildx-cache" >> $GITHUB_ENV
        echo "CACHE_TO=type=local,dest=/var/lib/docker/${{ env.NAME }}/.buildx-cache-new,mode=max" >> $GITHUB_ENV
      else
        echo "CACHE_FROM=type=gha" >> $GITHUB_ENV
        echo "CACHE_TO=type=gha,mode=max" >> $GITHUB_ENV
      fi

  - name: Build image
    uses: docker/build-push-action@v6
    with:
      push: ${{ inputs.push_image }}
      context: ./${{ inputs.image_path }}
      load: ${{ inputs.load }}
      provenance: false
      tags: |
        ${{ inputs.docker_repo }}/${{ env.NAME }}:rc-${{ env.TAG }}
      build-args: |
        GIT_BRANCH=${{ inputs.branch }}
        GIT_REPO=${{ github.repository }}
        GIT_TAG=${{ env.TAG }}
        ${{ inputs.args }}
      target: ${{ inputs.target }}
      shm-size: 4g
      file: ./${{ inputs.image_path }}/${{ inputs.dockerfile_name }}
      secret-files: |
        "pip-conf=${{ env.PYPI_HOME }}/.pip/pip.conf"
        "netrc=${{ env.PYPI_HOME }}/.netrc"
      cache-from: ${{ env.CACHE_FROM }}
      cache-to: ${{ env.CACHE_TO }}

  - name: 'Set up Cloud SDK'
    uses: 'google-github-actions/setup-gcloud@v2'
    with:
      version: '>= 363.0.0'

  # Temp fix
  # https://github.com/docker/build-push-action/issues/252
  # https://github.com/moby/buildkit/issues/1896
  - name: Move cache
    if: ${{ inputs.local_runner == 'true' }}
    shell: bash
    run: |
      sudo rm -rf /var/lib/docker/${{ env.NAME }}/.buildx-cache
      sudo mv /var/lib/docker/${{ env.NAME }}/.buildx-cache-new /var/lib/docker/${{ env.NAME }}/.buildx-cache 

  - name: error-handle
    if: ${{ failure() && steps.tag-exists.conclusion != 'failure' }}
    uses: dev-drprasad/delete-tag-and-release@v0.2.1
    with:
      delete_release: false
      tag_name: ${{ env.TAG }}

  - name: error-handle for latest tag
    if: ${{ inputs.image_tag_latest == 'true' }}
    uses: dev-drprasad/delete-tag-and-release@v0.2.1
    with:
      delete_release: false
      tag_name: latest

erez-levi · 2024-11-13T12:29:30Z

its a self hosted runner indeed, we use arc for that

tuxillo · 2024-11-14T16:37:30Z

Same here, we are using ARC and we're hitting this issue too.

tri-bao · 2024-11-14T17:52:32Z

Using hosted runners and got the same issue

crazy-max · 2024-11-14T18:07:09Z

@tri-bao Do you have a link to your repo please?

tri-bao · 2024-11-14T18:51:38Z

@crazy-max We can't share the link. It's a private repo of an org

erikburt · 2024-11-14T23:02:44Z

Error from our public repo: https://github.com/smartcontractkit/chainlink/actions/runs/11823026225/job/32941422115

ERROR: failed to solve: Internal: rpc error: code = Internal desc = received 4294967290-bytes data exceeding the limit 786732 bytes

crazy-max · 2024-11-19T15:11:47Z

This has been fixed in moby/buildkit#5530, will be available in next patch release of BuildKit.

crazy-max · 2024-11-22T11:22:02Z

Should be fixed with latest BuildKit stable (0.17.2)

tri-bao · 2024-11-22T15:02:56Z

Should be fixed with latest BuildKit stable (0.17.2)

Thanks @crazy-max To use this version on Github action, do we need to await for the new version of docker/setup-buildx-action action or we can use docker/setup-buildx-action@v3 and tell it to use this buildkit version?

crazy-max · 2024-11-22T15:06:59Z

Should be fixed with latest BuildKit stable (0.17.2)

Thanks @crazy-max To use this version on Github action, do we need to await for the new version of docker/setup-buildx-action action or we can use docker/setup-buildx-action@v3 and tell it to use this buildkit version?

Yes latest image has been updated so no changes required in the setup-buildx-action step.

You can look at Builder info group output to make sure of it like: https://github.com/docker/build-push-action/actions/runs/11975119937/job/33387774802#step:4:125

erez-levi added the status/triage label Nov 13, 2024

erez-levi mentioned this issue Nov 13, 2024

Data exceeding the limit of 491994 byte #1256

Closed

crazy-max added status/needs-investigation and removed status/triage labels Nov 13, 2024

crazy-max closed this as completed Nov 19, 2024

crazy-max removed the status/needs-investigation label Nov 19, 2024

crazy-max mentioned this issue Nov 20, 2024

Build cancellation when running loading to tar #1263

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failure due to data exceeding the limit of 491994 byte #1257

Failure due to data exceeding the limit of 491994 byte #1257

erez-levi commented Nov 13, 2024 •

edited by crazy-max

Loading

crazy-max commented Nov 13, 2024

erez-levi commented Nov 13, 2024

crazy-max commented Nov 13, 2024

erez-levi commented Nov 13, 2024

crazy-max commented Nov 13, 2024 •

edited

Loading

erez-levi commented Nov 13, 2024

crazy-max commented Nov 13, 2024

buildx-version:

erez-levi commented Nov 13, 2024 •

edited

Loading

erez-levi commented Nov 13, 2024

tuxillo commented Nov 14, 2024

tri-bao commented Nov 14, 2024

crazy-max commented Nov 14, 2024

tri-bao commented Nov 14, 2024

erikburt commented Nov 14, 2024

crazy-max commented Nov 19, 2024

crazy-max commented Nov 22, 2024

tri-bao commented Nov 22, 2024

crazy-max commented Nov 22, 2024 •

edited

Loading

Failure due to data exceeding the limit of 491994 byte #1257

Failure due to data exceeding the limit of 491994 byte #1257

Comments

erez-levi commented Nov 13, 2024 • edited by crazy-max Loading

Contributing guidelines

I've found a bug, and:

Description

Expected behaviour

Actual behaviour

Repository URL

Workflow run URL

YAML workflow

Workflow logs

BuildKit logs

Additional info

crazy-max commented Nov 13, 2024

erez-levi commented Nov 13, 2024

crazy-max commented Nov 13, 2024

erez-levi commented Nov 13, 2024

crazy-max commented Nov 13, 2024 • edited Loading

erez-levi commented Nov 13, 2024

Builder info

buildx-version:

docker info:

crazy-max commented Nov 13, 2024

buildx-version:

erez-levi commented Nov 13, 2024 • edited Loading

erez-levi commented Nov 13, 2024

tuxillo commented Nov 14, 2024

tri-bao commented Nov 14, 2024

crazy-max commented Nov 14, 2024

tri-bao commented Nov 14, 2024

erikburt commented Nov 14, 2024

crazy-max commented Nov 19, 2024

crazy-max commented Nov 22, 2024

tri-bao commented Nov 22, 2024

crazy-max commented Nov 22, 2024 • edited Loading

erez-levi commented Nov 13, 2024 •

edited by crazy-max

Loading

crazy-max commented Nov 13, 2024 •

edited

Loading

erez-levi commented Nov 13, 2024 •

edited

Loading

crazy-max commented Nov 22, 2024 •

edited

Loading