Unable to Push Large Docker Images to Harbor installed in an RKE2 Cluster

[aboualiaa]

Setup:

• RKE2 cluster managed with Rancher.
• ceph-csi as a CSI driver installed via Helm.
• Functioning Harbor installation (Version 2.9.0) also set up via Helm
• Persistent Volume Claims (PVCs) provided by ceph's csi-cephfs.

Objective:
I'm attempting to push large Docker images (e.g., freesurfer/freesurfer:7.2.0, which is almost 5 Gb) to Harbor. While pushing small images works seamlessly, I encounter difficulties with large images.

Attempts Made:

1. Increased the size of PVCs within the Kubernetes cluster (e.g. for the registry to 50 Gb, all other PVCs to 15 Gb)

2. Attempted pushing using both Docker and Podman; both retried several times before failing.

3. The only error message that could be useful I received from Skopeo:

   Error copying ref "docker://freesurfer/freesurfer:7.2.0": writing blob: Patch "https://my.local.harbor/v2/library/freesurfer/blobs/uploads/d19338fa-c32f-4cdc-8716-efdd84e0d19e?_state=3TqbKhk0jzmA5dmrhNWyb-aEdvTHOnBlSxWJOmTUEdp7Ik5hbWUiOiJsaWJyYXJ5L2ZyZWVzdXJmZXIiLCJVVUlEIjoiZDE5MzM4ZmEtYzMyZi00Y2RjLTg3MTYtZWZkZDg0ZTBkMTllIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTA5LTA3VDA4OjU0OjE4LjA0NTA4MTA2MloifQ%3D%3D": EOF
   

Any insights or guidance on this issue would be greatly appreciated, the Harbor installation is for my use case unusable because of this.

Am I missing something? Is there an image size limit? An upload timeout?

[stonezdj]

It maybe related to the network timeout, because I can upload the same image to Harbor installed with docker-compose.
see issue: #17908 or #15751. you should check the timeout setting in the LB/Ingress etc.

[aboualiaa]

Thank you! It was the timeouts from the load balancer (haproxy) in front of the kubernetes cluster. Closing as solved.