GC fails with "invalid checksum digest format" from registry when deleting manifests #15970

dkulchinsky · 2021-11-08T14:12:09Z

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:

GC should delete all manifests & blobs marked for removal and if an error is encountered should skip the offending manifest/blob and continue with the rest, logging the issue, instead GC fails to delete some manifest because registry returns 500 and the following error message:

err.message="invalid checksum digest format"

and the whole GC job fails and stops.

Steps to reproduce the problem:
Don't know what causes it, so not sure how to reproduce.

Versions:
Please specify the versions of following systems.

harbor version: 2.3.3
docker engine version: N/A
docker-compose version: N/A

Additional context:

Registry logs

time="2021-11-08T00:41:24.515110036Z" level=info msg="authorized request" go.version=go1.15.12 http.request.host="harbor-registry:5000" http.request.id=c5e9fa5c-804e-46b8-a610-1c523bda67ad http.request.method=DELETE http.request.remoteaddr="127.0.0.1:50242" http.request.uri="/v2/<project>/<repo>/manifests/sha256:875c0867bae86b06e7c4e098eff8877663c13859eb760f36b451354c755e591d" http.request.useragent=harbor-registry-client vars.name="<project>/<repo>" vars.reference="sha256:875c0867bae86b06e7c4e098eff8877663c13859eb760f36b451354c755e591d" 

time="2021-11-08T00:43:20.098404428Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.message="invalid checksum digest format" go.version=go1.15.12 http.request.host="harbor-registry:5000" http.request.id=c5e9fa5c-804e-46b8-a610-1c523bda67ad http.request.method=DELETE http.request.remoteaddr="127.0.0.1:50242" http.request.uri="/v2/<project>/<repo>/manifests/sha256:875c0867bae86b06e7c4e098eff8877663c13859eb760f36b451354c755e591d" http.request.useragent=harbor-registry-client http.response.contenttype="application/json; charset=utf-8" http.response.duration=1m55.650187865s http.response.status=500 http.response.written=70 vars.name="<project>/<repo>" vars.reference="sha256:875c0867bae86b06e7c4e098eff8877663c13859eb760f36b451354c755e591d"

jobservice log:

2021-11-08T00:43:20Z [ERROR] [/jobservice/runner/redis.go:113]: Job 'GARBAGE_COLLECTION:4d498cc8dea08115a5d2a92d' exit with error: run error: failed to delete manifest with v2 API: <project>/<repo>, sha256:875c0867bae86b06e7c4e098eff8877663c13859eb760f36b451354c755e591d: retry timeout: http status code: 500, body: {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{}}]}

The text was updated successfully, but these errors were encountered:

dkulchinsky · 2021-11-11T02:24:47Z

@wy65701436 came across this issue (distribution/distribution#3018) and PR (distribution/distribution#3019) in the upstream Docker Distribution that seem to have identified the same issue that is causing these errors we're seeing.

AFAICT the PR was abandoned and the issue is still open with now resolution, GitLab seem to have forked distribution and started using their own version due to inability to push changes.

With distribution still (?) using Google SDK from 2015, it's really worrying and looks like it's contributing to a lot of the issues we're seeing, including this one?

wy65701436 · 2021-11-11T07:52:18Z

thanks @dkulchinsky , I'll update the Google SDK for upstream distribution. Given this, Harbor still cannot leverage it till we get an new distribution release.

dkulchinsky · 2021-11-11T14:35:12Z

Thanks @wy65701436!

can we consider using gitlab's fork of the distribution? seems like it's in a much better shape in terms of reliability, perfromance and overall maintenance, Docker's distribution last release was in January 2019 😱

Is there anything you can suggest for my situation? we already have over 10,000 artifacts waiting for GC (and the number grows daily) and the GC job keeps failing either due to this issue or the other GC issues I've reported (mostly #15822)

What can we do? getting really desperate with this 😞

wy65701436 · 2021-11-11T17:28:23Z

We have no plan to leverage other forked distribution. We're(distribution maintainers) working on issue new release for upstream distribution, but I cannot give an date.

For this issue, I'll raise the PR to fix in the upstream. Harbor needs to wait for distribution 3.0 release.
For other issues on GC, I'll go through them and maybe we can do some enhancement on Harbor side. But, so far, most of the problems are raised by the backend storage.

dkulchinsky · 2021-11-11T18:09:32Z

Thanks again for replying @wy65701436! I appreciate it 👍🏼

We have no plan to leverage other forked distribution. We're(distribution maintainers) working on issue new release for upstream distribution, but I cannot give an date.

For this issue, I'll raise the PR to fix in the upstream. Harbor needs to wait for distribution 3.0 release.

Looking forward to it 👍🏼

For other issues on GC, I'll go through them and maybe we can do some enhancement on Harbor side. But, so far, most of the problems are raised by the backend storage.

I think allowing GC to skip blobs/manifests that fail to be removed due to persistent errors such as 404 & 500 can help mitigate this issue considerably, at least allow us to GC the majority of artifacts.

perhaps this behaviour can be an optional configuration, so that it won't be considered as a breaking change and will be opt-in.

wy65701436 · 2021-11-12T04:00:04Z

yes, to allow skip failure could be an option. BTW, for the performance issue, we maybe will do some enhancement on distribution side.

dkulchinsky · 2021-11-12T13:51:50Z

yes, to allow skip failure could be an option.

That would be great @wy65701436 🤝 this will be greatly appreciated as it would really help our current situation, I hope this can be implemented sooner than later 🙏🏼

BTW, for the performance issue, we maybe will do some enhancement on distribution side.

👏🏼

stale · 2022-04-16T19:14:44Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

dkulchinsky · 2022-04-17T13:19:18Z

this is still being tracked I believe?

github-actions · 2022-07-05T09:08:56Z