-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Registry failed on storage health check - s3aws.Stat("/") #4361
Comments
I would really appreciate if you formatted your logs using markdown code formatting. It's really hard to parse the context from your message. |
Hi @milosgajdos sorry about that, I have updated the log block. Thanks |
@milosgajdos, I just did a small test to prove the container can access s3 bucket run the following sdk to access s3 in container,
result:
Container can access the s3 bucket. does this mean the issue is from s3aws storage driver ? |
I have a feeling this is related to #3275. Note that |
Description
We had outage for existing docker registry server (2.8.3), now We just built a new docker registry:2.8.3 with same settings in EC2, however, we hit the following issues.
When container starts, passed ALB health check then failed on
s3aws.Stat("/")
The log when container starts, pass the ELB health check, check path registry:5000/v2/
Registry starts failed, ELB health check 503
If we set env variable
REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: "false"
The ELB health check is passed, but it shows this error
$ curl -X GET https://registry.sanbox.internal.com:5000/v2/images/node/tags/list {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"DriverName":"s3aws","Enclosed":{}}}]}
Reproduce
1: start container without "REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: "false", registry service failed on s3aws.Stat("/")
2: start container with "REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: "false", registry service return
{"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"DriverName":"s3aws","Enclosed":{}}}]}
Expected behavior
registry:2.8.3 should be run as normal.
registry version
registry:2.8.3
Additional Info
The docker compose file:
Searched a bit, it might be caused by s3 permission, but the execution role has full s3 access.
We have to use 2.8.3 for now to support many old images
The text was updated successfully, but these errors were encountered: