cri-o cannot pull from internal registry with default image-puller role

[csrwng]

On an OpenShift cluster running on cri-o, trying to run an pod with an image from the internal registry, I get the following event:

Failed to pull image "docker-registry.default.svc:5000/test-docker/origin-ruby-sample@sha256:c1fb4160ed2b43b9b8c6d93a5908ca22ccac8442a0876e1103da9493daa1e73d": rpc error: code = 2 desc = Error reading signatures: received unexpected HTTP status: 500 Internal Server Error

In the registry log, I see this:

time="2017-09-06T00:34:20.806312657Z" level=error msg="(*signatureHandler): unknown: unknown error" go.version=go1.7.6 http.request.host="docker-registry.default.svc:5000"
http.request.id=3739e0d4-252c-452a-9027-26ac2d344014 http.request.method=GET http.request.remoteaddr="10.129.0.1:47694" 
http.request.uri="/extensions/v2/test-docker/origin-ruby-sample/signatures/sha256:c1fb4160ed2b43b9b8c6d93a5908ca22ccac8442a0876e1103da9493daa1e73d" 
http.request.useragent="Go-http-client/1.1" instance.id=b95c4973-6916-4f02-8ad9-6c22952d927c openshift.auth.user="system:serviceaccount:test-docker:default" 
openshift.logger=registry vars.digest="sha256:c1fb4160ed2b43b9b8c6d93a5908ca22ccac8442a0876e1103da9493daa1e73d" 
vars.name="test-docker/origin-ruby-sample”

The default service account already has the /system:image-puller role. If I assign cluster-admin role to the same service account, then the pull succeeds.

It seems that the image-puller role doesn't have sufficient permissions for cri-o to pull the image.

[csrwng]

@mfojtik @openshift/devex @runcom

[bparees]

and/or cri-o is trying to do things it shouldn't have to do? (if docker doesn't need additional permissions, why does cri-o?)

[csrwng]

It looks like docker doesn't query signatures directly or querying for them in a different way

[runcom]

It looks like docker doesn't query signatures directly or querying for them in a different way

and/or cri-o is trying to do things it shouldn't have to do? (if docker doesn't need additional permissions, why does cri-o?)

Nope, we use containers/image in both afaict and the call to "get signatures" is the same :(

[runcom]

is there any way to debug this in the live cluster? just remove cluster-admin and see what happens?

[runcom]

/cc @mtrmac

[runcom]

and/or cri-o is trying to do things it shouldn't have to do? (if docker doesn't need additional permissions, why does cri-o?)

alright, I figured this, docker runs with --signature-verification=false so it never hit the GetSignature method...that's why it doesn't fail with docker.

[giuseppe]

@csrwng what was your workaround? I've oadm policy add-role-to-user cluster-admin default but I am still seeing the same error

[csrwng]

@giuseppe

oadm policy add-cluster-role-to-user cluster-admin system:serviceaccount:thenamespace:default

(Substitute thenamespace with whatever namespace your test is in)

[csrwng]

Or, if you want to test @mfojtik's latest fix, you can simply change the image ref in dc/docker-registry to cewong/origin-docker-registry:v3.6.0

[giuseppe]

@csrwng lovely, it works!