-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integration flake: TestV2RegistryGetTags #16012
Comments
Or could be a @deads2k bug in startup logic where the roles aren't being fully created I guess. |
Hrm, happening here too: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/15624/test_pull_request_origin_integration/6614/#githubcomopenshiftorigintestintegrationrunner-testpullthroughinsecure Probably not registry code problem, probably core roles problem. |
I see I don't see anywhere in the logic waiting for access on something. |
AFAIK only cluster admins can get images... I saw several docker.io flakes today where we failed to fetch stuff from dockerhub, maybe this one is related. |
@smarterclayton @deads2k what was the final assessment here? What I gathered is:
if (2) is true, this test would presumably never pass, so I don't buy that unless the 403 isn't the root cause of the 500 error that's being returned. |
Since the server reports healthy (eventually), the bootstrap rules were created |
are you saying we're running tests before the server has reported healthy? that seems like a fundamental flaw in our integration test model if true. |
The StartTestMasterAPI helper waits for the server to report healthy before proceeding, so the server was healthy, thus the bootstrap roles were created (And since this is a flake, not a permfail, those roles must be correct at least some of the time). So this still sounds like a flaw in RBAC to me. https://github.com/openshift/origin/blob/master/test/integration/v2_docker_registry_test.go#L80 @deads2k am i missing something? |
@bparees do you know which calls its responding to? I would expect that in normal operation the docker registry would make SARs or attempt to make calls which would be forbidden. I know that it does for deciding about the provisioning of imagestreams as a for instance. Are you sure that the denials aren't happening in "normal" code paths from the docker registry. |
the call is this (get on apis/image.openshift.io/v1/images as an admin user. Looks like perhaps that user is only admin for the project, not the cluster, but again if the permissions are insufficient why wouldn't this always fail?):
|
@deads2k bump. |
@bparees have you managed to figure out where in the docker registry you're failing? What its trying to do? This could very easily be a case of, "I'm trying to access A which failed so I try to access B". It could also be that your test needs to wait until the project access rules have synchronized (which I noted that it isn't doing back here: #16012 (comment) |
I turned on the audit trail and ran successful run. The only audit log says that on succeses, only system:admin is requesting images.
Either you're hitting a different path on failures or you're accidentally changing users. You need to figure out where the call is coming from. |
Thank you @deads2k |
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/16778/test_pull_request_origin_integration/9210/ shows two failures related to TestV2RegistryGetTags |
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/15624/test_pull_request_origin_integration/6615/#githubcomopenshiftorigintestintegrationrunner-testv2registrygettags
Needs to wait longer
@mfojtik @bparees can someone priority this so it gets fixed soon? Just need to wait longer I think
The text was updated successfully, but these errors were encountered: