Add retry import with no auth for IS if we get 401 error (public images case) #18012

mjudeikis · 2018-01-07T14:05:28Z

If we try to import any image (even public ones) and we have a bad secret (wrong password or expired cloud token) in our project imports are failing.

This leads to all imports in the same project to fail with a random unclear error.
This could be fixed in RoundTripper and Smarter Authenticator. This might end up bigger change than this fix...

This fix just retries import with empty authentication.

If you know better place for this fix let me know :)

https://bugzilla.redhat.com/show_bug.cgi?id=1506175

mjudeikis · 2018-01-10T20:37:47Z

@bparees this one is other card you sent me. Fix works, but to solve this properly would require changes in either docker API in upstream (docker distribution library) or RT in k8s code... Both of them I dont feel taking on as a challenge :D

bparees · 2018-01-10T20:45:35Z

pkg/image/registry/imagestreamimport/rest.go

+		if image.Status.Code == http.StatusUnauthorized {
+			importCtx := importer.NewContext(r.transport, r.insecureTransport).WithCredentials(nil)
+			imports := r.importFn(importCtx)
+			if err := imports.Import(ctx.(gocontext.Context), isi, stream); err != nil {


one problem I see with this is if the image really is private, does this turn what was a legitimate 401 into a 404 error when you make the import call with no credentials? (because docker returns 404 for private registries?)

That is, if the scenario is: i'm trying to import from a private registry and I provide invalid credentials, previously I would get a proper 401 and know my credentials were (probably) wrong. Is that what will still happen with this change, or will I now see a 404 error and be confused?

I did the bunch of tests and checking how this "awesome" docker registry API behaves and when we do this (spent way too much time on this...). This fix for me looked least intrusive and did the job.

I think your case is covered because it still gives 401 but it gives different error message so in importer.go it gets assigned different error message.

There is chance it might behave diferently if it will be used with non docker distribution based docker registry API's (Nexus in example) which will give different error messages. But in this case all our error handling logic is flawed.

The current behavior of our IS importer with private and public images with wrong secret:

-bash-4.4# oc create secret docker-registry dummy-secret1 --docker-server=docker.io --docker-username=dummy1 --docker-password=dummy1 --docker-email==dummy1@example.com secret "dummy-secret1" created ##### PUBLIC IMAGE -bash-4.4# oc import-image example --from=openshift/hello-openshift --confirm ! error: Import failed (InternalError): Internal error occurred: Get https://registry-1.docker.io/v2/openshift/hello-openshift/manifests/latest: unauthorized: incorrect username or password Less than a second ago error: tag latest failed: Internal error occurred: Get https://registry-1.docker.io/v2/openshift/hello-openshift/manifests/latest: unauthorized: incorrect username or password ##### PRIVATE IMAGE -bash-4.4# oc import-image example1 --from=mangirdas/kafka-producer:latest --confirm The import completed with errors. ! error: Import failed (InternalError): Internal error occurred: Get https://registry-1.docker.io/v2/mangirdas/kafka-producer/manifests/latest: unauthorized: incorrect username or password Less than a second ago error: tag latest failed: Internal error occurred: Get https://registry-1.docker.io/v2/mangirdas/kafka-producer/manifests/latest: unauthorized: incorrect username or password

With fix:

-bash-4.4# oc create secret docker-registry dummy-secret1 --docker-server=docker.io --docker-username=dummy1 --docker-password=dummy1 --docker-email==dummy1@example.com secret "dummy-secret1" created ##### PUBLIC IMAGE -bash-4.4# oc import-image example --from=openshift/hello-openshift --confirm The import completed successfully. latest tagged from openshift/hello-openshift * openshift/hello-openshift@sha256:1a7a40888035f95c10cde81f04c10aa5919a41766e4680452918fc5591d7b02a Less than a second ago ##### PRIVATE IMAGE -bash-4.4# oc import-image example1 --from=mangirdas/kafka-producer:latest --confirm The import completed with errors. latest tagged from mangirdas/kafka-producer:latest ! error: Import failed (Unauthorized): you may not have access to the Docker image "mangirdas/kafka-producer:latest" Less than a second ago error: tag latest failed: you may not have access to the Docker image "mangirdas/kafka-producer:latest"

Without secret:

##### PUBLIC IMAGE -bash-4.4# oc import-image example --from=openshift/hello-openshift --confirm The import completed successfully. latest tagged from openshift/hello-openshift * openshift/hello-openshift@sha256:1a7a40888035f95c10cde81f04c10aa5919a41766e4680452918fc5591d7b02a Less than a second ago

PRIVATE IMAGE

-bash-4.4# oc import-image example1 --from=mangirdas/kafka-producer:latest --confirm latest tagged from mangirdas/kafka-producer:latest ! error: Import failed (Unauthorized): you may not have access to the Docker image "mangirdas/kafka-producer:latest" Less than a second ago error: tag latest failed: you may not have access to the Docker image "mangirdas/kafka-producer:latest"

My secret is OK but the image is in the private repo of other person account:

oc create secret docker-registry dummy-secret1 --docker-server=docker.io --docker-username=mangirdas --docker-password=dummy1 --docker-email=mangirdas@example.com -bash-4.4# oc import-image example2 --from=juozasa/hellonode:latest --confirm latest tagged from juozasa/hellonode:latest ! error: Import failed (Unauthorized): you may not have access to the Docker image "juozasa/hellonode:latest" Less than a second ago error: tag latest failed: you may not have access to the Docker image "juozasa/hellonode:latest"

bparees · 2018-01-10T21:40:21Z

awesome, thank you for the detailed summary of the behavior!
/lgtm

mjudeikis · 2018-01-10T22:30:46Z

/retest

mjudeikis · 2018-01-10T22:49:28Z

looks like my local dev env is out of sync with our build environment. Investigating where is delta and why I can see different run results.

openshift-bot · 2018-01-11T02:24:45Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2018-01-11T06:26:45Z

/retest

Please review the full test history for this PR and help us cut down flakes.

mjudeikis · 2018-01-11T09:38:53Z

/retest

bparees · 2018-01-11T14:24:39Z

you've got a compile erorr:

pkg/image/registry/imagestreamimport/rest.go:215:17: undefined: importer.NewContext

mjudeikis · 2018-01-11T14:46:05Z

Problem is "Works on my laptop" type one :) will try to align my dev env to test env... ETA - dunno. few days.

openshift-bot · 2018-01-11T20:33:44Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2018-01-12T08:39:43Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2018-01-12T16:43:47Z

/retest

Please review the full test history for this PR and help us cut down flakes.

bparees · 2018-01-12T17:09:28Z

@mjudeikis i think the importer package was just restructured. Do a rebase from master.

mjudeikis · 2018-01-12T17:11:07Z

Thanks @bparees will do on the weekend or today in the evening... . Sev1 on the client site... "Life of the consultant..." :'(

openshift-bot · 2018-01-12T20:45:44Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2018-01-13T08:51:42Z

/retest

Please review the full test history for this PR and help us cut down flakes.

mjudeikis · 2018-01-14T16:31:22Z

pkg/image/registry/imagestreamimport/rest.go

+	// check imported images status. If we get authentication error (401), try import same image without authentication.
+	// Docker registry gives 401 on public images if you have wrong secret in your secret list.
+	for _, image := range isi.Status.Images {
+		if image.Status.Reason == metav1.StatusReasonUnauthorized && strings.Contains(image.Status.Message, "incorrect username or password") {


If I dont check for the specific message this breaks 'TestImageStreamImportAuthenticated` test. I still need to understand which behavior of the test I break and if this is the change for the test or change for logic. For now, by adding this it makes the test pass and at the same time solves the bug.

this sort of check is definitely fragile, i'd prefer to understand what might need to change about the test over adding this contains check.

we can put this on hold for now as I need to find few days to investigate this more.

i'd still like to revisit this, but it's an improvement for now so i'm going to merge.

mjudeikis · 2018-01-15T11:29:23Z

/hold

mjudeikis · 2018-02-06T19:26:43Z

will try to do some work this week and let you know on Monday. Got stuck with our awesome storage offering these days.

mjudeikis · 2018-02-07T14:26:51Z

/retest

mjudeikis · 2018-02-07T15:54:35Z

/retest extended_conformance_gce

bparees · 2018-02-07T16:14:57Z

/lgtm

@mjudeikis you can /hold cancel if you're happy with this.

mjudeikis · 2018-02-07T17:51:19Z

@bparees I dont know what changes form the first view, but after rebase tests passes. Will check the code base, but It might be harder to find :)

The only thing which I would like to add this to make a little bit more cleaner error message.
Consider a use case:

I have the wrong secret for my image (token or password expire).
Try import, import will fail, so logic will default to try "unauthenticated".
Import will image will fail due "secure image".

So now user see a message that cant access image as its secure, and no "hints" to check secret.

Suggested message flow to construct error messages:

If Image got "wrong passwod" and later tried unauthenticated and failed too I give something like:
err := fmt.Errorf("Image import failed using secret with incorrect username or password error. Defaulting to unauthenticated import. \n Failed import with error %v", err)

Still testing this is "chain of actions is even possible in real world"...

bparees · 2018-02-07T18:01:16Z

you're right. even if we try to import via unauthenticated, if we fail, we should return the original error we got when we tried via authentication, i think.

mjudeikis · 2018-02-07T20:02:51Z

the only change was now to use previous error message form image status field where its being saved.
return nil, kapierrors.NewInternalError(fmt.Errorf(image.Status.Message))

mjudeikis · 2018-02-07T20:03:31Z

/hold cancel

bparees · 2018-02-07T20:17:29Z

pkg/image/registry/imagestreamimport/rest.go

+			importCtx := registryclient.NewContext(r.transport, r.insecureTransport).WithCredentials(nil)
+			imports := r.importFn(importCtx)
+			if err := imports.Import(ctx.(gocontext.Context), isi, stream); err != nil {
+				return nil, kapierrors.NewInternalError(fmt.Errorf(image.Status.Message))


on further analysis i don't think this is correct. "err" will still be nil, even if the import failed due to credentials, no?

just as the credentials import check above expects err to be nil even if the import was disallowed by auth.

So you're turning a more serious error that occurred during import, into a credential message. I don't think that's right.

I think I overthinking this... I think let's leave previous error. Standart error when image credentials are wrong is for private image is:

! error: Import failed (Unauthorized): you may not have access to the Docker image "docker.io/juozasa/hellonode:latest" Less than a second ago

Only when public image is being imported it gives the wrong password when trying with auth header. So only in this case (as much as I manage to test all possible cases), it defaults to this condition. Its just how docker distribution api behaves...

or we can concatenate both errors to one string but I would better keep it simple... previous way.

i agree the error should be returned as is.

i'm less clear on what image.status.message is going to end up being in this case (where you provided invalid credentials and the image is not public, but we tried to access it with nil credentials)?

openshift-ci-robot · 2018-02-07T21:21:49Z

@mjudeikis: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/extended_conformance_gce	`43054c7`	link	`/test extended_conformance_gce`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

…authentication. This is to eliminate case when we try pull public images with wrong/expired secret and it blocks all imports

mjudeikis · 2018-02-08T10:26:08Z

Status now. I confirm that cases 2-3 got passed to "second loop to retry without a secret" and got old messages defaulted back after failure.
Private image with wrong secret:

 oc import-image example2 --from=docker.io/juozasa/hellonode:latest --confirm
The import completed with errors.

  ! error: Import failed (Unauthorized): incorrect username or password for image "docker.io/juozasa/hellonode:latest"
      Less than a second ago

error: tag latest failed: incorrect username or password for image "docker.io/juozasa/hellonode:latest"

Private image from different API:

[root@default origin]# oc import-image kafka-producer --from=quay.io/mangirdas/docker-debug-container --confirm                                                                                                    
                                                                                                                                                                                                   
latest                                                                                                                                                                                                             
  tagged from quay.io/mangirdas/docker-debug-container                                                                                                                                                             
                                                                                                                                                                                                                   
  ! error: Import failed (InternalError): Internal error occurred: Get https://quay.io/v2/mangirdas/docker-debug-container/manifests/latest: unauthorized: Invalid Username or Password
      Less than a second ago

Public image when we have a wrong secret:

oc import-image example --from=openshift/hello-openshift --confirm                
The import completed successfully.                  
  * openshift/hello-openshift@sha256:a1a35c43bf529df887dedbc63e85d62914de1df4f3b29f72422db0d06327aee2    
      Less than a second ago

mjudeikis · 2018-02-08T10:50:13Z

Did some more advanced use case test:
IS where one image is public, other is private:

apiVersion: v1                                                      
kind: ImageStream                                                   
metadata:                                                           
  annotations:                                                      
    openshift.io/display-name: Test                                                                                                                                                                                                                                             
  name: test-is                                                     
spec:                                                               
  lookupPolicy:                                                     
    local: false                                                    
  tags:                                                             
  - annotations:                                                    
      description: test1                                            
      iconClass: icon-datavirt                                      
      openshift.io/display-name: test1                                                                                                  
      tags: test                                                    
      version: "latest"                                             
    from:                                                           
      kind: DockerImage                                             
      name: docker.io/juozasa/hellonode:latest                                                                                          
    generation: 2                                                   
    importPolicy: {}                                                
    name: "1.0"                                                     
    referencePolicy:                                                
      type: Source                                                  
  - annotations:                                                    
      description: test1                                            
      iconClass: icon-datavirt                                      
      openshift.io/display-name: test2                              
      tags: test                                                    
      version: "latest"                                             
    from:                                                           
      kind: DockerImage                                             
      name: docker.io/openshift/hello-openshift                     
    generation: 2                                                   
    importPolicy: {}                                                
    name: "latest"                                                  
    referencePolicy:                                                
      type: Source                                                                                                                      
status:                                                             
  dockerImageRepository: ""

Before code:

[root@default ~]# oc describe is test-is                                                                                                                                                                                                                                        
Name:                   test-is                                                                                                                                                                                                                                                 
Namespace:              import-images                                                                                                                                                                                                                                           
Created:                49 seconds ago                                                                                                                                                                                                                                          
Labels:                 <none>                                                                                                                                                                                                                                                  
Annotations:            openshift.io/display-name=Test                                                                                                                                                                                                                          
                        openshift.io/image.dockerRepositoryCheck=2018-02-08T10:45:40Z                                                                                                                                                                                           
Docker Pull Spec:       172.30.11.158:5000/import-images/test-is                                                                                                                                                                                                                
Image Lookup:           local=false                                                                                                                                                                                                                                             
Unique Images:          0                                                                                                                                                                                                                                                       
Tags:                   2                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                
latest                                                                                                                                                                                                                                                                          
  tagged from docker.io/openshift/hello-openshift                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                
  test1                                                                                                                                                                                                                                                                         
  Tags: test                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                
  ! error: Import failed (Unauthorized): incorrect username or password for image "docker.io/openshift/hello-openshift:latest"                                                                                                                                                  
      1 second ago                                                                                                                                                                                                                                                              

1.0                                                                 
  tagged from docker.io/juozasa/hellonode:latest                    

  test1                                                             
  Tags: test                                                        

  ! error: Import failed (Unauthorized): incorrect username or password for image "docker.io/juozasa/hellonode:latest"                  
      1 second ago

after code:

[root@default ~]# oc describe is test-is
Name:                   test-is
Namespace:              import-images
Created:                About a minute ago
Labels:                 <none>
Annotations:            openshift.io/display-name=Test
                        openshift.io/image.dockerRepositoryCheck=2018-02-08T10:49:07Z
Docker Pull Spec:       172.30.113.250:5000/import-images/test-is
Image Lookup:           local=false
Unique Images:          1
Tags:                   2

latest
  tagged from docker.io/openshift/hello-openshift

  test1
  Tags: test

  * docker.io/openshift/hello-openshift@sha256:a1a35c43bf529df887dedbc63e85d62914de1df4f3b29f72422db0d06327aee2
      42 seconds ago

1.0
  tagged from docker.io/juozasa/hellonode:latest

  test1
  Tags: test

  ! error: Import failed (Unauthorized): incorrect username or password for image "docker.io/juozasa/hellonode:latest"
      42 seconds ago

mjudeikis · 2018-02-08T11:28:08Z

/hold
Not sure if approved label merges automatically, so putting this until review done.

bparees · 2018-02-08T17:18:18Z

/hold cancel
the approved label does not, it needs a lgtm label(and an approved label) to merge.
/lgtm

openshift-ci-robot · 2018-02-08T17:18:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees, mjudeikis

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~pkg/image/OWNERS~~ [bparees]
~~test/cmd/OWNERS~~ [bparees]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

openshift-merge-robot · 2018-02-08T17:36:43Z

Automatic merge from submit-queue.

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jan 7, 2018

openshift-ci-robot requested review from dcbw and liggitt January 7, 2018 14:18

bparees reviewed Jan 10, 2018

View reviewed changes

bparees self-assigned this Jan 10, 2018

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 10, 2018

mjudeikis force-pushed the bugzilla_1506175 branch from 917e9ee to 52ff14a Compare January 13, 2018 20:23

openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Jan 13, 2018

mjudeikis force-pushed the bugzilla_1506175 branch 2 times, most recently from ae8d1be to 43054c7 Compare January 14, 2018 16:29

mjudeikis commented Jan 14, 2018

View reviewed changes

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 15, 2018

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 7, 2018

bparees added the retest-not-required label Feb 7, 2018

mjudeikis force-pushed the bugzilla_1506175 branch from 43054c7 to a3c1f98 Compare February 7, 2018 20:01

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 7, 2018

bparees reviewed Feb 7, 2018

View reviewed changes

openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Feb 7, 2018

mjudeikis force-pushed the bugzilla_1506175 branch from a3c1f98 to e8c97a8 Compare February 8, 2018 10:10

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 8, 2018

add retry for 401 errors to image imported to try pull image without …

f3290cb

…authentication. This is to eliminate case when we try pull public images with wrong/expired secret and it blocks all imports

mjudeikis force-pushed the bugzilla_1506175 branch from e8c97a8 to f3290cb Compare February 8, 2018 10:23

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 8, 2018

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Feb 8, 2018

openshift-merge-robot merged commit f90b369 into openshift:master Feb 8, 2018

Add retry import with no auth for IS if we get 401 error (public images case) #18012

Add retry import with no auth for IS if we get 401 error (public images case) #18012

Conversation

mjudeikis commented Jan 7, 2018

mjudeikis commented Jan 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PRIVATE IMAGE

bparees commented Jan 10, 2018

mjudeikis commented Jan 10, 2018

mjudeikis commented Jan 10, 2018

openshift-bot commented Jan 11, 2018

openshift-bot commented Jan 11, 2018

mjudeikis commented Jan 11, 2018

bparees commented Jan 11, 2018

mjudeikis commented Jan 11, 2018

openshift-bot commented Jan 11, 2018

openshift-bot commented Jan 12, 2018

openshift-bot commented Jan 12, 2018

bparees commented Jan 12, 2018

mjudeikis commented Jan 12, 2018

openshift-bot commented Jan 12, 2018

openshift-bot commented Jan 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjudeikis commented Jan 15, 2018

mjudeikis commented Feb 6, 2018

mjudeikis commented Feb 7, 2018

mjudeikis commented Feb 7, 2018

bparees commented Feb 7, 2018

mjudeikis commented Feb 7, 2018

bparees commented Feb 7, 2018

mjudeikis commented Feb 7, 2018

mjudeikis commented Feb 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci-robot commented Feb 7, 2018 • edited Loading

mjudeikis commented Feb 8, 2018

mjudeikis commented Feb 8, 2018

mjudeikis commented Feb 8, 2018

bparees commented Feb 8, 2018

openshift-ci-robot commented Feb 8, 2018

openshift-merge-robot commented Feb 8, 2018

openshift-ci-robot commented Feb 7, 2018 •

edited

Loading