crane auth login fails when docker login works #861

kumprj · 2020-12-08T15:12:31Z

Steps to reproduce on macOS catalina bash. For what its worth I'm behind an enterprise proxy:

$ echo "acctid.dkr.ecr.us-east-2.amazonaws.com" | crane auth get | jq .password | sed 's/"//g'

long string password is output

$ crane auth login acctid.dkr.ecr.us-east-2.amazonaws.com -u AWS -p stringfromabove
$ crane pull acctid.dkr.ecr.us-east-2.amazonaws.com/reponame:tag tarball.tar

Error: 2020/12/07 13:27:15 GET https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/repo-name/manifests/tag-name: unexpected status code 401 Unauthorized: Not Authorized

$ docker login acctid.dkr.ecr.us-east-2.amazonaws.com -u AWS -p longstringfromabove
warning about --password via cli followed by Login Succeeded
docker pull succeeds

Config.json:

$ vim ~/.docker/config.json 
{
        "auths": {
                "acctid-that-I-want-to-pull-from.dkr.ecr.us-east-2.amazonaws.com": {},
                "different-actid.dkr.ecr.us-east-2.amazonaws.com": {},
                "jfrog.company.com": {},
                "jfrogdev.company.com": {}
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/19.03.13 (darwin)"
        },
        "credsStore": "desktop",
        "experimental": "disabled",
        "stackOrchestrator": "swarm"
}

I also tried adding a

"credHelpers": {
                "acct-id.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
        },

and setting the credsStore to ecr-login. But that didn't help my crane pull either. Running a docker pull with the ecr credHelpers set did work.

My AWS Environment Variables are set, as is the certificate path location.

The text was updated successfully, but these errors were encountered:

jonjohnsonjr · 2020-12-08T16:53:31Z

I also tried adding ...
and setting the credsStore to ecr-login

What happens if you just delete the global credsStore line? Same issue?

mattmoor · 2020-12-08T17:01:46Z

I have KO_DOCKER_REPO set to my ECR hostname, and I get:

echo $KO_DOCKER_REPO  | crane auth get | jq .password | sed 's/"//g' | crane auth login $KO_DOCKER_REPO -u AWS --password-stdin
2020/12/08 09:00:24 error storing credentials - err: exit status 1, out: `not implemented`

This is sunk to HEAD and freshly installed.

mattmoor · 2020-12-08T17:02:20Z

Ah, this is probably because the cred helper doesn't like being invoked to store, lemme futz with it.

jonjohnsonjr · 2020-12-08T17:02:52Z

Yeah I think you just use the ecr-login creds store as a read-only source of credentials, not a generic credential store. You shouldn't even need to log in, really.

mattmoor · 2020-12-08T17:04:01Z

If I move ~/.docker/config.json between invocations it completes, let's try a crane command.

mattmoor · 2020-12-08T17:05:02Z

# crane ls $KO_DOCKER_REPO/controller
latest
# crane ls $KO_DOCKER_REPO/webhook
v20201207-local-16baabad-dirty
latest

Seems to work for me.

mattmoor · 2020-12-08T17:06:18Z

I just did a crane pull $KO_DOCKER_REPO/controller foo.tar and things worked.

mattmoor · 2020-12-08T17:07:44Z

So the main thing I'd note here is that ECR is minting short-lived credentials, so given enough delay between the foo login command and any subsequent command you are likely to hit 401s, but I'm able to do stuff fine after immediately logging in.

kumprj · 2020-12-08T17:10:59Z

When you say move between invocations do you mean run a mv command to somehow refresh it?

I'm trying a login and then immediate pull and receiving the 401s. I too would have expected the ecrhelper to work that's why I'm puzzled. Its the same pw that is generated throughout the day for me, for what its worth.

kumprj · 2020-12-08T17:12:25Z

I also tried adding ...
and setting the credsStore to ecr-login

What happens if you just delete the global credsStore line? Same issue?

Same issue.

jonjohnsonjr · 2020-12-08T17:13:58Z

To eliminate any confounding variables, what happens if you config file is just this?

{
  "credHelpers": {
    "acctid-that-I-want-to-pull-from.dkr.ecr.us-east-2.amazonaws.com": "ecr-login"
  }
}

Don't login or anything, just try e.g. crane ls <repo>.

kumprj · 2020-12-08T17:19:29Z

To eliminate any confounding variables, what happens if you config file is just this?
{
  "credHelpers": {
    "acctid-that-I-want-to-pull-from.dkr.ecr.us-east-2.amazonaws.com": "ecr-login"
  }
}
Don't login or anything, just try e.g. crane ls <repo>.

I simplified the config.json to this and same 401 error.

jonjohnsonjr · 2020-12-08T17:27:33Z

If you use crane ls -v, does the 401 response have a WWW-Authenticate challenge header?

kumprj · 2020-12-08T17:32:43Z

If you use crane ls -v, does the 401 response have a WWW-Authenticate challenge header?

Www-Authenticate: Basic realm="https://acctid.dkr.ecr.us-east-2.amazonaws.com/",service="ecr.amazonaws.com"

edit: full log

$ crane ls acctid.dkr.ecr.us-east-2.amazonaws.com/v2/repo-name -v
2020/12/08 12:28:49 --> GET https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/
2020/12/08 12:28:49 GET /v2/ HTTP/1.1
Host: acctid.dkr.ecr.us-east-2.amazonaws.com
User-Agent: crane/0.2.1 go-containerregistry/(devel)
Authorization: <redacted>
Accept-Encoding: gzip


2020/12/08 12:28:50 <-- 401 https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/ (827.061998ms)
2020/12/08 12:28:50 HTTP/1.1 401 Unauthorized
Content-Length: 15
Cache-Control: proxy-revalidate
Connection: Keep-Alive
Content-Type: text/plain; charset=utf-8
Date: Tue, 08 Dec 2020 17:28:50 GMT
Docker-Distribution-Api-Version: registry/2.0
Proxy-Connection: Keep-Alive
Proxy-Support: Session-based-authentication
Set-Cookie: BCSI-CS-e2196ff800d4366a=1; Path=/
Www-Authenticate: Basic realm="https://acctid.dkr.ecr.us-east-2.amazonaws.com/",service="ecr.amazonaws.com"

Not Authorized

2020/12/08 12:28:50 --> GET https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/v2/repo-name/tags/list?n=1000
2020/12/08 12:28:50 GET /v2/v2/repo-name/tags/list?n=1000 HTTP/1.1
Host: acctid.dkr.ecr.us-east-2.amazonaws.com
User-Agent: crane/0.2.1 go-containerregistry/(devel)
Authorization: <redacted>
Accept-Encoding: gzip


2020/12/08 12:28:50 <-- 401 https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/v2/repo-name/tags/list?n=1000 (93.701822ms)
2020/12/08 12:28:50 HTTP/1.1 401 Unauthorized
Content-Length: 15
Cache-Control: proxy-revalidate
Connection: Keep-Alive
Content-Type: text/plain; charset=utf-8
Date: Tue, 08 Dec 2020 17:28:50 GMT
Docker-Distribution-Api-Version: registry/2.0
Proxy-Connection: Keep-Alive
Proxy-Support: Session-based-authentication
Set-Cookie: BCSI-CS-e2196ff800d4366a=1; Path=/
Www-Authenticate: Basic realm="https://acctid.dkr.ecr.us-east-2.amazonaws.com/",service="ecr.amazonaws.com"

Not Authorized

2020/12/08 12:28:50 reading tags for acctid.dkr.ecr.us-east-2.amazonaws.com/v2/repo-name: GET https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/v2/repo-name/tags/list?n=1000: unexpected status code 401 Unauthorized: Not Authorized

jonjohnsonjr · 2020-12-08T17:50:12Z

I wonder if your proxy (context: GoogleContainerTools/kaniko#676) is dropping headers or if the go http client (or maybe go-containerregistry) is somehow dropping them?

I'm obviously not privy to how your network is configured, but have you tried fiddling with HTTPS_PROXY environment variables?

kumprj · 2020-12-08T17:54:17Z

So I went into my Jenkins Docker Container (running Debian Linux) and this worked with the credential helper. I'm not sure what's going on but it must be an issue with my MacOS setup (enterprise configuration or proxy dropping headers are good theories). That CI environment where I need this longterm, so I guess it was a wasted effort trying to configure it locally first.

jonjohnsonjr · 2020-12-08T18:02:00Z

Glad it at least works where you need it, but now you've piqued my curiosity...

GET /v2/v2/repo-name

This is really suspicious. Is that /v2/v2/ really there? Or was this a copy/paste error during redaction?

Www-Authenticate: Basic realm="https://acctid.dkr.ecr.us-east-2.amazonaws.com/",service="ecr.amazonaws.com"

It may be that they expect us to do some token exchange at that realm even though it's basic auth? Hmm 😕

kumprj · 2020-12-08T18:56:58Z

Glad it at least works where you need it, but now you've piqued my curiosity...

GET /v2/v2/repo-name

This is really suspicious. Is that /v2/v2/ really there? Or was this a copy/paste error during redaction?

Www-Authenticate: Basic realm="https://acctid.dkr.ecr.us-east-2.amazonaws.com/",service="ecr.amazonaws.com"

It may be that they expect us to do some token exchange at that realm even though it's basic auth? Hmm 😕

Looks like that may have been a typo. I re-ran an ls and there's no duplicate of v2.

So I just now caught a tip that we moved towards a new proxy recently and I set that in my bash_profile. That proxy let the ls work. It must have been something about the old one (header stripping? I'm not sure).

I really appreciate your guys' help in investigating this with me.

edit: here's the verbose ls that worked

$ crane ls acctid.dkr.ecr.us-east-2.amazonaws.com/repo-name -v
2020/12/08 13:57:40 --> GET https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/
2020/12/08 13:57:40 GET /v2/ HTTP/1.1
Host: acctid.dkr.ecr.us-east-2.amazonaws.com
User-Agent: crane/0.2.1 go-containerregistry/(devel)
Authorization: <redacted>
Accept-Encoding: gzip


2020/12/08 13:57:41 <-- 401 https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/ (752.182363ms)
2020/12/08 13:57:41 HTTP/1.1 401 Unauthorized
Content-Length: 15
Content-Type: text/plain; charset=utf-8
Date: Tue, 08 Dec 2020 18:57:41 GMT
Docker-Distribution-Api-Version: registry/2.0
Proxy-Support: Session-Based-Authentication
Www-Authenticate: Basic realm="https://acctid.dkr.ecr.us-east-2.amazonaws.com/",service="ecr.amazonaws.com"

Not Authorized

2020/12/08 13:57:41 --> GET https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/repo-name/tags/list?n=1000
2020/12/08 13:57:41 GET /v2/repo-name/tags/list?n=1000 HTTP/1.1
Host: acctid.dkr.ecr.us-east-2.amazonaws.com
User-Agent: crane/0.2.1 go-containerregistry/(devel)
Authorization: <redacted>
Accept-Encoding: gzip


2020/12/08 13:57:41 <-- 200 https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/repo-name/tags/list?n=1000 (148.861939ms)
2020/12/08 13:57:41 HTTP/1.1 200 OK
Content-Length: 920
Content-Type: text/plain; charset=utf-8
Date: Tue, 08 Dec 2020 18:57:41 GMT
Docker-Distribution-Api-Version: registry/2.0

jonjohnsonjr · 2020-12-08T19:17:31Z

Fantastic! Glad to see it works. Going to close this out 🎉

jonjohnsonjr mentioned this issue Dec 8, 2020

Fix misleading Authorization: <redacted> #864

Merged

jonjohnsonjr closed this as completed Dec 8, 2020

to-bar mentioned this issue Aug 13, 2021

Add Internet connection test to download-requirements.sh (2124) hitachienergy/epiphany#2493

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crane auth login fails when docker login works #861

crane auth login fails when docker login works #861

kumprj commented Dec 8, 2020 •

edited

Loading

jonjohnsonjr commented Dec 8, 2020

mattmoor commented Dec 8, 2020

mattmoor commented Dec 8, 2020

jonjohnsonjr commented Dec 8, 2020

mattmoor commented Dec 8, 2020

mattmoor commented Dec 8, 2020

mattmoor commented Dec 8, 2020

mattmoor commented Dec 8, 2020

kumprj commented Dec 8, 2020 •

edited

Loading

kumprj commented Dec 8, 2020

jonjohnsonjr commented Dec 8, 2020

kumprj commented Dec 8, 2020

jonjohnsonjr commented Dec 8, 2020

kumprj commented Dec 8, 2020 •

edited

Loading

jonjohnsonjr commented Dec 8, 2020 •

edited

Loading

kumprj commented Dec 8, 2020

jonjohnsonjr commented Dec 8, 2020

kumprj commented Dec 8, 2020 •

edited

Loading

jonjohnsonjr commented Dec 8, 2020

crane auth login fails when docker login works #861

crane auth login fails when docker login works #861

Comments

kumprj commented Dec 8, 2020 • edited Loading

jonjohnsonjr commented Dec 8, 2020

mattmoor commented Dec 8, 2020

mattmoor commented Dec 8, 2020

jonjohnsonjr commented Dec 8, 2020

mattmoor commented Dec 8, 2020

mattmoor commented Dec 8, 2020

mattmoor commented Dec 8, 2020

mattmoor commented Dec 8, 2020

kumprj commented Dec 8, 2020 • edited Loading

kumprj commented Dec 8, 2020

jonjohnsonjr commented Dec 8, 2020

kumprj commented Dec 8, 2020

jonjohnsonjr commented Dec 8, 2020

kumprj commented Dec 8, 2020 • edited Loading

jonjohnsonjr commented Dec 8, 2020 • edited Loading

kumprj commented Dec 8, 2020

jonjohnsonjr commented Dec 8, 2020

kumprj commented Dec 8, 2020 • edited Loading

jonjohnsonjr commented Dec 8, 2020

kumprj commented Dec 8, 2020 •

edited

Loading

kumprj commented Dec 8, 2020 •

edited

Loading

kumprj commented Dec 8, 2020 •

edited

Loading

jonjohnsonjr commented Dec 8, 2020 •

edited

Loading

kumprj commented Dec 8, 2020 •

edited

Loading