Unable to push to private Docker Hub repository with default docker login and credential helper

[edthorne]

Environment:

• Jib version: 1.8.0
• Build tool: Maven 3.0.5
• OS: Linux

Description of the issue:

Changes introduced in #1958 changed the order of the credential store lookup and the method used to perform the lookup. The credential helpers utilize the repository address to retrieve the credentials. As documented in #586, the default value used by the docker CLI's login command is https://index.docker.io/v1/.

Versions prior to 1.6 worked as expected. Versions 1.6 and later exhibit the following output:

[WARNING] The credential helper (docker-credential-secretservice) has nothing for server URL: registry-1.docker.io
[WARNING] 
Got output:

credentials not found in native keychain

[WARNING] The credential helper (docker-credential-secretservice) has nothing for server URL: registry.hub.docker.com
[WARNING] 
Got output:

credentials not found in native keychain

[WARNING] The credential helper (docker-credential-secretservice) has nothing for server URL: index.docker.io
[WARNING] 
Got output:

credentials not found in native keychain

[WARNING] The credential helper (docker-credential-secretservice) has nothing for server URL: docker.io
[WARNING] 
Got output:

credentials not found in native keychain

The output demonstrates that the code is trying to use the values in RegistryAliasGroup.REGISTRY_ALIAS_GROUPS as keys into the credential store. Because the credential store has stored the login with the key https://index.docker.io/v1/, the credential helper is unable to find it.

The code previously used the auths values present to pattern match the registry values with DockerConfig.getRegistryMatchersFor(String registry) method. Because the credential key was matched from the partial index.docker.io in the auths value and that full value was used by the helper, the credentials were properly returned.

Expected behavior:

The default login credentials in the configured credential store should work when pushing to private Docker Hub repositories

Steps to reproduce:

1. Create a private Docker Hub repository target
2. docker login and enter your credentials
3. Use Quickstart on JAR/WAR project of your choosing.

Log output:

Output included above

Additional Information:

This comment acknowledges that #1958 was a breaking change. It also says:

credsStore is old enough to ditch and no real reason to use.

I couldn't find any foundation for this statement in any of the linked issues or the current Docker documentation for docker login. While there's a default behavior per operating system, nothing states that credsStore is no longer used or necessary. In my case I've explicitly configured secretservice in lieu of the default pass.

The thing that appears to be missing from the new code is usage of the getRegistryMatchersFor logic that pattern matches the repositories. In this linked issue there's discussion of client.List as a replacement for the blank auths in the configuration:

Modified the native credential store GetAll behavior to use the store's client.List as the source of truth, rather than the file store's "auths" list.

If this is in fact the case, the code should be listing the credentials stored by the credential helper using list and pattern matching those to identify the credential to retrieve.

Configuring a credential helper (credHelper) for registry-1.docker.io does not solve the problem. The only work around I was able to find was to do the following:

docker login registry-1.docker.io

This adds a second set of credentials to the credential helper that jib is able to find with the current code base. Here's the output of list:

$ docker-credential-secretservice list
{"https://index.docker.io/v1/":"edthorne","registry-1.docker.io":"edthorne"}

This case is not covered by the current FAQ entry. As you can expect, pushing using the docker CLI works.

[chanseokoh]

Hi @edthorne,

Configuring a credential helper (credHelper) for registry-1.docker.io does not solve the problem.

It is credHelpers, not credHelper. The following with the key https://index.docker.io/v1/ should work:

    "credHelpers": {
        "https://index.docker.io/v1/": "secretservice"
    },

It works for me:

$ docker-credential-gcr store <<< '{"ServerURL":"https://index.docker.io/v1/","Username":"francium25","Secret":"..."}'
$ docker-credential-gcr get <<< https://index.docker.io/v1/
{"ServerURL":"","Username":"francium25","Secret":"..."}
$ docker-credential-gcr list
{...,"https://index.docker.io/v1/":"francium25",...}
$ cat ~/.docker/config.json # Jib succeeds only when I have the following:
...
    "credHelpers": {
...
        "https://index.docker.io/v1/": "gcr"
    },
...

Basically, there's no reason to use credsStore anymore. To my understanding, Docker's previous interaction of credsStore with auths was a very weird and unintended behavior (which some people and tools depended on), which I think is fixed in Docker and no longer works in that way. If you don't want to put another key into your credential helpers, then just use credHelpers instead of the pre-1.6.0 credsStore + auths combination. credsStore is just inferior to credHelpers.

FYI, docker-credential-gcr no longer supports the generalized credsStore:

As of the 2.0 release, docker-credential-gcr no longer supports generalized credsStore functionality.

But make no mistake; Jib still supports credsStore (which credHelpers takes precedence over). But due to the fact that it is a generalized store that must hold all credentials you ever need to use, the key has to match what Jib requests. It is a valid idea for Jib to ask for a list of stored credentials and attempt to find a closest match. However, given that you can just use credHelpers and that we'd also like to tell people to use credHelpers instead of credsStore, I think implementing that idea is a very low priority. But if more people ask for this, we can certainly reconsider this.

And as you said, putting the key registry-1.docker.io is another workaround if you don't want to configure credHelpers for some reason (but frankly, I don't find any reason not to). I still suggest using the modern credHelpers instead of credsStore.

[chanseokoh]

I believe this issue is resolved. Feel free to re-open or update if you'd like to give us any feedback.