Amazon ECR token in config.json not picked up

[ocker-docker-gummi-klokker]

Description

I'm trying to build and push an image to an AWS repository using the spotify docker-maven-plugin.

In this issue olkulyk asks:
Is it be possible to use an auth token from /.docker/config.json

davidxia reply:
It's certainly possible

It sounds like it should be pretty straight forward to do, but it turns out not to be (for me that is).

To start of with I have successfully created a token, build and tagged and image, and pushed it to my AWS repository. This was done manually from a docker terminal (Docker tools for Windows) following the steps provided by Amazon when the repository was created (see below). Ok - the prerequisites should be in order. Now lets proceed to the actual task - building, tagging and pushing a docker image to Amazon ECR using the plugin.

=================== BEGIN AMAZON GUIDE ==================

Successfully created repository
https://myaccount.dkr.ecr.us-east-1.amazonaws.com/my-dockers

1. Retrieve the docker login command that you can use to authenticate your Docker client to your registry:

aws ecr get-login --region us-east-1

2. Run the docker login command that was returned in the previous step.

3. Build your Docker image using the following command. For information on building a Docker file from
   scratch see the instructions here. You can skip this step if your image is already built:

docker build -t my-dockers .

4. After the build completes, tag your image so you can push the image to this repository:

docker tag my-dockers:latest myaccount.dkr.ecr.us-east-1.amazonaws.com/my-dockers:latest 

5. Run the following command to push this image to your newly created AWS repository:

docker push myaccount.dkr.ecr.us-east-1.amazonaws.com/my-dockers:latest

==================== END AMAZON GUIDE ===================

To start of with, I have generated a token manually (step 1 and 2 in the above guide). This will create a config.json in C:\Users\me.docker dir, containing the token (later I have to figure out how to do this automatically from maven, but that is another story). The challenge is now to get the docker-maven-plugin to use this token to push to ECR.

In the pom.xml the three execution blocks correspond to 3, 4 and 5 in the Amazon guide shown above.

====================== START pom.xml ====================

<properties>
    <docker.image.name>my-dockers</docker.image.name>
    <docker.image.version>latest</docker.image.version>
    <docker.registry.host>https://myaccount.dkr.ecr.us-east-1.amazonaws.com</docker.registry.host>
</properties>

<plugin>
    <groupId>com.spotify</groupId>
    <artifactId>docker-maven-plugin</artifactId>
    <version>0.4.6</version>
    <configuration>
        <imageName>${docker.image.name}:${docker.image.version}</imageName>
        <!-- Define location of Dockerfile (only this file is located here) -->
        <dockerDirectory>${project.basedir}/docker/</dockerDirectory>
        <useConfigFile>true</useConfigFile>
        <dockerHost>${docker.registry.host}</dockerHost>
        <resources>
            <resource>
                <targetPath>files/</targetPath>
                <directory>${project.build.directory}</directory>
                <include>${project.build.finalName}.jar</include>
            </resource>
            <resource>
                <targetPath>files/libs/</targetPath>
                <directory>${project.build.directory}/libs</directory>
                <includes>
                    <include>*.jar</include>
                </includes>
            </resource>
        </resources>
    </configuration>
    <executions>
        <!-- Build the image. Corresponds to 3 in the Amazon guide -->
        <execution>
            <phase>package</phase>
            <goals>
                <goal>build</goal>
            </goals>
        </execution>
        <!-- Tag the image. Corresponds to 4 in the Amazon guide -->
        <execution>
            <id>tag-image</id>
            <phase>package</phase>
            <goals>
                <goal>tag</goal>
            </goals>
            <configuration>
                <forceTags>true</forceTags>
                <image>${docker.image.name}:${docker.image.version}</image>
                <newName>${docker.registry.host}/${docker.image.name}:${docker.image.version}</newName>
            </configuration>
        </execution>
        <!-- Push the image. Corresponds to 5 in the Amazon guide -->
        <execution>
            <id>push-image</id>
            <phase>deploy</phase>
            <goals>
                <goal>push</goal>
            </goals>
            <configuration>
                <useConfigFile>true</useConfigFile>
                <forceTags>true</forceTags>
                <imageName>${docker.registry.host}/${docker.image.name}:${docker.image.version}</imageName>
            </configuration>
        </execution>
    </executions>
</plugin>

======================= END pom.xml =====================

Using the <useConfigFile>true</useConfigFile> I would expect the plugin to use the token found in:

C:\Users\me.docker\config.json

I'm building the project with mvn clean install -X to give me debug output.
It looks like everything is build correctly, except I get some debug output that may hint to the problem:

[DEBUG] Auth Config null
[DEBUG] C:\Users\me.docker\ca.pem, C:\Users\me.docker\key.pem or C:\Users\me.docker\cert.pem does not exist, not using SSL
[DEBUG] Not using any build profile

After the maven run I end up with a docker folder in the target dir. The docker folder contains the Dockerfile and a files folder. The files folder contains the dependencies referenced from the Dockerfile i.e. the entrypoint jar, and all the lib-jars. I assume that the plugin builds from the target/docker folder (and not the dockerDirectory defined in the pom).

I would expect the build image to be pushed to Amazon ECR.

What happened instead

What I actually get is a 401 Unauthorized (se stacktrace below). Why is the plugin not picking up the token in config.json? Am I placing the config.json in the wrong location or am I missing something in the pom?

Note that I generate a token (which creates a new config.json in the .docker dir) right before building the project. Thus I don't think it's an invalid token.

[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:0.4.6:build (default) on project MyProject: Exception caught: Request error: GET https://myaccount.dkr.ecr.us-east-1.amazonaws.com/version: 401: HTTP 401 Unauthorized -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.spotify:docker-maven-plugin:0.4.6:build (default) on project MyProject: Exception caught
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Exception caught
at com.spotify.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:126)
at com.spotify.docker.BuildMojo.execute(BuildMojo.java:86)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 20 more
Caused by: com.spotify.docker.client.exceptions.DockerRequestException: Request error: GET https://myaccount.dkr.ecr.us-east-1.amazonaws.com/version: 401
at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1575)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1514)
at com.spotify.docker.client.DefaultDockerClient.version(DefaultDockerClient.java:394)
at com.spotify.docker.client.DefaultDockerClient.authRegistryHeader(DefaultDockerClient.java:1622)
at com.spotify.docker.client.DefaultDockerClient.build(DefaultDockerClient.java:1123)
at com.spotify.docker.client.DefaultDockerClient.build(DefaultDockerClient.java:1082)
at com.spotify.docker.BuildMojo.buildImage(BuildMojo.java:562)
at com.spotify.docker.BuildMojo.execute(BuildMojo.java:347)
at com.spotify.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:124)
... 23 more
Caused by: com.spotify.docker.client.shaded.javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:984)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:799)
at org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
at org.glassfish.jersey.client.JerseyInvocation$5.completed(JerseyInvocation.java:760)
at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:197)
at org.glassfish.jersey.client.ClientRuntime.access$300(ClientRuntime.java:78)
at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:179)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:209)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Software:

• docker version: Docker tools (on Windows 7). Docker version 1.11.1, build 5604cbe
• docker-maven-plugin version: 0.4.6
• maven version: Apache Maven 3.3.3

[mattnworb]

Can you try the most recent plugin version, 0.4.11?

[ocker-docker-gummi-klokker]

Hi mattnworb

Thanks for the quick reply.

I have changed the version to 0.4.11, and the token is now picked up (it seems). At least the
[DEBUG] Auth Config null
has been replaced with
[DEBUG] Auth Config AuthConfig{username=AWS, password=CiB...5Rt8cH, email=null, serverAddress=https://myaccount.dkr.ecr.us-east-1.amazonaws.com}

The problem still persist however (same stacktrace with the 401). If I copy the GET request from the stacktrace to a browser I'm prompted for a username and password. Using the username and password from the debug output i.e username=AWS and password=CiB...5Rt8cH, I can successfully log in. However I'm now getting a 404 page not found. OK - so it looks like there are two problems here:

1. The plugin picks up the token, but seems to be unable to login with the username and password in that token.

2. The GET request (see below) is not returning anything despite a successful login.

   GET https://966313488686.dkr.ecr.us-east-1.amazonaws.com/version

The problem with 2 might be the version path. I would expect it to be my-dockers as defined in the pom, and not version. However trying different GET requests still return a 404.

[mattnworb]

Copying and pasting the URL into a browser isn't really a good test since docker-client sends the credentials in a header in the request.

When the docker-client is deciding how to format the credentials header, it checks what version of docker your docker daemon is using.

I just noticed that you are configuring the plugin to use the registry as the docker host, which probably isn't what you want:

<dockerHost>${docker.registry.host}</dockerHost>

This value is meant to be the address where the docker daemon is running, i.e. where the docker instance that will build your image runs. This is likely the cause of the errors.

[ocker-docker-gummi-klokker]

If I use a local running docker instance I can get the dockerHost IP and port using:

docker-machine env

Additionally I have to add the following to the pom

dockerCertPath

When doing so, the maven build succeeds however the image is not pushed to the Amazon ECR. When I run a docker images command, I can see the newly created image locally:

REPOSITORY                                                TAG         CREATED
myaccount.dkr.ecr.us-east-1.amazonaws.com/my-dockers      latest      24 seconds ago

Any idea why it's not pushed to the ECR?

Doing a docker push manually on the image created by the plugin pushes the new image to the ECR successfully.

I also tried setting the with no success.

[AmazingDreams]

I had kind of the same problem. Authentication information not found. I just removed the https:// stuff from the url and it started to work.

Cat ~/.docker/config.json and ensure those values are the same.

So pom.xml:

<configuration>
    <serverId>docker-registry</serverId>
    <registryUrl>registry.something.com</registryUrl>
    [...]
</configuration>

And ~/.docker/config.json

{
     "auths": {
         "registry.something.com": {
            "auth": "blablaSomeThingSECRET="
         }
    }
}

Hope this helps.

[mdamir]

I am seeing similar issue.

[mdamir]

@mattnworb
It turns out that in my ~/.docker/config.json file, apart from my private registry, I also had docker.io registry which was before my private registry and mvn was picking up the credentials of the docker.io registry. When I deleted the docker.io registry config from that file, it started picking up the right credentials. It seems like a bug in the docker-maven-plugin plugin.

[deploymentking]

@mdamir I can confirm a similar issue (I'm using 0.4.13)

I had multiple entries in my auths nodes (1 for each of the 4 ECR repositories I have defined in different AWS accounts), all of them valid and all in the format

{
    "auths": {
        "https://account1.dkr.ecr.eu-west-1.amazonaws.com": {
            "auth": "...",
            "email": "none"
        },
        "https://account2.dkr.ecr.eu-west-1.amazonaws.com": {
            "auth": "...",
            "email": "none"
        },
        "https://account3.dkr.ecr.eu-west-1.amazonaws.com": {
            "auth": "...",
            "email": "none"
        },
        "https://account4.dkr.ecr.eu-west-1.amazonaws.com": {
            "auth": "...",
            "email": "none"
        }
    }
}

I have maven properties set up like this (note there is no https://)

<docker.image.prefix>account3.dkr.ecr.eu-west-1.amazonaws.com</docker.image.prefix>
<docker.image.name>${docker.image.prefix}/core-app-${project.artifactId}</docker.image.name>

Then in my plugin I have a push goal bound to the deploy phase with this configuration

<configuration>
    <useConfigFile>true</useConfigFile>
    <imageName>${docker.image.name}:${project.version}</imageName>
</configuration>

When doing my mvn deploy I got the following error
Failed to execute goal com.spotify:docker-maven-plugin:0.4.13:push (push-image) on project replication-service: Exception caught: error parsing HTTP 403 response body: invalid character 'Y' looking for beginning of value: "Your Authorization Token has expired. Please run 'aws ecr get-login' to fetch a new one."

I deleted my ~/.docker/config.json file and then ran a aws ecr get-login command for the appropriate account so my json now looked like this...

{
    "auths": {
        "https://account3.dkr.ecr.eu-west-1.amazonaws.com": {
            "auth": "...",
            "email": "none"
        }
    }
}

Bingo...it worked. So there's something about having multiple auths that causes issues

LATEST UPDATE
I have since found that if I put the following configuration in my plugin, I get successful pushes each time despite having multiple entries in my config.json

<plugin>
    <configuration>
...
        <registryUrl>https://${docker.image.prefix}</registryUrl>
...
    </configuration>
</plugin>

[emmaLP]

As @deploymentking stated in his previous comment, there is definitely something about having multiple auths and even specifying the registryUrl in the plugin doesn't work for me.

Our CI environment has 2 private repos in the config file but the plugin kept failing while the repo it was trying to reference was the second one in the config file.

For example

config.json file looks like this

{
    "auths": {
        "repo1.company.com" : {
            "auth": "...",
            "email": "none"
         },
        "https://repo2.dkr.ecr.eu-west-1.amazonaws.com" : {
            "auth": "...",
            "email": "none"
        }
    }
}

plugin code looks like this:

<plugin>
    <configuration>
...
        <imageName>https://repo2.dkr.ecr.eu-west-1.amazonaws.com/${project.artifactId}:${project.version}</imageName>
...
    </configuration>
</plugin>

This configuration fails unless the repos in the config.json file is the other way around.

Please could this be looked into and fixed so the order of the config.json does not matter to the plugin but finds the one it needs from the config.json

[ovelindstrom]

Thanks @deploymentking for the config information. It solved the issues I have struggled with for quite a while. :D

[jochenhebbrecht]

Hi,

I just ran into the same issue. I also figured out cleaning up the ~/.docker/config.json file solves the issue, but I do think the plugin should support multiple registries (I'm also using 0.4.13).

The workaround of AmazingDreams does work (specifying the registryUrl in the pom.xml itself)

Jochen

[mattnworb]

#339 revamps the authentication support in the plugin and will remove the bug around multiple auth configs / the behavior where the first one only is used.

[davidxia]

@ocker-docker-gummi-klokker Does the latest version work for you?

[davidxia]

Closing for now. Please comment here if it doesn't.