Nomad clients not honoring docker auth options?

[shilov]

Summary

Nomad clients are not able to pull private docker images from Google's container repository.

Clients seems to ignore the value provided in "docker.auth.config" or "docker.auth.helper".

For authorization, I'm using https://github.com/GoogleCloudPlatform/docker-credential-gcr which works as expected when running docker pull manually.

Nomad version

v0.6.0

Operating system and Environment details

OS: Debian 9

> docker version

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:17:22 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:16:12 2017
 OS/Arch:      linux/amd64
 Experimental: false

Docker credentials are handled via docker-credential-gcr v1.4.1 (current release)

Issue

Pulling a private docker image using docker pull works as anticipated.

Nomad client reports auth errors attempting to pull the same image.

Reproduction steps

Nomad client config:

client {
  enabled = true
  options {
    # Attempts to provide only one of these settings made no difference:
    "docker.auth.config" = "/root/.docker/config.json"
    "docker.auth.helper" = "gcr"
  }
}

Docker config (/root/.docker/config.json):

{
  "auths": {},
  "credHelpers": {
    "appengine.gcr.io": "gcr",
    "asia.gcr.io": "gcr",
    "eu.gcr.io": "gcr",
    "gcr.io": "gcr",
    "gcr.kubernetes.io": "gcr",
    "us.gcr.io": "gcr"
  }
}

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Recent Events:
Time                   Type            Description
08/01/17 23:07:13 PDT  Not Restarting  Error was unrecoverable
08/01/17 23:07:13 PDT  Driver Failure  failed to initialize task "foo-bar" for alloc "5f2ccfda-abd6-2ba7-c9a9-968ce98a33ee": Failed to find docker auth for repo "gcr.io/google-project-name/foo": docker-credential-gcr failed with stderr:
08/01/17 23:07:13 PDT  Task Setup      Building Task Directory
08/01/17 23:07:13 PDT  Received        Task received by client

Job file (if appropriate)

Job file consists of a bare-bones docker task. The image is private, hosted on gcr.io, Google's container repository.

I've tried specifying the docker image both as gcr.io/google-project/foo as well as https://gcr.io/google-project/foo, with and without the docker tag, but the end result was always the same (auth error).

The job file does not specify any docker auth values. I did try providing the docker auth server_address (using "gcr.io" as the value) but it made no difference.

Curiously, when I tried providing the docker auth server_address in the job description, the nomad client logs included the complete docker image name with tag. Without the server_address, the client logs only contained the image name, without the associated tag. Probably unrelated, but I figured it's worth mentioning.

[dadgar]

What is the name of the gcr helper binary. It should be named docker-credential-gcr and available on the path.

[shilov]

It's there, with that name and available in $PATH.

To rule out that issue, I renamed the binary as a test and sure enough Nomad reported an error about docker-credential-gcr not being available.

[dadgar]

@shilov Hmm, thanks. Will try to spin up a cluster on GCE and debug. Are there anything in the client logs that look useful?

[shilov]

Nothing insightful, other than the error I shared earlier. I'll try enabling debug logging and report back if anything valuable shows up.

[alxark]

Facing the same problem now. Docker config:

{
     "auths" : {
              "https://gcr.io": {
                          "auth" : "<BASE64DATA>"
               }
}

nomad client options:

   options {
      "driver.raw_exec.enable" = "1"
      "docker.privileged.enabled" = "true"
      "docker.auth.config" = "/root/.docker/config.json"
      "docker.endpoint" = "unix:///var/run/docker.sock"
   }

task config block:

config {
                    image = "gcr.io/hmnd-164915/frontend:current"
                    ssl = true
                    auth {
                                server_address = "gcr.io"
                    }
                   network_mode = "host"
}

Error message:

    2017/08/12 20:37:19.090444 [ERR] driver.docker: failed pulling container gcr.io/hmnd-164915/frontend:current: API error (404): {"message":"repository gcr.io/hmnd-164915/frontend not found: does not exist or no pull access"}
    2017/08/12 20:37:19.090505 [WARN] client: error from prestart: failed to initialize task "balancer" for alloc "15040581-5fc1-3122-88d1-743ddb23c397": Failed to pull `gcr.io/hmnd-164915/frontend:current`: API error (404): {"message":"repository gcr.io/hmnd-164915/frontend not found: does not exist or no pull access"}

[joe-boyce]

I've been seeing the same issue, anybody able to take a look?

[chelseakomlo]

Hi @joe-boyce, thanks for confirming this.

I picked up this issue today. I will keep this ticket updated when a fix is merged.

[joe-boyce]

Great thanks!

[joe-boyce]

Any further updates on this one?

Thanks,

Joe

[shilov]

• looking forward to this as well

[shilov]

@dadgar thanks!

[dadgar]

@shilov Of course! It would be great if you all could test with your setups using 0.6.3-rc1. I tested on GCE using their credential helper.

[alxark]

could you please attach your configuration sample ?

[dadgar]

@alxark It was something like this:

{
  "credHelpers": {
     "gcr.io": "gcr"
  }
}

[neidiom]

As far as I know credHelpers is added to .docker/config.json by running docker-credential-gcr configure-docker

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.