Full image repository sent to docker credential helpers

[whiteley]

Nomad version

Nomad v0.7.1 (0b295d399d00199cfab4621566babd25987ba06e)

Operating system and Environment details

GCE n1-standard-1 instance booted from https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1604-xenial-v20180126 with the following packages installed:

docker-ce_18.02.0~ce~rc1-0~ubuntu_amd64.deb
google-cloud-sdk_186.0.0-0_all.deb

Issue

Nomad is sending the full repository name to the credential helper https://github.com/hashicorp/nomad/blob/master/client/driver/docker.go#L2015

The helper is meant to take the "server address" as described in https://docs.docker.com/engine/reference/commandline/login/#credential-helper-protocol.

The implementation used in the docker cli itself appears to call registry.ConvertToHostname(serverAddress) before calling the credential helper.

This works fine with docker-credential-gcr but not with docker-credential-gcloud which comes in newer versions of the Google Cloud SDK packages.

Reproduction steps

$ echo https://us.gcr.io/PROJECTID/IMAGE/NAME | docker-credential-gcr get
{"ServerURL":"","Username":"oauth2accesstoken","Secret":"blah"}

$ echo https://us.gcr.io/PROJECTID/IMAGE/NAME | docker-credential-gcloud get
ERROR: (gcloud.auth.docker-helper) Repository url [https://us.gcr.io/PROJECTID/IMAGE/NAME] is not supported

Nomad Server logs (if appropriate)

2018/01/30 22:58:52.659702 [ERR] driver.docker: failed pulling container us.gcr.io/PROJECTID/IMAGE/NAME:latest: API error (404): {"message":"pull access denied for us.gcr.io/PROJECTID/IMAGE/NAME, repository does not exist or may require 'docker login'"}

I also opened a ticket about the difference in behavior on the two credential helpers with gcloud support to see if it was intentional and could be documented. I'm still getting familiar with the code base so I apologize if I've misdiagnosed things.

[whiteley]

A couple of other notes in case they help.

$ echo https://us.gcr.io | docker-credential-gcloud get
{"ServerURL":"","Username":"oauth2accesstoken","Secret":"blah"}

This works fine and is what docker pull itself will do successfully after setting up /root/.docker/config.json with the following content generated from gcloud beta auth configure-docker.

{
  "credHelpers": {
    "gcr.io": "gcloud", 
    "us.gcr.io": "gcloud", 
    "eu.gcr.io": "gcloud", 
    "asia.gcr.io": "gcloud", 
    "l.gcr.io": "gcloud", 
    "launcher.gcr.io": "gcloud", 
    "us-mirror.gcr.io": "gcloud", 
    "eu-mirror.gcr.io": "gcloud", 
    "asia-mirror.gcr.io": "gcloud", 
    "mirror.gcr.io": "gcloud", 
    "staging-k8s.gcr.io": "gcloud", 
    "k8s.gcr.io": "gcloud"
  }
}

Nomad fails to pull regardless of whether image is specified with https:// or not which makes sense to me following #3165.

I also tried adding the following to the task configuration without change after seeing it mentioned in #1119.

ssl = true
auth {
  server_address = "us.gcr.io"
}

[chelseakomlo]

Hey, thanks for reporting this issue and the detailed information. We've added this to our team's near-term backlog to fix.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.