-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad clients not honoring docker auth options? #2957
Comments
What is the name of the gcr helper binary. It should be named |
It's there, with that name and available in $PATH. To rule out that issue, I renamed the binary as a test and sure enough Nomad reported an error about |
@shilov Hmm, thanks. Will try to spin up a cluster on GCE and debug. Are there anything in the client logs that look useful? |
Nothing insightful, other than the error I shared earlier. I'll try enabling debug logging and report back if anything valuable shows up. |
Facing the same problem now. Docker config:
nomad client options:
task config block:
Error message:
|
I've been seeing the same issue, anybody able to take a look? |
Hi @joe-boyce, thanks for confirming this. I picked up this issue today. I will keep this ticket updated when a fix is merged. |
Great thanks! |
Any further updates on this one? Thanks, Joe |
|
This PR fixes the server url passed to docker credential helpers and fixes stderr capture. Fixes #2957
This PR fixes the server url passed to docker credential helpers and fixes stderr capture. Fixes #2957
@dadgar thanks! |
@shilov Of course! It would be great if you all could test with your setups using 0.6.3-rc1. I tested on GCE using their credential helper. |
could you please attach your configuration sample ? |
@alxark It was something like this:
|
As far as I know credHelpers is added to |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Summary
Nomad clients are not able to pull private docker images from Google's container repository.
Clients seems to ignore the value provided in "docker.auth.config" or "docker.auth.helper".
For authorization, I'm using https://github.com/GoogleCloudPlatform/docker-credential-gcr which works as expected when running
docker pull
manually.Nomad version
v0.6.0
Operating system and Environment details
OS: Debian 9
Docker credentials are handled via docker-credential-gcr v1.4.1 (current release)
Issue
Pulling a private docker image using
docker pull
works as anticipated.Nomad client reports auth errors attempting to pull the same image.
Reproduction steps
Nomad client config:
Docker config (/root/.docker/config.json):
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
Job file (if appropriate)
Job file consists of a bare-bones docker task. The image is private, hosted on
gcr.io
, Google's container repository.I've tried specifying the docker image both as
gcr.io/google-project/foo
as well ashttps://gcr.io/google-project/foo
, with and without the docker tag, but the end result was always the same (auth error).The job file does not specify any docker auth values. I did try providing the docker auth
server_address
(using "gcr.io" as the value) but it made no difference.Curiously, when I tried providing the docker auth
server_address
in the job description, the nomad client logs included the complete docker image name with tag. Without the server_address, the client logs only contained the image name, without the associated tag. Probably unrelated, but I figured it's worth mentioning.The text was updated successfully, but these errors were encountered: