-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to pull images from AWS ECR with docker driver in on-prem instances #10722
Comments
Hi @AlexITC! "Approach 4" is close, but What you'll need in the client configuration is the configuration of the credential helper. See the plugin "docker" {
config {
auth {
# Nomad will prepend "docker-credential-" to the helper value and call
# that script name.
helper = "ecr-login"
}
}
} |
Ok, that's another approach I forget to mention (which is documented in the official docs), this one fails to pull public images. nomad client log:
|
Can you try that with |
That seems to made the trick, thanks! By now it was difficult to know what I haven't tried, besides the way this got solved, shouldn't the |
It probably should, but keep in mind we're not running |
Thanks! |
Unfortunately, I tried replicating the setup in a new VM with the comments from this thread, it won't work, the problem seems to be the same, nomad doesn't invoke |
Actually, following the comments and using Approach 3 seems to do the trick, do you think this should be necessary? |
Yeah, it looks like Unit file:
Test script: #!/usr/bin/env bash
env Output:
|
Makes sense, I think it is worth documenting this behavior, I'm definitely not the only one being bitten by it, do you think that this part is adequate to highlight the issue? if so, I can submit a PR about it. |
Yeah I think the right place for it would be where we have the specific example of the ECR helper:
Maybe leave a parenthetical there saying something like "you may need to set $HOME in your Nomad environment, see awslabs/amazon-ecr-credential-helper#161" Also, weirdly I somehow missed that what I said here is wrong:
It's right at https://www.nomadproject.io/docs/drivers/docker#config |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.1.0 (2678c3604bc9530014208bc167415e167fd440fc)
Operating system and Environment details
On premise Ubuntu 20.04 AMD x64.
Issue
My jobs are unable to pull images from ECR, what seems to be the issue is that even after following the steps to configure aws cli, the credentials aren't being picked by nomad while pulling the image.
For example, I'm running the nomad client as
root
user, which has the aws credentials stored at/root/.aws/credentials
(HOME
being/root
).While reading from the hashicorp forum, github issues, and nomad's gitter, I have tried many different approaches but everything seems to lead to the same issue, nomad is not supporting the aws authentication mechanism.
I haven't checked nomad source to find why this is failing but I'm specified the approaches I have explored.
Approach 1
Use
docker login
with ECR credentials.Running this command as root gets a token, allowing
docker pull
but nomad fails due to missing credentials, it seems that it is not using the/root/.docker/config.json
settings which include the token:aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin [account].dkr.ecr.us-east-2.amazonaws.com
Approach 2
Set env variables that hold the AWS credentials in the systemd service that runs nomad
Update
nomad.service
to includeEnvironmentFile=/root/.env
, I found out this in comments from other users, unfortunately, it hasn't worked for me..env
:Approach 3
Set
HOME=/root
env variable in the systemd service that runs nomad.Update
nomad.service
to includeEnvironment=HOME=/root
, I found out this in comments from other users, unfortunately, it hasn't worked for my case.Approach 4
Use docker-credential-ecr-helper while storing the sdk credentials at
/root/.aws/credentials
, installing it by either of these:sudo apt install amazon-ecr-credential-helper
, printing0.6.0
as the version.https://github.com/awslabs/amazon-ecr-credential-helper/releases/tag/v0.5.0
, printing0.6.3
as the version.This requires the nomad client config (like
client.hcl
) to include this snippet:While
/etc/docker-auth.json
has:While
echo "[account].dkr.ecr.us-east-2.amazonaws.com/[image]" | docker-credential-ecr-login get
works, when nomad tries pulling the image it fails with:Reproduction steps
It is difficult to provide these steps because a private registry is involved.
Expected Result
The job involving images from private ECR would work.
Actual Result
The job always fails due to the images not being pulled from ECR.
Job file (if appropriate)
The text was updated successfully, but these errors were encountered: