-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to connect Vault API in 0.9.2 log showing wrong address #5816
Comments
This error is blocking new jobs from starting: |
I rolled back 2 other servers and let them lead. I was not seeing this failure when I tested it on a single server/client instance with a single .hcl config. So I started playing with the 0.9.2 version, first I attempted to just reload the config but I was still seeing the issue in the logs. So then I added the vault stanza to the last file in the config directory exactly how it is in the first file and reloaded it. I checked the logs and it is working
checking nomad agent-info now shows the vault stanza with a token
So I went back to my test environment which is a single nomad instance running in client/server mode. I created a new config2.hcl in the folder with just meta data and reloaded nomad. Immediately I get the error I was expecting:
Just to verify my results I deleted config2.hcl and
|
So looking through the code when we merge we are looking to see if the address is an empty string as seen here. But when we initialize the config we always initialize it as DefaultVaultConfig() which has a value of "https://vault.service.consul:8200". This most likely would apply to ie: file | file vault stanza | config vault value after parsing config_1.vault.merge(config_2.vault) => {addr: "https://vault.service.consul:8200" } I think this would also affect any values that are default loaded across multiple files. And I half verified this via my config. In the primary config I have:
to check:
|
Thank you so much for reporting this issue as well as the detailed investigation and reporting . This was fixed by #5817. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
If you have a question, prepend your issue with
[question]
or preferably use the nomad mailing list.If filing a bug please include the following:
Nomad version
Output from
nomad version
Nomad v0.9.2 (0283266)
Operating system and Environment details
Linux ip-10-REMOVED 4.4.0-1069-aws #79-Ubuntu SMP Mon Sep 24 15:01:41 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Issue
It is logging that it cannot connect to vault.service.consul, but I do not have it configured to use that address at all.
Reproduction steps
Install nomad 0.9.2 following the upgrade guide. I upgrade each instance by installing 0.9.2 then restarting them one at a time.
its run with:
/bin/nomad agent -config /etc/nomad/config.d
There are 3 config files but only 1 with vault stanza in it (first in order)
edit: The vault stanza can be in any file except for the last file that loads in order. If it is in the last file it will work.
I have tried changing it to "https://vault.someaddress.loadbalancer.com:443" as I thought maybe that could be the problem.
Job file (if appropriate)
Nomad Client logs (if appropriate)
I have not updated the clients yet as I was afraid this might cause an issue.
If possible please post relevant logs in the issue.
Logs and other artifacts may also be sent to: nomad-oss-debug@hashicorp.com
Please link to your Github issue in the email and reference it in the subject
line:
Emails sent to that address are readable by all HashiCorp employees but are not publicly visible.
Nomad Server logs (if appropriate)
Jun 11 18:10:34 ip-10-REMOVED nomad[22466]: 2019-06-11T18:10:34.881Z [WARN ] nomad.vault: failed to contact Vault API: retry=30s error="Get https://vault.service.consul:8200/v1/sys/init: dial tcp: lookup vault.service.consul on 127.0.0.1:53: no such host"
The text was updated successfully, but these errors were encountered: