Not reaching vault makes nomad re-schedule all allocations #2689
Comments
jorgemarey
changed the title
|
Hey @jorgemarey, Looking at the logs it appears that the allocations are being restarted because the nodes are unreachable by the Nomad servers (we use heartbeating to detect if clients are alive). You can see this in the server logs: When Vault is down, Nomad will not start rescheduling allocations. What the client will do is attempt to keep renewing the token until it expires. After that it will attempt to retrieve new token and once it successfully has it will take action based on the config vault change_mode configured. I just tried this and you get continued renewal attempts as expected. The logs start right after taking Vault down: So I think something else is going on in your test environment! |
|
Hi @dadgar, thank you for your answer. At first I thought that too. That allocations are being restarted due to the nodes being unreachable (as is seen in the logs), but it felt weird that all of the nodes had lost contact at the same time. Then I tried to cut the commucation between the servers and vault and it happen again. The behaviour that I was expecting is what you explain. I'll try to see if there's something else going on (maybe is something weird in my environment). Thanks a lot. |
|
Hi @dadgar, Sorry to bother you again with this, but I keep seing this behaviour on the cluster that I have set up. As you said, allocations are being restarted because the nodes are unreachable by the Nomad servers. That is what is happening with the allocations. The thing is that I don't know why all nodes are set as unreachable at the same time, when there's a problem reaching vault. If I block the connection to vault, all nodes are marked as down by the server. I was able to reproduce this (althought not on my laptop with the dev server). Every time I do that all nodes are being set as unreachable within seconds. Some logs (afther blocking the connection with vault). The nomad server leader: These messages keep happening until I enable the connection with vault. Node 283e35b3-d42c-4b38-8864-4cc12c9dfa63: Node 40625c55-98b6-4e4e-8a45-2b9ac171fdd1: The rest of the nodes have the same kind of logs. I found weird the following: Maybe I have some problem on my configuration, the set up, or other stuff, but keep seing this (nodes unreacheable) every time there's network problem reaching vault. Can you think of something that could trigger this kind of behaviour? Thanks! |
|
@jorgemarey I will reopen and try to reproduce post 0.6.0 and if it is happening, I will get a fix for 0.6.1! |
|
Thanks a lot! I'll upgrade when it's released and test this. |
|
Hi @dadgar I just noticed this again on a cluster with nomad v0.6.2, consul 0.9.2 and vault 0.8.2. We just had a problem with consul (used as storage for vault) and at that time we started seeing all allocations being re-scheduled. Some logs: It seems the same problem. When our nomad cluster can't reach vault correctly it marks all nodes as down. I don't know why this happens, maybe it's a problem of our set-up... |
|
@jorgemarey Sorry haven't had time to reproduce this yet. Would you mind trying to distill it down to as simple of reproduction steps as possible? |
|
@jorgemarey I got this to reproduce! Thanks! |
dadgar
added a commit
that referenced
this issue
Sep 12, 2017
This PR removes locking around commonly accessed node attributes that do not need to be locked. The locking could cause nodes to TTL as the heartbeat code path was acquiring a lock that could be held for an excessively long time. An example of this is when Vault is inaccessible, since the fingerprint is run with a lock held but the Vault fingerprinter makes the API calls with a large timeout. Fixes #2689
|
Awesome! Thanks a lot @dadgar ! |
dadgar
added a commit
that referenced
this issue
Sep 14, 2017
This PR removes locking around commonly accessed node attributes that do not need to be locked. The locking could cause nodes to TTL as the heartbeat code path was acquiring a lock that could be held for an excessively long time. An example of this is when Vault is inaccessible, since the fingerprint is run with a lock held but the Vault fingerprinter makes the API calls with a large timeout. Fixes #2689
dadgar
added a commit
that referenced
this issue
Sep 14, 2017
This PR removes locking around commonly accessed node attributes that do not need to be locked. The locking could cause nodes to TTL as the heartbeat code path was acquiring a lock that could be held for an excessively long time. An example of this is when Vault is inaccessible, since the fingerprint is run with a lock held but the Vault fingerprinter makes the API calls with a large timeout. Fixes #2689
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Hi,
I noticed that sometimes all the allocations that are placed on my nomad cluster are re-scheduled simultaniously. This happens very rarely, but it does. I suspect that is related to the Vault integration.
The setup is the next:
3 instances with nomad server and consul agent
3 instances with consul server and nomad client
a buch of other instances with consul agent and nomad client.
Im running consul 0.7.5 and nomad 0.5.6
On other instance I have vault (just one server) and I have nomad configured to use that vault.
The log output when this happens:
logs-client1.txt (client with some allocations)
logs-sched1.txt
logs-sched2.txt
logs-sched3.txt (leader)
This happens when vault is not reachable, maybe due to some kind of network issue. Then nomad starts reescheduling all allocations (all of them were runnning fine).
I'm able to reproduce this behaviour, just by setting a security group on the vault machine denying the access to the port where vault is running.
I don't know if this behaviour is expected, but it doesn't feel right that due to a moment of not reaching vault correctly nomad should start moving allocations around.
The text was updated successfully, but these errors were encountered: