-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
clients require node:read on anon ACL policy to join cluster via Consul discovery (1.5.1) #16470
Comments
Hi @CarelvanHeerden! I'm pretty sure this has to do with #16217 where the node is now hitting our If you tighten the anonymous policy to just the following: node {
policy = "read"
} That will tighten up the policy a bit. But obviously we need to fix that. I'm at the end of my day here but I'll circle back to this tomorrow to see what the best approach to fix is. |
Thanks so much @tgross |
JFTR for those Googling, this problem happens when updating from 1.5.0 to 1.5.1, too. |
I've retitled the issue to reflect the specific problem and workaround; it's not related to upgrades per se and if you start a new cluster from scratch on 1.5.1 you'll hit the same problem. I'll have a patch up later today and we'll get that out in the next patch release of Nomad. Some notes:
|
I've got a draft PR #16490 up with the fix. It turned out to be a little more complicated than I expected, because the problem wasn't simply that we didn't add the client secret but that the client secret isn't meaningful to the server at that point -- we haven't registered yet! I've got another round of end-to-end testing to wrap up tomorrow morning with this and then it should be ready to land. |
For what its worth, attempting to manually join a 1.5.1 client to my 1.5.1 server cluster using either Defining a |
Still working on #16490, as it's just been a bit more complicated than we wanted. The resulting changes should actually reduce round-trips to the server as part of client startup though.
The |
#16490 has been merged and will ship in the next patch version. We're still working on figuring out the schedule for that. |
Nomad version
v1.5.1
Operating system and Environment details
Ubuntu 20.04
Issue
Our Test system was running v1.4.3, with ACL system configured. Upgraded to V1.5.1 using a rolling upgrade (replacing all servers and clients)
The servers started up correctly and joined the cluster.
The clients however could not join, getting a RPC error: Permission denied
The only way to get the clients to Join, was to add back the Anonymous policy
Client Config
etc/nomad.d/acl.hcl
/etc/nomad.d/client.hcl
/etc/nomad.d/nomad.hcl
Server Config
etc/nomad.d/acl.hcl
/etc/nomad.d/server.hcl
/etc/nomad.d/nomad.hcl
CLI output
nomad acl token list
Reproduction steps
Upgrade working Nomad Cluster from 1.4.3 to 1.5.1
Expected Result
Clients should be able to join the cluster without the need for the Anonymous policy
Actual Result
Clients unable to join the cluster
Nomad Client logs (if appropriate)
The text was updated successfully, but these errors were encountered: