-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improvement: flag to wait for node to be drained #2736
Comments
Would be super nice indeed |
That is the behaviour I actually expect when draining a cluster node of some sort. Nomad surprised me the first time I drained a node, when it stopped all jobs and re-allocated them. |
@pznamensky What do you mean by "terminate |
I'd like to be able to tell a node not to take any new task allocations. As almost all our services are deployed continously, within a couple of days the node set to drain would be empty and ready for maintenance work. |
What about existing allocations on the draining node? For long-running services, don't you need to terminate them? |
No, no action except moving allocations on job update/scheduling. Say there's a couple of tasks left closing in on a maint. window, we'll just re-schedule those jobs again to empty the node and complete the drain. Personally I would like to see:
2 "node-drain -stop" would pull the plug right away and re-allocate task immediately i.e current behaviour. That's more in line with my experience of resource/connection "brokering" products like loadbalancers and such. Does it make sense? :) |
I've done something over in seatgeek/nomad-helper that is similar to whats being asked here.. The code basically wait for all allocations on the node to be in a state that isn't binaries are available for linux, darwin and windows |
I mean enable drainage and keep the command
Great! Thanks! |
I'm handling it like this:
and here is the nomad-drain-wait:
|
Mine's a little more … ah … complicated. https://gist.github.com/blalor/246eaf5755e784b353ab756a36a1142e I make a best effort to ensure that allocations running on the local node are started up elsewhere, so that any Even with all of this jiggery-pokery, systemd still doesn't always shut Nomad down cleanly when stopping the instance. |
Also, this is basically the same thing as #2052. |
@blalor thats a lot of code! :P feel free to use my Go binary instead :) |
@jippi It is, but it works, and I wrote it 7 months ago… :-) Yours doesn't wait for evals triggered by the node draining to complete, and that's an important piece of making sure the |
@blalor I had the understanding that once the allocation is in stop mode, all migrations has been completed too - that |
My experience is that that is not the case. The old allocation will become stopped, but the new one is in pending until the data's migrated. [edit] as of 0.5.6. |
This would be very useful for my use case as well. I think a workaround using |
@jippi I also have noticed that it does not wait for the other allocations to hit run state. The new allocations are in a pending as they pull from Docker Hub, and there is a period where it stops the allocs before the new ones come up. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
I guess it would be useful not only for us:
what do you think about adding a flag like
-wait-for-drain
that will terminatenomad node-drain
only when there were no more jobs on a node?Something like:
nomad node-drain -enable -self -wait-for-drain
It would be useful as a part of start/stop scripts at a pre stop step:
The text was updated successfully, but these errors were encountered: