v0.10.4 - Shutdown Delay not working as expected #7251
Comments
|
@drewbailey this is related to the work in #6746, I don't think I'm missing anything in my job config, but yea, it does not seem to be working as expected, I'm not sure why. |
|
Hey @djenriquez thanks for reporting, There seems to be a few things going on so I wanted to share a reproduction job file to discuss the different scenarios around the reproduction file. repro.hcljob "delay" {
datacenters = ["dc1"]
group "api" {
# shutdown_delay = "30s"
meta {
foo = "bar"
}
service {
tags = ["group"]
}
spread {
attribute = "${node.datacenter}"
target "dc1" {
percent = 100
}
}
task "web" {
driver = "docker"
# shutdown_delay = "10s"
config {
image = "hashicorpnomad/counter-api:v1"
}
service {
tags = ["awesome"]
}
}
task "dashboard" {
driver = "docker"
shutdown_delay = "60s"
env {
COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
}
config {
image = "hashicorpnomad/counter-dashboard:v1"
}
service {
tags = ["dashboard-service"]
}
}
}
}
edit: edit 2: |
|
Hi @drewbailey, yes, sorry there might have been some confusion in my reporting. The However, you clarified that Does your edit no. 2 allow for tasks' shutdown_delay to go into effect regardless of service registration? |
|
Interestingly, I'm looking at your |
It would be great to have |
|
@danlsgiga a really really bad hack around right for now would actually be to set the shutdown signal as something that wont terminate the process, then use the Edit: Nevermind, you said batch job. So that'll run to completion by itself. |
|
yeah, didn't think about that option. Thanks for that... but I prefer to wait for the fix because... you know... if I set that hack, it will be there forever 😄 |
|
Also having the same issue, does shutdown_delay only apply to service type jobs? I have a system job which is not using the shutdown_delay |
|
Hey there Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this. Thanks! |
tgross
moved this from Needs Triage
to Needs Roadmapping
in Nomad - Community Issues Triage
|
Having the same issue. UPD shutdown_delay at group level works though |
|
UPD shutdown_delay at group level does not work as expected. After canary promotion instead of:
I have:
That is a huge disappointment since shutdown_delay is vital for us to let external LBs to update configs with consul template. |
|
@tgross Sorry to bother you, but is there anything to do with this issue? I am willing to help with any investigations if needed |
|
@sashayakovtseva this issue probably should have been closed when #7663 was merged for 0.11.1. What you're describing isn't what this ticket is about (not respecting |
|
I've double checked this issue again. Task level |
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Server and Clients both on v0.10.4.
Operating system and Environment details
Issue
ShutDownDelay as defined in the Nomad docs is not being considered during deployments. Below is a jobfile that describes two tasks within a taskgroup with different ShutDownDelays. However, during deployment, it doesn't seem like they are considered based on two pieces of evidence:
Here we have two allocations representing a deployment. Version 18 was created at 13:28:51, so I could expect that the task with the 60s ShutDownDelay will not exit until 13:29:51 at the earliest.
However, if we look at the allocation, this one is the proxy specifically, which has the 60s shutdown delay:
This shows that the task received the shutdown signal 10s after 18 was created. This means that it completely ignored the task's shutdown delay OR it used the app's shutdown delay, which was 10s.
Now here's the app's task in that same allocation:
This should not be happening if the tasks' shutdown delays are truly considered.
Also, it's worth mentioning, when the TaskGroup's shutdown delay is updated, job plans do not detect the change:
When changing the TaskGroup's ShutdownDelay from
nullto60000000000, the plan shows this:Lastly, even though the tasks have shutdowndelays, shutdowndelays seem to be completely ignored until the group's shutdowndelay was defined, allocations were getting their shutdown signal immediately. After only adding a 10s shutdowndelay to the group did I notice delays, but it did not trickle down to the tasks. Reverting it back to null kept the allocations with the 10s before i reverted it back to
null.Reproduction steps
Metamap)Throughout all of this, watch as the task's shutdown delays are never considered.
Job file (if appropriate)
The text was updated successfully, but these errors were encountered: