You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The update stanza controls rolling updates and canary deployments. A task group's reschedule stanza does not take affect during a deployment
However, with the reproduction steps & file listed, a failed set of canary allocations get rescheduled according to the reschedule stanza, but do not register as canary allocations. This causes the previous deploys healthy tasks to be marked as stop during reconciliation.
Reproduction steps
using a local cluster with 3 server nodes, 2 client nodes
# run first job
nomad job run repro.hcl
# allow deploy to become healthy
nomad job run repro-fail.hcl
# observe rescheduled eval eventually kill off healthy allocations
ID = failing-nomad-test01
Name = failing-nomad-test01
Submit Date = 2020-01-13T15:35:51-05:00
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
group 0 0 4 4 0 0
Future Rescheduling Attempts
Task Group Eval ID Eval Time
group d8ddc12d 2s from now
Latest Deployment
ID = 86d8d83f
Status = running
Description = Deployment is running but requires manual promotion
Deployed
Task Group Promoted Desired Canaries Placed Healthy Unhealthy Progress Deadline
group false 4 4 4 0 4 2020-01-13T15:45:51-05:00
Allocations
ID Node ID Task Group Version Desired Status Created Modified
78bc2bb6 303bd631 group 1 run failed 30s ago 26s ago
164b81e5 303bd631 group 1 run failed 30s ago 26s ago
203a01c6 303bd631 group 1 run failed 30s ago 26s ago
497ae378 303bd631 group 1 run failed 30s ago 26s ago
18df0793 27ce5ad0 group 0 run running 55s ago 40s ago
2949e33b 27ce5ad0 group 0 run running 55s ago 36s ago
0e191e2a 27ce5ad0 group 0 run running 55s ago 38s ago
e5f6b54c 27ce5ad0 group 0 run running 55s ago 37s ago
Newly placed allocations cause original healthy allocs to be marked as stop
ID = failing-nomad-test01
Name = failing-nomad-test01
Submit Date = 2020-01-13T15:35:51-05:00
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = pending
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
group 0 0 1 8 3 0
Future Rescheduling Attempts
Task Group Eval ID Eval Time
group 289de16b 58s from now
Latest Deployment
ID = 86d8d83f
Status = running
Description = Deployment is running but requires manual promotion
Deployed
Task Group Promoted Desired Canaries Placed Healthy Unhealthy Progress Deadline
group false 4 4 8 0 8 2020-01-13T15:45:51-05:00
Allocations
ID Node ID Task Group Version Desired Status Created Modified
5824276a 303bd631 group 1 run failed 4s ago 0s ago
19cb409a 303bd631 group 1 run failed 4s ago 0s ago
8c70b8cc 303bd631 group 1 run failed 4s ago 0s ago
8c5e464e 27ce5ad0 group 1 run failed 4s ago 0s ago
497ae378 303bd631 group 1 stop failed 36s ago 4s ago
78bc2bb6 303bd631 group 1 stop failed 36s ago 4s ago
164b81e5 303bd631 group 1 stop failed 36s ago 4s ago
203a01c6 303bd631 group 1 stop failed 36s ago 4s ago
2949e33b 27ce5ad0 group 0 stop running 1m1s ago 0s ago
0e191e2a 27ce5ad0 group 0 stop complete 1m1s ago 0s ago
18df0793 27ce5ad0 group 0 stop complete 1m1s ago 0s ago
e5f6b54c 27ce5ad0 group 0 stop complete 1m1s ago 0s ago
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Nomad version
0.10.3-dev
Operating system and Environment details
ubuntu 19.10
Issue
Rescheduling during deployments doc claims that
However, with the reproduction steps & file listed, a failed set of canary allocations get rescheduled according to the reschedule stanza, but do not register as canary allocations. This causes the previous deploys healthy tasks to be marked as stop during reconciliation.
Reproduction steps
using a local cluster with 3 server nodes, 2 client nodes
Failed canaries, future reschedule occurring shortly
Newly placed allocations cause original healthy allocs to be marked as stop
Job file (if appropriate)
repro.hcl
repro-fail.hcl
originally reported issue #6864
The text was updated successfully, but these errors were encountered: