You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
10 Clients (3 are also servers) with the below spec:
$ nomad node status <NODE_ID>
Allocated Resources
CPU Memory Disk
169698/332800 MHz 91 GiB/126 GiB 1.5 TiB/3.2 TiB
Allocation Resource Utilization
CPU Memory
77229/332800 MHz 36 GiB/126 GiB
Host Resource Utilization
CPU Memory Disk
77324/332800 MHz 47 GiB/126 GiB 291 GiB/3.4 TiB
Issue
I have a job with 780 groups with 2 tasks inside each group.
When all the tasks enter a restart loop (all crashing together) Nomad doesn't handle it well.
Reproduction steps
Run the below Job file and enter all tasks into a restart loop.
Expected Result
Nomad should handle the restart loop and not get stuck.
Actual Result
UI - all tabs are stuck and often show server error message
CLI - no operations work with the below errors:
Unexpected response code: 500 (rpc error: Not ready to serve consistent reads
evaluation reached delivery limit (3)
[ERROR] nomad.raft: failed to take snapshot: error="cannot take snapshot now, wait until the configuration entry at 125550 has been applied (have applied 123884)"
[ERROR] worker: error waiting for Raft index: worker_id=04faa162-292c-a943-b8ec-ca3b9af37be1 error="timed out after 5s waiting for index=125843" index=125843
error fetching node stats: actual resource usage not present
nomad: memberlist: Refuting a suspect message
nomad: yamux: failed to send ping reply: session shutdown
I suspect the size of the final job HCL containing 780 groups is the issue.
I'm not sure how the HCL size is related to the raft indices but many errors are raft-related.
Worth Mentioning
I removed the Connect blocks from the groups to reduce the number of containers, health-checks and Consul-related operations
When tasks are not restarting it takes a few minutes to schedule and finally works well.
All operations including $ nomad system gc, $ nomad system reconcile summaries, $ nomad stop <job_id> -purge are throwing the mentioned errors with no ability to repair the state (even manually).
I was under the assumption that this would fix it.
I know Nomad was tested in the 2M containers challenge, so in terms of the containers count I think I'm good 🙂
but I wonder if a restart loop of multiple containers is something that was tested and the scheduler should be able to handle.
Questions
Is there a limit for the number of groups in a single job spec? if so, what is the limit?
In case I reach this vicious cycle, is there a way to ask the cluster to stop all scheduling operations (mainly evaluations) - some "super admin" action so my operator actions take precedence and bypass all existing requests in the queue?
The text was updated successfully, but these errors were encountered:
Thanks for reporting, @Dgotlieb. There is no hard limit on the number of groups that can exist in a job spec, however as you've found you may end up overwhelming the ability to process the amount of evaluations that get created in a pathological case like all of those tasks entering a crash loop. There's been some work around load shedding of evaluations recently; we'll want to see if any of that applies to this case or if we need to do something extra.
OK, I will also add that once the job has stopped, I still saw errors in the cluster logs, servers disappearing from the UI/CLI, Failures in Consul health checks for ports 4646/4647/4648, and other issues which could take hours to get resolved on their own.
With regard to my second question, is there any way to bypass all eval/schedule requests in the cluster queue to prevent the cluster from entering this vicious cycle?
Can you please explain what is the relation between the evals size/frequency and the raft as most of the errors are raft-related?
Can you suggest a version/branch to try out?
Is there any way to set max parallel restarts (cluster-wide), so even in a case when 1000's of tasks are restarting together, something in the cluster will limit those restarts (e.g: server_max_parallel_restarts)
Nomad version
Nomad v1.4.3 (f464aca)
Operating system and Environment details
Infra resources
10 Clients (3 are also servers) with the below spec:
Issue
I have a job with 780 groups with 2 tasks inside each group.
When all the tasks enter a restart loop (all crashing together) Nomad doesn't handle it well.
Reproduction steps
Run the below Job file and enter all tasks into a restart loop.
Expected Result
Nomad should handle the restart loop and not get stuck.
Actual Result
UI - all tabs are stuck and often show server error message
![image](https://user-images.githubusercontent.com/12662834/208384403-2eaeb4fc-6e00-47a8-b412-2c1952b13ef3.png)
CLI - no operations work with the below errors:
[ERROR] nomad.raft: failed to take snapshot: error="cannot take snapshot now, wait until the configuration entry at 125550 has been applied (have applied 123884)"
Job file
Server.hcl
Client.hcl
Suspect
I suspect the size of the final job HCL containing 780 groups is the issue.
I'm not sure how the HCL size is related to the raft indices but many errors are raft-related.
Worth Mentioning
Connect
blocks from the groups to reduce the number of containers, health-checks and Consul-related operations$ nomad system gc
,$ nomad system reconcile summaries
,$ nomad stop <job_id> -purge
are throwing the mentioned errors with no ability to repair the state (even manually).but I wonder if a restart loop of multiple containers is something that was tested and the scheduler should be able to handle.
Questions
The text was updated successfully, but these errors were encountered: