System jobs are not rescheduled when resources become available

[maihde]

Nomad version

0.7.1

Operating system and Environment details

Linux

Issue

When a system job is evaluated, if a node has no free resources the task is queued. Later, if resources on the node become available the system task should be allocated.

Reproduction steps

1. Start a Nomad cluster
2. Run jobs to consume all of the CPU resources
3. Run a system job
4. Notice that the system jobs are Queued
5. Stop the job started at step 2
6. Notice the the system jobs remain Queued

Expected behaviour:
System jobs are removed from the Queued state and go into the Running state

Observed behaviour:
System jobs remain in the Queued state.

[maihde]

By porting the blocked evaluation logic from generic_scheduler.go into system_scheduler.go I'm able to get things to work as expected.

https://github.com/maihde/nomad/tree/issue-4072

[dadgar]

@maihde I think you identified the problem correctly. We do want to bring a lot of the improvements from the generic scheduler to the system scheduler. Hopefully we can use some of your work!

[maihde]

@dadgar thanks for the feedback. If you would like a pull-request for my patch, let me know.

[jippi]

@maihde would be nice to get that PR up

[maihde]

@jippi just opened up. Thanks.

[mwalters-workmarket]

Is there a timeline for the system scheduler rework that will include the fix for this?

[maihde]

@jippi @mwalters-workmarket my original pull request was closed because there was a major refactor planned to the schedulers and it was deemed easier to start afresh. If this feature is still on your roadmap I'd be happy to implement the feature against the current master and provide a new pull request.

[burdandrei]

would be cool to revive this

[notnoop]

Following up here late, sorry. Is this still a relevant issue? I believe Nomad 0.9.4 fixed this issue with #5900 . Can someone confirm?

[notnoop]

Closing this ticket, as it seems fixed and I'm unable to reproduce it now. Please re-open or open a new one if you believe this to be an error. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.