CPU hard limit in Nomad - Fix

[jaininshah9]

2 weeks ago, I created a PR which allows users to pass a flag via which you can enable cpu_hard_limit.

PR: #3825

At that time, the change looked good to me and it was merged into master.

We added this commit in our Nomad version we have running and found out that it is not working as expected. The CPU usage gets constraint and never uses the full capacity.

Reproduction steps

1. Run nomad in a multi-core (4 or 8) system and try to add a task that requires 200% CPU. Ideally, you should be able to create that, but currently, you cannot do that.

We have identified a fix for this

Fix:

Below is the change we need in the formula for calculating the CPUQuota to fix that issue.

With the fix, if we have a 16 CPU @ 2.7 GHz and want to add multiple tasks which use 100% CPU, 200%, CPU, 400% CPU and 50% CPU, we make the CPU value as below:

100% CPU - 2700
200% CPU - 5400
400 CPU - 10800
50% CPU - 1350

Nomad version

master branch

Operating system and Environment details

Linux - Ubuntu and Centos

Thought??

[jaininshah9]

cc: @schmichael @diptanu

[jaininshah9]

Here is the doc for cpu resources:
Ref link: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cpu

the cpu.cfs_period_us

Note that the quota and period parameters operate on a CPU basis. To allow a process to fully utilize two CPUs, for example, set cpu.cfs_quota_us to 200000 and cpu.cfs_period_us to 100000.

[schmichael]

Thanks again @jaininshah9 !

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.