core: Change the CPUPolicyUnit for dedicated cpu pinning #299

ljelinkova · 2022-04-22T12:33:37Z

Before the introduction of the exclusively pinned CPUs, the logic of CPUPolicyUnit allowed to check the filtering
constraints for each VM in the vm group individually. The constraint was that the number of VM's CPUs had to be <=
host CPUs.

With the introduction of the exclusively pinned CPUs, that is no more possible - if the group contained VMs with both, shared and exclusively pinned CPUs, we need to calculate the CPU count constraints for the whole group (similar to huge pages in HugePagesFilterPolicyUnit).

Now the algorithm for calculating if the vm group fits into
the host is as follows:

Calculate the host CPU count
Calculate the currently exclusively pinned CPUs (including pending)
Calculate the sum of all dedicated CPUs of the vm group
For all VMs with shared CPUs, find the max requested shared CPUs

The host can fit the VMs if:
hostCpuCount - takenCpus - addedExclusivelyPinnedCpus - requestedMaxSharedCpuCount >= 0

Note that the calculation of the previous values may differ based on the cluster setting of "Count threads as cores".

ljelinkova · 2022-04-22T13:01:15Z

/ost

ahadas · 2022-04-25T07:20:26Z

Before the introduction of the exclusively pinned CPUs, the logic of CPUPolicyUnit allowed to check the filtering constraints for each VM in the vm group individually. The constraint was that the number of VM's CPUs had to be >= host CPUs.

<=, right?

ahadas

overall looks good, it's much more than refactoring ;)

...es/bll/src/test/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnitTest.java

...es/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/utils/VdsCpuUnitPinningHelper.java

...odules/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnit.java

ljelinkova · 2022-04-25T08:18:01Z

Before the introduction of the exclusively pinned CPUs, the logic of CPUPolicyUnit allowed to check the filtering constraints for each VM in the vm group individually. The constraint was that the number of VM's CPUs had to be >= host CPUs.

<=, right?

Yes

ljelinkova · 2022-04-27T11:38:25Z

/ost

ljelinkova · 2022-04-27T17:00:51Z

/ost

ahadas · 2022-04-28T11:08:24Z

@ljelinkova can you please rebase?

ljelinkova · 2022-04-28T13:13:51Z

/ost

ljelinkova · 2022-04-28T13:30:50Z

/ost

...odules/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnit.java

...les/common/src/main/java/org/ovirt/engine/core/common/businessentities/CpuPinningPolicy.java

...odules/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnit.java

liranr23

Looked on the CpuPolicyUnit and the calls from there. Overall looks good to me :)

liranr23 · 2022-05-12T15:24:53Z

...es/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/utils/VdsCpuUnitPinningHelper.java

+                if (allocatedCpus.isEmpty()) {
+                    return Integer.MAX_VALUE;


why MAX_VALUE?? if it's empty it means the vm can't allocate on this host (maybe it's the first vm or the second). This means these VMs can't run together in this order of scheduling on this specific host.

yeah so I think that's why we return max value here - so the caller will realize that everything is unavailable and the host would be filtered out. worth adding a comment though in that case

@liranr23 Yes, you're right, the MAX_VALUE is just to notify the caller about that. We can throw an exception, but we would need to handle that. Saying that Integer.MAX_VALUE number of CPUs are unavailable allows the caller to use the same calculation for successful and unsuccessful allocations.

@ahadas adding a comment is a good idea

...odules/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnit.java

ahadas

nice!

...ll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CpuTopologyPolicyUnit.java

...es/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/utils/VdsCpuUnitPinningHelper.java

ahadas · 2022-05-15T20:28:29Z

...es/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/utils/VdsCpuUnitPinningHelper.java

+                if (allocatedCpus.isEmpty()) {
+                    return Integer.MAX_VALUE;


yeah so I think that's why we return max value here - so the caller will realize that everything is unavailable and the host would be filtered out. worth adding a comment though in that case

...ll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CpuTopologyPolicyUnit.java

liranr23 · 2022-05-18T13:07:32Z

...es/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/utils/VdsCpuUnitPinningHelper.java

@@ -127,8 +138,9 @@ public List<VdsCpuUnit> updatePhysicalCpuAllocations(VM vm, Map<Guid, List<VdsCp
            return cpusToBeAllocated;
        }

-        filterSocketsWithInsufficientMemoryForNumaNode(cpuTopology, vm, hostId);


we still need it, don't we?

yes, but not for the calculation of the free cpus (because CpuPolicyUnit is not about numa memory). I've just moved it to line 113.

However, your question made me think, if we need two separate cpu policies (cpu count, cpu pinning) now when we are actually doing the same thing in both of them (preview the "pinning"). I think we could merge them into one.

given the changes i did now in #380, we run this allocation: VMs * hosts * 2 (once pinning policy, once CPU policy) + VMs(schedule) .

but: in terms of logical thinking - i think the current implementation make sense, also it saves us big scheduler change.
in terms of code and performance - we should do all the filtering once to reduce the *2 part.

I think it won't be possible to merge CpuUnitPolicy and CpuPinningPolicyUnit into one unit after all. It would be possible if we could rely that all of the hosts do report cpuTopology and we can do manual pinning on the cpuToplogy directly but that might not be true for older hosts. For those, we still need to count with the currently reported online cpus and it makes sense to have separated in CpuPinningPolicyUnit.

ljelinkova

I've updated the patch with your comments and I've also changed the preview pinning in VdsCpuUnitPinningHelper to be as close to the actual pinning as possible. This allowed to get rid of Integer.MAX_VALUE. Now the algorithm is not concerned if the pinning succeeds or not (that should filter out the CpuPinningPolicyUnit.

...odules/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnit.java

Before the introduction of the exclusively pinned CPUs, the logic of CPUPolicyUnit allowed to check the filtering constraints for each VM in the vm group individually. The constraint was that the number of VM's CPUs had to be >= host CPUs. With the introduction of the exclusively pinned CPUs, that is no more possible - if the group contained VMs with both, shared and exclusively pinned CPUs, we need to calculate the CPU count constraints for the whole group (similar to huge pages in HugePagesFilterPolicyUnit). Now the algorithm for calculating if the vm group fits into the host is as follows: 1. Calculate the host CPU count 2. Calculate the currently exclusively pinned CPUs (including pending) 3. Pin the vms that are being schedulled and count how many CPUs will be taken 4. Calculate how many shared CPUs are required to be left on the host as the maximum of required shared CPUs for vmGroup, pending VMs and running VMs. The host can fit the VMs if: hostCpuCount - exclusiveCpus - maxSharedCpuCount >= 0 Note that the calculation of the previous values may differ based on the cluster setting of "Count threads as cores".

ljelinkova · 2022-05-24T12:12:20Z

I've updated the patch with your comments and I've also changed the preview pinning in VdsCpuUnitPinningHelper to be as close to the actual pinning as possible. This allowed to get rid of Integer.MAX_VALUE. Now the algorithm is not concerned if the pinning succeeds or not (that should filter out the CpuPinningPolicyUnit.

... and I put the Integer.MAX_VALUE back to the patch as we need to know if the pinning of the exclusive cpus does not succeed otherwise we can end up taking more shared CPUs than we should.

ljelinkova · 2022-05-24T12:13:50Z

/ost

...odules/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnit.java

...ll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CpuTopologyPolicyUnit.java

ahadas · 2022-05-25T14:34:28Z

@liranr23 anything else from your side?

ljelinkova added the virt label Apr 22, 2022

ljelinkova requested review from ahadas and liranr23 April 22, 2022 12:33

ljelinkova requested review from bennyz, emesika, mwperina, michalskrivanek, oliel, sgratch and didib as code owners April 22, 2022 12:33

ahadas reviewed Apr 25, 2022

View reviewed changes

ljelinkova force-pushed the fix-dedicated-migration branch 2 times, most recently from f7fde23 to 80b73ac Compare April 26, 2022 13:36

ljelinkova changed the title ~~core: Refactoring of CPUPolicyUnit~~ core: Change the CPUPolicyUnit for dedicated cpu pinning Apr 26, 2022

ljelinkova force-pushed the fix-dedicated-migration branch from 80b73ac to effbd00 Compare April 27, 2022 10:33

ljelinkova force-pushed the fix-dedicated-migration branch from effbd00 to d0705be Compare April 27, 2022 16:22

ljelinkova force-pushed the fix-dedicated-migration branch from d0705be to 317c10f Compare April 28, 2022 11:51

liranr23 reviewed May 2, 2022

View reviewed changes

ljelinkova force-pushed the fix-dedicated-migration branch 2 times, most recently from e695ae8 to c8c5afb Compare May 4, 2022 07:53

liranr23 mentioned this pull request May 9, 2022

core: consider dedicated cpus when scoring hosts #252

Merged

ljelinkova force-pushed the fix-dedicated-migration branch from c8c5afb to a42e0fa Compare May 12, 2022 08:23

ljelinkova force-pushed the fix-dedicated-migration branch from a42e0fa to 758498b Compare May 12, 2022 11:08

liranr23 reviewed May 12, 2022

View reviewed changes

ahadas reviewed May 15, 2022

View reviewed changes

ljelinkova force-pushed the fix-dedicated-migration branch from 758498b to 625b774 Compare May 17, 2022 10:16

liranr23 reviewed May 18, 2022

View reviewed changes

ljelinkova force-pushed the fix-dedicated-migration branch from 625b774 to 4a82698 Compare May 23, 2022 13:28

ljelinkova commented May 23, 2022

View reviewed changes

...odules/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnit.java Show resolved Hide resolved

ljelinkova force-pushed the fix-dedicated-migration branch from 4a82698 to bfc1395 Compare May 24, 2022 11:43

ahadas approved these changes May 25, 2022

View reviewed changes

...odules/bll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CPUPolicyUnit.java Show resolved Hide resolved

...ll/src/main/java/org/ovirt/engine/core/bll/scheduling/policyunits/CpuTopologyPolicyUnit.java Show resolved Hide resolved

liranr23 approved these changes May 26, 2022

View reviewed changes

ahadas merged commit a6c17bd into oVirt:master May 26, 2022

ljelinkova deleted the fix-dedicated-migration branch June 15, 2022 11:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: Change the CPUPolicyUnit for dedicated cpu pinning #299

core: Change the CPUPolicyUnit for dedicated cpu pinning #299

ljelinkova commented Apr 22, 2022 •

edited

Loading

ljelinkova commented Apr 22, 2022

ahadas commented Apr 25, 2022

ahadas left a comment

ljelinkova commented Apr 25, 2022

ljelinkova commented Apr 27, 2022

ljelinkova commented Apr 27, 2022

ahadas commented Apr 28, 2022

ljelinkova commented Apr 28, 2022

ljelinkova commented Apr 28, 2022

liranr23 left a comment

liranr23 May 12, 2022

ahadas May 15, 2022

ljelinkova May 17, 2022

ahadas left a comment

ahadas May 15, 2022

liranr23 May 18, 2022

ljelinkova May 18, 2022

liranr23 May 18, 2022

ljelinkova May 23, 2022

ljelinkova left a comment

ljelinkova commented May 24, 2022

ljelinkova commented May 24, 2022

ahadas commented May 25, 2022

core: Change the CPUPolicyUnit for dedicated cpu pinning #299

core: Change the CPUPolicyUnit for dedicated cpu pinning #299

Conversation

ljelinkova commented Apr 22, 2022 • edited Loading

ljelinkova commented Apr 22, 2022

ahadas commented Apr 25, 2022

ahadas left a comment

Choose a reason for hiding this comment

ljelinkova commented Apr 25, 2022

ljelinkova commented Apr 27, 2022

ljelinkova commented Apr 27, 2022

ahadas commented Apr 28, 2022

ljelinkova commented Apr 28, 2022

ljelinkova commented Apr 28, 2022

liranr23 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahadas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ljelinkova left a comment

Choose a reason for hiding this comment

ljelinkova commented May 24, 2022

ljelinkova commented May 24, 2022

ahadas commented May 25, 2022

ljelinkova commented Apr 22, 2022 •

edited

Loading