Add the AlignCPUs enabler #2695

orelmisan · 2023-12-27T14:50:32Z

What this PR does / why we need it:
PR kubevirt/kubevirt#10872 introduced a cluster-wide setting to enable KubeVirt to request up to two additional dedicated CPUs in order to complete the total CPU count to an even parity when using emulator thread isolation.

In order to enable this functionality, in the Kubevirt CR:

The AlignCPUs FG should be enabled.
The alpha.kubevirt.io/EmulatorThreadCompleteToEvenParity annotation should be set with a value of an empty string.

Introduce a new enabler to HCO's API in order to toggle this behavior in KubeVirt.

The new API field is optional and defaults to false.

This PR replaces #2685.

Reviewer Checklist

Reviewers are supposed to review the PR for every aspect below one by one. To check an item means the PR is either "OK" or "Not Applicable" in terms of that item. All items are supposed to be checked before merging a PR.

Jira Ticket:

https://issues.redhat.com/browse/CNV-31584

Release note:

Add the AlignCPUs enabler

PR kubevirt/kubevirt#10872 introduced a cluster-wide setting to enable KubeVirt to request up to two additional dedicated CPUs in order to complete the total CPU count to an even parity when using emulator thread isolation. Introduce a new enabler to HCO's API in order to toggle this behavior in KubeVirt. The new API field is optional and defaults to false. The logic behind it will be introduced in the following commits. Signed-off-by: Orel Misan <omisan@redhat.com> Signed-off-by: Ram Lavi <ralavi@redhat.com>

kubevirt-bot · 2023-12-27T14:50:34Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2023-12-27T14:50:35Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

orelmisan · 2023-12-27T14:51:28Z

/cc @nunnatsa

coveralls · 2023-12-27T14:56:05Z

Pull Request Test Coverage Report for Build 7346808755

22 of 22 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.05%) to 86.482%

Totals
Change from base Build 7346384711:	0.05%
Covered Lines:	5310
Relevant Lines:	6140

💛 - Coveralls

nunnatsa

Thanks for the PR @orelmisan!

Added some inline comments.

nunnatsa · 2023-12-27T15:02:18Z

api/v1beta1/hyperconverged_types.go

+	// Note: this feature is in Developer Preview.
+	// +optional
+	// +kubebuilder:default=false
+	// +default=false


Don't we want it true by default?

Since its an alpha "feature" in KubeVirt, IMO it should be explicitly enabled.

Furthermore, users of this feature probably also set the defaultRuntimeClass field, so they already need to edit the HCO manifest.
Both changes could be done in a single patch command.

nunnatsa · 2023-12-27T15:05:19Z

controllers/operands/kubevirt.go

@@ -303,6 +310,30 @@ func NewKubeVirt(hc *hcov1beta1.HyperConverged, opts ...string) (*kubevirtcorev1
 	return kv, nil
 }

+func isAnnotationStateMeetingRequirements(hc *hcov1beta1.HyperConverged, annotations map[string]string) bool {


Not sure about the name of this function. I find the name both a bit too long, and too generic.

Are there alternative names you have in mind?

nunnatsa · 2023-12-27T15:13:11Z

controllers/operands/kubevirt.go

+	_, found := annotations[kubevirtcorev1.EmulatorThreadCompleteToEvenParity]
+
+	if (hc.Spec.FeatureGates.AlignCPUs == nil || !*hc.Spec.FeatureGates.AlignCPUs) && found {
+		return false
+	} else if (hc.Spec.FeatureGates.AlignCPUs != nil && *hc.Spec.FeatureGates.AlignCPUs) && !found {
+		return false
+	}
+
+	return true


I think what we really want to check here is that (required KV's annotation exists) XOR (found KV's annotation exists). I mean - here I don't care if the annotation is reuired or not. This is already done in NewKubeVirt (bysetAnnotationsToReqState).

Good idea!
Done.

nunnatsa · 2023-12-27T15:15:21Z

controllers/operands/kubevirt.go

-	if !reflect.DeepEqual(found.Spec, virt.Spec) ||
-		!reflect.DeepEqual(found.Labels, virt.Labels) {
+
+	isUpdateRequired := !reflect.DeepEqual(found.Spec, virt.Spec) ||


Since we only check the EmulatorThreadCompleteToEvenParity annotation, and not all the annotations, I think we don't need a new variable here, and we can use a regular if condition. It is not been used after the if.

orelmisan · 2023-12-27T15:48:06Z

Change: Answered review comments.

hco-bot · 2023-12-27T16:58:12Z

okd-hco-e2e-operator-sdk-gcp lane succeeded.
/override ci/prow/okd-hco-e2e-operator-sdk-aws
hco-e2e-upgrade-prev-operator-sdk-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-operator-sdk-sno-azure

kubevirt-bot · 2023-12-27T16:58:16Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-prev-operator-sdk-sno-azure, ci/prow/okd-hco-e2e-operator-sdk-aws

In response to this:

okd-hco-e2e-operator-sdk-gcp lane succeeded.
/override ci/prow/okd-hco-e2e-operator-sdk-aws
hco-e2e-upgrade-prev-operator-sdk-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-operator-sdk-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2023-12-27T17:17:52Z

hco-e2e-operator-sdk-sno-azure lane succeeded.
/override ci/prow/hco-e2e-operator-sdk-sno-aws
okd-hco-e2e-upgrade-operator-sdk-gcp lane succeeded.
/override ci/prow/okd-hco-e2e-upgrade-operator-sdk-aws

kubevirt-bot · 2023-12-27T17:17:56Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-operator-sdk-sno-aws, ci/prow/okd-hco-e2e-upgrade-operator-sdk-aws

In response to this:

hco-e2e-operator-sdk-sno-azure lane succeeded.
/override ci/prow/hco-e2e-operator-sdk-sno-aws
okd-hco-e2e-upgrade-operator-sdk-gcp lane succeeded.
/override ci/prow/okd-hco-e2e-upgrade-operator-sdk-aws

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2023-12-27T17:48:05Z

hco-e2e-upgrade-prev-operator-sdk-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-operator-sdk-sno-azure

kubevirt-bot · 2023-12-27T17:48:09Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-prev-operator-sdk-sno-azure

In response to this:

hco-e2e-upgrade-prev-operator-sdk-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-operator-sdk-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2023-12-27T18:39:27Z

hco-e2e-kv-smoke-gcp lane succeeded.
/override ci/prow/hco-e2e-kv-smoke-azure

kubevirt-bot · 2023-12-27T18:39:31Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-kv-smoke-azure

In response to this:

hco-e2e-kv-smoke-gcp lane succeeded.
/override ci/prow/hco-e2e-kv-smoke-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa

Added inline comments

nunnatsa · 2023-12-28T10:01:09Z

controllers/operands/kubevirt_test.go

+
+				It("should not add the AlignCPUs feature gate if AlignCPUs is not set in HyperConverged CR", func() {
+					hco.Spec.FeatureGates = hcov1beta1.HyperConvergedFeatureGates{
+						AlignCPUs: nil, //nolint SA1019


nit: the nolint comment here is not needed.

This linter alert is about deprecated fields, as in the NonRoot field, but here the linter won't trigger any alert.

nunnatsa · 2023-12-28T10:02:54Z

controllers/operands/kubevirt_test.go

+				It("should add the AlignCPUs feature gate if AlignCPUs is true in HyperConverged CR", func() {
+					hco.Spec.FeatureGates = hcov1beta1.HyperConvergedFeatureGates{
+						AlignCPUs: ptr.To(true),
+					}
+
+					existingResource, err := NewKubeVirt(hco)
+					Expect(err).ToNot(HaveOccurred())
+					By("KV CR should contain the AutoResourceLimitsGate feature gate", func() {
+						Expect(existingResource.Spec.Configuration.DeveloperConfiguration).NotTo(BeNil())
+						Expect(existingResource.Spec.Configuration.DeveloperConfiguration.FeatureGates).To(ContainElement(kvAlignCPUs))
+					})
+				})
+
+				It("should not add the AlignCPUs feature gate if AlignCPUs is false in HyperConverged CR", func() {
+					hco.Spec.FeatureGates = hcov1beta1.HyperConvergedFeatureGates{
+						AlignCPUs: ptr.To(false),
+					}
+
+					existingResource, err := NewKubeVirt(hco)
+					Expect(err).ToNot(HaveOccurred())
+					By("KV CR should not contain the AlignCPUs feature gate", func() {
+						Expect(existingResource.Spec.Configuration.DeveloperConfiguration).NotTo(BeNil())
+						Expect(existingResource.Spec.Configuration.DeveloperConfiguration.FeatureGates).ToNot(ContainElement(kvAlignCPUs))
+					})
+				})
+
+				It("should not add the AlignCPUs feature gate if AlignCPUs is not set in HyperConverged CR", func() {
+					hco.Spec.FeatureGates = hcov1beta1.HyperConvergedFeatureGates{
+						AlignCPUs: nil, //nolint SA1019


Please also check that the annotation does/doesn't exist in all 3 new tests.

Use the `AlignCPUs` enabler in order to control KubeVirt's `AlignCPUs` feature gate. Signed-off-by: Orel Misan <omisan@redhat.com>

When the FG is enabled ensure the Kubevirt CR will have: 1. The `AlignCPUs` feature gate enabled. 2. The `alpha.kubevirt.io/EmulatorThreadCompleteToEvenParity` annotation set with a value of an empty string. Following PR kubevirt/kubevirt#10872, it will enable KubeVirt to request up to two additional dedicated CPUs in order to complete the total CPU count to an even parity when using emulator thread isolation. When the FG is disabled, do the opposite: 1. Make sure the `AlignCPUs` feature gate is disabled. 2. Make sure the above annotation is not set. The logic introduced in this commit is allowing users to freely annotate the Kubevirt CR, but HCO will ensure the required state of the above annotation. Signed-off-by: Orel Misan <omisan@redhat.com>

hco-bot · 2023-12-28T22:48:14Z

hco-e2e-upgrade-prev-operator-sdk-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-operator-sdk-azure

kubevirt-bot · 2023-12-28T22:48:18Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-prev-operator-sdk-azure

In response to this:

hco-e2e-upgrade-prev-operator-sdk-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-operator-sdk-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

orelmisan · 2023-12-28T23:06:51Z

/retest

orelmisan · 2023-12-29T07:17:50Z

/retest

openshift-ci · 2023-12-29T09:20:50Z

@orelmisan: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/hco-e2e-operator-sdk-aws	`429e4dc`	link	true	`/test hco-e2e-operator-sdk-aws`
ci/prow/okd-hco-e2e-operator-sdk-aws	`429e4dc`	link	true	`/test okd-hco-e2e-operator-sdk-aws`
ci/prow/hco-e2e-kv-smoke-azure	`429e4dc`	link	true	`/test hco-e2e-kv-smoke-azure`
ci/prow/hco-e2e-upgrade-operator-sdk-azure	`429e4dc`	link	true	`/test hco-e2e-upgrade-operator-sdk-azure`
ci/prow/hco-e2e-consecutive-operator-sdk-upgrades-aws	`429e4dc`	link	true	`/test hco-e2e-consecutive-operator-sdk-upgrades-aws`
ci/prow/hco-e2e-upgrade-prev-operator-sdk-azure	`429e4dc`	link	true	`/test hco-e2e-upgrade-prev-operator-sdk-azure`
ci/prow/hco-e2e-upgrade-operator-sdk-sno-azure	`429e4dc`	link	false	`/test hco-e2e-upgrade-operator-sdk-sno-azure`
ci/prow/hco-e2e-operator-sdk-sno-azure	`429e4dc`	link	false	`/test hco-e2e-operator-sdk-sno-azure`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

orelmisan · 2023-12-29T09:44:35Z

@nunnatsa The two remaining Azure tests are still failing after multiple retries.
Could you please have a look?

nunnatsa · 2023-12-29T13:10:57Z

/override-bot

nunnatsa · 2023-12-29T13:13:21Z

hco-e2e-operator-sdk-sno-aws lane passed.
/override ci/prow/hco-e2e-operator-sdk-sno-azure

kubevirt-bot · 2023-12-29T13:13:27Z

@nunnatsa: Overrode contexts on behalf of nunnatsa: ci/prow/hco-e2e-operator-sdk-sno-azure

In response to this:

hco-e2e-operator-sdk-sno-aws lane passed.
/override ci/prow/hco-e2e-operator-sdk-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa · 2023-12-29T13:15:28Z

hco-e2e-upgrade-operator-sdk-sno-aws lane passed

/override
ci/prow/hco-e2e-upgrade-operator-sdk-sno-azure

nunnatsa · 2023-12-29T13:16:37Z

/override ci/prow/hco-e2e-upgrade-operator-sdk-sno-azure

kubevirt-bot · 2023-12-29T13:16:43Z

@nunnatsa: Overrode contexts on behalf of nunnatsa: ci/prow/hco-e2e-upgrade-operator-sdk-sno-azure

In response to this:

/override ci/prow/hco-e2e-upgrade-operator-sdk-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

orelmisan · 2023-12-29T13:17:01Z

Thank you @nunnatsa

Following the changes made on HCO [0], the SMT alignment fix is configurable via HCO, thus making the fix enablelment more user freindly. [0] kubevirt/hyperconverged-cluster-operator#2695 Signed-off-by: Ram Lavi <ralavi@redhat.com>

In case the user's worker nodes have SMT [1] enabled, they will probably encounter CNV-31584 [2]. Add an instruction to enable the `AlignCPUs` enabler in HCO [3], so it will enable [4] KubeVirt to request up to two additional dedicated CPUs in order to complete the total CPU count to an even parity when using emulator thread isolation. [1] https://en.wikipedia.org/wiki/Simultaneous_multithreading [2] https://issues.redhat.com/browse/CNV-31584 [3] kubevirt/hyperconverged-cluster-operator#2695 [4] kubevirt/kubevirt#10872 Signed-off-by: Orel Misan <omisan@redhat.com>

kubevirt-bot requested review from sradco and tiraboschi December 27, 2023 14:50

kubevirt-bot added the size/L label Dec 27, 2023

kubevirt-bot requested a review from nunnatsa December 27, 2023 14:51

orelmisan marked this pull request as ready for review December 27, 2023 14:56

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 27, 2023

kubevirt-bot requested a review from machadovilaca December 27, 2023 14:56

nunnatsa reviewed Dec 27, 2023

View reviewed changes

orelmisan force-pushed the aligncpus branch from dc5c356 to c71e982 Compare December 27, 2023 15:45

nunnatsa reviewed Dec 28, 2023

View reviewed changes

Control KubeVirt AlignCPUs FG

3064fd0

Use the `AlignCPUs` enabler in order to control KubeVirt's `AlignCPUs` feature gate. Signed-off-by: Orel Misan <omisan@redhat.com>

orelmisan force-pushed the aligncpus branch from c71e982 to 6cb2965 Compare December 28, 2023 10:20

kubevirt-bot merged commit 4d4a6d9 into kubevirt:main Dec 29, 2023
36 checks passed

orelmisan deleted the aligncpus branch December 29, 2023 13:16

orelmisan mentioned this pull request Jan 1, 2024

[virt] DPDK cluster config: Enable AlignCPUs openshift/openshift-docs#69713

Merged

1 task

Add the AlignCPUs enabler #2695

Add the AlignCPUs enabler #2695

Conversation

orelmisan commented Dec 27, 2023

kubevirt-bot commented Dec 27, 2023

openshift-ci bot commented Dec 27, 2023

orelmisan commented Dec 27, 2023

coveralls commented Dec 27, 2023 • edited Loading

Pull Request Test Coverage Report for Build 7346808755

💛 - Coveralls

nunnatsa left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orelmisan Dec 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orelmisan commented Dec 27, 2023

hco-bot commented Dec 27, 2023

kubevirt-bot commented Dec 27, 2023

hco-bot commented Dec 27, 2023

kubevirt-bot commented Dec 27, 2023

hco-bot commented Dec 27, 2023

kubevirt-bot commented Dec 27, 2023

hco-bot commented Dec 27, 2023

kubevirt-bot commented Dec 27, 2023

nunnatsa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hco-bot commented Dec 28, 2023

kubevirt-bot commented Dec 28, 2023

orelmisan commented Dec 28, 2023

orelmisan commented Dec 29, 2023

openshift-ci bot commented Dec 29, 2023

orelmisan commented Dec 29, 2023

nunnatsa commented Dec 29, 2023

nunnatsa commented Dec 29, 2023

kubevirt-bot commented Dec 29, 2023

nunnatsa commented Dec 29, 2023

nunnatsa commented Dec 29, 2023

kubevirt-bot commented Dec 29, 2023

orelmisan commented Dec 29, 2023

coveralls commented Dec 27, 2023 •

edited

Loading

nunnatsa left a comment •

edited

Loading

orelmisan Dec 27, 2023 •

edited

Loading