Kubevirt: Enforce limits & requests by a configurable ratio #2206

iholder101 · 2023-01-15T10:31:05Z

This PR brings a mechanism to align Kubevirt virt-launcher Pods with ResourceQuotas that are applied to the namespace.

As an example, let's look at the following VMI:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  labels:
    special: vmi-fedora
  name: vmi-fedora
spec:
  domain:
    devices:
      disks:
      - disk:
          bus: virtio
        name: containerdisk
      - disk:
          bus: virtio
        name: cloudinitdisk
      rng: {}
    resources:
      requests:
        memory: 1024M
  terminationGracePeriodSeconds: 0
  volumes:
  - containerDisk:
      image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:20230115_2baf5303d 
    name: containerdisk
  - cloudInitNoCloud:
      userData: |-
        #cloud-config
        password: fedora
        chpasswd: { expire: False }
    name: cloudinitdisk

When this VMI is created, the following virt-launcher pod is created (some details are omitted for simplicity):

kind: Pod
metadata:
  name: virt-launcher-vmi-fedora-lzzn6
  namespace: kubevirt-hyperconverged
spec:
  containers:
    name: compute
    resources:
      limits:
        devices.kubevirt.io/kvm: "1"
        devices.kubevirt.io/tun: "1"
        devices.kubevirt.io/vhost-net: "1"
      requests:
        cpu: 100m
        devices.kubevirt.io/kvm: "1"
        devices.kubevirt.io/tun: "1"
        devices.kubevirt.io/vhost-net: "1"
        ephemeral-storage: 50M
        memory: "1279755392"

As can be seen, this virt-launcher has only CPU and memory requests - but not limits. This means that if this VMI is being created in a namespace that has a ResourceQuota defined in it - the virt-launcher Pod won't be able to start. This now can be solved by the mechanism that's presented in this PR.

To enable this mechanism, first a ratio between memory/CPU limits to request needs to be defined as an annotation in HCO object:

kind: HyperConverged
metadata:
  annotations:
    kubevirt.io/cpu-limit-to-request-ratio: "2"
    kubevirt.io/memory-limit-to-request-ratio: "1.5"

In addition, a ResourceQuota needs to exist on the relevant namespace. As an example, it's possible to create the following object:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: test-rq 
spec:
  hard:
    limits.cpu: "200"
    limits.memory: "2000G"

Please take into account that if a ResourceQuota only sets a limit on limits.cpu or limits.cpu, then CPU/memory limits will be set accordingly. If multiple ResourceQuota exist within the relevant namespace, it takes only one of then to limit CPU/memory limits in order to enforce these limits.

When these annotations are enabled along with a ResourceQuota object, a mutating webhook that's targeted to virt-launcher pods will enforce limits on the pod. It would now look like the following:

    resources:
      limits:
        cpu: 200m
        devices.kubevirt.io/kvm: "1"
        devices.kubevirt.io/tun: "1"
        devices.kubevirt.io/vhost-net: "1"
        memory: "1919633088"
      requests:
        cpu: 100m
        devices.kubevirt.io/kvm: "1"
        devices.kubevirt.io/tun: "1"
        devices.kubevirt.io/vhost-net: "1"
        ephemeral-storage: 50M
        memory: "1279755392"

Bear in mind that the new webhook would be active only if a request is set but a limit is not.

Reviewer Checklist

Reviewers are supposed to review the PR for every aspect below one by one. To check an item means the PR is either "OK" or "Not Applicable" in terms of that item. All items are supposed to be checked before merging a PR.

Release note:

Kubevirt: Enforce limits & requests by a configurable ratio

kubevirt-bot · 2023-01-15T10:31:07Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2023-01-15T10:31:08Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

coveralls · 2023-01-15T11:01:57Z

Pull Request Test Coverage Report for Build 3981586663

127 of 181 (70.17%) changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.6%) to 85.013%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/webhooks/mutator/virt-launcher-mutator.go	98	152	64.47%

Totals
Change from base Build 3940385214:	-0.6%
Covered Lines:	4850
Relevant Lines:	5705

💛 - Coveralls

nunnatsa

Added three comments about context, but they are all the same one, actually.

nunnatsa · 2023-01-15T12:22:28Z

pkg/webhooks/mutator/virt-launcher-mutator.go

+	}
+}
+
+func (m *VirtLauncherMutator) Handle(_ context.Context, req admission.Request) admission.Response {


Please use a real parameter for the context. You'll need it below.

nunnatsa · 2023-01-15T12:23:00Z

pkg/webhooks/mutator/virt-launcher-mutator.go

+	}
+	originalPod := launcherPod.DeepCopy()
+
+	hco, err := getHcoObject(m.cli, m.hcoNamespace)


please use the context here.

nunnatsa · 2023-01-15T12:25:02Z

pkg/webhooks/mutator/utils.go

+func getHcoObject(cli client.Client, namespace string) (*v1beta1.HyperConverged, error) {
+	ctx := context.TODO()


Please define the context as the first function parameter and use it.

Done.
FYI, as can be seen here, I was simply using the same logic that already exists and following the current pattern.

If you don't mind to fix it, please do. You're refactoring this code anyway.

iholder101 · 2023-01-15T16:28:43Z

/cc @enp0s3 @fabiand @stu-gott @acardace

nunnatsa

Added a few comments

nunnatsa · 2023-01-15T17:09:31Z

pkg/webhooks/mutator/utils.go

+	hcoutil "github.com/kubevirt/hyperconverged-cluster-operator/pkg/util"
+)
+
+func getHcoObject(cli client.Client, ctx context.Context, namespace string) (*v1beta1.HyperConverged, error) {


The golang convention is that the context is always the first parameter in the function. Could you please reorder?

nunnatsa · 2023-01-15T17:10:35Z

pkg/webhooks/mutator/utils.go

+	if err != nil {
+		if apierrors.IsNotFound(err) {
+			logger.Info("HCO CR doesn't not exist, allow hcoNamespace deletion")
+			return nil, err
+		}
+
+		logger.Error(err, "failed getting HyperConverged CR")
+		return nil, err
+	}


Is that the right error handling and error message, now that it's a generic function?

nunnatsa · 2023-01-15T17:12:33Z

pkg/webhooks/mutator/virt-launcher-mutator.go

+
+	hco, err := getHcoObject(m.cli, ctx, m.hcoNamespace)
+	if err != nil {
+		m.logErr(err, "cannot get hco object")


maybe

Suggested change

m.logErr(err, "cannot get hco object")

m.logErr(err, "cannot get the HyperConverged object")

nunnatsa · 2023-01-15T17:22:10Z

pkg/webhooks/mutator/virt-launcher-mutator_test.go

+		Expect(resources.Limits[v1.ResourceCPU].Equal(expectedResources.Limits[v1.ResourceCPU])).To(BeTrue())
+		Expect(resources.Requests[v1.ResourceCPU].Equal(expectedResources.Requests[v1.ResourceCPU])).To(BeTrue())
+		Expect(resources.Limits[v1.ResourceMemory].Equal(expectedResources.Limits[v1.ResourceMemory])).To(BeTrue())
+		Expect(resources.Requests[v1.ResourceMemory].Equal(expectedResources.Requests[v1.ResourceMemory])).To(BeTrue())


Isn't the gomega's Equals matcher work here?

Unfortunately not :(
With Gomega's Equals I get errors that are similar to:

[FAILED] Expected <resource.Quantity>: { i: {value: 150000000000, scale: -3}, d: {Dec: nil}, s: "150M", Format: "DecimalSI", } to equal <resource.Quantity>: { i: {value: 150, scale: 6}, d: {Dec: nil}, s: "150M", Format: "DecimalSI", }

Although the two quantities are effectively equal (to 150M).

nunnatsa · 2023-01-15T17:24:22Z

pkg/webhooks/mutator/virt-launcher-mutator_test.go

+	podNamespace = "fake-namespace"
+)
+
+var _ = Describe("virt-launcher webhook mutator", func() {


Please add error cases; e.g. no HyperConverged, negative ratio, zero ratio, wrongly formatted inputs etc.

Done!
Added tests for no HCO and invalid ratio. IMHO we don't need to test the library that parses the annotation as it's beyond this "unit's" scope.

nunnatsa · 2023-01-15T17:28:33Z

pkg/webhooks/mutator/utils.go

+func getHcoObject(cli client.Client, namespace string) (*v1beta1.HyperConverged, error) {
+	ctx := context.TODO()


If you don't mind to fix it, please do. You're refactoring this code anyway.

hco-bot · 2023-01-15T18:06:50Z

okd-hco-e2e-image-index-aws lane succeeded.
/override ci/prow/okd-hco-e2e-image-index-gcp

kubevirt-bot · 2023-01-22T23:21:32Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-azure

In response to this:

hco-e2e-image-index-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

orenc1 · 2023-01-23T06:38:33Z

/retest

hco-bot · 2023-01-23T08:08:34Z

hco-e2e-upgrade-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-sno-azure

kubevirt-bot · 2023-01-23T08:08:41Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-index-sno-azure

In response to this:

hco-e2e-upgrade-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-01-23T08:49:12Z

@iholder101: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-hco-e2e-upgrade-index-gcp	`56f2190`	link	true	`/test okd-hco-e2e-upgrade-index-gcp`
ci/prow/hco-e2e-upgrade-index-azure	`56f2190`	link	true	`/test hco-e2e-upgrade-index-azure`
ci/prow/hco-e2e-image-index-gcp	`56f2190`	link	true	`/test hco-e2e-image-index-gcp`
ci/prow/hco-e2e-image-index-sno-azure	`56f2190`	link	false	`/test hco-e2e-image-index-sno-azure`
ci/prow/hco-e2e-image-index-azure	`56f2190`	link	true	`/test hco-e2e-image-index-azure`
ci/prow/hco-e2e-upgrade-index-sno-azure	`56f2190`	link	false	`/test hco-e2e-upgrade-index-sno-azure`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

hco-bot · 2023-01-23T08:51:26Z

hco-e2e-upgrade-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-sno-azure

kubevirt-bot · 2023-01-23T08:51:33Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-index-sno-azure

In response to this:

hco-e2e-upgrade-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

orenc1 · 2023-01-23T09:08:38Z

since it is a temporary solution for the resource limits problem in VirtualMachines, and it is going to get a permanent fix in kubevirt/kubevirt side, i'm ok with disregarding the coverage decrease.
/override coverage/coveralls
/lgtm
/approve

kubevirt-bot · 2023-01-23T09:08:42Z

@orenc1: Overrode contexts on behalf of orenc1: coverage/coveralls

In response to this:

since it is a temporary solution for the resource limits problem in VirtualMachines, and it is going to get a permanent fix in kubevirt/kubevirt side, i'm ok with disregarding the coverage decrease.
/override coverage/coveralls
/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubevirt-bot · 2023-01-23T09:08:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: orenc1

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [orenc1]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

iholder101 · 2023-01-31T09:34:08Z

/cherrypick release-1.6

kubevirt-bot · 2023-01-31T09:34:48Z

@iholder101: #2206 failed to apply on top of branch "release-1.6":

Applying: Refactor: rename mutator to namespace mutator
Applying: Introduce basic virt-launcher mutator
Using index info to reconstruct a base tree...
A	deploy/index-image/community-kubevirt-hyperconverged/1.9.0/manifests/kubevirt-hyperconverged-operator.v1.9.0.clusterserviceversion.yaml
A	deploy/olm-catalog/community-kubevirt-hyperconverged/1.9.0/manifests/kubevirt-hyperconverged-operator.v1.9.0.clusterserviceversion.yaml
M	go.mod
M	pkg/components/components.go
M	pkg/util/consts.go
M	pkg/webhooks/mutator/namespace_mutator.go
M	pkg/webhooks/mutator/namespace_mutator_test.go
M	pkg/webhooks/setup.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/webhooks/setup.go
CONFLICT (content): Merge conflict in pkg/webhooks/setup.go
Auto-merging pkg/webhooks/mutator/namespace_mutator_test.go
Auto-merging pkg/webhooks/mutator/namespace_mutator.go
CONFLICT (content): Merge conflict in pkg/webhooks/mutator/namespace_mutator.go
Auto-merging pkg/util/consts.go
CONFLICT (content): Merge conflict in pkg/util/consts.go
Auto-merging pkg/components/components.go
Auto-merging go.mod
CONFLICT (content): Merge conflict in go.mod
Auto-merging deploy/olm-catalog/community-kubevirt-hyperconverged/1.6.0/manifests/kubevirt-hyperconverged-operator.v1.6.0.clusterserviceversion.yaml
Auto-merging deploy/index-image/community-kubevirt-hyperconverged/1.6.0/manifests/kubevirt-hyperconverged-operator.v1.6.0.clusterserviceversion.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 Introduce basic virt-launcher mutator
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fabiand · 2023-02-14T09:07:16Z

@iholder101 wrt to the annotation, can we change it from

    kubevirt.io/cpu-limit-to-request-ratio: "2"
    kubevirt.io/memory-limit-to-request-ratio: "1.5"

to

    kubevirt.io/launcher-resource-cpu-limit-to-request-ratio: "2"
    kubevirt.io/launcher-resource-memory-limit-to-request-ratio: "1.5"

That's quite long, agreed, but otoh the current name does not really say WHAT cpu and memory values are adjusted, and we have at least two places (.resources and .cpu.count or .memory.guest). At the same time it is not clear on what entity is being operated, vm, vmi, pod, which pod?
Thus the ask to make the annotation more specific, and a little more expressive and future proof.

iholder101 · 2023-03-08T09:51:17Z

/cherrypick release-1.8

kubevirt-bot · 2023-03-08T09:51:56Z

@iholder101: #2206 failed to apply on top of branch "release-1.8":

Applying: Refactor: rename mutator to namespace mutator
Applying: Introduce basic virt-launcher mutator
Using index info to reconstruct a base tree...
A	deploy/index-image/community-kubevirt-hyperconverged/1.9.0/manifests/kubevirt-hyperconverged-operator.v1.9.0.clusterserviceversion.yaml
A	deploy/olm-catalog/community-kubevirt-hyperconverged/1.9.0/manifests/kubevirt-hyperconverged-operator.v1.9.0.clusterserviceversion.yaml
M	go.mod
M	pkg/components/components.go
M	pkg/webhooks/mutator/namespace_mutator_test.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/webhooks/mutator/namespace_mutator_test.go
Auto-merging pkg/components/components.go
Auto-merging go.mod
CONFLICT (content): Merge conflict in go.mod
Auto-merging deploy/olm-catalog/community-kubevirt-hyperconverged/1.8.0/manifests/kubevirt-hyperconverged-operator.v1.8.0.clusterserviceversion.yaml
Auto-merging deploy/index-image/community-kubevirt-hyperconverged/1.8.0/manifests/kubevirt-hyperconverged-operator.v1.8.0.clusterserviceversion.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 Introduce basic virt-launcher mutator
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release-1.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

…evirt#2341) Remove the support of the kubevirt.io/cpu-limit-to-request-ratio and the kubevirt.io/memory-limit-to-request-ratio annotations, as this workaround does now work as expected. Revert PR kubevirt#2206 as it's not needed anymore This is a manual cherry-pick of: kubevirt#2341 Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com> Signed-off-by: Nahshon Unna Tsameret <60659093+nunnatsa@users.noreply.github.com>

Remove the support of the kubevirt.io/cpu-limit-to-request-ratio and the kubevirt.io/memory-limit-to-request-ratio annotations, as this workaround does now work as expected. Revert PR #2206 as it's not needed anymore This is a manual cherry-pick of: #2341 Signed-off-by: Nahshon Unna Tsameret <60659093+nunnatsa@users.noreply.github.com> Co-authored-by: Nahshon Unna Tsameret <60659093+nunnatsa@users.noreply.github.com>

kubevirt-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jan 15, 2023

kubevirt-bot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Jan 15, 2023

kubevirt-bot requested review from sradco and tiraboschi January 15, 2023 10:31

kubevirt-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L labels Jan 15, 2023

iholder101 force-pushed the limit-enforcement branch from b6b5a88 to 69fd669 Compare January 15, 2023 10:48

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 15, 2023

iholder101 force-pushed the limit-enforcement branch from 69fd669 to 28b970b Compare January 15, 2023 10:56

iholder101 force-pushed the limit-enforcement branch from 28b970b to 7df714f Compare January 15, 2023 11:42

nunnatsa reviewed Jan 15, 2023

View reviewed changes

iholder101 changed the title ~~Limit enforcement~~ Kubevirt: Enforce limits & requests by a configurable ratio Jan 15, 2023

kubevirt-bot added size/XL and removed size/L labels Jan 15, 2023

iholder101 marked this pull request as ready for review January 15, 2023 16:24

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 15, 2023

kubevirt-bot requested a review from nunnatsa January 15, 2023 16:24

kubevirt-bot requested review from acardace, enp0s3, fabiand and stu-gott January 15, 2023 16:28

iholder101 force-pushed the limit-enforcement branch 2 times, most recently from 3ef06b8 to 24b848c Compare January 15, 2023 16:40

nunnatsa reviewed Jan 15, 2023

View reviewed changes

kubevirt-bot assigned orenc1 Jan 23, 2023

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Jan 23, 2023

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 23, 2023

kubevirt-bot merged commit ac6c4e1 into kubevirt:main Jan 23, 2023

iholder101 mentioned this pull request Feb 1, 2023

[release-1.7]: Kubevirt: Enforce limits & requests by a configurable ratio #2225

Merged

iholder101 mentioned this pull request Feb 14, 2023

Kubevirt: Add guest-to-request memory headroom to virt-launchers #2245

Closed

11 tasks

iholder101 mentioned this pull request Feb 27, 2023

Add guest-to-request memory headroom ratio kubevirt/kubevirt#9322

Merged

iholder101 mentioned this pull request Mar 8, 2023

[release-1.8]: Kubevirt: Enforce limits & requests by a configurable ratio #2280

Merged

nunnatsa added a commit to nunnatsa/hyperconverged-cluster-operator that referenced this pull request May 15, 2023

Revert PR kubevirt#2206 as it's not needed anymore

8d4c75e

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

kubevirt-bot pushed a commit that referenced this pull request May 15, 2023

Revert PR #2206 as it's not needed anymore (#2341)

b4f6a41

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

tiraboschi mentioned this pull request May 18, 2023

[release-1.9] Revert PR #2206 as it's not needed anymore (#2341) #2347

Merged

11 tasks

		func getHcoObject(cli client.Client, namespace string) (*v1beta1.HyperConverged, error) {
		ctx := context.TODO()

	m.logErr(err, "cannot get hco object")
	m.logErr(err, "cannot get the HyperConverged object")

Kubevirt: Enforce limits & requests by a configurable ratio #2206

Kubevirt: Enforce limits & requests by a configurable ratio #2206

Conversation

iholder101 commented Jan 15, 2023 • edited Loading

kubevirt-bot commented Jan 15, 2023

openshift-ci bot commented Jan 15, 2023

coveralls commented Jan 15, 2023 • edited Loading

Pull Request Test Coverage Report for Build 3981586663

💛 - Coveralls

nunnatsa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nunnatsa Jan 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iholder101 Jan 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iholder101 commented Jan 15, 2023

nunnatsa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nunnatsa Jan 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hco-bot commented Jan 15, 2023

kubevirt-bot commented Jan 22, 2023

orenc1 commented Jan 23, 2023

hco-bot commented Jan 23, 2023

kubevirt-bot commented Jan 23, 2023

openshift-ci bot commented Jan 23, 2023

hco-bot commented Jan 23, 2023

kubevirt-bot commented Jan 23, 2023

orenc1 commented Jan 23, 2023

kubevirt-bot commented Jan 23, 2023

kubevirt-bot commented Jan 23, 2023

iholder101 commented Jan 31, 2023

kubevirt-bot commented Jan 31, 2023

fabiand commented Feb 14, 2023

iholder101 commented Mar 8, 2023

kubevirt-bot commented Mar 8, 2023

iholder101 commented Jan 15, 2023 •

edited

Loading

coveralls commented Jan 15, 2023 •

edited

Loading

nunnatsa Jan 15, 2023 •

edited

Loading

iholder101 Jan 15, 2023 •

edited

Loading

nunnatsa Jan 15, 2023 •

edited

Loading