Start smart clone controller from datavolume controller when needed #2265

awels · 2022-05-06T15:57:05Z

Signed-off-by: Alexander Wels awels@redhat.com

What this PR does / why we need it:
It was possible for the CSI snapshot CRD check to fail silently and prevent the smart clone controller from starting during the cdi deployment pod start up. This would prevent smart clone from working properly.

This moves the creation of the smart clone controller into the datavolume controller when someone attempts to do a smart clone. Until that happens the smart clone controller is not started at all. To ensure we don't have multiple smart clone controllers started, we use a channel to synchronize creation, and once created the function will be a no-op.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

BugFix: Fix smart clone controller not starting if an error occurred during startup

mhenriks · 2022-05-06T17:24:31Z

cmd/cdi-controller/controller.go

@@ -288,7 +289,7 @@ func deleteReadyFile() {
 }

 func addCrdInformerEventHandlers(crdInformer cache.SharedIndexInformer, extclient extclientset.Interface, mgr manager.Manager, log logr.Logger) {
-	crdInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
+	crdInformer.AddEventHandlerWithResyncPeriod(cache.ResourceEventHandlerFuncs{
 		AddFunc: func(obj interface{}) {


Should handle UpdateFunc as well as AddFunc as I believe UpdateFunc is called on resync

mhenriks · 2022-05-06T17:26:10Z

cmd/cdi-controller/controller.go

 }

 func startSmartController(extclient extclientset.Interface, mgr manager.Manager, log logr.Logger) {
-	if controller.IsCsiCrdsDeployed(extclient) {
+	if controller.IsCsiCrdsDeployed(extclient, log) {
 		log.Info("CSI CRDs detected, starting smart clone controller")
 		if _, err := controller.NewSmartCloneController(mgr, log, installerLabels); err != nil {


Have to make sure to not do this multiple times I think

Yeah, this probably worked because we never tried to do it more than once.

akalenyu · 2022-05-09T14:19:55Z

What about panicking when this GET fails? just putting it out there, curious as to why you ruled it out?

This GET failing is an indication of pretty bad things going on right?

awels · 2022-05-09T15:07:20Z

It could be a transient network issue for instance, no reason to panic. In essence a panic would cause the control plane of CDI to disappear for some period of time, which could add to whatever causing the transient failure. So why not handle it properly.

awels · 2022-05-10T12:28:52Z

/retest

mhenriks · 2022-05-10T19:59:03Z

pkg/controller/datavolume-controller.go

@@ -291,6 +294,7 @@ func NewDatavolumeController(
 	if err := addDatavolumeControllerWatches(mgr, datavolumeController); err != nil {
 		return nil, err
 	}
+	go reconciler.startSnapshotCsiCrdDetection()


I think it would be nicer, rather than creating our own goroutine, let the controller runtime manager deal with it. See here: https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/manager/manager.go#L61

Basically just pass a Runnable and when the manager is started, it will create/manage the goroutine.

I believe this will clean up all the funniness in the unit test with the dummy goroutine and passing the additional context

We take advantage of this in cdi-operator here: https://github.com/kubevirt/containerized-data-importer/blob/main/pkg/operator/controller/certrotation.go#L80

It was possible for the CSI snapshot CRD check to fail silently and prevent the smart clone controller from starting during the cdi deployment pod start up. This would prevent smart clone from working properly. This adds a periodic sync of 1 minute for checking the CRDs. We also log failures that are not is not found so we can more easily detect this situation as humans. Signed-off-by: Alexander Wels <awels@redhat.com>

mhenriks · 2022-05-11T18:02:41Z

pkg/controller/datavolume-controller.go

 		log.Info("Missing CSI snapshotter CRDs, falling back to host assisted clone")
 		return "", nil
 	}
+	r.sccs.StartController()


This seems a little too hidden in this function, how about here:

containerized-data-importer/pkg/controller/datavolume-controller.go

Line 482 in f68376f

Check if strategy is smart clone and start controller if so. In this location (main reconcile func) it is more explicit, no weird side effects

But we should only start the smart clone controller if the snapshot CRDs are there. I guess us creating the controller will fail because the watches won't work, and we can try again later.

We should not choose smart clone strategy without checking the CRDs. Not sure what you mean about failing.

I mean if we do manage to attempt to start the smart clone controller, it will fail when we add the watches in the controller because the CRDs don't exist. but you make a good point that we won't pick that strategy without checking if the CRDs exist. I already fixed it btw.

Signed-off-by: Alexander Wels <awels@redhat.com>

mhenriks · 2022-05-11T23:27:32Z

/retest

mhenriks · 2022-05-11T23:32:13Z

/lgtm
/approve

kubevirt-bot · 2022-05-11T23:32:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mhenriks

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mhenriks]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

awels · 2022-05-13T12:21:20Z

/cherrypick release-v1.43

kubevirt-bot · 2022-05-13T12:21:58Z

@awels: #2265 failed to apply on top of branch "release-v1.43":

Applying: periodic sync CSI snapshot CRD check
Using index info to reconstruct a base tree...
M	Makefile
M	cmd/cdi-controller/BUILD.bazel
M	cmd/cdi-controller/controller.go
M	pkg/controller/BUILD.bazel
M	pkg/controller/datavolume-controller.go
M	pkg/controller/datavolume-controller_test.go
M	pkg/controller/util.go
M	tests/datavolume_test.go
Falling back to patching base and 3-way merge...
Auto-merging tests/datavolume_test.go
Auto-merging pkg/controller/util.go
CONFLICT (content): Merge conflict in pkg/controller/util.go
Auto-merging pkg/controller/datavolume-controller_test.go
CONFLICT (content): Merge conflict in pkg/controller/datavolume-controller_test.go
Auto-merging pkg/controller/datavolume-controller.go
Auto-merging pkg/controller/BUILD.bazel
Auto-merging cmd/cdi-controller/controller.go
CONFLICT (content): Merge conflict in cmd/cdi-controller/controller.go
Auto-merging cmd/cdi-controller/BUILD.bazel
CONFLICT (content): Merge conflict in cmd/cdi-controller/BUILD.bazel
Auto-merging Makefile
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 periodic sync CSI snapshot CRD check
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release-v1.43

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…ubevirt#2265) * periodic sync CSI snapshot CRD check It was possible for the CSI snapshot CRD check to fail silently and prevent the smart clone controller from starting during the cdi deployment pod start up. This would prevent smart clone from working properly. This adds a periodic sync of 1 minute for checking the CRDs. We also log failures that are not is not found so we can more easily detect this situation as humans. Signed-off-by: Alexander Wels <awels@redhat.com> * Change location of the start controller call. Signed-off-by: Alexander Wels <awels@redhat.com>

…2265) (#2271) * periodic sync CSI snapshot CRD check It was possible for the CSI snapshot CRD check to fail silently and prevent the smart clone controller from starting during the cdi deployment pod start up. This would prevent smart clone from working properly. This adds a periodic sync of 1 minute for checking the CRDs. We also log failures that are not is not found so we can more easily detect this situation as humans. Signed-off-by: Alexander Wels <awels@redhat.com> * Change location of the start controller call. Signed-off-by: Alexander Wels <awels@redhat.com>

…ubevirt#2265) * periodic sync CSI snapshot CRD check It was possible for the CSI snapshot CRD check to fail silently and prevent the smart clone controller from starting during the cdi deployment pod start up. This would prevent smart clone from working properly. This adds a periodic sync of 1 minute for checking the CRDs. We also log failures that are not is not found so we can more easily detect this situation as humans. Signed-off-by: Alexander Wels <awels@redhat.com> * Change location of the start controller call. Signed-off-by: Alexander Wels <awels@redhat.com>

…er when needed #2265 (#2307) * Start smart clone controller from datavolume controller when needed (#2265) * periodic sync CSI snapshot CRD check It was possible for the CSI snapshot CRD check to fail silently and prevent the smart clone controller from starting during the cdi deployment pod start up. This would prevent smart clone from working properly. This adds a periodic sync of 1 minute for checking the CRDs. We also log failures that are not is not found so we can more easily detect this situation as humans. Signed-off-by: Alexander Wels <awels@redhat.com> * Change location of the start controller call. Signed-off-by: Alexander Wels <awels@redhat.com> * Modifications needed for backport. In particular added check for both betav1 and v1 crds. Signed-off-by: Alexander Wels <awels@redhat.com>

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels May 6, 2022

kubevirt-bot requested review from aglitke and maya-r May 6, 2022 15:57

kubevirt-bot added the size/S label May 6, 2022

mhenriks reviewed May 6, 2022

View reviewed changes

awels force-pushed the add_refresh_on_snapshot_crd_check branch from fdec6b2 to c751542 Compare May 6, 2022 21:58

kubevirt-bot added size/M and removed size/S labels May 6, 2022

awels force-pushed the add_refresh_on_snapshot_crd_check branch from c751542 to 4d9f1a3 Compare May 9, 2022 22:12

kubevirt-bot added size/L and removed size/M labels May 9, 2022

awels changed the title ~~periodic sync CSI snapshot CRD check~~ Start smart clone controller from datavolume controller when needed May 10, 2022

mhenriks reviewed May 10, 2022

View reviewed changes

awels force-pushed the add_refresh_on_snapshot_crd_check branch from 4d9f1a3 to f68376f Compare May 11, 2022 15:40

mhenriks reviewed May 11, 2022

View reviewed changes

Change location of the start controller call.

3a4f985

Signed-off-by: Alexander Wels <awels@redhat.com>

kubevirt-bot assigned mhenriks May 11, 2022

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label May 11, 2022

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 11, 2022

kubevirt-bot merged commit b3d0826 into kubevirt:main May 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start smart clone controller from datavolume controller when needed #2265

Start smart clone controller from datavolume controller when needed #2265

awels commented May 6, 2022 •

edited

Loading

mhenriks May 6, 2022

mhenriks May 6, 2022

awels May 6, 2022

akalenyu commented May 9, 2022 •

edited

Loading

awels commented May 9, 2022

awels commented May 10, 2022

mhenriks May 10, 2022

mhenriks May 10, 2022

mhenriks May 11, 2022

awels May 11, 2022

mhenriks May 11, 2022 •

edited

Loading

awels May 11, 2022

mhenriks commented May 11, 2022

mhenriks commented May 11, 2022

kubevirt-bot commented May 11, 2022

awels commented May 13, 2022

kubevirt-bot commented May 13, 2022

Start smart clone controller from datavolume controller when needed #2265

Start smart clone controller from datavolume controller when needed #2265

Conversation

awels commented May 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akalenyu commented May 9, 2022 • edited Loading

awels commented May 9, 2022

awels commented May 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhenriks May 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhenriks commented May 11, 2022

mhenriks commented May 11, 2022

kubevirt-bot commented May 11, 2022

awels commented May 13, 2022

kubevirt-bot commented May 13, 2022

awels commented May 6, 2022 •

edited

Loading

akalenyu commented May 9, 2022 •

edited

Loading

mhenriks May 11, 2022 •

edited

Loading