Import populator #2690

alromeros · 2023-04-12T12:01:01Z

What this PR does / why we need it:

The import populator is a controller that handles the import of data in PVCs (still taking advantage of the import-controller flow) without the need for DataVolumes.

This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVC and deletes the PVC'.

Special notes for your reviewer: Still need to commit the unit tests.

Release note:

Include a new controller "import-populator" to populate PVCs with the standard CDI Import flow using a volume populator

kubevirt-bot · 2023-04-12T12:01:04Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

alromeros · 2023-04-14T17:28:29Z

@ShellyKa13 I cherry-picked some of your commits and tried to refactor common code so it's shared between populators in f51f2f7. Let me know what you think.

mhenriks

Great start! Here's my first quick pass...

pkg/controller/populators/populator-base.go

mhenriks · 2023-04-17T00:35:03Z

pkg/controller/populators/populator-base.go

+	// We still return the nil PVC' as we'll get called again once PVC' exists.
+	// If target PVC is bound, we don't really need to populate anything.
+	if pvcPrime == nil && cc.IsUnbound(pvc) {
+		_, err := r.createPVCPrime(pvc, populationSource, waitForFirstConsumer, populator.updateAnnotations)


why pass populator.updateAnnotations instead of just populator?

In the case of createPVCPrime I think it makes sense that we pass a single pvcModifier (as we do in the base DV controller) instead of the whole populator.

mhenriks · 2023-04-17T00:37:11Z

staging/src/kubevirt.io/containerized-data-importer-api/pkg/apis/core/v1beta1/types.go

+}
+
+// ImportSourceSpec defines the Spec field for ImportSource
+type ImportSourceSpec struct {


Just like UploadSource I think this is a bad name. Source of what? Source to what? VolumeImportSource is better but still not great IMO

Yeah I get your point... VolumeImportSource works for me but I'm easy to convince. I'll try to think of some names with @ShellyKa13 so we can further discuss them.

pkg/controller/populators/util.go

pkg/controller/populators/populator-base.go

mhenriks

Couple little things

tests/import_test.go

mhenriks · 2023-04-25T00:51:44Z

tests/import_test.go

+		}, timeout, pollingInterval).Should(BeTrue())
+	})
+})
+


Can we add a test to make sure that the static binding works as expected? Create a PV with claimref set for PVC. Then create the PVC with populator. Make sure the PVC gets bound correctly and data is not overwritten

mhenriks · 2023-04-25T00:54:23Z

pkg/controller/populators/import-populator_test.go

+	dvPopulatorLog = logf.Log.WithName("import-populator-test")
+)
+
+var _ = Describe("Import populator tests", func() {


Maybe I missed it, but how about a unit test that ignores/errors cross namespace datasource?

alromeros · 2023-04-25T14:04:25Z

staging/src/kubevirt.io/containerized-data-importer-api/pkg/apis/core/v1beta1/types.go

+// VolumeImportSourceSpec defines the Spec field for VolumeImportSource
+type VolumeImportSourceSpec struct {
+	ContentType DataVolumeContentType `json:"contentType,omitempty"`
+	// Import sources


@mhenriks want to know your opinion on this, do you think it's better to have the sources embedded directly in the spec? Or should we add a new field to store them as we do with DataVolumes.

Good question! Yeah I think it makes sense to have this pseudo-union defined in it's own struct

mhenriks · 2023-04-26T20:08:07Z

Great work @alromeros!

/approve

Let''s let @ShellyKa13 do a final once over and lgtm

kubevirt-bot · 2023-04-26T20:08:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mhenriks

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mhenriks]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

alromeros · 2023-04-27T07:34:21Z

/test pull-containerized-data-importer-e2e-ceph

ShellyKa13

looks really good!
have several comments, mostly cosmetics, once fixed ill lgtm

ShellyKa13 · 2023-04-27T07:44:42Z

pkg/controller/common/util.go

@@ -79,6 +85,8 @@ const (
 	AnnPriorityClassName = AnnAPIGroup + "/storage.pod.priorityclassname"
 	// AnnExternalPopulation annotation marks a PVC as "externally populated", allowing the import-controller to skip it
 	AnnExternalPopulation = AnnAPIGroup + "/externalPopulation"
+	// AnnOwnedByDataVolume annotation has the owner DataVolume name
+	AnnOwnedByDataVolume = AnnAPIGroup + "/ownedByDataVolume"


same for this annotation I guess

ShellyKa13 · 2023-04-27T07:46:17Z

pkg/controller/common/util.go

+
+// HasAnnOwnedByDataVolume returns if the obj has annotation
+// it is owned by DataVolume
+func HasAnnOwnedByDataVolume(obj metav1.Object) bool {


so eventually since we decided not to check if the pvc was owned by a dv to and we always inflate we dont actually use this function outside of the dv controllers, so might be worth it to put it back where it used to..

ShellyKa13 · 2023-04-27T07:52:21Z

pkg/controller/populators/import-populator.go

+}
+
+// Import-specific implementation of preparePVCForPopulation
+func (r *ImportPopulatorReconciler) preparePVCForPopulation(pvc *corev1.PersistentVolumeClaim, source client.Object) {


I dont really like this name, it doesnt 'prepare' anything, it really just updates annotation on the pvc prime..
If you dont want to mention specifically the annotation part then can just call it modifyPVCPrime or something like that

ShellyKa13 · 2023-04-27T07:54:45Z

pkg/controller/populators/import-populator_test.go

+	})
+
+	// Common environment requirements
+	scName := "testsc"


should be in the consts

ShellyKa13 · 2023-04-27T07:54:56Z

pkg/controller/populators/import-populator_test.go

+		Kind:     cdiv1.VolumeImportSourceRef,
+		Name:     samplePopulatorName,
+	}
+	nsName := "test-import"


should be in the consts

This one needs to be a variable to do &nsName.

ShellyKa13 · 2023-04-27T07:56:51Z

pkg/controller/populators/import-populator_test.go

+			reconciler = createImportPopulatorReconciler(targetPvc, pvcPrime, pod)
+			err = reconciler.updateImportProgress(string(corev1.PodRunning), targetPvc, pvcPrime)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(targetPvc.Annotations[AnnImportProgressReporting]).To(BeEquivalentTo("13.45%"))


something the indentation of this Expect line seems wrong, maybe its just in the view in github

Yeah I'm sure it's a github thing, looks good when you open the file.

ShellyKa13 · 2023-04-27T08:05:34Z

pkg/controller/util.go

+	selectedNode := pvc.Annotations[cc.AnnSelectedNode]
+	if isPopulator && selectedNode != "" {
+		annotations[cc.AnnSelectedNode] = selectedNode
+	}


maybe also put in a function? big chunk of code for this function which is very specific to populators :)
not a must though..

hmmm I know I'm repeating some code here but I've tried some alternatives to reuse SetNodeNameIfPopulator and I think it looks uglier.

ShellyKa13 · 2023-04-27T08:09:43Z

tests/import_test.go

+			Expect(err).ToNot(HaveOccurred())
+		}
+		Eventually(func() bool {
+			// Make sure pvcPrime was deleted after upload population


comment line wrong: "upload" population instead of import

This commit introduces the basic reconciler for populators with common function that can be used by the different populators. Signed-off-by: Shelly Kagan <skagan@redhat.com>

Signed-off-by: Shelly Kagan <skagan@redhat.com>

This commit adds the VolumeImportSource CRD into CDI. CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator. Signed-off-by: Alvaro Romero <alromero@redhat.com>

This commit introduces and modifies several functions so we can reuse common code between all populators. Other than having a common reconcile function, a new populatorController interface has been introduced so we are able to call populator-specific methods from the populator-base reconciler. Signed-off-by: Alvaro Romero <alromero@redhat.com>

The import populator is a controller that handles the import of data in PVCs without the need of DataVolumes while still taking advantage of the import-controller flow. This controller creates an additional PVC' with import annotations. After the import process succeeds, the controller rebinds the PV to the original target PVc and deletes the PVC prime. Signed-off-by: Alvaro Romero <alromero@redhat.com>

This commit updates the import tests to cover the new import populator flow. Signed-off-by: Alvaro Romero <alromero@redhat.com>

Signed-off-by: Alvaro Romero <alromero@redhat.com>

* Modify indexes and other related code to support namespaced dataSourceRefs. Cross-namespace population is still not supported as it depends on alpha feature gates. * Add functional test to cover static binding. * Fix selected node annotation bug in scratch space PVCs * Fix linter alerts Signed-off-by: Alvaro Romero <alromero@redhat.com>

alromeros · 2023-04-27T11:44:13Z

/test pull-cdi-linter

ShellyKa13 · 2023-04-27T12:05:02Z

great work!
/lgtm

alromeros · 2023-04-27T16:12:58Z

/test pull-containerized-data-importer-e2e-nfs

kubevirt-bot requested review from aglitke and mhenriks April 12, 2023 12:01

kubevirt-bot added the size/XXL label Apr 12, 2023

alromeros force-pushed the import-populator branch 2 times, most recently from b7d10c9 to d6e096b Compare April 14, 2023 16:44

alromeros marked this pull request as ready for review April 14, 2023 16:49

kubevirt-bot requested a review from maya-r April 14, 2023 16:49

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 14, 2023

alromeros force-pushed the import-populator branch from d6e096b to 770834f Compare April 14, 2023 17:01

alromeros force-pushed the import-populator branch 5 times, most recently from ef256b6 to 644d6a4 Compare April 15, 2023 17:32

mhenriks reviewed Apr 17, 2023

View reviewed changes

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 17, 2023

alromeros force-pushed the import-populator branch 3 times, most recently from 11938e4 to d37236d Compare April 17, 2023 17:35

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 17, 2023

alromeros force-pushed the import-populator branch 4 times, most recently from de6316f to 8cf8840 Compare April 18, 2023 08:03

alromeros force-pushed the import-populator branch from b17f57a to 07879e4 Compare April 24, 2023 16:44

mhenriks reviewed Apr 25, 2023

View reviewed changes

alromeros force-pushed the import-populator branch from 07879e4 to 1143aae Compare April 25, 2023 13:38

alromeros commented Apr 25, 2023

View reviewed changes

alromeros force-pushed the import-populator branch 6 times, most recently from 6cce0ac to 3d4487a Compare April 26, 2023 15:23

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 26, 2023

ShellyKa13 reviewed Apr 27, 2023

View reviewed changes

ShellyKa13 and others added 8 commits April 27, 2023 10:58

Create populators package to be used for all populators

05ee29d

This commit introduces the basic reconciler for populators with common function that can be used by the different populators. Signed-off-by: Shelly Kagan <skagan@redhat.com>

unite getcontenttype func across code

75d9d62

Signed-off-by: Shelly Kagan <skagan@redhat.com>

Add VolumeImportSource CRD for import populator

96b00d9

This commit adds the VolumeImportSource CRD into CDI. CRs created from this CRD will be referenced in the dataSourceRef field to populate PVCs with the import populator. Signed-off-by: Alvaro Romero <alromero@redhat.com>

Add functional tests to cover the import populator flow

8f68f0f

This commit updates the import tests to cover the new import populator flow. Signed-off-by: Alvaro Romero <alromero@redhat.com>

Add unit tests for import populator

3bf06a8

Signed-off-by: Alvaro Romero <alromero@redhat.com>

alromeros force-pushed the import-populator branch from 3d4487a to 702ee39 Compare April 27, 2023 11:22

kubevirt-bot assigned ShellyKa13 Apr 27, 2023

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Apr 27, 2023

kubevirt-bot merged commit c5f767d into kubevirt:main Apr 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Import populator #2690

Import populator #2690

alromeros commented Apr 12, 2023 •

edited

Loading

kubevirt-bot commented Apr 12, 2023

alromeros commented Apr 14, 2023

mhenriks left a comment

mhenriks Apr 17, 2023

alromeros Apr 17, 2023

mhenriks Apr 17, 2023

alromeros Apr 17, 2023

mhenriks left a comment

mhenriks Apr 25, 2023

mhenriks Apr 25, 2023

alromeros Apr 25, 2023

mhenriks Apr 26, 2023

mhenriks commented Apr 26, 2023

kubevirt-bot commented Apr 26, 2023

alromeros commented Apr 27, 2023

ShellyKa13 left a comment

ShellyKa13 Apr 27, 2023

ShellyKa13 Apr 27, 2023

ShellyKa13 Apr 27, 2023

ShellyKa13 Apr 27, 2023

ShellyKa13 Apr 27, 2023

alromeros Apr 27, 2023

ShellyKa13 Apr 27, 2023 •

edited

Loading

alromeros Apr 27, 2023

ShellyKa13 Apr 27, 2023

alromeros Apr 27, 2023

ShellyKa13 Apr 27, 2023

alromeros commented Apr 27, 2023

ShellyKa13 commented Apr 27, 2023

alromeros commented Apr 27, 2023

Import populator #2690

Import populator #2690

Conversation

alromeros commented Apr 12, 2023 • edited Loading

kubevirt-bot commented Apr 12, 2023

alromeros commented Apr 14, 2023

mhenriks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhenriks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhenriks commented Apr 26, 2023

kubevirt-bot commented Apr 26, 2023

alromeros commented Apr 27, 2023

ShellyKa13 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShellyKa13 Apr 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alromeros commented Apr 27, 2023

ShellyKa13 commented Apr 27, 2023

alromeros commented Apr 27, 2023

alromeros commented Apr 12, 2023 •

edited

Loading

ShellyKa13 Apr 27, 2023 •

edited

Loading