OADP-694 oadp-1.1: Add data mover to e2e #863

kaovilai · 2022-11-10T17:01:46Z

Signed-off-by: Tiger Kaovilai tkaovila@redhat.com

override images

comment image overrides

Use 1.1 images + emcmulla/csi-plugin:latest

Signed-off-by: Tiger Kaovilai tkaovila@redhat.com

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> override images comment image overrides Use 1.1 images + emcmulla/csi-plugin:latest Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

codecov-commenter · 2022-11-10T17:07:13Z

Codecov Report

Base: 32.65% // Head: 32.65% // No change to project coverage 👍

Coverage data is based on head (f19a66c) compared to base (165e436).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff            @@
##           oadp-1.1     #863   +/-   ##
=========================================
  Coverage     32.65%   32.65%           
=========================================
  Files            17       17           
  Lines          3237     3237           
=========================================
  Hits           1057     1057           
  Misses         2088     2088           
  Partials         92       92

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

kaovilai · 2022-11-11T06:51:32Z

4.9 aws succeeded.

/retest

kaovilai · 2022-11-11T06:53:45Z

one timed out on datamover
one timed out on non data mover entry

kaovilai · 2022-11-11T09:01:43Z

/retest

kaovilai · 2022-11-11T13:56:30Z

/retest

weshayutin · 2022-11-15T00:55:52Z

/retest

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

kaovilai · 2022-11-16T13:10:04Z

/retest

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

kaovilai · 2022-11-17T03:14:31Z

/retest

kaovilai · 2022-11-17T13:40:20Z

/retest

kaovilai · 2022-11-17T16:13:26Z

/test 4.10-operator-e2e-azure

eemcmullan · 2022-11-17T20:24:19Z

tests/e2e/backup_restore_suite_test.go

@@ -186,7 +261,7 @@ var _ = Describe("AWS backup restore tests", func() {
 			Expect(err).ToNot(HaveOccurred())

 			// wait for backup to not be running
-			Eventually(IsBackupDone(dpaCR.Client, namespace, backupName), timeoutMultiplier*time.Minute*12, time.Second*10).Should(BeTrue())
+			Eventually(IsBackupDone(dpaCR.Client, namespace, backupName), timeoutMultiplier*time.Minute*20, time.Second*10).Should(BeTrue())


Since this is data mover test, would it be possible to also check if VSB is completed here?

Velero backup status should be reliable. VSB status should really only rarely be checked when things go wrong.

Let's leave it out. Could add on failure. Follow up todo?

Actually VSB/VSR is already checked on failure

*logged on failure.

right here https://github.com/openshift/oadp-operator/pull/863/files#diff-a9f567f9b6083a8f8e52d9318806042225c729fa99bea2354e148eead33bdf9fR119-R124

Sounds good

eemcmullan · 2022-11-17T20:24:34Z

tests/e2e/backup_restore_suite_test.go

 			// run restore
 			log.Printf("Creating restore %s for case %s", restoreName, brCase.Name)
 			restore, err := CreateRestoreFromBackup(dpaCR.Client, namespace, backupName, restoreName)
 			Expect(err).ToNot(HaveOccurred())
-			Eventually(IsRestoreDone(dpaCR.Client, namespace, restoreName), timeoutMultiplier*time.Minute*4, time.Second*10).Should(BeTrue())
+			Eventually(IsRestoreDone(dpaCR.Client, namespace, restoreName), timeoutMultiplier*time.Minute*60, time.Second*10).Should(BeTrue())


Same here for VSR

Actually VSB/VSR is already checked on failure

tests/e2e/backup_restore_suite_test.go

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

kaovilai · 2022-11-18T03:23:25Z

/test 4.9-operator-e2e-gcp

kaovilai · 2022-11-18T05:09:24Z

/retest

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

tests/e2e/lib/dpa_helpers.go

weshayutin · 2022-11-18T14:27:07Z

tests/e2e/lib/dpa_helpers.go

@@ -171,14 +193,26 @@ func (v *DpaCustomResource) CreateOrUpdateWithRetries(spec *oadpv1alpha1.DataPro
 	)
 	for i := 0; i < retries; i++ {
 		if cr, err = v.Get(); apierrors.IsNotFound(err) {


was there repeated errors creating the dpa here or is this just an improvement?

Ensuring the content of v.CustomResource that's all.

weshayutin · 2022-11-18T14:28:42Z

tests/e2e/lib/kube_helpers.go

+	return buf.String(), nil
+}
+
+func GetDeploymentPodContainerLogs(namespace, deploymentName, containerName string) (string, error) {


oh snap.. am I reading the correctly? We have to verify the that restore worked w/ datamover by scraping the deployment log?

It's used here in case of failure for dumping out logs

weshayutin

looks awesome but I do have some questions

weshayutin · 2022-11-18T14:38:44Z

tests/e2e/backup_restore_suite_test.go

+				GinkgoWriter.Println("Printing volume-snapshot-mover deployment pod logs")
+				GinkgoWriter.Print(GetDeploymentPodContainerLogs(namespace, common.DataMover, common.DataMoverControllerContainer))


can we get some code comments as to why we're pulling the log

weshayutin

please add code comments as to why

weshayutin

spoke to Tiger about a few things, will land improvements in jira. Approved well done!

openshift-ci · 2022-11-18T16:35:14Z

@kaovilai: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

eemcmullan · 2022-11-18T14:41:41Z

tests/e2e/backup_restore_suite_test.go

@@ -186,7 +261,7 @@ var _ = Describe("AWS backup restore tests", func() {
 			Expect(err).ToNot(HaveOccurred())

 			// wait for backup to not be running
-			Eventually(IsBackupDone(dpaCR.Client, namespace, backupName), timeoutMultiplier*time.Minute*12, time.Second*10).Should(BeTrue())
+			Eventually(IsBackupDone(dpaCR.Client, namespace, backupName), timeoutMultiplier*time.Minute*20, time.Second*10).Should(BeTrue())


Sounds good

eemcmullan · 2022-11-18T16:28:38Z

tests/e2e/backup_restore_suite_test.go

+				err = dpaCR.Client.List(context.Background(), &pvcList, &client.ListOptions{Namespace: namespace})
+				Expect(err).NotTo(HaveOccurred())
+				GinkgoWriter.Printf("PVC oadp ns list %v\n", pvcList)
+


Worth listing volumeSnapshotContents? wdyt

hhpatel14 · 2023-04-10T19:56:29Z

/cherry-pick master

openshift-cherrypick-robot · 2023-04-10T19:57:07Z

@hhpatel14: #863 failed to apply on top of branch "master":

Applying: Add data mover to e2e
Using index info to reconstruct a base tree...
M	Makefile
M	go.mod
M	go.sum
M	tests/e2e/backup_restore_suite_test.go
M	tests/e2e/lib/apps.go
M	tests/e2e/lib/dpa_helpers.go
Falling back to patching base and 3-way merge...
Auto-merging tests/e2e/lib/dpa_helpers.go
Auto-merging tests/e2e/lib/apps.go
Auto-merging tests/e2e/backup_restore_suite_test.go
CONFLICT (content): Merge conflict in tests/e2e/backup_restore_suite_test.go
Auto-merging go.sum
CONFLICT (content): Merge conflict in go.sum
Auto-merging go.mod
CONFLICT (content): Merge conflict in go.mod
Auto-merging Makefile
CONFLICT (content): Merge conflict in Makefile
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Add data mover to e2e
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick master

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Co-authored-by: Tiger Kaovilai <tiger+github@redhat.com>

* master: Add data mover to e2e cherrypick of #863 Co-authored-by: Tiger Kaovilai <tiger+github@redhat.com> * workaround to fix existing PVC of the same name found.. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * 1.2 update * add restorePhase async plugin waiting phases Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Remove duplicate application installs, add CSIDataMover failure deletion handling, volsync channel stable-0.7 Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Create app, and therefore namespace, before creating PVC into namespace Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * FEntry typo Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * enable volsync MoverSecurityContext Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Update mysql/mariadb pod securityContext to match Dockerfile USER for volsync to pick up. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * add vsmv1alpha1 to client scheme for failure cleanup Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> --------- Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Co-authored-by: Tiger Kaovilai <tkaovila@redhat.com> Co-authored-by: Tiger Kaovilai <tiger+github@redhat.com>

Co-authored-by: Tiger Kaovilai <tiger+github@redhat.com>

* master: Add data mover to e2e cherrypick of #863 Co-authored-by: Tiger Kaovilai <tiger+github@redhat.com> * workaround to fix existing PVC of the same name found.. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * 1.2 update * add restorePhase async plugin waiting phases Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Remove duplicate application installs, add CSIDataMover failure deletion handling, volsync channel stable-0.7 Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Create app, and therefore namespace, before creating PVC into namespace Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * FEntry typo Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * enable volsync MoverSecurityContext Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Update mysql/mariadb pod securityContext to match Dockerfile USER for volsync to pick up. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * add vsmv1alpha1 to client scheme for failure cleanup Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> --------- Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Co-authored-by: Tiger Kaovilai <tkaovila@redhat.com> Co-authored-by: Tiger Kaovilai <tiger+github@redhat.com> Co-authored-by: hhpatel14 <hitpatel@redhat.com>

Add data mover to e2e

a94ad42

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> override images comment image overrides Use 1.1 images + emcmulla/csi-plugin:latest Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

openshift-ci bot requested review from savitharaghunathan and sseago November 10, 2022 17:02

go1.17 pin gomega to v1.22.1

ff32807

kaovilai mentioned this pull request Nov 10, 2022

Check for empty VSR.Status phase openshift/velero-plugin-for-csi#54

Closed

use ecmculla csi-plugin+dataMover

60e58d2

Up timeouts

458a792

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

kaovilai force-pushed the datamovere2e-1.1 branch from a96db3e to 458a792 Compare November 16, 2022 11:39

up ginkgo/v2 to v2.4

29f8718

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

kaovilai force-pushed the datamovere2e-1.1 branch from 5c85335 to 29f8718 Compare November 16, 2022 15:02

Set mysql datamover to pending

95775f5

add back mysql and increase timeout

8e386bb

kaovilai force-pushed the datamovere2e-1.1 branch from 42ac489 to 213fb9e Compare November 17, 2022 17:45

eemcmullan reviewed Nov 17, 2022

View reviewed changes

add more logs

f19a66c

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

kaovilai force-pushed the datamovere2e-1.1 branch from 213fb9e to f19a66c Compare November 18, 2022 01:36

kaovilai added 2 commits November 18, 2022 12:47

Add GCP Data Mover

7c0beff

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

Update e2e comment on added gcp support

8e9c240

weshayutin reviewed Nov 18, 2022

View reviewed changes

tests/e2e/lib/dpa_helpers.go Show resolved Hide resolved

weshayutin reviewed Nov 18, 2022

View reviewed changes

weshayutin requested changes Nov 18, 2022

View reviewed changes

weshayutin approved these changes Nov 18, 2022

View reviewed changes

weshayutin merged commit 19c4952 into openshift:oadp-1.1 Nov 18, 2022

eemcmullan reviewed Nov 18, 2022

View reviewed changes

hhpatel14 pushed a commit to hhpatel14/oadp-operator that referenced this pull request Apr 10, 2023

master: Add data mover to e2e cherrypick of openshift#863

9c75e34

Co-authored-by: Tiger Kaovilai <tiger+github@redhat.com>

openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/oadp-operator that referenced this pull request Apr 13, 2023

master: Add data mover to e2e cherrypick of openshift#863

7b48fcd

Co-authored-by: Tiger Kaovilai <tiger+github@redhat.com>

		GinkgoWriter.Println("Printing volume-snapshot-mover deployment pod logs")
		GinkgoWriter.Print(GetDeploymentPodContainerLogs(namespace, common.DataMover, common.DataMoverControllerContainer))

OADP-694 oadp-1.1: Add data mover to e2e #863

OADP-694 oadp-1.1: Add data mover to e2e #863

Conversation

kaovilai commented Nov 10, 2022

codecov-commenter commented Nov 10, 2022 • edited Loading

Codecov Report

kaovilai commented Nov 11, 2022

kaovilai commented Nov 11, 2022

kaovilai commented Nov 11, 2022

kaovilai commented Nov 11, 2022

weshayutin commented Nov 15, 2022

kaovilai commented Nov 16, 2022

kaovilai commented Nov 17, 2022

kaovilai commented Nov 17, 2022

kaovilai commented Nov 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaovilai commented Nov 18, 2022

kaovilai commented Nov 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weshayutin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weshayutin left a comment

Choose a reason for hiding this comment

weshayutin left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Nov 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hhpatel14 commented Apr 10, 2023

openshift-cherrypick-robot commented Apr 10, 2023

codecov-commenter commented Nov 10, 2022 •

edited

Loading