test: speed up e2e test by running in parallel #861

cvvz · 2023-03-29T03:26:52Z

What type of PR is this?
/kind test

What this PR does / why we need it:
Currently e2e test is running by go test in serial mode, change it to parallel mode to fast the pipeline and detect flaky failure early.

Which issue(s) this PR fixes:

Fixes #

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
tested upgrade from previous version

Special notes for your reviewer:

Release note:

none

cvvz · 2023-03-29T03:27:04Z

/ok-to-test

cvvz · 2023-04-01T01:17:09Z

/test pull-blob-csi-driver-e2e-capz

cvvz · 2023-04-04T06:15:47Z

Suite will probably be panic if there is any spec failed, the problem will be solved either by onsi/ginkgo#1178 or kubernetes/kubernetes#112923 (kubernetes v1.27).

I've handled the panic properly, so it won't cause any real problem, we can still get the logs and other information to dig into the failed specs. However, we should upgrade kubernetes package to v1.27 or upgrade ginkgo after onsi/ginkgo#1178 has been merged to solve the panic problem. Tracked by #865

andyzhangx

could you squash the commits? thanks.

commit d9fcdae Merge: 6569b29 07f79d4 Author: weizhichen <weizhichen@microsoft.com> Date: Tue Apr 4 08:36:31 2023 +0000 Merge branch 'master' of https://github.com/kubernetes-sigs/blob-csi-driver into e2e-test commit 6569b29 Author: weizhichen <weizhichen@microsoft.com> Date: Mon Apr 3 11:37:40 2023 +0000 parallel again commit 9ed8a55 Merge: 551c409 a47bc07 Author: weizhichen <weizhichen@microsoft.com> Date: Mon Apr 3 10:27:26 2023 +0000 Merge branch 'master' of https://github.com/kubernetes-sigs/blob-csi-driver into e2e-test commit 551c409 Author: weizhichen <weizhichen@microsoft.com> Date: Mon Apr 3 10:21:04 2023 +0000 another flaky test commit 38e0b6a Author: weizhichen <weizhichen@microsoft.com> Date: Mon Apr 3 08:46:01 2023 +0000 fix panic commit cce9102 Author: weizhichen <weizhichen@microsoft.com> Date: Mon Apr 3 07:40:40 2023 +0000 flaky: e2e: fix pre-provisioned test commit 3b5a6ee Author: weizhichen <weizhichen@microsoft.com> Date: Mon Apr 3 03:19:55 2023 +0000 framework init commit 6fa9cd4 Author: weizhichen <weizhichen@microsoft.com> Date: Sun Apr 2 08:31:37 2023 +0000 flake attempts commit 7c752ae Author: weizhichen <weizhichen@microsoft.com> Date: Sun Apr 2 03:29:32 2023 +0000 cancel parallel commit cffd14a Author: weizhichen <weizhichen@microsoft.com> Date: Sun Apr 2 00:31:23 2023 +0000 flakeattempts commit c6a0266 Author: weizhichen <weizhichen@microsoft.com> Date: Sat Apr 1 20:10:30 2023 +0000 make nfs test serial commit 9bba102 Author: weizhichen <weizhichen@microsoft.com> Date: Sat Apr 1 01:46:32 2023 +0000 make private endpoint test serial commit fe1e3b8 Author: weizhichen <weizhichen@microsoft.com> Date: Sat Apr 1 00:52:47 2023 +0000 output-interceptor-mode=none commit c4a3a91 Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 10:05:40 2023 +0000 no flake-attempts commit a94726a Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 09:59:57 2023 +0000 gomega success commit f9638e2 Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 09:27:02 2023 +0000 pass project root to e2e test commit 6b795c6 Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 07:48:30 2023 +0000 fix commit 2a14b30 Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 07:07:39 2023 +0000 fix restart driver commit 3553af8 Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 06:37:30 2023 +0000 fix pre_provisioned_provided_credentials_tester commit 0aa27dd Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 06:10:13 2023 +0000 move verify examples to ginkgo Node container commit 7ebd065 Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 04:58:18 2023 +0000 add flake-attempts commit 02a48ba Author: weizhichen <weizhichen@microsoft.com> Date: Fri Mar 31 04:57:40 2023 +0000 Revert "use seed to repro" This reverts commit 1c5fea8. commit 1c5fea8 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 16:27:03 2023 +0000 use seed to repro commit c52b495 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 13:57:37 2023 +0000 fix: container name commit f1f55e4 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 13:23:25 2023 +0000 fix dynamic inline volume and byok volume commit 0e4b11e Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 12:24:47 2023 +0000 revert --delete-namespace commit 7df5df4 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 10:53:45 2023 +0000 fix: set framework flags commit 103840c Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 09:52:05 2023 +0000 set delete-namespace=false to avoid deleting ns which is used by other specs during parallel testing commit cda91f8 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 08:36:47 2023 +0000 fix: should notify all goroutine channel by close commit 0aa42e6 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 07:08:10 2023 +0000 fix NodeTimeout, need context commit 9c1fb1d Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 06:49:52 2023 +0000 fix defer cleanup order commit b4d44b2 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 06:43:29 2023 +0000 add 10min GracePeriod for AfterSuite to avoid exit too quick commit 49fdec3 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 05:22:55 2023 +0000 workaround the issue: kubernetes/kubernetes#107065 commit 30369fb Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 05:19:35 2023 +0000 fix restartDriverTest panic commit 67ff546 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 05:00:44 2023 +0000 fix: dump namespace info commit 54a2d2f Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 04:25:58 2023 +0000 add log after driver pod is restarted commit fd32cab Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 03:04:35 2023 +0000 fix commit 7074288 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 02:37:11 2023 +0000 adjust AccountCreationLeak check threshold commit b86e385 Author: weizhichen <weizhichen@microsoft.com> Date: Thu Mar 30 02:08:51 2023 +0000 fix: reduce csi driver daemon restart times commit 63e820f Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 18:35:54 2023 +0000 fix pwd commit 65c8c04 Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 17:44:04 2023 +0000 fix: no log print out after blob daemonset is recreated commit b9754e4 Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 15:03:27 2023 +0000 fix commit a9b913d Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 14:55:11 2023 +0000 fix: createvolume and initialize volumeID in beforeeach commit e6aa3a8 Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 11:04:30 2023 +0000 fix: set azidentity.EnvironmentCredential for each process commit 2dd2015 Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 09:19:49 2023 +0000 fix init k8s client error commit b319406 Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 08:33:25 2023 +0000 fix BeforeSuite and AfterSuite commit 1496638 Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 07:49:07 2023 +0000 1. use e2e.test v1.26.0 2. upgrade ginkgo to v2.9.2 to use GinkgoHelper 3. add --fast-fail commit d6d05ea Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 03:49:51 2023 +0000 fix commit 6e6d71f Author: weizhichen <weizhichen@microsoft.com> Date: Wed Mar 29 03:22:14 2023 +0000 test: speed up e2e test by running in parallel

cvvz · 2023-04-04T10:39:54Z

/retest

cvvz · 2023-04-04T12:02:12Z

/retest

andyzhangx · 2023-04-04T12:03:15Z

I found there are two driver restart cases, will it affect the test runs?

blob-csi-driver/test/e2e/pre_provisioning_test.go

Lines 337 to 345 in 07f79d4

    
           ginkgo.It("nfs volume mount is still valid after driver restart [blob.csi.azure.com]", func() { 
        
           	// print driver logs before driver restart 
        
           	blobLog := testCmd{ 
        
           		command:  "bash", 
        
           		args:     []string{"test/utils/blob_log.sh"}, 
        
           		startLog: "===================blob log (before restart)===================", 
        
           		endLog:   "====================================================================", 
        
           	} 
        
           	execTestCmd([]testCmd{blobLog})

cvvz · 2023-04-04T22:40:57Z

I found there are two driver restart cases, will it affect the test runs?

Actually, there is only one place to print out log before restart driver, which is in restart script. The others have been removed.

andyzhangx

what's current parallel num and is there a switch to make it serial run if there is issue?

andyzhangx · 2023-04-06T08:59:26Z

/retest

cvvz · 2023-04-06T12:22:58Z

what's current parallel num and is there a switch to make it serial run if there is issue?

It's based on the number of cores on the machine. And we can simply remove -p opition in ginkgo to switch back to serial.

andyzhangx

/lgtm

k8s-ci-robot · 2023-04-08T02:34:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx, cvvz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andyzhangx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cvvz · 2023-04-08T22:33:21Z

/retest

k8s-ci-robot added kind/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 29, 2023

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 29, 2023

k8s-ci-robot requested review from andyzhangx and ZeroMagic March 29, 2023 03:27

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 29, 2023

cvvz force-pushed the e2e-test branch from 852d68f to 103840c Compare March 30, 2023 10:08

cvvz force-pushed the e2e-test branch from 4c3c7ed to 6569b29 Compare April 4, 2023 06:05

cvvz mentioned this pull request Apr 4, 2023

Solve e2e suite panic issue when running in parallel #865

Closed

andyzhangx reviewed Apr 4, 2023

View reviewed changes

cvvz force-pushed the e2e-test branch from d9fcdae to f190dde Compare April 4, 2023 09:07

fix bash

7386e44

andyzhangx reviewed Apr 6, 2023

View reviewed changes

andyzhangx approved these changes Apr 8, 2023

View reviewed changes

k8s-ci-robot assigned andyzhangx Apr 8, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 8, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 8, 2023

k8s-ci-robot merged commit c5db670 into kubernetes-sigs:master Apr 8, 2023

umagnus mentioned this pull request Jul 5, 2024

test: speed up e2e test by running in parallel kubernetes-sigs/azurefile-csi-driver#1956

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: speed up e2e test by running in parallel #861

test: speed up e2e test by running in parallel #861

cvvz commented Mar 29, 2023 •

edited

Loading

cvvz commented Mar 29, 2023

cvvz commented Apr 1, 2023

cvvz commented Apr 4, 2023 •

edited

Loading

andyzhangx left a comment

cvvz commented Apr 4, 2023

cvvz commented Apr 4, 2023

andyzhangx commented Apr 4, 2023

cvvz commented Apr 4, 2023

andyzhangx left a comment

andyzhangx commented Apr 6, 2023

cvvz commented Apr 6, 2023

andyzhangx left a comment

k8s-ci-robot commented Apr 8, 2023

cvvz commented Apr 8, 2023

test: speed up e2e test by running in parallel #861

test: speed up e2e test by running in parallel #861

Conversation

cvvz commented Mar 29, 2023 • edited Loading

cvvz commented Mar 29, 2023

cvvz commented Apr 1, 2023

cvvz commented Apr 4, 2023 • edited Loading

andyzhangx left a comment

Choose a reason for hiding this comment

cvvz commented Apr 4, 2023

cvvz commented Apr 4, 2023

andyzhangx commented Apr 4, 2023

cvvz commented Apr 4, 2023

andyzhangx left a comment

Choose a reason for hiding this comment

andyzhangx commented Apr 6, 2023

cvvz commented Apr 6, 2023

andyzhangx left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 8, 2023

cvvz commented Apr 8, 2023

cvvz commented Mar 29, 2023 •

edited

Loading

cvvz commented Apr 4, 2023 •

edited

Loading