🏃 Add common utilities for provider-based testing #1474

detiber · 2019-10-01T19:51:16Z

What this PR does / why we need it:

Starts to build out common testing utilities for use across providers

k8s-ci-robot · 2019-10-01T19:51:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: detiber

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [detiber]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

detiber · 2019-10-03T15:25:11Z

/assign @chuckha ptal

k8s-ci-robot · 2019-10-03T15:25:13Z

@detiber: GitHub didn't allow me to assign the following users: ptal.

Note that only kubernetes-sigs members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @chuckha ptal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

chuckha

this looks fine to me, were you looking for any particular feedback?

I'd probably factor out the ginkgo.GinkgoWriter because I hate globals but it really doesn't matter for test code. Same with all the var flags, but again, this is totally fine since it's test code.

Also lately i've been favoring three sections of go imports, stdlib, external imports, current project imports.

chuckha · 2019-10-03T15:49:27Z

test/e2e/util/kind/setup.go

+	buffer := &bytes.Buffer{}
+	defer pipe.Close()
+	for {
+		n, _ := buffer.ReadFrom(pipe)


should this be reading until io.EOF or is n==0 the right thing?

Oy, this rabbit hole led me down another, which is that we shouldn't be using exec.Cmd.Run with pipes, and instead should be using Start/Wait... working on cleaning up.

Updated this, but still need to test the changes.

hold off on reviewing the changes, it won't work as is 😂

not a problem. i'm just starting to work on some of the trickier technical bits of the test proposal and this PR is a fairly crucial aspect :D

detiber · 2019-10-03T16:03:04Z

Also lately i've been favoring three sections of go imports, stdlib, external imports, current project imports.

This has been my preference as well, but as a group we've been prioritizing for two groups, so I've been trying to follow the group preference.

ncdc · 2019-10-03T17:03:38Z

Aside from the visual aspect, there isn't any difference if you order imports one way or another, right? My personal preference is to ignore import grouping and let the tools do it however they want. IMHO there are more important things to be reviewing 😄

chuckha · 2019-10-04T15:27:07Z

@ncdc yep, I guess I was making small talk in the PR. I didn't intend for that comment to be blocking nor do I have any interest in changing how we do things. I was asked for a review and since the code is test code and written in a readable way there really wasn't much to review.

chuckha · 2019-10-04T16:13:30Z

test/e2e/util/kind/setup.go

+
+// Cluster represents the running state of a KIND cluster.
+// An empty struct is enough to call Setup() on.
+type Cluster struct {


I can see this becoming a ManagementCluster concept and we end up with constructors such as NewManagementClusterFromKind() or similar that allows us to keep the cluster concept and abstract out kind. I don't always want kind to be my management cluster

I should mention this is future work, i think this is totally fine as is.

detiber · 2019-10-04T20:39:59Z

/test pull-cluster-api-integration

detiber · 2019-10-07T20:41:23Z

@chuckha I believe this should be good to go now.

vincepri

We should add a doc.go example of how to use these test utilities, or a README for developers

test/e2e/e2e_suite_test.go

vincepri · 2019-10-07T21:10:30Z

test/e2e/capi_test.go

+	"github.com/onsi/ginkgo"
+)
+
+var _ = ginkgo.Describe("functional tests", func() {


Do we need this file?

I mainly added this to facilitate testing of the utilities, but after CAPD and CABPK are in tree I think we'd want to have E2E testing.

Not directly related, but after this is merged I want to refactor the integration tests to use these utilities as well, so I think I'll move them up a directory to avoid confusion for why the integration tests are importing a package from the e2e dir.

detiber · 2019-10-08T12:28:04Z

We should add a doc.go example of how to use these test utilities, or a README for developers

Agreed, I can do that after I've updated CAPA to use these utilities and create an example based off of that.

- Add common kind utility for provider-based testing - Add flag helper - Add scheme helper - Add component helper

chuckha · 2019-10-08T13:09:50Z

test/e2e/e2e_suite_test.go

+	suiteTmpDir string
+)
+
+var _ = ginkgo.BeforeSuite(func() {


Is there a way to opt out of the before and after suites?

Not that I'm aware of, but I wouldn't expect other providers to inherit this suite directly, but instead we'd continue to move common logic into the helpers, and have each provider implement their own suites.

chuckha · 2019-10-08T13:19:39Z

test/helpers/kind/setup.go

+	}
+
+	gomega.Expect(cmd.Start()).NotTo(gomega.HaveOccurred())
+	wg.Wait()


the wait groups and go routines can be avoided completely by:

getting a reference to the stderr and stdout pipes

Running start

Running ioutil.ReadAll on the pipes

waiting for the command to finish

This example is a decent start: https://golang.org/pkg/os/exec/#Cmd.StderrPipe

The ioutil.ReadAll example seems geared towards a single pipe, if we used it here with both pipes, wouldn't it output the streams separately instead of attempting to interleave them?

chuckha · 2019-10-08T13:22:02Z

test/helpers/kind/setup.go

+limitations under the License.
+*/
+
+package kind


Is there any room for non-kind based e2e tests? it would be good if we didn't bind ourselves to kind in the package name when it's an implementation detail. The one way I could see a package name of kind being good is if we have multiple ways to spin up a management cluster and kind is one of those ways.

I believe there may be, but I would expect them to implement their own helper.

vincepri

/lgtm
/hold

vincepri · 2019-10-08T14:08:35Z

/hold cancel

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 1, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 1, 2019

k8s-ci-robot requested review from chuckha and ncdc October 1, 2019 19:51

detiber changed the title ~~[WIP] 🏃 Add common kind utility for provider-based testing~~ [WIP] 🏃 Add common utilities for provider-based testing Oct 1, 2019

k8s-ci-robot assigned chuckha Oct 3, 2019

chuckha reviewed Oct 3, 2019

View reviewed changes

detiber force-pushed the commonE2EUtils branch from de4f9da to dddb2cf Compare October 4, 2019 16:12

chuckha reviewed Oct 4, 2019

View reviewed changes

detiber force-pushed the commonE2EUtils branch 2 times, most recently from 2f126ef to 42c6627 Compare October 4, 2019 20:19

detiber force-pushed the commonE2EUtils branch from 42c6627 to deb1375 Compare October 7, 2019 19:31

detiber changed the title ~~[WIP] 🏃 Add common utilities for provider-based testing~~ 🏃 Add common utilities for provider-based testing Oct 7, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 7, 2019

vincepri reviewed Oct 7, 2019

View reviewed changes

detiber force-pushed the commonE2EUtils branch from deb1375 to 1134f61 Compare October 8, 2019 12:26

Add helper utilities for testing

fe32969

- Add common kind utility for provider-based testing - Add flag helper - Add scheme helper - Add component helper

detiber force-pushed the commonE2EUtils branch from 1134f61 to fe32969 Compare October 8, 2019 12:30

chuckha reviewed Oct 8, 2019

View reviewed changes

vincepri approved these changes Oct 8, 2019

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 8, 2019

k8s-ci-robot assigned vincepri Oct 8, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 8, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 8, 2019

k8s-ci-robot merged commit 7f1890e into kubernetes-sigs:master Oct 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🏃 Add common utilities for provider-based testing #1474

🏃 Add common utilities for provider-based testing #1474

detiber commented Oct 1, 2019

k8s-ci-robot commented Oct 1, 2019

detiber commented Oct 3, 2019

k8s-ci-robot commented Oct 3, 2019

chuckha left a comment

chuckha Oct 3, 2019

detiber Oct 4, 2019

detiber Oct 4, 2019

detiber Oct 4, 2019

chuckha Oct 4, 2019

detiber commented Oct 3, 2019

ncdc commented Oct 3, 2019

chuckha commented Oct 4, 2019

chuckha Oct 4, 2019

chuckha Oct 4, 2019

detiber commented Oct 4, 2019

detiber commented Oct 7, 2019

vincepri left a comment

vincepri Oct 7, 2019

detiber Oct 8, 2019

detiber commented Oct 8, 2019

chuckha Oct 8, 2019

detiber Oct 8, 2019

chuckha Oct 8, 2019

detiber Oct 8, 2019

chuckha Oct 8, 2019

detiber Oct 8, 2019

vincepri left a comment

vincepri commented Oct 8, 2019

🏃 Add common utilities for provider-based testing #1474

🏃 Add common utilities for provider-based testing #1474

Conversation

detiber commented Oct 1, 2019

k8s-ci-robot commented Oct 1, 2019

detiber commented Oct 3, 2019

k8s-ci-robot commented Oct 3, 2019

chuckha left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

detiber commented Oct 3, 2019

ncdc commented Oct 3, 2019

chuckha commented Oct 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

detiber commented Oct 4, 2019

detiber commented Oct 7, 2019

vincepri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

detiber commented Oct 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincepri left a comment

Choose a reason for hiding this comment

vincepri commented Oct 8, 2019