🏃Add Etcd e2e tests #2785

wfernandes · 2020-03-25T22:11:59Z

What this PR does / why we need it:
This PR adds an e2e test for testing etcd upgrades.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Ref #2753

test/infrastructure/docker/e2e/docker_test.go

wfernandes · 2020-03-26T15:19:14Z

test/framework/workload_cluster.go

+
+// WaitForPodListCondition waits for the specified condition to be true for all
+// pods returned from the list filter.
+func WaitForPodListCondition(ctx context.Context, input WaitForPodListConditionInput, intervals ...interface{}) {


We can use this to also test the CoreDNS images having the correct image tags after a coredns upgrade.

@gab-satchi If you'd like you can also just do the image check on the coredns deployment.spec.containers[0].Image similar to how @sedefsavas did for kube-proxy.
I had to do a pod list because etcd is deployed as static pods.

Since we are creating the workload clusters within the tests, let's delete them in the AfterEach instead of making the "mgmtClient" and "cluster" vars global.

Also pass in prefix to cluster generator and improve test output

wfernandes · 2020-03-26T16:47:36Z

@gab-satchi @sedefsavas @vincepri If y'all got some time for a review or even a local test run.
I don't mind doing a code walkthrough since the changes "look" substantial.
The tests are quite solid and I'm confident in their stability 🙂

sedefsavas · 2020-03-26T17:04:01Z

test/infrastructure/docker/e2e/docker_upgrade_test.go

+	SetDefaultEventuallyTimeout(10 * time.Minute)
+	SetDefaultEventuallyPollingInterval(10 * time.Second)
+
+	BeforeEach(func() {


Since this part is mostly same with Basic test, instead of separating them to multiple files, why not keep them as is so that beforeach and after each code can be reused?

I made it separate because the original test had failure domains configured and I didn't want to couple the failure domain assertions with the other upgrade tests.
TBH I'm currently unfamiliar with failure domains and as I was trying to write these tests up I wanted to do the upgrade against a cluster that was "as vanilla as I could get".

Having failure domains should not cause issue and checking that logic is important too. I think this is fine for this PR, but we should consider combining these files for not maintaining before/after suites in two places and also AFAIK to run it in parallel, we will need to have them together.

We have a single before/after suites to create the management cluster.
The Before/After each is responsible for creating workload clusters (with different identifiers) and because of that isolation I believe they can run in parallel.

I also think that keeping these tests separate would make them more readable and provide a better understanding of what the expectations are.
Initially, as someone who did not have context, when I came to these tests I didn't know if the upgrades somehow relied on the failure domain configuration because the original full upgrade test was running against a cluster with failure domains configured.

That being said, if others feel that it would be better to couple these tests together I don't mind doing that in a follow up PR.

+1 to separating them but mitigating the speed hit with running them in parallel.

I can create a separate issue to track the work of making these tests run in parallel.

If we're duplicating code, I'd suggest to make the similar code in a different function and call it from multiple places

Issue for running CAPD e2e in parallel #2795

gab-satchi

I'm still running it locally but it's failing to bring up 3 control plane nodes.

vincepri · 2020-03-26T19:27:02Z

test/infrastructure/docker/Makefile

@@ -92,7 +92,7 @@ test-e2e: ## Run the end-to-end tests
 E2E_CONF_FILE ?= e2e/local-e2e.conf
 SKIP_RESOURCE_CLEANUP ?= false
 run-e2e:
-	go test ./e2e -v -ginkgo.v -ginkgo.trace -count=1 -timeout=20m -tags=e2e -e2e.config="$(abspath $(E2E_CONF_FILE))" -skip-resource-cleanup=$(SKIP_RESOURCE_CLEANUP)
+	go test ./e2e -v -ginkgo.v -ginkgo.trace -count=1 -timeout=35m -tags=e2e -e2e.config="$(abspath $(E2E_CONF_FILE))" -skip-resource-cleanup=$(SKIP_RESOURCE_CLEANUP)


Sidenote: we keep increasing the timeout value 😬

heh I noticed the same. But we're also adding more assertions that wait on Pod's running and their attributes

Yeah I know. 😐sigh.
But that's why I suggested opening an issue to run the tests in parallel! 🙂

I can't think of any other way to reduce running time as we add more tests other than compounding all the behaviors under a single test. But at that point, we are writing an e2e test that says "test everything". I don't know the parameters of our CI but maybe we can tweak docker params to make it run faster if we start to see failures due to timeouts.

Issue for running CAPD e2e in parallel #2795

gab-satchi · 2020-03-27T15:13:27Z

I've got the CoreDNS upgrade part ready. I'll wait for this to get merged in

vincepri · 2020-03-27T15:15:11Z

Are we good to merge this in?

vincepri · 2020-03-27T15:15:21Z

/milestone v0.3.3

sedefsavas · 2020-03-27T16:11:53Z

/lgtm

vincepri · 2020-03-27T16:54:36Z

/approve

vincepri · 2020-03-27T16:54:43Z

/milestone v0.3.3

k8s-ci-robot · 2020-03-27T16:55:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri, wfernandes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wfernandes · 2020-03-27T17:52:18Z

Yikes. I should've squashed those commits. My bad.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 25, 2020

k8s-ci-robot requested review from justinsb and ncdc March 25, 2020 22:12

vincepri reviewed Mar 25, 2020

View reviewed changes

test/infrastructure/docker/e2e/docker_test.go Outdated Show resolved Hide resolved

wfernandes force-pushed the etcd-e2e-tests branch from e04a288 to 91c02bb Compare March 26, 2020 04:02

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 26, 2020

wfernandes force-pushed the etcd-e2e-tests branch from b6449c9 to 016483e Compare March 26, 2020 05:17

wfernandes commented Mar 26, 2020

View reviewed changes

Warren Fernandes added 5 commits March 26, 2020 09:43

Add test for etcd upgrade

21146ec

Remove outdated comments

5676636

Isolate upgrade etcd tests

25dd126

Delete workload cluster in AfterEach

314a00b

Since we are creating the workload clusters within the tests, let's delete them in the AfterEach instead of making the "mgmtClient" and "cluster" vars global.

Move etcd upgrade test into separate file

e977a76

Also pass in prefix to cluster generator and improve test output

wfernandes force-pushed the etcd-e2e-tests branch 2 times, most recently from 213ffd7 to e55bad0 Compare March 26, 2020 15:56

Refactor upgrade tests

d205b68

wfernandes force-pushed the etcd-e2e-tests branch from e55bad0 to d205b68 Compare March 26, 2020 16:43

wfernandes changed the title ~~WIP: 🏃Add Etcd e2e tests~~ 🏃Add Etcd e2e tests Mar 26, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 26, 2020

sedefsavas reviewed Mar 26, 2020

View reviewed changes

gab-satchi reviewed Mar 26, 2020

View reviewed changes

vincepri reviewed Mar 26, 2020

View reviewed changes

wfernandes mentioned this pull request Mar 26, 2020

Run CAPD e2e tests in parallel #2795

Closed

k8s-ci-robot added this to the v0.3.3 milestone Mar 27, 2020

k8s-ci-robot assigned sedefsavas Mar 27, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 27, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 27, 2020

k8s-ci-robot merged commit 2cc8570 into kubernetes-sigs:master Mar 27, 2020

wfernandes deleted the etcd-e2e-tests branch March 27, 2020 17:52

gab-satchi mentioned this pull request Mar 30, 2020

KCP: Add e2e tests for ETCD upgrade #2610

Closed

enxebre mentioned this pull request May 4, 2022

🌱 test: tolerate zero pods in WaitForPodListCondition #6478

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🏃Add Etcd e2e tests #2785

🏃Add Etcd e2e tests #2785

wfernandes commented Mar 25, 2020

wfernandes Mar 26, 2020

wfernandes Mar 26, 2020

wfernandes commented Mar 26, 2020 •

edited

Loading

sedefsavas Mar 26, 2020

wfernandes Mar 26, 2020

sedefsavas Mar 26, 2020

wfernandes Mar 26, 2020

gab-satchi Mar 26, 2020

wfernandes Mar 26, 2020

vincepri Mar 26, 2020

wfernandes Mar 26, 2020

gab-satchi left a comment

vincepri Mar 26, 2020

gab-satchi Mar 26, 2020

wfernandes Mar 26, 2020

wfernandes Mar 26, 2020

gab-satchi commented Mar 27, 2020

vincepri commented Mar 27, 2020

vincepri commented Mar 27, 2020

sedefsavas commented Mar 27, 2020

vincepri commented Mar 27, 2020

vincepri commented Mar 27, 2020

k8s-ci-robot commented Mar 27, 2020

wfernandes commented Mar 27, 2020

🏃Add Etcd e2e tests #2785

🏃Add Etcd e2e tests #2785

Conversation

wfernandes commented Mar 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wfernandes commented Mar 26, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gab-satchi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gab-satchi commented Mar 27, 2020

vincepri commented Mar 27, 2020

vincepri commented Mar 27, 2020

sedefsavas commented Mar 27, 2020

vincepri commented Mar 27, 2020

vincepri commented Mar 27, 2020

k8s-ci-robot commented Mar 27, 2020

wfernandes commented Mar 27, 2020

wfernandes commented Mar 26, 2020 •

edited

Loading