Add more time and logging to extensions test #2996

chiayi · 2023-02-24T01:10:39Z

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation
/kind feature

/kind hotfix

What this PR does / Why we need it:
It attempts to fix the flaky e2e extensions test.

Which issue(s) this PR fixes:
Maybe #2992

Special notes for your reviewer:
I ran my test locally around 20+ times and it all passed. I'm not 100% sure what the issue is with the

test/e2e/extensions/high_availability_test.go

agones-bot · 2023-02-24T02:39:07Z

Build Succeeded 👏

Build Id: 7b1f2edd-7bc3-4101-9736-c0f7da24cd1c

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-622f8d3-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-622f8d3-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-622f8d3-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-622f8d3-amd64
Linux C++ SDK (build): agonessdk-1.30.0-622f8d3-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-622f8d3-amd64.zip

A preview of the website (the last 30 builds are retained):

https://622f8d3-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2996/head:pr_2996 && git checkout pr_2996
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-622f8d3-amd64

agones-bot · 2023-02-24T03:17:58Z

Build Succeeded 👏

Build Id: d2d17ea8-7c98-4724-88ac-5651b27599ca

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-8fbf897-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-8fbf897-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-8fbf897-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-8fbf897-amd64
Linux C++ SDK (build): agonessdk-1.30.0-8fbf897-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-8fbf897-amd64.zip

A preview of the website (the last 30 builds are retained):

https://8fbf897-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2996/head:pr_2996 && git checkout pr_2996
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-8fbf897-amd64

agones-bot · 2023-02-24T03:25:23Z

Build Failed 😱

Build Id: c5ad3f89-1f2e-4edc-8696-51573116735f

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-02-24T04:51:07Z

Build Succeeded 👏

Build Id: 0f3b29f1-0bd5-47b4-bf99-41e42397ff04

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-13a1725-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-13a1725-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-13a1725-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-13a1725-amd64
Linux C++ SDK (build): agonessdk-1.30.0-13a1725-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-13a1725-amd64.zip

A preview of the website (the last 30 builds are retained):

https://13a1725-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2996/head:pr_2996 && git checkout pr_2996
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-13a1725-amd64

markmandel · 2023-02-24T17:18:26Z

test/e2e/extensions/high_availability_test.go

+		}
+
+		return false
+	}, 30*time.Second, 20*time.Millisecond)


Suggested change

}, 30*time.Second, 20*time.Millisecond)

}, 5 * time.Minute, time.Second)

We don't need to hammer the K8s API and slow down other e2e tests. Once a second should be sufficient.

markmandel · 2023-02-24T17:20:20Z

test/e2e/extensions/high_availability_test.go


-	assert.NoError(t, waitForAgonesExtensionsRunning(ctx))
+		if len(extList.Items) == 2 {


A check I would add - is the len(2) but also is the list above different from the list we just got in extList - that way we definitely know we have a new replacement pod, and it's up and running.

Otherwise we could get 1 pod running 1 terminating.

You could likely just do a comparison of sorted pod names in both lists. A bit of code, but then we know we're in the place we expect to be.

https://pkg.go.dev/sort#Strings

I think this would be easier to integrate into deleteAgonesExtensionsPods, and here's why: that function is misnamed. It's deleting exactly one pod, and that pod has a name, known to that function as list.Items[1].ObjectMeta.Name.

It's much easier just to manually use the KubeClient.Delete, then require.Eventually() that the KubeClient.Get on the Pod fails - i.e. manually delete the single pod and just wait on that single pod to be gone.

And rename the function as deleteAgonesExtensionPod.

I would maybe then change waitForAgonesExtensionsRunning to actually make sure there are 2 running.

Works for me 👍🏻

Added a comparison to make sure that they are not equal, please take a look.

This looks fine to me - it's more code than what I suggested above but it'll get the job done for sure. We can clean it up later if needed.

test/e2e/extensions/high_availability_test.go

agones-bot · 2023-02-24T20:09:16Z

Build Failed 😱

Build Id: b6f35e71-db1e-456c-8535-e358b49ab69a

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2023-02-24T20:37:04Z

Look like the linter got you, but also looks good to me!

test/e2e/extensions/high_availability_test.go:121:2: rangeValCopy: each iteration copies 1008 bytes (consider pointers or indexing) (gocritic)
	for i, e := range list.Items {

chiayi · 2023-02-24T21:53:43Z

Had to remove the unused function. Should be good now, please take a look.

agones-bot · 2023-02-24T21:54:44Z

Build Failed 😱

Build Id: e18568e1-2194-423d-ad1a-c21944efb233

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2023-02-24T21:56:51Z

Linter got you again on the same item 😄

test/e2e/extensions/high_availability_test.go:117:2: rangeValCopy: each iteration copies 1008 bytes (consider pointers or indexing) (gocritic)
	for i, e := range list.Items {
	^

if you run make lint you can see the issue locally.

chiayi · 2023-02-24T22:04:22Z

Yeah, that was part of the function I removed on the latest push 😅

google-oss-prow · 2023-02-24T22:10:58Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chiayi, zmerlynn
Once this PR has been reviewed and has the lgtm label, please assign alekser for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

agones-bot · 2023-02-24T22:43:10Z

Build Succeeded 👏

Build Id: 5cf90c26-0994-4ade-b1b0-63ed1d2dc040

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-b8b9cce-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-b8b9cce-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-b8b9cce-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-b8b9cce-amd64
Linux C++ SDK (build): agonessdk-1.30.0-b8b9cce-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-b8b9cce-amd64.zip

A preview of the website (the last 30 builds are retained):

https://b8b9cce-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2996/head:pr_2996 && git checkout pr_2996
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-b8b9cce-amd64

google-oss-prow bot requested review from aLekSer and dzlier-gcp February 24, 2023 01:10

google-oss-prow bot added the size/XS label Feb 24, 2023

chiayi requested review from zmerlynn and markmandel and removed request for aLekSer and dzlier-gcp February 24, 2023 01:11

chiayi force-pushed the clean branch from 622f8d3 to 8fbf897 Compare February 24, 2023 01:43

google-oss-prow bot added size/S and removed size/XS labels Feb 24, 2023

markmandel reviewed Feb 24, 2023

View reviewed changes

chiayi force-pushed the clean branch 2 times, most recently from 7c7d050 to 13a1725 Compare February 24, 2023 03:08

markmandel reviewed Feb 24, 2023

View reviewed changes

zmerlynn reviewed Feb 24, 2023

View reviewed changes

test/e2e/extensions/high_availability_test.go Outdated Show resolved Hide resolved

chiayi force-pushed the clean branch from 13a1725 to 1944b29 Compare February 24, 2023 20:03

google-oss-prow bot added size/M and removed size/S labels Feb 24, 2023

chiayi force-pushed the clean branch from 1944b29 to 8f038a2 Compare February 24, 2023 21:49

Add more time and loggint to extensions test

b8b9cce

chiayi force-pushed the clean branch from 8f038a2 to b8b9cce Compare February 24, 2023 21:53

zmerlynn approved these changes Feb 24, 2023

View reviewed changes

google-oss-prow bot assigned zmerlynn Feb 24, 2023

google-oss-prow bot added the lgtm label Feb 24, 2023

zmerlynn enabled auto-merge (squash) February 24, 2023 22:34

zmerlynn merged commit 4c47cac into googleforgames:main Feb 24, 2023

Kalaiselvi84 added this to the 1.30.0 milestone Feb 28, 2023

Kalaiselvi84 added the kind/feature New features for Agones label Feb 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more time and logging to extensions test #2996

Add more time and logging to extensions test #2996

chiayi commented Feb 24, 2023 •

edited

Loading

agones-bot commented Feb 24, 2023

agones-bot commented Feb 24, 2023

agones-bot commented Feb 24, 2023

agones-bot commented Feb 24, 2023

markmandel Feb 24, 2023

markmandel Feb 24, 2023

markmandel Feb 24, 2023 •

edited

Loading

zmerlynn Feb 24, 2023

markmandel Feb 24, 2023

chiayi Feb 24, 2023

zmerlynn Feb 24, 2023

agones-bot commented Feb 24, 2023

markmandel commented Feb 24, 2023

chiayi commented Feb 24, 2023

agones-bot commented Feb 24, 2023

markmandel commented Feb 24, 2023

chiayi commented Feb 24, 2023

google-oss-prow bot commented Feb 24, 2023

agones-bot commented Feb 24, 2023

	}, 30time.Second, 20time.Millisecond)
	}, 5 * time.Minute, time.Second)


		assert.NoError(t, waitForAgonesExtensionsRunning(ctx))
		if len(extList.Items) == 2 {

Add more time and logging to extensions test #2996

Add more time and logging to extensions test #2996

Conversation

chiayi commented Feb 24, 2023 • edited Loading

agones-bot commented Feb 24, 2023

agones-bot commented Feb 24, 2023

agones-bot commented Feb 24, 2023

agones-bot commented Feb 24, 2023

markmandel Feb 24, 2023

Choose a reason for hiding this comment

markmandel Feb 24, 2023

Choose a reason for hiding this comment

markmandel Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

zmerlynn Feb 24, 2023

Choose a reason for hiding this comment

markmandel Feb 24, 2023

Choose a reason for hiding this comment

chiayi Feb 24, 2023

Choose a reason for hiding this comment

zmerlynn Feb 24, 2023

Choose a reason for hiding this comment

agones-bot commented Feb 24, 2023

markmandel commented Feb 24, 2023

chiayi commented Feb 24, 2023

agones-bot commented Feb 24, 2023

markmandel commented Feb 24, 2023

chiayi commented Feb 24, 2023

google-oss-prow bot commented Feb 24, 2023

agones-bot commented Feb 24, 2023

chiayi commented Feb 24, 2023 •

edited

Loading

markmandel Feb 24, 2023 •

edited

Loading