Add e2e test for leader election #3076

chiayi · 2023-04-06T20:23:21Z

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation

/kind feature

/kind hotfix

What this PR does / Why we need it:
This is the e2e test that will test the controller leader election feature

Which issue(s) this PR fixes:
Work on #2797

Special notes for your reviewer:

agones-bot · 2023-04-06T20:29:47Z

Build Failed 😱

Build Id: 25733a05-2a80-4946-b0b0-cc4b63467068

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

zmerlynn · 2023-04-06T20:42:03Z

test/e2e/controller/crash_test.go

+		require.NoError(t, err, "Could not delete controller pod")
+	}
+
+	err = deleteAgonesControllerPod(ctx, willBeLeader)


It looks like you are:

killing all replicas except the first

then killing the first

the waiting for everything to stabilize

Why? The way you've done it here, willBeLeader is pretty much irrelevant - you could have just killed all the pods.

The point of deleing the other pods is to shepherd the leader to willBeLeader. But you can't just kill it after doing that, otherwise you might as well have just restarted the whole replicaset.

What we talked about offoine was close to this, but not exactly:

delete all but one as you do in line 82-86

wait for all running (separately, just add a parameter to waitForAgonesControllerRunning if you want to ensure the count - you can add a wantReplicas parameter and then only check it if it's non-negative or something)

then delete willBeLeader

then proceed to verify the controller till works, without waiting again

the point is that shepherding it into one replica, waiting for the others to come back, then killing that one replica, is a pretty solid guarantee that you're killing the leader. The way you have it here also guarantees you killed the leader but it then waits for everything to come back up - we ideally want to show that right after the leader dies, we can still function. Make sense?

Ooh I see I see. I think I have a better understanding now. I will try this.

agones-bot · 2023-04-06T23:36:11Z

Build Succeeded 👏

Build Id: 3f6bed01-131c-4cc1-9b70-2d73a9133faa

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-a18846b-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-a18846b-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-a18846b-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-a18846b-amd64
Linux C++ SDK (build): agonessdk-1.31.0-a18846b-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-a18846b-amd64.zip

A preview of the website (the last 30 builds are retained):

https://a18846b-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3076/head:pr_3076 && git checkout pr_3076
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-a18846b-amd64

agones-bot · 2023-04-06T23:52:02Z

Build Failed 😱

Build Id: 3fadbcbd-18a0-418f-88e1-e725a1ba6137

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-04-07T03:34:21Z

Build Succeeded 👏

Build Id: 2b6a8bd5-43ad-45bb-9a0d-16a409e7d991

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-9b077fb-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-9b077fb-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-9b077fb-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-9b077fb-amd64
Linux C++ SDK (build): agonessdk-1.31.0-9b077fb-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-9b077fb-amd64.zip

A preview of the website (the last 30 builds are retained):

https://9b077fb-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3076/head:pr_3076 && git checkout pr_3076
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-9b077fb-amd64

agones-bot · 2023-04-07T18:20:52Z

Build Succeeded 👏

Build Id: 96227e0e-5e7b-4178-ba39-2c61d580ec6e

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-2eaed5a-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-2eaed5a-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-2eaed5a-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-2eaed5a-amd64
Linux C++ SDK (build): agonessdk-1.31.0-2eaed5a-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-2eaed5a-amd64.zip

A preview of the website (the last 30 builds are retained):

https://2eaed5a-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3076/head:pr_3076 && git checkout pr_3076
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-2eaed5a-amd64

zmerlynn

I think you can take it out of WIP, LGTM!

google-oss-prow · 2023-04-07T19:36:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chiayi, zmerlynn
Once this PR has been reviewed and has the lgtm label, please assign markmandel for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

agones-bot · 2023-04-07T20:46:49Z

Build Failed 😱

Build Id: 893bb6fb-da2d-49ea-bc4d-ebb65e76218f

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

chiayi · 2023-04-07T20:51:13Z

Seemed to have timed out waiting for condition during test: TestGameServerAllocationDuringGameServerDeletion/rolling_update. Going to re-run to see if it's a flake.

agones-bot · 2023-04-08T00:03:42Z

Build Succeeded 👏

Build Id: 3fd23d4d-215f-4ce1-9485-cd541d9fc4b9

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-95118f8-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-95118f8-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-95118f8-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-95118f8-amd64
Linux C++ SDK (build): agonessdk-1.31.0-95118f8-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-95118f8-amd64.zip

A preview of the website (the last 30 builds are retained):

https://95118f8-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3076/head:pr_3076 && git checkout pr_3076
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-95118f8-amd64

google-oss-prow · 2023-04-08T04:08:02Z

New changes are detected. LGTM label has been removed.

agones-bot · 2023-04-08T05:46:42Z

Build Succeeded 👏

Build Id: d4a59815-523b-4704-b465-276da360d8c7

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-bca2839-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-bca2839-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-bca2839-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-bca2839-amd64
Linux C++ SDK (build): agonessdk-1.31.0-bca2839-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-bca2839-amd64.zip

A preview of the website (the last 30 builds are retained):

https://bca2839-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3076/head:pr_3076 && git checkout pr_3076
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-bca2839-amd64

agones-bot · 2023-04-08T21:24:50Z

Build Succeeded 👏

Build Id: 2c4f6376-fd22-4bbb-80b0-299b1fe7cbef

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-5dcaf98-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-5dcaf98-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-5dcaf98-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-5dcaf98-amd64
Linux C++ SDK (build): agonessdk-1.31.0-5dcaf98-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-5dcaf98-amd64.zip

A preview of the website (the last 30 builds are retained):

https://5dcaf98-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3076/head:pr_3076 && git checkout pr_3076
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-5dcaf98-amd64

Add e2e test for leader election

google-oss-prow bot added the do-not-merge/work-in-progress label Apr 6, 2023

google-oss-prow bot requested review from aLekSer and markmandel April 6, 2023 20:23

google-oss-prow bot added the size/M label Apr 6, 2023

chiayi requested review from zmerlynn and removed request for aLekSer April 6, 2023 20:23

zmerlynn reviewed Apr 6, 2023

View reviewed changes

chiayi force-pushed the agoneshabranch branch from a4e86b3 to a18846b Compare April 6, 2023 20:47

chiayi force-pushed the agoneshabranch branch from a18846b to 6547f20 Compare April 6, 2023 23:46

chiayi force-pushed the agoneshabranch branch from 6547f20 to 9b077fb Compare April 6, 2023 23:54

chiayi force-pushed the agoneshabranch branch from 9b077fb to 2eaed5a Compare April 7, 2023 17:24

chiayi force-pushed the agoneshabranch branch from 2eaed5a to 95118f8 Compare April 7, 2023 18:33

zmerlynn approved these changes Apr 7, 2023

View reviewed changes

google-oss-prow bot assigned zmerlynn Apr 7, 2023

google-oss-prow bot added the lgtm label Apr 7, 2023

chiayi marked this pull request as ready for review April 7, 2023 20:13

google-oss-prow bot removed the do-not-merge/work-in-progress label Apr 7, 2023

google-oss-prow bot requested review from aLekSer and EricFortin April 7, 2023 20:14

Add e2e test for leader election

bca2839

chiayi force-pushed the agoneshabranch branch from 95118f8 to bca2839 Compare April 8, 2023 04:07

google-oss-prow bot removed the lgtm label Apr 8, 2023

Fix minor typo

5dcaf98

zmerlynn enabled auto-merge (squash) April 8, 2023 20:36

zmerlynn merged commit f2dd4c4 into googleforgames:main Apr 8, 2023

Kalaiselvi84 added the kind/feature New features for Agones label Apr 10, 2023

Kalaiselvi84 added this to the 1.31.0 milestone Apr 10, 2023

Kalaiselvi84 pushed a commit to Kalaiselvi84/agones that referenced this pull request Apr 11, 2023

Add e2e test for leader election (googleforgames#3076)

2b5ce6f

Add e2e test for leader election

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add e2e test for leader election #3076

Add e2e test for leader election #3076

chiayi commented Apr 6, 2023

agones-bot commented Apr 6, 2023

zmerlynn Apr 6, 2023

chiayi Apr 6, 2023

agones-bot commented Apr 6, 2023

agones-bot commented Apr 6, 2023

agones-bot commented Apr 7, 2023

agones-bot commented Apr 7, 2023

zmerlynn left a comment

google-oss-prow bot commented Apr 7, 2023

agones-bot commented Apr 7, 2023

chiayi commented Apr 7, 2023

agones-bot commented Apr 8, 2023

google-oss-prow bot commented Apr 8, 2023

agones-bot commented Apr 8, 2023

agones-bot commented Apr 8, 2023

Add e2e test for leader election #3076

Add e2e test for leader election #3076

Conversation

chiayi commented Apr 6, 2023

agones-bot commented Apr 6, 2023

zmerlynn Apr 6, 2023

Choose a reason for hiding this comment

chiayi Apr 6, 2023

Choose a reason for hiding this comment

agones-bot commented Apr 6, 2023

agones-bot commented Apr 6, 2023

agones-bot commented Apr 7, 2023

agones-bot commented Apr 7, 2023

zmerlynn left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Apr 7, 2023

agones-bot commented Apr 7, 2023

chiayi commented Apr 7, 2023

agones-bot commented Apr 8, 2023

google-oss-prow bot commented Apr 8, 2023

agones-bot commented Apr 8, 2023

agones-bot commented Apr 8, 2023