Leader Election in Custom Controller

[Kalaiselvi84]

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind hotfix
/kind release

What this PR does / Why we need it:

Which issue(s) this PR fixes:

Closes #3694

Special notes for your reviewer:

[Kalaiselvi84]

I tried with 2 Replicas and I am able to see that only one custom-controller pod is serving the Info logs(https://screenshot.googleplex.com/999yiFkiWDKtVKj) and the other custom-controller pod is awaiting to become a leader (https://screenshot.googleplex.com/72j8n4PhtkMJu89). Does this look good to you?

[agones-bot]

Build Failed 😱

Build Id: c9f488db-5a3c-4b6d-a760-42e2f0dba502

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[markmandel]

🔥 Nice!

[agones-bot]

Build Succeeded 👏

Build Id: 8c65ef28-a178-4d6f-9b4f-57fd1168442e

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.40.0-dev-fe191ea-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.40.0-dev-fe191ea-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.40.0-dev-fe191ea-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.40.0-dev-fe191ea-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.40.0-dev-fe191ea-amd64
• Linux C++ SDK (build): agonessdk-1.40.0-dev-fe191ea-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.40.0-dev-fe191ea-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://fe191ea-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/3696/head:pr_3696 && git checkout pr_3696
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.40.0-dev-fe191ea-amd64

[Kalaiselvi84]

After testing the code with 3 replicas, one controller became the leader while the other two controllers awaited their turn. When I deleted the leader pod, one of the awaiting controllers took over as the new leader and logged the event. Additionally, a new controller was automatically created.

Resources that I followed:
go-leaderElection, Leader-with-lease-election, and resolved the cannot get resource "leases" in api group "coordination.k8s.io" issue by adding leases resources in the ClusterRole(deployment.yaml) following this documentation.

Please review the code and share your thoughts. Log - https://gist.github.com/Kalaiselvi84/958f4280cb1f2202d62471e049c03aec

Please let me know if we need a quick call to show the testing.