Optimise GameServer Sub-Controller Queues

[markmandel]

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking

/kind bug

/kind cleanup
/kind documentation
/kind feature
/kind hotfix
/kind release

What this PR does / Why we need it:

Looking at pprof profiles of the Agones Controller, we see a lot memory in the Migration controller and the Missing Controller queues.

This implements improved checking at the event layer, before placing items in the workerqueue, which drops the memory footprint, and also allows for faster processing of the edge cases that Missing and Migration Controllers implement.

Also did a review of the Health Controller and made a small improvement there as well.

Looking at the memory pprof profile and load tests, we see a decrease in memory usage, and also less of a slow "step up" of memory over time.

Which issue(s) this PR fixes:

Closes #3748

Special notes for your reviewer:

Here's some pretty pictures!

Here's the two tests run side by side - I only did 30m, but you can see a drop in memory, and less of a "step up" in memory usage you slowly see the beginnings of with the original code

Also updated pprof, it's all basically JSON de/serialisation, which is pretty much what we would want to see.

[zmerlynn]

total nit on a log line but o/w LGTM

[agones-bot]

Build Failed 😱

Build Id: 3ee3fd37-5b52-4519-93dd-24db3d90c475

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[markmandel]

Aaah, didn't have enough time to become leader!

23: e2e-wait-to-become-leader

[agones-bot]

Build Succeeded 👏

Build Id: 19c72705-1899-45c0-9eb1-96d46abfe65f

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.40.0-dev-13d2be1-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.40.0-dev-13d2be1-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.40.0-dev-13d2be1-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.40.0-dev-13d2be1-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.40.0-dev-13d2be1-amd64
• Linux C++ SDK (build): agonessdk-1.40.0-dev-13d2be1-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.40.0-dev-13d2be1-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://13d2be1-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/3781/head:pr_3781 && git checkout pr_3781
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.40.0-dev-13d2be1-amd64

[agones-bot]

Build Succeeded 👏

Build Id: 39adee09-897b-4873-8465-f1d098eb616d

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.40.0-dev-7b51704-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.40.0-dev-7b51704-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.40.0-dev-7b51704-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.40.0-dev-7b51704-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.40.0-dev-7b51704-amd64
• Linux C++ SDK (build): agonessdk-1.40.0-dev-7b51704-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.40.0-dev-7b51704-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://7b51704-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/3781/head:pr_3781 && git checkout pr_3781
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.40.0-dev-7b51704-amd64