fix: rollout strategy issues

[nrwiersma]

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind hotfix
/kind release

What this PR does / Why we need it:

This PR fixes issues with the fleet rollout strategy that have been observed in production. It seems in clusters that are massively allocation heavy, rollout does not behave as expected or as documented. These patches have been confirmed to work in out production environment, using a custom build.

Each issue is in a separate commit, to allow them to be evaluated individually.

Which issue(s) this PR fixes:

Closes #3688

Special notes for your reviewer:

Additional to the 2 bug fixes (first 2 commits), we observe that once an inactive GSS gets to the point of Allocated == Replicas, it is still possible for it to spawn GSes, if the new GS comes up fast enough and there is enough churn in the controller. Once this point is reached, there is no real reason for the GSS to have Spec Replicas, but rather it would be better for it to get its Spec Replicas set to 0 and be allowed to loose it allocations until it is cleaned up. This patch makes up the last commit.

[agones-bot]

Build Failed 😱

Build Id: 04b542f9-9a13-4947-bbd7-a3b8364d9b4c

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[markmandel]

I have some suggestions for how to feature gate this change.

Is there a e2e type scenario (akin to what we see in #3438) you can think of that would highlight the issues you were running into - that would also mean we don't run into it again in the future.

[agones-bot]

Build Failed 😱

Build Id: 0ce99505-3c74-45ec-aaa4-a8033f1d7f7d

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Failed 😱

Build Id: 1193d3f4-b99a-4bc9-9bf4-a4896d0f6ac3

• Cloud Build view
• Cloud Build log download

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

[agones-bot]

Build Succeeded 👏

Build Id: 621017df-61f9-4a38-a663-85d710f0cbc4

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.40.0-dev-1c0bc18-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.40.0-dev-1c0bc18-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.40.0-dev-1c0bc18-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.40.0-dev-1c0bc18-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.40.0-dev-1c0bc18-amd64
• Linux C++ SDK (build): agonessdk-1.40.0-dev-1c0bc18-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.40.0-dev-1c0bc18-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://1c0bc18-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/3762/head:pr_3762 && git checkout pr_3762
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.40.0-dev-1c0bc18-amd64

[markmandel]

Thanks for making these changes - I'm probably going to opt to let this one slide past the next release on Tuesday, and give me (someone? 😄 ) some time to sit down with this, and make sure everything is good (which really means running some rolling updates and watching them happen 😄 )

[nrwiersma]

We have production data of this patch if that would help.

[markmandel]

We have production data of this patch if that would help.

TBH - totally trust that you do. I just want to know that I've spent some time with this as an extra verification step, and give it time to back between releases as well.

[agones-bot]

Build Succeeded 👏

Build Id: 8981dcfc-9e41-418c-b309-09dffc754cc9

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.41.0-dev-a2f39cf-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.41.0-dev-a2f39cf-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.41.0-dev-a2f39cf-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.41.0-dev-a2f39cf-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.41.0-dev-a2f39cf-amd64
• Linux C++ SDK (build): agonessdk-1.41.0-dev-a2f39cf-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.41.0-dev-a2f39cf-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://a2f39cf-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/3762/head:pr_3762 && git checkout pr_3762
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.41.0-dev-a2f39cf-amd64

[agones-bot]

Build Succeeded 👏

Build Id: 1d95bf2c-5e7e-4b13-ab31-569d414ea084

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.41.0-dev-3a8aee7-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.41.0-dev-3a8aee7-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.41.0-dev-3a8aee7-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.41.0-dev-3a8aee7-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.41.0-dev-3a8aee7-amd64
• Linux C++ SDK (build): agonessdk-1.41.0-dev-3a8aee7-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.41.0-dev-3a8aee7-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://3a8aee7-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/3762/head:pr_3762 && git checkout pr_3762
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.41.0-dev-3a8aee7-amd64

[markmandel]

Just a heads up - will take this for a spin this week 👍🏻

[markmandel]

Small thing, can you make a copy of this table, one with the new feature and one without, and use the feature shortcodes to keep this feature flag hidden until the next release please.

See: https://agones.dev/site/docs/contribute/#within-a-page for details

[agones-bot]

Build Succeeded 👏

Build Id: 5fca4a12-50e0-4401-9193-f5a92d9137a2

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.41.0-dev-06b190f-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.41.0-dev-06b190f-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.41.0-dev-06b190f-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.41.0-dev-06b190f-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.41.0-dev-06b190f-amd64
• Linux C++ SDK (build): agonessdk-1.41.0-dev-06b190f-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.41.0-dev-06b190f-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://06b190f-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/3762/head:pr_3762 && git checkout pr_3762
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.41.0-dev-06b190f-amd64

[markmandel]

Finally took this for a spin, basically just wanted to see it in action - look great, let's go ahead with it!

(Rolling update on a Fleet of 50 gameservers)

[agones-bot]

Build Succeeded 👏

Build Id: 819f3e3b-9f05-4011-b38c-22c9dbb01880

The following development artifacts have been built, and will exist for the next 30 days:

• image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.41.0-dev-26860d7-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.41.0-dev-26860d7-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.41.0-dev-26860d7-linux-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.41.0-dev-26860d7-amd64
• image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.41.0-dev-26860d7-amd64
• Linux C++ SDK (build): agonessdk-1.41.0-dev-26860d7-amd64-linux-arch_64.tar.gz
• SDK Server: agonessdk-server-1.41.0-dev-26860d7-amd64.zip

A preview of the website (the last 30 builds are retained):

• https://26860d7-dot-preview-dot-agones-images.appspot.com/

To install this version:

• git fetch https://github.com/googleforgames/agones.git pull/3762/head:pr_3762 && git checkout pr_3762
• helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.41.0-dev-26860d7-amd64