GameServer: Implement (immutable) scale subresource, add pdb #2807

zmerlynn · 2022-11-11T16:30:57Z

This PR:

Adds a scale resource to GameServer by adding an immutableReplicas field, which has a default, min and max of 1. Having a scale subresource lets us define a PodDisruptionBudget that can be set to maxUnavailable: 0%. (Without the scale subresource on GameServer, we can only set minAvailable, and it applies across the label selector - we would be forced to use one PDB per pod.)
- The only artifact of this addition is a new field in the Spec/Status named immutableReplicas, in the Kubernetes object. This field is not present in the in-memory representation for GameServer, nor is it present in etcd (by defaulting rules). The field is visible on describe or get -oyaml, but is otherwise ignored.
Adds a PDB per namespace with label selector agones.dev/safe-to-evict: "false".
- Note: Nothing uses this label selector yet. I plan to use it in Support Agones on GKE Autopilot #2777 as a replacement for safe-to-evict, but this PR currently has no effect except allowing the use of the PDB by explicit label in the Spec.

After this PR, it's possible to define a fleet with the agones.dev/safe-to-evict: "false" label and have all GameServer pods protected by a PodDisruptionBudget, e.g.:

$ cat tmp/fleet.yaml
[...]
      template:
        metadata:
          labels:
            agones.dev/safe-to-evict: "false"
$ kubectl scale fleet/fleet-example --replicas=5
fleet.agones.dev/fleet-example scaled
$ kubectl describe pdb
Name:             agones-gameserver-safe-to-evict-false
Namespace:        default
Max unavailable:  0%
Selector:         agones.dev/safe-to-evict=false
Status:
    Allowed disruptions:  0
    Current:              5
    Desired:              5
    Total:                5
Events:                   <none>

Additionally, because min/max/default are 1, Kubernetes enforces the immutability for us:

$ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=1
gameserver.agones.dev/fleet-example-k6dfs-6m5nq scaled
$ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=2
The GameServer "fleet-example-k6dfs-6m5nq" is invalid: spec.immutableReplicas: Invalid value: 2: spec.immutableReplicas in body should be less than or equal to 1
$ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=0
The GameServer "fleet-example-k6dfs-6m5nq" is invalid: spec.immutableReplicas: Invalid value: 0: spec.immutableReplicas in body should be greater than or equal to 1

Closes #2806
See also: #553
/kind feature

agones-bot · 2022-11-11T16:39:24Z

Build Failed 😱

Build Id: 96836bd3-0d19-4719-b0b9-3e769c4056f9

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2022-11-11T17:07:52Z

Build Failed 😱

Build Id: ce7f46fc-2661-4ad0-bad6-7397c6906856

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2022-11-11T17:47:24Z

Build Succeeded 👏

Build Id: e3921135-a01f-4154-a28c-c6b10cf97e00

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-e0adeab-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-e0adeab-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-e0adeab-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-e0adeab-amd64
Linux C++ SDK (build): agonessdk-1.28.0-e0adeab-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.28.0-e0adeab-amd64.zip

A preview of the website (the last 30 builds are retained):

https://e0adeab-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2807/head:pr_2807 && git checkout pr_2807
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-e0adeab-amd64

install/helm/agones/templates/crds/_gameserverspecschema.yaml

install/helm/agones/templates/pdb.yaml

agones-bot · 2022-11-14T17:44:52Z

Build Succeeded 👏

Build Id: 1b3f716a-74ae-4043-9512-7dd9fa15cb0d

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-0aef7c4-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-0aef7c4-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-0aef7c4-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-0aef7c4-amd64
Linux C++ SDK (build): agonessdk-1.28.0-0aef7c4-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.28.0-0aef7c4-amd64.zip

A preview of the website (the last 30 builds are retained):

https://0aef7c4-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2807/head:pr_2807 && git checkout pr_2807
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-0aef7c4-amd64

agones-bot · 2022-11-14T22:21:04Z

Build Failed 😱

Build Id: 2c21c2b2-781c-4e6d-b50a-28d8dc19e05c

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2022-11-14T23:01:43Z

Build Succeeded 👏

Build Id: 6511a1b8-7685-4812-baf2-01ce8e29655b

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-ce28721-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-ce28721-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-ce28721-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-ce28721-amd64
Linux C++ SDK (build): agonessdk-1.28.0-ce28721-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.28.0-ce28721-amd64.zip

A preview of the website (the last 30 builds are retained):

https://ce28721-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2807/head:pr_2807 && git checkout pr_2807
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-ce28721-amd64

agones-bot · 2022-11-19T00:08:04Z

Build Succeeded 👏

Build Id: 525c9e71-a0e8-426d-b4c2-ec777bae6f66

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-2d5e9f4-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-2d5e9f4-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-2d5e9f4-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-2d5e9f4-amd64
Linux C++ SDK (build): agonessdk-1.28.0-2d5e9f4-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.28.0-2d5e9f4-amd64.zip

A preview of the website (the last 30 builds are retained):

https://2d5e9f4-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2807/head:pr_2807 && git checkout pr_2807
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-2d5e9f4-amd64

roberthbailey · 2022-11-19T05:52:32Z

This looks great! And I think it might be the first example of using a feature gate in the helm templating, which is really cool.

One last thing before we merge: Can you update https://agones.dev/site/docs/guides/feature-stages/ to mention the new feature gate? I also want to make sure that we capture this appropriately in the release notes so I'm wondering if we should change the PR title to mention that this is an alpha feature? WDYT?

markmandel · 2022-11-19T07:20:17Z

Since we are feature gating, please add the feature gate to:

agones/cloudbuild.yaml

Line 247 in 4ab41be

    
           - 'PlayerTracking=true&StateAllocationFilter=false&PlayerAllocationFilter=true&SDKGracefulTermination=true&CustomFasSyncInterval=false&ResetMetricsOnDelete=true'

and

agones/build/Makefile

Line 70 in 4ab41be

    
           ALPHA_FEATURE_GATES ?= "PlayerTracking=true&PlayerAllocationFilter=true&SDKGracefulTermination=true&ResetMetricsOnDelete=true"

Which will make it easier to test locally, and also will run under the e2e tests with it turned on and off.

(I swore I wrote these steps in features.go, but now cannot find them).

You'll have to also add your gate here:

agones/pkg/util/runtime/features.go

Lines 27 to 67 in 4ab41be

    
           const ( 
        
           	// FeatureGateFlag is a name of a command line flag, which turns on specific tests for FeatureGates 
        
           	FeatureGateFlag = "feature-gates" 
        
           	// FeatureExample is an example feature gate flag, used for testing and demonstrative purposes 
        
           	FeatureExample Feature = "Example" 
        
           	// FeaturePlayerTracking is a feature flag to enable/disable player tracking features. 
        
           	FeaturePlayerTracking Feature = "PlayerTracking" 
        
           	// FeatureStateAllocationFilter is a feature flag that enables state filtering on Allocation. 
        
           	FeatureStateAllocationFilter Feature = "StateAllocationFilter" 
        
           	// FeaturePlayerAllocationFilter is a feature flag that enables the ability for Allocations to filter based on 
        
           	// player capacity. 
        
           	FeaturePlayerAllocationFilter Feature = "PlayerAllocationFilter" 
        
           	// FeatureCustomFasSyncInterval is a feature flag that enables custom the FleetAutoscaler rsync interval 
        
           	FeatureCustomFasSyncInterval Feature = "CustomFasSyncInterval" 
        
           	// FeatureSDKGracefulTermination is a feature flag that enables SDK to support gracefulTermination 
        
           	FeatureSDKGracefulTermination Feature = "SDKGracefulTermination" 
        
           	// FeatureResetMetricsOnDelete is a feature flag that tells the metrics service to unregister and register 
        
           	// relevant metric views to reset their state immediately when an Agones resource is deleted. 
        
           	FeatureResetMetricsOnDelete Feature = "ResetMetricsOnDelete" 
        
           ) 
        
           var ( 
        
           	// featureDefaults is a map of all Feature Gates that are 
        
           	// operational in Agones, and what their default configuration is. 
        
           	// alpha features are disabled. 
        
           	featureDefaults = map[Feature]bool{ 
        
           		FeatureExample:                true, 
        
           		FeaturePlayerTracking:         false, 
        
           		FeatureStateAllocationFilter:  true, 
        
           		FeaturePlayerAllocationFilter: false, 
        
           		FeatureCustomFasSyncInterval:  true, 
        
           		FeatureSDKGracefulTermination: false, 
        
           		FeatureResetMetricsOnDelete:   false, 
        
           	}

Otherwise the parsing will fail, as it validates flags.

zmerlynn · 2022-11-29T01:01:40Z

Ok! I made this more into a formal mechanism for feature gates in Helm and added more documentation of the process. PTAL!

agones-bot · 2022-11-29T01:24:33Z

Build Succeeded 👏

Build Id: 2b75ab5f-7149-4939-8955-829170350a80

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-7954fac-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-7954fac-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-7954fac-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-7954fac-amd64
Linux C++ SDK (build): agonessdk-1.28.0-7954fac-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.28.0-7954fac-amd64.zip

A preview of the website (the last 30 builds are retained):

https://7954fac-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2807/head:pr_2807 && git checkout pr_2807
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-7954fac-amd64

install/helm/agones/templates/pdb.yaml

… pdb This PR, under the `LifecycleContract` feature gate suggested in googleforgames#2794 * Adds a scale resource to `GameServer` by adding an `immutableReplicas` field, which has a `default`, `min` and `max` of 1. Having a `scale` subresource lets us define a `PodDisruptionBudget` that can be set to `maxUnavailable: 0%` [1]. * Adds a PDB per namespace with label selector `agones.dev/safe-to-evict: "false"`, which nothing yet adds. * Adds a mechanism to get feature gate values in Helm by using: {{- $featureGates := include "agones.featureGates" . | fromYaml }} * Cleanup / documentation of feature gate mechanisms After this PR, it's possible to define a fleet with the label and have all `GameServer` pods protected by a `PodDisruptionBudget`, e.g.: ``` $ kubectl scale fleet/fleet-example --replicas=5 fleet.agones.dev/fleet-example scaled $ kubectl describe pdb Name: agones-gameserver-safe-to-evict-false Namespace: default Max unavailable: 0% Selector: agones.dev/safe-to-evict=false Status: Allowed disruptions: 0 Current: 4 Desired: 5 Total: 5 Events: <none> ``` Additionally, because min/max/default are 1, Kubernetes enforces the immutability for us: ``` $ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=1 gameserver.agones.dev/fleet-example-k6dfs-6m5nq scaled $ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=2 The GameServer "fleet-example-k6dfs-6m5nq" is invalid: spec.immutableReplicas: Invalid value: 2: spec.immutableReplicas in body should be less than or equal to 1 $ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=0 The GameServer "fleet-example-k6dfs-6m5nq" is invalid: spec.immutableReplicas: Invalid value: 0: spec.immutableReplicas in body should be greater than or equal to 1 ``` The only artifact of this addition is a new field in the Spec/Status named `immutableReplicas`, in the Kubernetes object. This field is not present in the in-memory representation for `GameServer`, nor is it present in `etcd` (by defaulting rules). The field is visible on `describe` or `get -oyaml`, but is otherwise ignored. [1] https://kubernetes.io/docs/tasks/run-application/configure-pdb/#identify-an-application-to-protect

zmerlynn · 2022-11-29T16:45:36Z

Fixed, PTAL!

agones-bot · 2022-11-29T17:01:45Z

Build Failed 😱

Build Id: 26887251-551b-496a-b3f8-f6c1f15bad9e

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2022-11-29T17:10:04Z

Build Succeeded 👏

Build Id: a2936572-794e-41ff-8603-2563203790da

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-23ebcc6-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-23ebcc6-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.28.0-23ebcc6-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.28.0-23ebcc6-amd64
Linux C++ SDK (build): agonessdk-1.28.0-23ebcc6-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.28.0-23ebcc6-amd64.zip

A preview of the website (the last 30 builds are retained):

https://23ebcc6-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2807/head:pr_2807 && git checkout pr_2807
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.tag=1.28.0-23ebcc6-amd64

google-oss-prow · 2022-11-29T17:11:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: roberthbailey, zmerlynn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [roberthbailey]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot requested review from aLekSer and dzlier-gcp November 11, 2022 16:31

google-oss-prow bot added the size/M label Nov 11, 2022

zmerlynn removed request for aLekSer and dzlier-gcp November 11, 2022 16:32

zmerlynn force-pushed the add-scale-to-gs branch from 4df070f to 6af80d0 Compare November 11, 2022 16:50

google-oss-prow bot added size/L and removed size/M labels Nov 11, 2022

zmerlynn force-pushed the add-scale-to-gs branch from 6af80d0 to e0adeab Compare November 11, 2022 17:17

zmerlynn requested review from markmandel and roberthbailey November 11, 2022 17:55

zmerlynn assigned roberthbailey Nov 11, 2022

zmerlynn commented Nov 11, 2022

View reviewed changes

install/helm/agones/templates/crds/_gameserverspecschema.yaml Show resolved Hide resolved

zmerlynn commented Nov 11, 2022

View reviewed changes

install/helm/agones/templates/pdb.yaml Show resolved Hide resolved

zmerlynn added the kind/feature New features for Agones label Nov 11, 2022

zmerlynn force-pushed the add-scale-to-gs branch from e0adeab to 0aef7c4 Compare November 14, 2022 17:13

zmerlynn force-pushed the add-scale-to-gs branch from 0aef7c4 to adb85f3 Compare November 14, 2022 21:54

zmerlynn force-pushed the add-scale-to-gs branch from adb85f3 to ce28721 Compare November 14, 2022 22:34

zmerlynn mentioned this pull request Nov 18, 2022

Explicit flag for "can be disrupted": eviction.safe #2794

Closed

zmerlynn force-pushed the add-scale-to-gs branch from ce28721 to 2d5e9f4 Compare November 18, 2022 23:40

google-oss-prow bot added size/M and removed size/L labels Nov 18, 2022

zmerlynn force-pushed the add-scale-to-gs branch from 2d5e9f4 to 7954fac Compare November 29, 2022 00:58

google-oss-prow bot added size/L and removed size/M labels Nov 29, 2022

roberthbailey approved these changes Nov 29, 2022

View reviewed changes

install/helm/agones/templates/pdb.yaml Outdated Show resolved Hide resolved

google-oss-prow bot added lgtm approved labels Nov 29, 2022

zmerlynn force-pushed the add-scale-to-gs branch from 7954fac to fc602c3 Compare November 29, 2022 16:43

google-oss-prow bot removed the lgtm label Nov 29, 2022

zmerlynn force-pushed the add-scale-to-gs branch from fc602c3 to 23ebcc6 Compare November 29, 2022 16:44

roberthbailey approved these changes Nov 29, 2022

View reviewed changes

google-oss-prow bot added the lgtm label Nov 29, 2022

roberthbailey merged commit e700d5b into googleforgames:main Nov 29, 2022

mangalpalli added this to the 1.28.0 milestone Nov 29, 2022

zmerlynn mentioned this pull request Nov 29, 2022

Request for zmerlynn to become Approver #2833

Closed

5 tasks

zmerlynn deleted the add-scale-to-gs branch January 25, 2023 22:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GameServer: Implement (immutable) scale subresource, add pdb #2807

GameServer: Implement (immutable) scale subresource, add pdb #2807

zmerlynn commented Nov 11, 2022 •

edited

Loading

agones-bot commented Nov 11, 2022

agones-bot commented Nov 11, 2022

agones-bot commented Nov 11, 2022

agones-bot commented Nov 14, 2022

agones-bot commented Nov 14, 2022

agones-bot commented Nov 14, 2022

agones-bot commented Nov 19, 2022

roberthbailey commented Nov 19, 2022

markmandel commented Nov 19, 2022

zmerlynn commented Nov 29, 2022

agones-bot commented Nov 29, 2022

zmerlynn commented Nov 29, 2022

agones-bot commented Nov 29, 2022

agones-bot commented Nov 29, 2022

google-oss-prow bot commented Nov 29, 2022

GameServer: Implement (immutable) scale subresource, add pdb #2807

GameServer: Implement (immutable) scale subresource, add pdb #2807

Conversation

zmerlynn commented Nov 11, 2022 • edited Loading

agones-bot commented Nov 11, 2022

agones-bot commented Nov 11, 2022

agones-bot commented Nov 11, 2022

agones-bot commented Nov 14, 2022

agones-bot commented Nov 14, 2022

agones-bot commented Nov 14, 2022

agones-bot commented Nov 19, 2022

roberthbailey commented Nov 19, 2022

markmandel commented Nov 19, 2022

zmerlynn commented Nov 29, 2022

agones-bot commented Nov 29, 2022

zmerlynn commented Nov 29, 2022

agones-bot commented Nov 29, 2022

agones-bot commented Nov 29, 2022

google-oss-prow bot commented Nov 29, 2022

zmerlynn commented Nov 11, 2022 •

edited

Loading