LifecycleContract alpha: Implement (immutable) scale subresource, add…

… pdb (#2807)

This PR, under the `LifecycleContract` feature gate suggested in #2794

* Adds a scale resource to `GameServer` by adding an
`immutableReplicas` field, which has a `default`, `min` and `max` of 1.
Having a `scale` subresource lets us define a `PodDisruptionBudget`
that can be set to `maxUnavailable: 0%` [1].

* Adds a PDB per namespace with label selector
`agones.dev/safe-to-evict: "false"`, which nothing yet adds.

* Adds a mechanism to get feature gate values in Helm by using:
  {{- $featureGates := include "agones.featureGates" . | fromYaml }}

* Cleanup / documentation of feature gate mechanisms

After this PR, it's possible to define a fleet with the label and have
all `GameServer` pods protected by a `PodDisruptionBudget`, e.g.:

```
$ kubectl scale fleet/fleet-example --replicas=5
fleet.agones.dev/fleet-example scaled
$ kubectl describe pdb
Name:             agones-gameserver-safe-to-evict-false
Namespace:        default
Max unavailable:  0%
Selector:         agones.dev/safe-to-evict=false
Status:
    Allowed disruptions:  0
    Current:              4
    Desired:              5
    Total:                5
Events:                   <none>
```

Additionally, because min/max/default are 1, Kubernetes enforces the
immutability for us:

```
$ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=1
gameserver.agones.dev/fleet-example-k6dfs-6m5nq scaled
$ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=2
The GameServer "fleet-example-k6dfs-6m5nq" is invalid: spec.immutableReplicas: Invalid value: 2: spec.immutableReplicas in body should be less than or equal to 1
$ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=0
The GameServer "fleet-example-k6dfs-6m5nq" is invalid: spec.immutableReplicas: Invalid value: 0: spec.immutableReplicas in body should be greater than or equal to 1
```

The only artifact of this addition is a new field in the Spec/Status
named `immutableReplicas`, in the Kubernetes object. This field is not
present in the in-memory representation for `GameServer`, nor is it
present in `etcd` (by defaulting rules). The field is visible on
`describe` or `get -oyaml`, but is otherwise ignored.

[1] https://kubernetes.io/docs/tasks/run-application/configure-pdb/#identify-an-application-to-protect

build/Makefile

@@ -67,7 +67,8 @@ KIND_CONTAINER_NAME=$(KIND_PROFILE)-control-plane
 # Game Server image to use while doing end-to-end tests
 GS_TEST_IMAGE ?= gcr.io/agones-images/simple-game-server:0.14
 
-ALPHA_FEATURE_GATES ?= "PlayerTracking=true&PlayerAllocationFilter=true&SDKGracefulTermination=true&ResetMetricsOnDelete=true"
+# Enable all alpha feature gates. Keep in sync with `false` (alpha) entries in pkg/util/runtime/features.go:featureDefaults
+ALPHA_FEATURE_GATES ?= "LifecycleContract=true&PlayerAllocationFilter=true&PlayerTracking=true&ResetMetricsOnDelete=true&SDKGracefulTermination=true&Example=true"
 
 # Build with Windows support
 WITH_WINDOWS=1

cloudbuild.yaml

@@ -241,10 +241,11 @@ steps:
 #
 # Run the e2e tests with FeatureGates inverted compared to Stable
 #
+# Keep in sync with (the inverse of) pkg/util/runtime/features.go:featureDefaults
 
 - name: 'e2e-runner'
   args:
-    - 'PlayerTracking=true&StateAllocationFilter=false&PlayerAllocationFilter=true&SDKGracefulTermination=true&CustomFasSyncInterval=false&ResetMetricsOnDelete=true'
+    - 'CustomFasSyncInterval=false&StateAllocationFilter=false&LifecycleContract=true&PlayerAllocationFilter=true&PlayerTracking=true&ResetMetricsOnDelete=true&SDKGracefulTermination=true&Example=true'
     - 'e2e-test-cluster'
     - "${_REGISTRY}"
   id: e2e-feature-gates

install/helm/agones/defaultfeaturegates.yaml

@@ -0,0 +1,29 @@
+# Copyright 2022 Google LLC All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Default values for feature gates. Keep in sync with pkg/util/runtime/features.go:featureDefaults
+
+# Beta features
+CustomFasSyncInterval: true
+StateAllocationFilter: true
+
+# Alpha features
+LifecycleContract: false
+PlayerAllocationFilter: false
+PlayerTracking: false
+ResetMetricsOnDelete: false
+SDKGracefulTermination: false
+
+# Example feature
+Example: false

install/helm/agones/templates/_helpers.tpl

@@ -46,3 +46,27 @@ Create chart name and version as used by the chart label.
 {{- define "agones.chart" -}}
 {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
 {{- end -}}
+
+{{/*
+Creates a YAML object representing known feature gates. Can then be used as:
+  {{- $featureGates := include "agones.featureGates" . | fromYaml }}
+then you can check a feature gate with e.g. $featureGates.Example
+
+Implemented by intentionally duplicating YAML - later keys take precedence.
+So we start with defaultfeaturegates.yaml and then splat intentionally set
+feature gates. In the process, we validate that the feature gate is known.
+*/}}
+{{- define "agones.featureGates" -}}
+{{- .Files.Get "defaultfeaturegates.yaml" -}}
+{{- if $.Values.agones.featureGates }}
+{{- $gates := .Files.Get "defaultfeaturegates.yaml" | fromYaml }}
+{{- range splitList "&" $.Values.agones.featureGates }}
+{{- $f := splitn "=" 2 . -}}
+{{- if hasKey $gates $f._0 }}
+{{$f._0}}: {{$f._1}}
+{{- else -}}
+{{- printf "Unknown feature gate %q" $f._0 | fail -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}

install/helm/agones/templates/crds/_gameserverspecschema.yaml

@@ -153,4 +153,12 @@ properties:
             type: integer
             title: The initial player capacity of this Game Server
             minimum: 0
+{{- if .featureLifecycleContract }}
+      immutableReplicas:
+        type: integer
+        title: Immutable count of Pods to a GameServer. Always 1. (Implementation detail of implementing the Scale subresource.)
+        default: 1
+        minimum: 1
+        maximum: 1
+{{- end }}
 {{- end }}

install/helm/agones/templates/crds/_gameserverstatus.yaml

@@ -65,4 +65,12 @@ status:
           nullable: true
           items:
             type: string
+{{- if .featureLifecycleContract }}
+    immutableReplicas:
+      type: integer
+      title: Immutable count of Pods to a GameServer. Always 1. (Implementation detail of implementing the Scale subresource.)
+      default: 1
+      minimum: 1
+      maximum: 1
+{{- end}}
 {{- end}}

install/helm/agones/templates/crds/fleet.yaml

@@ -13,6 +13,7 @@
 # limitations under the License.
 
 {{- if .Values.agones.crds.install }}
+{{- $featureGates := include "agones.featureGates" . | fromYaml }}
 
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
@@ -99,7 +100,7 @@ spec:
                             - type: integer
                             - type: string
                 template:
-                  {{- $data := dict "metadata" true "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields }}
+                  {{- $data := dict "metadata" true "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields "featureLifecycleContract" $featureGates.LifecycleContract }}
                   {{- include "gameserver.schema" $data | indent 17 }}
             status:
               description: 'FleetStatus is the status of a Fleet. More info:

install/helm/agones/templates/crds/gameserver.yaml

@@ -13,6 +13,7 @@
 # limitations under the License.
 
 {{- if .Values.agones.crds.install }}
+{{- $featureGates := include "agones.featureGates" . | fromYaml }}
 
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
@@ -55,7 +56,17 @@ spec:
           type: date
       schema:
         openAPIV3Schema:
-          {{- $data := dict "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields }}
-          {{- include "gameserver.schema" . | indent 9 }}
-          {{- include "gameserver.status" . | indent 11 }} # in an include, as it's easier to align
+          {{- $data := dict "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields "featureLifecycleContract" $featureGates.LifecycleContract }}
+          {{- include "gameserver.schema" $data | indent 9 }}{{- /* include the schema then status, as it's easier to align */ -}}
+          {{- include "gameserver.status" $data | indent 11 }}
+{{- if $featureGates.LifecycleContract }}
+      subresources:
+        # scale enables the scale subresource. We can't actually scale GameServers, but this allows
+        # for the use of PodDisruptionBudget (PDB) without having to use a PDB per Pod.
+        scale:
+          # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
+          specReplicasPath: .spec.immutableReplicas
+          # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
+          statusReplicasPath: .status.immutableReplicas
+{{- end }}
 {{- end }}

install/helm/agones/templates/crds/gameserverset.yaml

@@ -13,6 +13,7 @@
 # limitations under the License.
 
 {{- if .Values.agones.crds.install }}
+{{- $featureGates := include "agones.featureGates" . | fromYaml }}
 
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
@@ -79,7 +80,7 @@ spec:
                     - Packed
                     - Distributed
                 template:
-                  {{- $data := dict "metadata" true "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields }}
+                  {{- $data := dict "metadata" true "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields "featureLifecycleContract" $featureGates.LifecycleContract }}
                   {{- include "gameserver.schema" $data | indent 18 }}
             status:
               description: 'GameServerSetStatus is the status of a GameServerSet. More info:

install/helm/agones/templates/pdb.yaml

@@ -0,0 +1,30 @@
+# Copyright 2022 Google LLC All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- $featureGates := include "agones.featureGates" . | fromYaml }}
+{{- if $featureGates.LifecycleContract }}
+{{- range .Values.gameservers.namespaces }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: agones-gameserver-safe-to-evict-false
+  namespace: {{ . }}
+spec:
+  maxUnavailable: 0%
+  selector:
+    matchLabels:
+      agones.dev/safe-to-evict: "false"
+{{- end }}
+{{- end }}

install/yaml/install.yaml

@@ -9518,7 +9518,7 @@ spec:
                      type: array
                      nullable: true
                      items:
-                       type: string # in an include, as it's easier to align
+                       type: string
 ---
 # Source: agones/templates/crds/gameserverallocationpolicy.yaml
 # Copyright 2019 Google LLC All Rights Reserved.
@@ -15009,6 +15009,21 @@ spec:
 # See the License for the specific language governing permissions and
 # limitations under the License.
 ---
+# Source: agones/templates/pdb.yaml
+# Copyright 2022 Google LLC All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+---
 # Source: agones/templates/extensions.yaml
 apiVersion: admissionregistration.k8s.io/v1
 kind: MutatingWebhookConfiguration

pkg/apis/agones/v1/gameserver.go

@@ -161,6 +161,7 @@ type GameServerSpec struct {
 	// (Alpha, PlayerTracking feature flag) Players provides the configuration for player tracking features.
 	// +optional
 	Players *PlayersSpec `json:"players,omitempty"`
+	// immutableReplicas is present in gameservers.agones.dev but omitted here (it's always 1).
 }
 
 // PlayersSpec tracks the initial player capacity
@@ -233,6 +234,7 @@ type GameServerStatus struct {
 	// [FeatureFlag:PlayerTracking]
 	// +optional
 	Players *PlayerStatus `json:"players"`
+	// immutableReplicas is present in gameservers.agones.dev but omitted here (it's always 1).
 }
 
 // GameServerStatusPort shows the port that was allocated to a

pkg/util/runtime/features.go

@@ -28,42 +28,82 @@ const (
 	// FeatureGateFlag is a name of a command line flag, which turns on specific tests for FeatureGates
 	FeatureGateFlag = "feature-gates"
 
-	// FeatureExample is an example feature gate flag, used for testing and demonstrative purposes
-	FeatureExample Feature = "Example"
+	////////////////
+	// Beta features
 
-	// FeaturePlayerTracking is a feature flag to enable/disable player tracking features.
-	FeaturePlayerTracking Feature = "PlayerTracking"
+	// FeatureCustomFasSyncInterval is a feature flag that enables a custom FleetAutoscaler resync interval
+	FeatureCustomFasSyncInterval Feature = "CustomFasSyncInterval"
 
 	// FeatureStateAllocationFilter is a feature flag that enables state filtering on Allocation.
 	FeatureStateAllocationFilter Feature = "StateAllocationFilter"
 
+	////////////////
+	// Alpha features
+
+	// FeatureLifecycleContract enables the `lifecycleContract` API to specify disruption tolerance.
+	FeatureLifecycleContract Feature = "LifecycleContract"
+
 	// FeaturePlayerAllocationFilter is a feature flag that enables the ability for Allocations to filter based on
 	// player capacity.
 	FeaturePlayerAllocationFilter Feature = "PlayerAllocationFilter"
 
-	// FeatureCustomFasSyncInterval is a feature flag that enables custom the FleetAutoscaler rsync interval
-	FeatureCustomFasSyncInterval Feature = "CustomFasSyncInterval"
-
-	// FeatureSDKGracefulTermination is a feature flag that enables SDK to support gracefulTermination
-	FeatureSDKGracefulTermination Feature = "SDKGracefulTermination"
+	// FeaturePlayerTracking is a feature flag to enable/disable player tracking features.
+	FeaturePlayerTracking Feature = "PlayerTracking"
 
 	// FeatureResetMetricsOnDelete is a feature flag that tells the metrics service to unregister and register
 	// relevant metric views to reset their state immediately when an Agones resource is deleted.
 	FeatureResetMetricsOnDelete Feature = "ResetMetricsOnDelete"
+
+	// FeatureSDKGracefulTermination is a feature flag that enables SDK to support gracefulTermination
+	FeatureSDKGracefulTermination Feature = "SDKGracefulTermination"
+
+	////////////////
+	// Example feature
+
+	// FeatureExample is an example feature gate flag, used for testing and demonstrative purposes
+	FeatureExample Feature = "Example"
 )
 
 var (
 	// featureDefaults is a map of all Feature Gates that are
 	// operational in Agones, and what their default configuration is.
-	// alpha features are disabled.
+	// alpha features are disabled by default; beta features are enabled.
+	//
+	// To add a new alpha feature:
+	// * add a const above
+	// * add it to `featureDefaults`
+	// * add it to install/helm/agones/defaultfeaturegates.yaml
+	// * add it to `ALPHA_FEATURE_GATES` in build/Makefile
+	// * add the inverse to the e2e-runner config in cloudbuild.yaml
+	// * add it to site/content/en/docs/Guides/feature-stages.md
+	//
+	// To promote a feature from alpha->beta:
+	// * move from `false` to `true` in `featureDefaults`
+	// * move from `false` to `true` in install/helm/agones/defaultfeaturegates.yaml
+	// * remove from `ALPHA_FEATURE_GATES` in build/Makefile
+	// * invert in the e2e-runner config in cloudbuild.yaml
+	// * change the value in site/content/en/docs/Guides/feature-stages.md
+	//
+	// To promote a feature from beta->GA:
+	// * remove all places consuming the feature gate and fold logic to true
+	//   * consider cleanup - often folding a gate to true allows refactoring
+	// * invert the "new alpha feature" steps above
+	//
+	// In each of these, keep the feature sorted by descending maturity then alphabetical
 	featureDefaults = map[Feature]bool{
-		FeatureExample:                true,
-		FeaturePlayerTracking:         false,
-		FeatureStateAllocationFilter:  true,
+		// Beta features
+		FeatureCustomFasSyncInterval: true,
+		FeatureStateAllocationFilter: true,
+
+		// Alpha features
+		FeatureLifecycleContract:      false,
 		FeaturePlayerAllocationFilter: false,
-		FeatureCustomFasSyncInterval:  true,
-		FeatureSDKGracefulTermination: false,
+		FeaturePlayerTracking:         false,
 		FeatureResetMetricsOnDelete:   false,
+		FeatureSDKGracefulTermination: false,
+
+		// Example feature
+		FeatureExample: false,
 	}
 
 	// featureGates is the storage of what features are enabled

site/content/en/docs/Guides/feature-stages.md

@@ -22,17 +22,18 @@ A feature within Agones can be in `Alpha`, `Beta` or `Stable` stage.
 `Alpha` and `Beta` features can be enabled or disabled through the `agones.featureGates` configuration option 
 that can be found in the [Helm configuration]({{< ref "/docs/Installation/Install Agones/helm.md#configuration" >}}) documentation.
 
-The current set of `alpha` and `beta` feature gates are:
+The current set of `alpha` and `beta` feature gates:
 
 | Feature Name                                                                                                          | Gate                     | Default  | Stage   | Since  |
 |-----------------------------------------------------------------------------------------------------------------------|--------------------------|----------|---------|--------|
-| Example Gate (not in use)                                                                                             | `Example`                | Disabled | None    | 0.13.0 |
-| [Player Tracking]({{< ref "/docs/Guides/player-tracking.md" >}})                                                      | `PlayerTracking`         | Disabled | `Alpha` | 1.6.0  |
 | [Custom resync period for FleetAutoscaler](https://github.com/googleforgames/agones/issues/1955)                      | `CustomFasSyncInterval`  | Enabled  | `Beta`  | 1.25.0 |
 | [GameServer state filtering on GameServerAllocations](https://github.com/googleforgames/agones/issues/1239)           | `StateAllocationFilter`  | Enabled  | `Beta`  | 1.26.0 |
+| [Lifecycle Contracts](https://github.com/googleforgames/agones/issues/2794)                                           | `LifecycleContract`      | Disabled | `Alpha` | 1.28.0 |
 | [GameServer player capacity filtering on GameServerAllocations](https://github.com/googleforgames/agones/issues/1239) | `PlayerAllocationFilter` | Disabled | `Alpha` | 1.14.0 |
-| [Graceful Termination for GameServer SDK](https://github.com/googleforgames/agones/pull/2205)                         | `SDKGracefulTermination` | Disabled | `Alpha` | 1.18.0 |
+| [Player Tracking]({{< ref "/docs/Guides/player-tracking.md" >}})                                                      | `PlayerTracking`         | Disabled | `Alpha` | 1.6.0  |
 | [Reset Metric Export on Fleet / Autoscaler deletion]({{% relref "./metrics.md#dropping-metric-labels" %}})            | `ResetMetricsOnDelete`   | Disabled | `Alpha` | 1.26.0 |
+| [Graceful Termination for GameServer SDK](https://github.com/googleforgames/agones/pull/2205)                         | `SDKGracefulTermination` | Disabled | `Alpha` | 1.18.0 |
+| Example Gate (not in use)                                                                                             | `Example`                | Disabled | None    | 0.13.0 |
 
 {{< alert title="Note" color="info" >}}
 If you aren't sure if Feature Flags have been set correctly, have a look at the