Estafette CI GKE extension

This extension makes it easy to deploy an application to GKE, and automatically handles getting the right credentials, implementing a number of best practices and more.

• Usage
• Parameters
• Visibility

Usage

To use it you can pass a minimal number of parameters and have it create a deployment for you.

releases:
  dev:
    stages:
      deploy:
        image: extensions/gke:stable
        container:
          repository: extensions
        hosts:
        - gke.estafette.io

But there's many more parameters to control the kind of resource it creates - deployment, cronjob, job, statefulset - and many other things to tune.

Parameters

Global parameters

These parameters apply to any of the kind values.

Parameter	Description	Allowed values	Default value
credentials	Is automatically generated from the release name prefixed by gke-	string	gke-${ESTAFETTE_RELEASE_NAME}
action	Controls what action is taken; can take values from Estafette release actions	deploy-simple, deploy-canary, deploy-stable, restart-simple, restart-canary, restart-stable, diff-simple, diff-canary, diff-stable, rollback-canary	deploy-simple
kind	Determines the type of Kubernetes resource to get created	deployment, headless-deployment, statefulset, job, cronjob, config, config-to-file	deployment
dryrun	Controls whether the changes generated by this extension will be applied	bool	false
app	The name used to deploy the application	string	${ESTAFETTE_LABEL_APP} if set, ${ESTAFETTE_GIT_NAME} otherwise
namespace	Sets the kubernetes namespace to deploy to	string	empty, but usually set in the credential defaults

Note: the action should preferably not be set directly on the stage, but as actions on the stage, so you can trigger every action from estafette using the same stage:

releases:
  prd:
    actions:
    - name: deploy-canary
    - name: deploy-stable
    - name: rollback-canary
      hideBadge: true
    - name: restart-stable
      hideBadge: true
    stages:
      deploy:
        image: extensions/gke:stable
        kind: deployment

Application container parameters

For any of the kind values except for config and config-to-file these set the values for the main application container with sensible defaults. Try to match your application port and endpoints as much as possible to the defaults so you have to override the bare minimum.

Parameter	Description	Allowed values	Default value
container.repository	Path for the image repository minus the last part, for example extensions for extensions/gke image, or gcr.io/<project id>	string
container.name	The name of the container image, usually ${ESTAFETTE_LABEL_APP} or ${ESTAFETTE_GIT_NAME}	string	app
container.tag	The container image tag, usually the build version	string	${ESTAFETTE_BUILD_VERSION}
container.imagePullPolicy	The image pull policy for the main container image	IfNotPresent or Always	IfNotPresent
container.port	The port the main container listens on; preferably port 5000 so you don't have to explicitly set it	int	5000
container.env	A map of environment variable keys and values, passed on to the container	map[string]interface{}
container.secretEnv	Same as env but the values are stored in a secret instead and referenced with secretKeyRef automatically; no need to base64 encode	map[string]interface{}
container.cpu.request	The cpu request value; this ensures the application has at least the cpu required to operate under normal circumstances and is used for calculating the autoscaling cpu load	string	100m
container.cpu.limit	The cpu limit; no need to set, it can lead to cpu throttling, but in case your application turns out to be a noisy neighbour can be set	string
container.memory.request	The requested memory; setting it lower than the limit can lead to out of memory kill before hitting the limit if the node it runs on is short on memory	string	128Mi
container.memory.limit	The memory limit; when a container hits this limit it's killed with an out of memory kill; set equal to request for guaranteed Quality of Service	string	128Mi
container.liveness.enabled	Toggles the liveness probe on the application container, which determines whether the application is still healthy	bool	true
container.liveness.path	Sets the path for the liveness probe	string	/liveness
container.liveness.port	Sets the port for the liveness probe	int	container.port
container.liveness.delay	Sets the number of seconds to wait before running the liveness probe first; increase if it takes longer than 30 seconds for the application to be up and running	int	30
container.liveness.timeout	Time to wait for a response from the liveness path	int	1
container.liveness.period	Interval between liveness probes	int	10
container.liveness.failureThreshold	Number of failures before liveness probe is considered to have failed and the container is killed	int	3
container.liveness.successThreshold	Number of consecutive successes for liveness probe to be considered successful	int	1
container.readiness.enabled	Toggles the readiness probe on the application container, which determines whether the application is ready to receive requests	bool	true
container.readiness.path	Sets the path for the readiness probe	string	/readiness
container.readiness.port	Sets the port for the readiness probe	int	container.port
container.readiness.delay	Sets the number of seconds to wait before running the readiness probe first	int	0
container.readiness.timeout	Time to wait for a response from the readiness path	int	1
container.readiness.period	Interval between readiness probes	int	10
container.readiness.failureThreshold	Number of failures before readiness probe is considered to have failed and removed as an endpoint from the service, so it no longer receives requests	int	3
container.readiness.successThreshold	Number of consecutive successes for readiness probe to be considered successful	int	1
container.metrics.scrape	Toggles whether Prometheus metrics are exposed and need to be scraped	bool	true
container.metrics.path	The path to the Prometheus metrics endpoint	string	/metrics
container.metrics.port	The port at which the Prometheus metrics are exposed	int	container.port
container.lifecycle.prestopsleep	To reduce the risk of failing requests for terminating pods a prestop sleep is used; disable if the container has no sleep command because there's no os (scratch image)	bool	true for os: linux, false for os: windows
container.lifecycle.prestopsleepseconds	Number of seconds to sleep; 15 to 20 should be enough in the majority of cases	int	20
container.containerLifecycle	To set custom lifecycle as yaml if this set it will disable the container.lifecycle.prestopsleep, so make sure to set the sleep with this custom hook	map[string]interface{}	nil
container.additionalports[].name	To configure any other ports than the usual http/https ports the application can communicate through	string
container.additionalports[].port	The port number for an additional port	int
container.additionalports[].protocol	Can be any of the Kubernetes supported protocols	TCP or UDP	TCP
container.additionalports[].visibility	Can be set differently from the main visibility if it needs to be different (more restrictive for example)	see visibility	visibility

Deployment parameters

Specific to kind deployment

Parameter	Description	Allowed values	Default value
replicas	The number of pods to run	int	1
visibility	Determines how the application can be reached	private, public, public-whitelist, esp, espv2, iap, apigee	private
workloadIdentity	Enable workload identity to access Google Cloud services from applications running within GKE due to its improved security properties and manageability.	bool	false
iapOauthClientID	Needs a Google OAuth Client ID encoded in base64 when using visibility: iap; has to be created in advance	string (base64 encoded)
iapOauthClientSecret	Needs a Google OAuth Client Secret encoded in base64 when using visibility: iap; has to be created in advance	string (base64 encoded)
espEndpointsProjectID	When Google Cloud Endpoints are set up in a centralized project set it's ID with this parameter	string
espConfigID	When you want to pin the version of the openapi spec uploaded as a Google Cloud Endpoint config it can be set	string	Takes the latest openapi spec uploaded by this extension
espOpenapiYamlPath	Path to openapi.yaml file to use for creating the endpoint config; use separate ones per environment	string
espServiceTypeClusterIP	Creates ClusterIP service instead of LoadBalancer for ESP. Doesn't destroy already existing LB (if exists), but will point DNS record to new IP	bool	false
whitelist	A list of [CIDRs][https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing] to allow access to the application	[]string	The default configured in the nginx-office controller
progressDeadlineSeconds	Sets the number of seconds for Kubernetes to wait for a deployment to lack progress before treating it as a failure	int	600
os	The operating system to deploy to	linux, windows	linux
chaosproof	Determines whether it's okay to run the application on preemptibles	bool	false
manifests.files	To set additional template files to apply	[]string
manifests.data	To provide extra data to the additional templates beyond what's already set by the extension	map[string]interface{}
trustedips	To set loadBalancerSourceRanges on the service of type LoadBalancer for visibility: public or visibility: esp or visibility: espv2	[]string	Cloudflare's origin ip addresses, see https://www.cloudflare.com/ips-v4
labels	To set labels that are use on all kubernetes resources	map[string]string	The labels set in the .estafette.yaml manifest
containerNativeLoadBalancing	To use Google Cloud container-native load balancing	bool
hosts	The public hostnames associated with this application	[]string
hostsrouteonly	Additional hostnames listened to by the app and its ingresses, but not set in DNS records	[]string
internalhosts	The internal hostnames associated with this application	[]string
internalhostsrouteonly	Additional internal hostnames listened to by the app and its ingresses, but not set in DNS records	[]string
apigeesuffix	Suffix for the hostnames when using visibility: apigee	string	apigee
dns.useCloudflareEstafetteExtension	Add annotations used by estafette-cloudflare-dns-extension	bool	false
dns.useExternalDNS	Add annotations used by external-dns	bool	true
basepath	Base path in the ingresses to route to this application	string	/
autoscale.enabled	Enables Horizontal Pod Autoscaler	bool	true
autoscale.min	The minimum replicas set in the HPA	int	3
autoscale.max	The maximum replicas set in the HPA	int	100
autoscale.cpu	Target CPU percentage set in the HPA	int	80
autoscale.behavior	https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior	map[string]interface{}
autoscale.safety.enabled	Enabled use of estafette-k8s-hpa-scaler as a safety net	bool	false
autoscale.safety.promquery	The Prometheus query to get the request rate to this or a downstream application in the call stack	string	sum(rate(nginx_http_requests_total{app='%v'}[5m])) by (app)
autoscale.safety.ratio	A divider to get from request rate to number of pods; equals the desired requests per pod	string	1
autoscale.safety.delta	A constant to increase or lower the function minReplicas = Ceiling ( delta + ( promquery / ratio ) )	string
autoscale.safety.scaledownratio	Sets the fraction the min replicas is allowed to scale down compared to the last value in order to ease scaling	string	1
vpa.enabled	Enables Vertical Pod Autoscaler	bool	false
vpa.updateMode	The update mode for VPA	"Off", "Initial", "Recreate", "Auto"	"Off"
request.ingressbackendprotocol	The backend protocol (HTTPS or GRPCS) to be used by the Ingress	string	HTTPS
request.timeout	Maximum time for a response, set at the ingresses and openresty sidecar	string	60s
request.maxbodysize	Maximum body size for a request, set at the ingresses and openresty sidecar	string	128m
request.proxybuffersize	Buffer size for proxying, set at the ingresses and openresty sidecar	string	4k
request.proxybuffersnumber	Number of buffers for proxying, set at the ingresses and openresty sidecar	int	4
request.clientbodybuffersize	Buffer size for request body, set at the ingresses and openresty sidecar	string	8k
request.loadbalance	Loadbalancing algorithm used by the ingress controller	ewma, round_robin	round_robin
request.authsecret	Secret name in the form of namespace/secret used for client certificate authentication	string
request.verifydepth	The validation depth between the provided client certificate and the Certification Authority chain	int	3
request.configuration-snippet	Configration snippet for all nginx ingresses	string
secrets.keys	Map of filenames and base64 encoded values stored in a secret, mounted into the application container	map[string]interface{}
secrets.mountpath	Path to where the secret is mounted	string
configs.files	Files in the repository to include in a configmap, mounted into the application container	[]string
configs.data	Key/value map to replace any gotemplate placeholders in the config files set with configs.files	map[string]interface{}
configs.inline	Key/value map to set config files for the configmap without using templates on disk	map[string]string
configs.mountpath	Path to where the configmap is mounted	string
volumemounts[].name	Additional volumes to mount into the application container	string
volumemounts[].mountpath	Path to where the volume is mounted	string
volumemounts[].volume	Yaml snippet for the volume spec; can be used to mount secrets, configmaps, persistentvolumeclaims, etc	map[string]interface{}
certificatesecret	If set use a pre-existing secret with TLS certificate instead of automatically creating one from the host and internalhosts using a secret with estafette-letsencrypt-certificate annotations	string
allowhttp	If the application needs to be available on http, besides the default https	bool	false
enablePayloadLogging	Mounts a host path into the container that's used for an internal Travix payload log shipper	bool	false
useGoogleCloudCredentials	Uses a estafette-gcp-service-account annotated secret to get a service account keyfile and mount it into the application container	bool	false
disableServiceAccountKeyRotation	Keeps the retrieved service account keyfile without rotating it regularly	bool	true
legacyGoogleCloudServiceAccountKeyFile	Base64 encoded keyfile stored in the application secret	string
googleCloudCredentialsApp	When a shared service account needs to be used set the name of the app the service account is generated for	string	app
probeService	Configures a prometheus probe on the service using blackbox-exporter to check for availability	bool	false for visibility: esp and visibility: espv2, true otherwise
topologyAwareHints	Enables Topology Aware Hints, which provides a mechanism to help keep traffic within the zone it originated fromand reduce the extra costs generated from egress traffic	bool	true
tolerations	Yaml snippets to configure Kubernetes tolerations	[]yaml snippet
affinity	Map of pod and node (anti)affinity config to configure Kubernetes affinity	map[string]interface{}	Pre-defined PodAntiAffinity and NodeAffinity based on type of OS
injecthttpproxysidecar	Indicates whether the openresty sidecar should be injected	bool	true
initcontainers	Yaml snippets to configure Kubernetes init containers	[]yaml snippet
sidecar	deprecated, use sidecars parameter instead
sidecars[].type	Can be used to configure a couple of sidecars known by this extension	openresty, esp, espv2, cloudsqlproxy, istio
sidecars[].image	The full container image path for the sidecar	string
sidecars[].env	Environment variables passed into the sidecar	map[string]interface{}
sidecars[].secretEnv	Secret values passed into the sidecar as environment variables by storing them in a secret and using secretKeyRef	map[string]interface{}
sidecars[].cpu.request	The cpu request value	string	50m
sidecars[].cpu.limit	The cpu limit value	string
sidecars[].memory.request	The memory request value	string	30m
sidecars[].memory.limit	The memory limit value	string	50m
sidecars[].healthcheckpath	Can be set for the health check through the openresty sidecar towards the main application	string	container.readiness.path
sidecars[].dbinstanceconnectionname	A Cloud SQL connection name to be used in the Cloud SQL proxy sidecar	string
sidecars[].sqlproxyport	The port the cloud sql proxy listens on	int	5432
sidecars[].sqlproxyterminationtimeoutseconds	The cloud sql proxy termination timeout	int	60
customsidecars	Yaml snippets to pass in additional sidecars	[]yaml snippet
strategytype	Configures the upgrade strategy for kind: deployment; augments the Kubernetes strategyType with AtomicUpdate	RollingUpdate, Recreate, AtomicUpdate
rollingupdate.maxsurge	Maximum percentage of pods to surge during a rolling update	string	25%
rollingupdate.maxunavailable	Maximum number of unavailable pods during a rolling update	string	0
rollingupdate.timeout	Timeout during a rolling update before considering it as failed	string	5m
defaultOpenrestySidecarImage	Allows the default OpenResty sidecar image to be overridden via defaults in kubernetes-engine credentials	string	estafette/openresty-sidecar@sha256:2aa9f2c8c3f506e0f6cc70871701b5ac81aa0f12e8574c7b8213e4d0379d2ddd
defaultESPSidecarImage	Allows the default ESP sidecar image to be overridden via defaults in kubernetes-engine credentials	string	gcr.io/endpoints-release/endpoints-runtime:1.57.0
defaultESPv2SidecarImage	Allows the default ESP v2 sidecar image to be overridden via defaults in kubernetes-engine credentials	string	gcr.io/endpoints-release/endpoints-runtime:2.29.1
defaultCloudSQLProxySidecarImage	Allows the default Cloud SQL proxy sidecar image to be overridden via defaults in kubernetes-engine credentials	string	eu.gcr.io/cloudsql-docker/gce-proxy:1.35.0
imagePullSecretUser	When the application image is stored in a private registry not accessible for the GKE cluster set a username	string
imagePullSecretPassword	Password for the private registry	string

Note: for visibility: esp a release needs access to the openapi spec, so combine with clone: true on the release target, for example:

releases:
  dev:
    clone: true
    stages:
      deploy:
        image: extensions/gke:stable
        visibility: esp
        espOpenapiYamlPath: openapi.dev.yaml

Statefulset parameters

Specific to kind statefulset

Parameter	Description	Allowed values	Default value
storageclass	Determines the type of persistent disk created	standard	standard, pd-standard, pd-ssd
storagesize	The size of the persistent disk	string	1Gi
storagemountpath	The path where the persistent disk is mounted	string	/data

Cronjob parameters

Specific to kind cronjob

Parameter	Description	Allowed values	Default value
schedule	Sets the schedule at which the cronjob spawns a new job	string
concurrencypolicy	Indicates whether concurrent jobs are allowed or forbidden	Allow, Forbid	Allow

Cronjob/Job parameters

Specific to kind cronjob and job

Parameter	Description	Allowed values	Default value
completions	The amount of times the job needs to complete	int	1
parallelism	How many jobs to run in parallel	int	1
backoffLimit	After how many failures to stop retrying	int	6
restartPolicy	Controls whether a container should be restarted when it stops	Always, OnFailure	OnFailure

Visibility

How the customers of an application - whether it's an actual user connecting or another service - can communicate with your application is set by the visibility parameter.

Value	Description
private	A Kubernetes service of type ClusterIP is created and depending on the values for hosts and internalhosts it also sets up ingresses using ingress class nginx-office and/or nginx-internal
public	A LoadBalancer Kubernetes service is created which gives direct public access to the application; this is no longer allowed for new deployments, esp should be used in that case to avoid accidental exposure of endpoints like /metrics, /swagger, etc
public-whitelist	This uses ingress controller nginx-office but sets a different list of allowed ip addresses than the default by using the whitelist parameter
esp	This creates a Google Cloud Endpoint, adds the esp sidecar container to the deployment and exposes it through a LoadBalancer service
espv2	Sames as esp but uses the envoy-based version 2
iap	Sets up the application behind Identity Aware Proxy; requires parameters iapOauthClientID and iapOauthClientSecret to be set
apigee	Routes requests through the nginx-open ingress controller; requires parameters request.authsecret and request.verifydepth to be set

Note: all of the above set up an internal ingress if parameter internalhosts is set; for esp this cannot be used to connect to the application since internally since it's limited to only a single hostname