Kubernetes Update Manager

This code allows to update deployments in a kubernetes cluster without needing to give cluster admin access to the updating entity. Instead an API is provided which handles the update process via pre-defined annotations in the specified deployment structure.

Motivation

We wrote a blog article on our website to explain why we think this feature is needed. Check it out.

Installation

Currently it is expected for the update manager to run in the same cluster in which the upgrades should be applied. The image is available on Docker Hub for installation. Example deployments can be found in the kube folder. You must adjust the secret to set an api key and additionally adjust the namespace where you want to update-manager to run in. Additionally, if you want the update-manager to be accessible externally you should expose the manager. This can be for example done with an Ingress configuration.

Additionally, if you are using calico for network policies an example policy can be found here.

Configuration

The server can additionally be configured with the following environment variables:

Name	Description	Default	Required
UPDATE_MANAGER_LISTENING_HOST	The host to bind the server to.	0.0.0.0	true
UPDATE_MANAGER_LISTENING_PORT	The port which the update manager should be available at.	9000	true
UPDATE_MANAGER_AUTOLOAD_NAMESPACES	If all namespaces should be scanned when applying an update. This is per default true and can be turned off if only specific namespaces should be scanned for update information.	true	true
UPDATE_MANAGER_NAMESPACES	A list of namespaces which should be scanned for an update. This is only used if autload of namespaces has been switched off. The namespaces should be added comma separated.		false
UPDATE_MANAGER_API_KEY	The pre-shared API key used to authenticate API calls. This is a required field and must be set.		true
SENTRY_DSN	The sentry dsn which should be used when reporting errors from the server.		false

Usage

The update manager scans the specified namespaces for deployments and jobs which have the image attached. They then search for the update classifier and update the images for the specified deployments. This can be either done via the REST API provided by the script which swagger documentation can be accessed by going to the updates-managers URL and add /swagger (so for example https://up.xcnt.io/swagger). Alternatively, it is also possible to use the update command provided by the update manager itself:

docker run --rm -t xcnt/kubernetes-update-manager:stable update --url https://up.xcnt.io/updates --image xcnt/kubernetes-update-manager:1.0.0 --update-classifier stable

This would notify the update manager to update itself, if the following annotation has been put on the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: update-manager
  annotations:
    xcnt.io/update-classifier: stable
  labels:
    app: update-manager
spec:
  replicas: 1
  selector:
    matchLabels:
      app: update-manager
  template:
    metadata:
      labels:
        app: update-manager
    spec:
      containers:
      - name: update-manager
        image:  xcnt/kubernetes-update-manager:stable

Additionally, it is possible to specify migration jobs for the image which should be updated. This can be done by adding a job with the given image and also adding a xcnt.io/update-classifier annotation. For example:

apiVersion: batch/v1
kind: Job
metadata:
  name: migration-job
  annotations:
    xcnt.io/update-classifier: stable
spec:
  template:
    spec:
      containers:
      - name: migration-job
        image: xcnt/test:develop
        imagePullPolicy: Always
        command: ["python", "run.py", "run-migrations"]
      restartPolicy: Never
  backoffLimit: 4

If a update call is done with the classifier stable and the image xcnt/test it will execute a copy of this job once during the update process.

Error Handling

If a deployment doesn't start or a job fails, a rollback of the deployments will be attempted. However, this does not reverse any jobs which have already been executed, meaning the state of the application might need manual work to be restored to a previously compatible version.

License

The application is published under the MIT License.