Creating and rotating snapshots of Hetzner cloud servers

This script can perform the following tasks for selected servers of a Hetzner cloud project:

• Creating a new snapshot
• Shutting down the server before taking a snapshot and restarting it afterwards
• Rotating snapshots, retaining a limited number of quarter-hourly, hourly, daily, weekly, monthly, quarterly and yearly snapshots
• Generating names of new and rotated snapshots from templates

These tasks can be configured independently per server in a JSON configuration file.

Additional features:

• The secret API token can be read from the configuration file, from an environment variable or from stdin
• Log messages are sent to the console or to syslog

Content

• Generating an API token
• Installing this script
• Creating the configuration file
  • Taking snapshots
  • Rotating snapshots
  • Snapshot name templates
• Running the script natively
  • Command line options
  • Passing the API token
  • Creating and rotating snapshots in a cron job
  • Snapshots seem to be missing after rotation
• Running the script in a container
  • Passing the configuration file to the container
  • Environment variables
  • Examples
• Licenses

Generating an API token

This document describes in detail how to generate an API token for your Hetzner cloud project. This script requires a token with "Read & Write" permission.

Guard your API token well since it provides unlimited read/write/delete access to all servers, snapshots, backups etc. of your Hetzner cloud project!

Installing this script

This script is available as a PyPI project. It can be installed by:

python3 -m pip install hetzner-snap-and-rotate

Creating the configuration file

The configuration file is expected to be a JSON document in UTF-8 encoding with the following structure:

{
  "api-token": "...",          // optional, can also be passed via the command line;
                               // see section "Command line options" below

  "defaults": {                // optional defaults, can be overriden per server
      "create-snapshot":       // create a new snapshot whenever the script is invoked
        true,
      "snapshot-timeout":      // timeout (in s) after which snapshot creation is considered to have failed
         120,
      "snapshot-name":         // template for the snapshot name; see section "Snapshot name templates" below
        "{server}-{label[VERSION]}_{period_type}#{period_number}_{timestamp:%Y-%m-%d_%H:%M:%S}_by_{env[USER]}",
      "shutdown-and-restart":  // shut down the server before taking a snapshot
        true,                  // and restart it afterwards
      "shutdown-timeout":      // timeout (in s) after which graceful shutdown is considered to have failed
         15,
      "allow-poweroff":        // power off the server if it cannot be shut down gracefully
         false,
      "rotate":                // rotate the existing snapshots and the new one, if any 
         true,
      "quarter-hourly":        // number of quarter-hourly snapshots to retain (intended for testing)
         0,
      "hourly":                // number of hourly snapshots to retain
         2,
      "daily":                 // number of daily snapshots to retain
         3,
      "weekly":                // number of weekly snapshots to retain
         4,
      "monthly":               // number of monthly snapshots to retain
         3,
      "quarter-yearly":        // number of quarter-yearly snapshots to retain
         2,
      "yearly":                // number of yearly snapshots to retain
         1
    },

  "servers": {                 // one entry per server
    "server-1": {              // Hetzner cloud server name
                               // uses only "defaults" settings
    },

    "server-2": {              // another server; override a few defaults
      "snapshot-name": "snapshot_{timestamp:%Y-%m-%d_%H:%M:%S}",
      "snapshot-timeout": 300,
      "shutdown-timeout": 30,
      "allow-poweroff": true,

      "hourly": 0,
      "daily": 0,
      "monthly": 6,
      "quarter-yearly": 0,
      "yearly": 0
    }
  }
}

Omitted defaults default to 0, false or '' except for:

• snapshot-timeout: defaults to 300
• shutdown-timeout: defaults to 30

Please note that this is not valid JSON due to the comments. File resources/config-example.json contains the same example as a valid JSON file without comments.

Taking snapshots

This script takes a snapshot of every server in the configuration file for which create-snapshot is true. Each of these snapshots also stores a copy of the current server labels.

If taking the snapshot takes longer than snapshot-timeout then that operation is considered to have failed.

If shutdown-and-restart is true and the server is running then the script attempts to shut down the server gracefully before taking the snapshot. If the server cannot be shut down gracefully within the shutdown-timeout then it will be powered down instead if allow-poweroff is true, or else the snapshot operation will fail. If the server was running before taking the snapshot then it is restarted afterwards.

Rotating snapshots

This script rotates the snapshots of every server in the configuration file for which rotate-snapshots is true.

"Rotating" means that existing snapshots will be renamed according to snapshot-name, or will be deleted if they are not contained in any of the configured quarter-hourly, hourly, ... yearly periods. Those settings determine for how many such periods the snapshots will be retained.

Snapshots that have been protected are neither renamed nor deleted during rotation. Nevertheless, they are taken into account in the rotation process.

Rotating snapshots is controlled by the following rules:

• Rotation period types (quarter-hourly, hourly, ... yearly) that were set to zero are ignored.
• Periods are counted backwards from the present into the past.
• The first period is the shortest period immediately preceding the instant of rotation.
• Other periods immediately precede the next shorter periods without gaps.
• If a period contains multiple snapshots then only the oldest one will be retained for that period.
• Rotated snapshots are renamed according to the template snapshot-name. This allows the server name and labels, the period, the snapshot timestamp and environment variables to become part of the snapshot name.
  See section Snapshot name templates below for details.

Snapshot name templates

snapshot-name must be a string and may contain Python format strings. The following field names are available for formatting:

Field name	Type	Rendered as
server	str	Server name, same as server in the configuration file
timestamp	datetime.datetime	Creation instant of the snapshot (not changed by rotation), expressed in the timezone of the system running this script. datetime-specific formatting may be used for this field.
label	dict[str]	Value of a server label at the creation instant of the snapshot (not changed by rotation), may be referred to as e.g. label[VERSION]
period_type	str	Type of period: quarter-hourly, hourly, ... yearly, or latest for a new snapshot that is not contained in any period
period_number	int	Rotation number of the period: 1 = latest, 2 = next to latest and so on; always 0 for a new snapshot with period latest
env	dict[str]	Value of an environment variable at the creation or rotation instant of the snapshot, may be referred to as e.g. env[USER]

Running the script natively

python3 -m hetzner_snap_and_rotate [options ...]

Command line options

Option	Description
--config config_file -c config_file	Read the configuration from config_file. Default: config.json
--api-token-from env_var -t env_var	Read the API token from environment variable env_var, or read it from stdin if '-' is specified. Default: get the API token from config_file.
--facility syslog_facility -f syslog_facility	Send the log messages to syslog_facility (SYSLOG, USER, DAEMON, CRON, etc.). Default: send log messages to stdout.
--priority pri -p pri	Log only messages up to syslog priority pri (ERR, WARNING, NOTICE, INFO, DEBUG, or OFF to disable logging). Default: NOTICE.
--dry-run -n	Perform a trial run with no changes made. This requires only an API token with "Read" permission.
--version -v	Display the version number and exit.
--help -h	Display a help message and exit.

Passing the API token

The API token provides complete control over your Hetzner cloud project, therefore it must be protected against unauthorized access. The following methods are available for passing the API token to the script:

• Inlcuding it as api-token in the configuration file. The configuration file then must be protected adequately.
• Passing it via an environent variable and specifying the variable name on the command line as api-token-from.
• Piping it to the script through stdin and specifying api-token-from - on the command line.

Creating and rotating snapshots in a cron job

As a cron job, this script should run at least once per the shortest period for which snapshots are to be retained.
If, for example, the shortest retention period has been set to daily then the script should run at least daily.

Snapshots seem to be missing after rotation

If several types of retention period (e.g. daily, weekly and monthly) have been defined then the latest snapshot will be contained in the latest period of each type at the same time.
Since the snapshot will be named after the longest period (monthly), there will not be any snapshots named after the latest ones of the shorter retention periods (daily and weekly).

Running the script in a container

This image on Docker Hub runs the script in a Docker or Podman container (for Podman, substitute podman for docker in the following commands):

docker run [Docker options] undecaf/hetzner-snap-and-rotate:latest [script command line options]

The same command line options are available as for the native script.

Passing the configuration file to the container

The configuration file needs to be prepared on the host. It can be passed to the container in various ways:

• As a bind mount, e.g.

  docker run --mount type=bind,source=/path/to/your/config.json,target=/config.json [other Docker options] undecaf/hetzner-snap-and-rotate:latest [script command line options]

• As a secret; this requires Podman or Docker swarm.
  First, your configuration file needs to be saved as a secret (e.g. called config_json):

  docker secret create config_json /path/to/your/config.json

  That secret becomes part of your Docker/Podman configuration and then can be passed to the container:

  docker run --secret=config_json,target=/config.json [other Docker options] undecaf/hetzner-snap-and-rotate:latest [script command line options]

Environment variables

Environment variables referenced in your snapshot name templates must be passed to the container as --env options, e.g.

docker run --env USER=your_username [other Docker options] undecaf/hetzner-snap-and-rotate:latest [script command line options]

Examples

Display the script version and exit:

# does not require a configuration file
docker run --rm  undecaf/hetzner-snap-and-rotate:latest --version

Dry run with the API token in the configuration file, log priority DEBUG:

# option --tty/-t displays log output in real time
docker run \
  --rm \
  --tty \
  --mount type=bind,source=/path/to/your/config.json,target=/config.json \
  undecaf/hetzner-snap-and-rotate:latest --dry-run --priority DEBUG

Live run with the API token in the configuration file:

# option --tty/-t displays log output in real time
docker run \
  --rm \
  --tty \
  --mount type=bind,source=/path/to/your/config.json,target=/config.json \
  undecaf/hetzner-snap-and-rotate:latest

Passing the API token through stdin:

# requires option --interactive/-i, adding --tty/-t would display the API token :-(
cat /your/api/token/file | docker run \
  --rm \
  --interactive \
  --mount type=bind,source=/path/to/your/config.json,target=/config.json \
  undecaf/hetzner-snap-and-rotate:latest --api-token-from -

Licenses

Software: MIT

Documentation: CC-BY-SA 4.0