Skip to content

Demonstrate 2 different approaches for backing up a Kafka cluster with Kasten

Notifications You must be signed in to change notification settings

michaelcourcy/kafka-kasten

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Kafka backup with Kasten

Kafka Kasten

Demonstrate 2 different approaches for backing up a Kafka cluster with Kasten :

  • Hot snapshot backup using Kasten
  • Application consistent backup using mirror maker and Kasten with kanister blueprint

This labs is deployed on openshift 4.6 on Azure and we extensivly use the Strimzi operator to manage the kafka clusters and kafka mirror maker 2.

Install the kafka operator

Go on Operator hub > Strimzi > install in all namespaces, which will install on the openshift-opertors namespace.

For this presentation we used the 0.23.0 version of strimzi.

A cluster and a mirror cluster

env

We mirror the primary cluster. This setting is very important and allow us to :

  • Validate that a restore does not lose data because we have a mirror to compare
  • Provide an application consistent method for backing up kafka by operating the backup on the mirror rather than on the primary

Three test-plan

For each parts we created a video to illustrate the detailed information that you will find in each test plan.

1. Setting up the env

In this part we install the env and introduce the different scripts that we'll use to control and compare the topics :

2. Hot snapshot

In this part we show that when taking a hot snapshot with kasten and restore it, the head is not anymore application consistent, even if there is no data loss at the tail level.

Blogs topic before snap :

Partition 0: ---------------------------
Partition 1: ---------------------------
Partition 2: ---------------------------

Blogs topic snapshotted :

Partition 0: ---------------------------
                                       ^ capture at time                                     
Partition 1: ---------------------------
                                      ^ capture at time but last segment not flushed
Partition 2: ------------------------------
                                          ^ capture at time + epsilon because even if snap are scheduled in 
                                          the same time storage may not be immediatly available for snap

Blogs topic restored :

Partition 0: ---------------------------
Partition 1: --------------------------
Partition 2: ------------------------------

3. Application consistent backup

In this part we run the backup on the mirror not on the source itself. But we are able to perform operation that can't be done on the primary like :

  • Stop all mirrormaker productions before snapshot
  • Call sync to flush all file to the disk
  • Stop the brokers and zookeepers before snapshot
  • Restart the brokers and zookeepers after snapshot
  • Restart mirrormaker production after snapshot

We show that with this method the restored topic has its head also application consistent.

About

Demonstrate 2 different approaches for backing up a Kafka cluster with Kasten

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages