Add Functionality to use RestoreData using Kopia Server

[r4rajat]

Change Overview

Add New Kanister function to use RestoreData using Kopia Server

This function RestoreDataUsingKopiaServer uses Kopia Repository Server as a Datamover to perform restore action

Pull request type

Please check the type of change your PR introduces:

• 🚧 Work in Progress
• 🌈 Refactoring (no functional changes, no api changes)
• 🐹 Trivial/Minor
• 🐛 Bugfix
• 🌻 Feature
• 🗺️ Documentation
• 🤖 Test

Issues

• fixes #issue-number

Test Plan

• 💪 Manual
• ⚡ Unit test
• 💚 E2E

Manual Testing Steps

1) Create Images for Kanister and Repo Server controller

git tag -fa v21-repo-server-rajat -m "Testing"

bash build/gorelease.sh

docker tag ghcr.io/kanisterio/controller:v21-repo-server-rajat r4rajat/controller:v21-repo-server-rajat

docker tag ghcr.io/kanisterio/repo-server-controller:v21-repo-server-rajat r4rajat/repo-server-controller:v21-repo-server-rajat

docker push r4rajat/controller:v21-repo-server-rajat && docker push r4rajat/repo-server-controller:v21-repo-server-rajat

2) Install Kanister

helm install kanister ./helm/kanister-operator \
--namespace kanister \
--set image.repository=r4rajat/controller \
--set image.tag=v21-repo-server-rajat \
--set repositoryServerImage.repository=r4rajat/repo-server-controller \
--set repositoryServerImage.tag=v21-repo-server-rajat \
--set controller.parallelism=10 \
--create-namespace

3) Apply Repo Server CRD

kubectl apply -f pkg/customresource/repositoryserver.yaml -n kanister

4) Create Test Application [Time Logger]

kubectl create namespace time-logger

kubectl create -f ./examples/time-log/time-logger-deployment.yaml -n time-logger

5) Create OpenSSL Certificate

openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem

6) Create S3 Location and Location Secret Config Files

• S3 Location Secret

vi s3_location_creds.yaml

apiVersion: v1
kind: Secret
metadata:
   name: s3-creds
   namespace: kanister
   labels:
      repo.kanister.io/target-namespace: monitoring
type: secrets.kanister.io/aws
data:
   # required: base64 encoded value for key with proper permissions for the bucket
   aws_access_key_id: <base64 encoded access key>
   # required: base64 encoded value for the secret corresponding to the key above
   aws_secret_access_key: <base64 encoded secret key>

• S3 Location

vi s3_location.yaml

apiVersion: v1
kind: Secret
metadata:
   name: s3-location
   namespace: kanister
   labels:
      repo.kanister.io/target-namespace: monitoring
type: Opaque
data:
   # required: specify the type of the store
   # supported values are s3, gcs, azure, and file-store
   type: czM=
   bucket: cmFqYXQtaW5mcmFjbG91ZA==
   # optional: used as a sub path in the bucket for all backups
   path: L3JlcG8tY29udHJvbGxlci8=
   # required, if supported by the provider
   region: dXMtZWFzdC0x
   # optional: if set to true, do not verify SSL cert.
   # Default, when omitted, is false
   #skipSSLVerify: false
   # required: if type is `file-store`
   # optional, otherwise
   #claimName: store-pvc

7) Apply Secrets

kubectl create secret tls repository-server-tls-cert --cert=certificate.pem --key=key.pem -n kanister

kubectl create secret generic repository-server-user-access -n kanister --from-literal=localhost=test1234

kubectl create secret generic repository-admin-user -n kanister --from-literal=username=admin@testpod1 --from-literal=password=test1234

kubectl create secret generic repo-pass -n kanister --from-literal=repo-password=test1234

kubectl apply -f s3_location_creds.yaml -n kanister

kubectl apply -f s3_location.yaml -n kanister

8) Create Repository

kopia --log-level=error --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-cache repository create --no-check-for-updates --cache-directory=/tmp/cache.dir --content-cache-size-mb=0 --metadata-cache-size-mb=500 --override-hostname=mysql.app --override-username=kanisterAdmin s3 --bucket=rajat-infracloud --prefix=/repo-controller/ --region=us-east-1 --access-key=<ACCESS_KEY> --secret-access-key=<SECRET_ACCESS_KEY>

9) Create Repository Server CR

vi repo-server-cr.yaml

apiVersion: cr.kanister.io/v1alpha1
kind: RepositoryServer
metadata:
  labels:
    app.kubernetes.io/name: repositoryserver
    app.kubernetes.io/instance: repositoryserver-sample
    app.kubernetes.io/part-of: kanister
    app.kuberentes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kanister
  name: kopia-repo-server-1
  namespace: kanister
spec:
  storage:
    secretRef:
      name: s3-location
      namespace: kanister
    credentialSecretRef:
      name: s3-creds
      namespace: kanister
  repository:
    rootPath: /repo-controller/
    passwordSecretRef:
      name: repo-pass
      namespace: kanister
    username: kanisterAdmin
    hostname: mysql.app
  server:
    adminSecretRef:
      name: repository-admin-user
      namespace: kanister
    tlsSecretRef:
      name: repository-server-tls-cert
      namespace: kanister
    userAccess:
      userAccessSecretRef:
        name: repository-server-user-access
        namespace: kanister
      username: kanisteruser

kubectl apply -f repo-server-cr.yaml -n kanister

Wait till the status of Repository Server CR gets to ServerReady , You could check it by running following command

kubectl describe -n kanister repositoryserver.cr.kanister.io/kopia-repo-server-1

10) Create Blueprint

vi test-blueprint.yaml

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: backupdate-bp
  namespace: kanister
actions:
  backup:
    outputArtifacts:
      timeLog:
        keyValue:
          path: '/repo-controller/time-logger/'
      backupIdentifier:
        keyValue:
          id: "{{ .Phases.backupToS3.Output.backupID }}"
    phases:
    - func: BackupDataUsingKopiaServer
      name: backupToS3
      args:
        namespace: "{{ .Deployment.Namespace }}"
        pod: "{{ index .Deployment.Pods 0 }}"
        container: test-container
        includePath: /var/log

  restore:
    inputArtifactNames:
    - timeLog
    - backupIdentifier
    phases:
    - func: ScaleWorkload
      name: shutdownPod
      args:
        namespace: "{{ .Deployment.Namespace }}"
        name: "{{ .Deployment.Name }}"
        kind: Deployment
        replicas: 0
    - func: RestoreDataUsingKopiaServer
      name: restoreFromS3
      args:
        namespace: "{{ .Deployment.Namespace }}"
        pod: "{{ index .Deployment.Pods 0 }}"
        image: ghcr.io/kanisterio/kanister-tools:0.89.0
        backupIdentifier: "{{ .ArtifactsIn.backupIdentifier.KeyValue.id }}"
        restorePath: /var/log
    - func: ScaleWorkload
      name: bringupPod
      args:
        namespace: "{{ .Deployment.Namespace }}"
        name: "{{ .Deployment.Name }}"
        kind: Deployment
        replicas: 1

kubectl create -f test-blueprint.yaml -n kanister

11) Build kanctl with latest changes

go build -o kanctl cmd/kanctl/main.go 

12) Take Backup of the Application

./kanctl create actionset --action backup --namespace kanister --blueprint backupdate-bp --deployment time-logger/time-logger --repository-server=kopia-repo-server-1

actionset backup-7m5lh created

Check Status of the actionset

kubectl describe actionsets -n kanister backup-7m5lh

Events:
  Type    Reason           Age   From                 Message
  ----    ------           ----  ----                 -------
  Normal  Started Action   9s    Kanister Controller  Executing action backup
  Normal  Started Phase    9s    Kanister Controller  Executing phase backupToS3
  Normal  Ended Phase      3s    Kanister Controller  Completed phase backupToS3
  Normal  Update Complete  3s    Kanister Controller  Updated ActionSet 'backup-7m5lh' Status->complete

13) Restore the Application

• Delete the files in /var/log folder in time-logger pod

kubectl exec -n time-logger time-logger-6c95887764-lrzjg -it sh

sh-5.1# cd /var/log/

sh-5.1# rm -rf time.log

• Run Restore Action

./kanctl --namespace kanister create actionset --action restore --from  "backup-7m5lh"

Check status

kubectl describe actionsets -n kanister restore-backup-7m5lh-crwlw

Events:
  Type    Reason           Age   From                 Message
  ----    ------           ----  ----                 -------
  Normal  Started Action   43s   Kanister Controller  Executing action restore
  Normal  Started Phase    43s   Kanister Controller  Executing phase shutdownPod
  Normal  Ended Phase      11s   Kanister Controller  Completed phase shutdownPod
  Normal  Started Phase    11s   Kanister Controller  Executing phase restoreFromS3
  Normal  Ended Phase      5s    Kanister Controller  Completed phase restoreFromS3
  Normal  Started Phase    5s    Kanister Controller  Executing phase bringupPod
  Normal  Ended Phase      1s    Kanister Controller  Completed phase bringupPod
  Normal  Update Complete  1s    Kanister Controller  Updated ActionSet 'restore-backup-7m5lh-crwlw' Status->complete

• Verify the restore

kubectl exec -n time-logger time-logger-7z87759646-mvtkb -it sh

sh-5.1# cd /var/log/

sh-5.1# ls
time.log

[github-actions]

Thanks for submitting this pull request 🎉. The team will review it soon and get back to you.

If you haven't already, please take a moment to review our project contributing guideline and Code of Conduct document.