New Parameter *insecureTLS* for kanister functions using restic

[r4rajat]

Change Overview

This PR will enable the support for the insecureTLS flags for Kanister Functions which are using restic as the Datamover.

• Why do we need this PR ?
  Since we support the s3Compliant profiles and sometimes those locations are not secured by a TLS so when we create an actionset using named profile, we get the following error: Fatal: unable to open config file: Stat: Get \"https://test-bucket/s3-cxk41004-kanister-backup/?location=\": x509: certificate signed by unknown authority"

Changes have also been made for some of the Dockerfiles so that CI builds the images with latest version of restic since the current in use version doesn't support --insecure-tls flag.

Pull request type

Please check the type of change your PR introduces:

• 🚧 Work in Progress
• 🌈 Refactoring (no functional changes, no api changes)
• 🐹 Trivial/Minor
• 🐛 Bugfix
• 🌻 Feature
• 🗺️ Documentation
• 🤖 Test

Issues

• fixes Add a way to disable TLS for BackupData* and RestoreData* functions #2585

Test Plan

• 💪 Manual
• ⚡ Unit test
• 💚 E2E

Manual Testing Steps

1) Build the kanister-controller and cassandra images using goreleaser and push the images

2) Deployed cassandra and kanister applications on the cluster using helm

helm install kanister ./helm/kanister-operator \                        
--namespace kanister \
--set image.repository=r4rajat/controller \
--set image.tag=v8.0.0 \
--create-namespace

helm install cassandra bitnami/cassandra --namespace cassandra --set image.repository=r4rajat/cassandra --set image.tag=v8.0.0 --set cluster.replicaCount=2 --set image.registry=docker.io --set image.pullPolicy=Always --create-namespace

3) Created Blueprint and Profile

• Blueprint

> vi cassandra-blueprint.yaml

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: cassandra-blueprint
actions:
  backup:
    outputArtifacts:
      params:
        keyValue:
          backupPrefixLocation: "{{ .Phases.getBackupPrefixLocation.Output.backupPrefixLocation }}"
          snapshotPrefix: "{{ .Phases.getBackupPrefixLocation.Output.localSnapshotPrefixLocation }}"
          replicaCount: "{{ .Phases.getBackupPrefixLocation.Output.replicaCount }}"
          backupInfo: "{{ .Phases.backupToObjectStore.Output.BackupAllInfo }}"
          restorePathPrefix: "/"
    phases:
    - func: KubeExecAll
      name: getBackupPrefixLocation
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        pods: "{{ range .StatefulSet.Pods }} {{.}}{{end}}"
        containers: "cassandra"
        command:
          - bash
          - -o
          - errexit
          - -o
          - xtrace
          - -o
          - pipefail
          - -c
          - |
            BACKUP_PREFIX_LOCATION={{ .Profile.Location.Bucket }}/cassandra_backups/{{ .StatefulSet.Namespace }}/{{ .StatefulSet.Name }}
            LOCAL_SNAPSHOT_PREFIX_LOCATION=/bitnami/cassandra/cassandra_data/kanister_backups
            kando output backupPrefixLocation $BACKUP_PREFIX_LOCATION
            kando output localSnapshotPrefixLocation $LOCAL_SNAPSHOT_PREFIX_LOCATION
            kando output replicaCount {{ len .StatefulSet.Pods }}
    - func: KubeExecAll
      name: takeSnapshots
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        pods: "{{ range .StatefulSet.Pods }} {{.}}{{end}}"
        containers: "cassandra"
        command:
          - bash
          - -o
          - errexit
          - -o
          - xtrace
          - -o
          - pipefail
          - -c
          - |
            nodetool cleanup
            nodetool clearsnapshot
            nodetool snapshot -t ${HOSTNAME}
            snapshot_prefix="{{ .Phases.getBackupPrefixLocation.Output.localSnapshotPrefixLocation }}"
            mkdir -p ${snapshot_prefix}/${HOSTNAME}
            cd /bitnami/cassandra/data/data/
            if [ -n "$(ls -A | grep -v  -w "system" | grep -v  -w "system_traces")" ]
            then
              cp -r `ls -A | grep -v  -w "system" | grep -v  -w "system_traces"` ${snapshot_prefix}/${HOSTNAME}/
              cd ${snapshot_prefix}/${HOSTNAME}/
              cqlsh -u cassandra -p $CASSANDRA_PASSWORD -e "DESCRIBE SCHEMA" > schema.cql
            fi
            nodetool clearsnapshot
    - func: BackupDataAll
      name: backupToObjectStore
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        pods: "{{ range .StatefulSet.Pods }} {{.}}{{end}}"
        container: "cassandra"
        includePath: "{{ .Phases.getBackupPrefixLocation.Output.localSnapshotPrefixLocation }}"
        backupArtifactPrefix: "{{ .Phases.getBackupPrefixLocation.Output.backupPrefixLocation }}"
        insecureTLS: true
    - func: KubeExec
      name: deleteLocalBackup
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        pod: "{{ index .StatefulSet.Pods 0}}"
        command:
          - bash
          - -o
          - errexit
          - -o
          - xtrace
          - -o
          - pipefail
          - -c
          - |
            rm -rf {{ .Phases.getBackupPrefixLocation.Output.localSnapshotPrefixLocation }}
  restore:
    inputArtifactNames:
      - params
    phases:
    - func: ScaleWorkload
      name: shutdownPod
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        name: "{{ .StatefulSet.Name }}"
        kind: StatefulSet
        replicas: 0
    - func: RestoreDataAll
      name: restoreFromObjectStore
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        image: ghcr.io/kanisterio/kanister-tools:0.103.0
        backupArtifactPrefix: "{{ .ArtifactsIn.params.KeyValue.backupPrefixLocation }}"
        pods: "{{ range .StatefulSet.Pods }} {{.}}{{end}}"
        restorePath: "{{ .ArtifactsIn.params.KeyValue.restorePathPrefix }}"
        backupInfo: "{{ .ArtifactsIn.params.KeyValue.backupInfo }}"
        insecureTLS: true
    - func: ScaleWorkload
      name: bringupPod
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        name: "{{ .StatefulSet.Name }}"
        kind: StatefulSet
        replicas: "{{ .ArtifactsIn.params.KeyValue.replicaCount }}"
    - func: KubeExec
      name: restoreSnapshot
      args:
        namespace: "{{ .StatefulSet.Namespace }}"
        pod: "{{ index .StatefulSet.Pods 0 }}"
        command:
          - bash
          - -o
          - xtrace
          - -o
          - pipefail
          - -o
          - errexit
          - -c
          - |
            local_snapshot_prefix=/bitnami/cassandra/cassandra_data/kanister_backups/${HOSTNAME}
            rm -rf ${local_snapshot_prefix}/\?/
            if [ -n "$(ls ${local_snapshot_prefix}/)" ]
            then
              timeout=300
              while true
              do
                VAR=$((cqlsh -u cassandra -p $CASSANDRA_PASSWORD -e "DESCRIBE keyspaces;" --request-timeout=300) 2>&1)
                if [[ $VAR != *"Could not connect to localhost"* ]]
                then
                  break
                fi
                if [[ $timeout -le 0 ]]
                then
                   echo "Timed out waiting for cqlsh to configure.."
                   exit 1
                fi
                sleep 2
                timeout=$((timeout-2))
              done
              allkeyspaces=$(cqlsh -u cassandra -p $CASSANDRA_PASSWORD -e "DESCRIBE keyspaces" --request-timeout=300)
              keyspacestodel=$(echo $allkeyspaces | xargs -n1 echo | grep -v ^system || true)
              for ks in $keyspacestodel; do
                cqlsh -u cassandra -p $CASSANDRA_PASSWORD -e "drop keyspace if exists $ks;" --request-timeout=300
              done
              cqlsh -u cassandra -p $CASSANDRA_PASSWORD -e "$(cat ${local_snapshot_prefix}/schema.cql)" --request-timeout=300
              rm ${local_snapshot_prefix}/schema.cql
              list="$(ls ${local_snapshot_prefix}/)"
              cp -r ${local_snapshot_prefix}/. /bitnami/cassandra/data/data/
              cd /bitnami/cassandra/data/data/
              for keyspace in $list
              do
                cd $keyspace
                for table in *
                do
                  sstableloader -u cassandra -pw $CASSANDRA_PASSWORD -d ${HOSTNAME} $table/
                done
                cd ..
              done
            fi
            rm -rf {{ .ArtifactsIn.params.KeyValue.snapshotPrefix }}
  delete:
    inputArtifactNames:
      - params
    phases:
    - func: DeleteDataAll
      name: deleteFromObjectStore
      args:
        namespace: "{{ .Object.metadata.name }}"
        backupArtifactPrefix: "{{ .ArtifactsIn.params.KeyValue.backupPrefixLocation }}"
        backupInfo: "{{ .ArtifactsIn.params.KeyValue.backupInfo }}"
        reclaimSpace: true
        insecureTLS: true


> kubectl create -f cassandra-blueprint.yaml -n kanister

• Profile

> kanctl create profile s3compliant --access-key $ACCESS_KEY \              
--secret-key $SECRET_KEY \
--bucket $BUCKET \
--endpoint $ENDPOINT \
--namespace cassandra \
--skip-validation

secret 's3-secret-l9dg3q' created
profile 's3-profile-4h74x' created

4) Create backup, restore and delete Actionsets

• backup

> kanctl create actionset --action backup --namespace kanister --blueprint cassandra-blueprint --statefulset cassandra/cassandra --profile cassandra/s3-profile-sk7x6

actionset backup-5vxgv created


> k describe actionsets.cr.kanister.io -n kanister backup-5vxgv              


Name:         backup-5vxgv
Namespace:    kanister
Labels:       <none>
Annotations:  <none>
API Version:  cr.kanister.io/v1alpha1
Kind:         ActionSet
...
...
...
Events:
  Type    Reason           Age   From                 Message
  ----    ------           ----  ----                 -------
  Normal  Started Action   33s   Kanister Controller  Executing action backup
  Normal  Started Phase    33s   Kanister Controller  Executing phase getBackupPrefixLocation
  Normal  Ended Phase      33s   Kanister Controller  Completed phase getBackupPrefixLocation
  Normal  Started Phase    33s   Kanister Controller  Executing phase takeSnapshots
  Normal  Ended Phase      23s   Kanister Controller  Completed phase takeSnapshots
  Normal  Started Phase    23s   Kanister Controller  Executing phase backupToObjectStore
  Normal  Ended Phase      20s   Kanister Controller  Completed phase backupToObjectStore
  Normal  Started Phase    20s   Kanister Controller  Executing phase deleteLocalBackup
  Normal  Ended Phase      19s   Kanister Controller  Completed phase deleteLocalBackup
  Normal  Update Complete  19s   Kanister Controller  Updated ActionSet 'backup-5vxgv' Status->complete

• restore

> kanctl create actionset --action restore --namespace kanister --from backup-5vxgv

Warning: Neither --profile nor --repository-server flag is provided.
Action might fail if blueprint is using these resources.
actionset restore-backup-5vxgv-z62z5 created

> k describe actionsets.cr.kanister.io -n kanister restore-backup-5vxgv-z62z5                                                                                                                                                        

Name:         restore-backup-5vxgv-z62z5
Namespace:    kanister
Labels:       <none>
Annotations:  <none>
API Version:  cr.kanister.io/v1alpha1
Kind:         ActionSet
...
...
...
Events:
  Type    Reason           Age    From                 Message
  ----    ------           ----   ----                 -------
  Normal  Started Action   4m2s   Kanister Controller  Executing action restore
  Normal  Started Phase    4m2s   Kanister Controller  Executing phase shutdownPod
  Normal  Ended Phase      3m55s  Kanister Controller  Completed phase shutdownPod
  Normal  Started Phase    3m55s  Kanister Controller  Executing phase restoreFromObjectStore
  Normal  Ended Phase      3m41s  Kanister Controller  Completed phase restoreFromObjectStore
  Normal  Started Phase    3m41s  Kanister Controller  Executing phase bringupPod
  Normal  Ended Phase      2m28s  Kanister Controller  Completed phase bringupPod
  Normal  Started Phase    2m28s  Kanister Controller  Executing phase restoreSnapshot
  Normal  Ended Phase      21s    Kanister Controller  Completed phase restoreSnapshot
  Normal  Update Complete  21s    Kanister Controller  Updated ActionSet 'restore-backup-5vxgv-z62z5' Status->complete

• delete

> kanctl create actionset --action delete --namespace kanister --from backup-5vxgv --namespacetargets kanister                                                                                                                       

Warning: Neither --profile nor --repository-server flag is provided.
Action might fail if blueprint is using these resources.
actionset delete-backup-5vxgv-6wkns created


> k describe actionsets.cr.kanister.io -n kanister delete-backup-5vxgv-6wkns                                                                                                                                                         

Name:         delete-backup-5vxgv-6wkns
Namespace:    kanister
Labels:       <none>
Annotations:  <none>
API Version:  cr.kanister.io/v1alpha1
Kind:         ActionSet
...
...
...
Events:
  Type    Reason           Age   From                 Message
  ----    ------           ----  ----                 -------
  Normal  Started Action   45s   Kanister Controller  Executing action delete
  Normal  Started Phase    45s   Kanister Controller  Executing phase deleteFromObjectStore
  Normal  Ended Phase      41s   Kanister Controller  Completed phase deleteFromObjectStore
  Normal  Update Complete  41s   Kanister Controller  Updated ActionSet 'delete-backup-5vxgv-6wkns' Status->complete

[hairyhum]

@r4rajat can you also update related docs?

[pavannd1]

We were planning to deprecate these functions in favor of kopia-based functions. Not sure how much value this will add going forward.
@PrasadG193 @r4rajat

[r4rajat]

@hairyhum Will add the documentation in a follow up PR

[r4rajat]

Documentation for same #2610

[PrasadG193]

We were planning to deprecate these functions in favor of kopia-based functions. Not sure how much value this will add going forward.

@pavannd1 we've got requests from customers to support this. The kopia-based functions cannot be used unless we have server setup in place. Since this was straightforward to fix, and not much efforts were needed, I think we should add this feature.

[PrasadG193]

Seems straightforward. Let's wait for @pavannd1's feedback

[PrasadG193]

@hairyhum could you please review and approve? The merge kueue is blocking because you have "requested changes".