Skip to content

Commit

Permalink
New Parameter *insecureTLS* for kanister functions using restic (#2589)
Browse files Browse the repository at this point in the history 
* Update Dockerfiles to use latest version of the restic to support --insecure-tls flag

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update restic wrappers to support insecureTLS Flag

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update function BackupData for insecureTLS support

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update function BackupDataAll for insecureTLS support

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update function CheckRepository for insecureTLS support

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update function CopyVolumeData for insecureTLS support

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update function DeleteData for insecureTLS support

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update function DeleteDataAll for insecureTLS support

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update function RestoreData for insecureTLS support

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update function RestoreDataAll for insecureTLS support

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update tests for Restore Data

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update Documentation for Kanister Functions (#2610)

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

---------

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>
Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com>
  • Loading branch information
@r4rajat @pavannd1
r4rajat and pavannd1 committed Jan 30, 2024
1 parent aec073b commit 3c41939
Show file tree
Hide file tree
Showing 12 changed files with 148 additions and 63 deletions.
2 changes: 1 addition & 1 deletion docker/kanister-mongodb-replicaset/image/Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ ADD . /kanister

RUN /kanister/install.sh && rm -rf /kanister && rm -rf /tmp && mkdir /tmp

COPY --from=restic/restic:0.11.0 /usr/bin/restic /usr/local/bin/restic
COPY --from=restic/restic:0.16.2 /usr/bin/restic /usr/local/bin/restic

CMD ["tail", "-f", "/dev/null"]
7 changes: 7 additions & 0 deletions docs/functions.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,7 @@ Arguments:
`includePath`, Yes, `string`, path of the data to be backed up
`backupArtifactPrefix`, Yes, `string`, path to store the backup on the object store
`encryptionKey`, No, `string`, encryption key to be used for backups
`insecureTLS`, No, `bool`, enables insecure connection for data mover

Outputs:

Expand Down Expand Up @@ -368,6 +369,7 @@ Arguments:
`includePath`, Yes, `string`, path of the data to be backed up
`backupArtifactPrefix`, Yes, `string`, path to store the backup on the object store appended by pod name later
`encryptionKey`, No, `string`, encryption key to be used for backups
`insecureTLS`, No, `bool`, enables insecure connection for data mover

Outputs:

Expand Down Expand Up @@ -430,6 +432,7 @@ and restores data to the specified path.
`pod`, No, `string`, pod to which the volumes are attached
`volumes`, No, `map[string]string`, Mapping of `pvcName` to `mountPath` under which the volume will be available
`encryptionKey`, No, `string`, encryption key to be used during backups
`insecureTLS`, No, `bool`, enables insecure connection for data mover
`podOverride`, No, `map[string]interface{}`, specs to override default pod specs with

.. note::
Expand Down Expand Up @@ -505,6 +508,7 @@ respective PVCs and restores data to the specified path.
`pods`, No, `string`, pods to which the volumes are attached
`encryptionKey`, No, `string`, encryption key to be used during backups
`backupInfo`, Yes, `string`, snapshot info generated as output in BackupDataAll function
`insecureTLS`, No, `bool`, enables insecure connection for data mover
`podOverride`, No, `map[string]interface{}`, specs to override default pod specs with

.. note::
Expand Down Expand Up @@ -575,6 +579,7 @@ Arguments:
`volume`, Yes, `string`, name of the source PVC
`dataArtifactPrefix`, Yes, `string`, path on the object store to store the data in
`encryptionKey`, No, `string`, encryption key to be used during backups
`insecureTLS`, No, `bool`, enables insecure connection for data mover
`podOverride`, No, `map[string]interface{}`, specs to override default pod specs with

Outputs:
Expand Down Expand Up @@ -620,6 +625,7 @@ This function deletes the snapshot data backed up by the :ref:`backupdata` funct
`backupID`, No, `string`, (required if backupTag not provided) unique snapshot id generated during backup
`backupTag`, No, `string`, (required if backupID not provided) unique tag added during the backup
`encryptionKey`, No, `string`, encryption key to be used during backups
`insecureTLS`, No, `bool`, enables insecure connection for data mover
`podOverride`, No, `map[string]interface{}`, specs to override default pod specs with

Example:
Expand Down Expand Up @@ -657,6 +663,7 @@ BackupDataAll function.
`backupInfo`, Yes, `string`, snapshot info generated as output in BackupDataAll function
`encryptionKey`, No, `string`, encryption key to be used during backups
`reclaimSpace`, No, `bool`, provides a way to specify if space should be reclaimed
`insecureTLS`, No, `bool`, enables insecure connection for data mover
`podOverride`, No, `map[string]interface{}`, specs to override default pod specs with

Example:
Expand Down
15 changes: 11 additions & 4 deletions pkg/function/backup_data.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ const (
BackupDataOutputBackupSize = "size"
// BackupDataOutputBackupPhysicalSize is the key used for returning physical size taken by the snapshot
BackupDataOutputBackupPhysicalSize = "phySize"
// InsecureTLS is the key name which provides an option to a user to disable tls
InsecureTLS = "insecureTLS"
)

func init() {
Expand All @@ -83,6 +85,7 @@ func (b *backupDataFunc) Exec(ctx context.Context, tp param.TemplateParams, args

var namespace, pod, container, includePath, backupArtifactPrefix, encryptionKey string
var err error
var insecureTLS bool
if err = Arg(args, BackupDataNamespaceArg, &namespace); err != nil {
return nil, err
}
Expand All @@ -101,6 +104,9 @@ func (b *backupDataFunc) Exec(ctx context.Context, tp param.TemplateParams, args
if err = OptArg(args, BackupDataEncryptionKeyArg, &encryptionKey, restic.GeneratePassword()); err != nil {
return nil, err
}
if err = OptArg(args, InsecureTLS, &insecureTLS, false); err != nil {
return nil, err
}

if err = ValidateProfile(tp.Profile); err != nil {
return nil, errors.Wrapf(err, "Failed to validate Profile")
Expand All @@ -114,7 +120,7 @@ func (b *backupDataFunc) Exec(ctx context.Context, tp param.TemplateParams, args
}
ctx = field.Context(ctx, consts.PodNameKey, pod)
ctx = field.Context(ctx, consts.ContainerNameKey, container)
backupOutputs, err := backupData(ctx, cli, namespace, pod, container, backupArtifactPrefix, includePath, encryptionKey, tp)
backupOutputs, err := backupData(ctx, cli, namespace, pod, container, backupArtifactPrefix, includePath, encryptionKey, insecureTLS, tp)
if err != nil {
return nil, errors.Wrapf(err, "Failed to backup data")
}
Expand Down Expand Up @@ -147,6 +153,7 @@ func (*backupDataFunc) Arguments() []string {
BackupDataIncludePathArg,
BackupDataBackupArtifactPrefixArg,
BackupDataEncryptionKeyArg,
InsecureTLS,
}
}

Expand All @@ -158,19 +165,19 @@ type backupDataParsedOutput struct {
phySize string
}

func backupData(ctx context.Context, cli kubernetes.Interface, namespace, pod, container, backupArtifactPrefix, includePath, encryptionKey string, tp param.TemplateParams) (backupDataParsedOutput, error) {
func backupData(ctx context.Context, cli kubernetes.Interface, namespace, pod, container, backupArtifactPrefix, includePath, encryptionKey string, insecureTLS bool, tp param.TemplateParams) (backupDataParsedOutput, error) {
pw, err := GetPodWriter(cli, ctx, namespace, pod, container, tp.Profile)
if err != nil {
return backupDataParsedOutput{}, err
}
defer CleanUpCredsFile(ctx, pw, namespace, pod, container)
if err = restic.GetOrCreateRepository(cli, namespace, pod, container, backupArtifactPrefix, encryptionKey, tp.Profile); err != nil {
if err = restic.GetOrCreateRepository(cli, namespace, pod, container, backupArtifactPrefix, encryptionKey, insecureTLS, tp.Profile); err != nil {
return backupDataParsedOutput{}, err
}

// Create backup and dump it on the object store
backupTag := rand.String(10)
cmd, err := restic.BackupCommandByTag(tp.Profile, backupArtifactPrefix, backupTag, includePath, encryptionKey)
cmd, err := restic.BackupCommandByTag(tp.Profile, backupArtifactPrefix, backupTag, includePath, encryptionKey, insecureTLS)
if err != nil {
return backupDataParsedOutput{}, err
}
Expand Down
12 changes: 9 additions & 3 deletions pkg/function/backup_data_all.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ func (b *backupDataAllFunc) Exec(ctx context.Context, tp param.TemplateParams, a

var namespace, pods, container, includePath, backupArtifactPrefix, encryptionKey string
var err error
var insecureTLS bool
if err = Arg(args, BackupDataAllNamespaceArg, &namespace); err != nil {
return nil, err
}
Expand All @@ -99,6 +100,9 @@ func (b *backupDataAllFunc) Exec(ctx context.Context, tp param.TemplateParams, a
if err = OptArg(args, BackupDataAllEncryptionKeyArg, &encryptionKey, restic.GeneratePassword()); err != nil {
return nil, err
}
if err = OptArg(args, InsecureTLS, &insecureTLS, false); err != nil {
return nil, err
}

if err = ValidateProfile(tp.Profile); err != nil {
return nil, errors.Wrapf(err, "Failed to validate Profile")
Expand All @@ -124,7 +128,7 @@ func (b *backupDataAllFunc) Exec(ctx context.Context, tp param.TemplateParams, a
ps = strings.Fields(pods)
}
ctx = field.Context(ctx, consts.ContainerNameKey, container)
return backupDataAll(ctx, cli, namespace, ps, container, backupArtifactPrefix, includePath, encryptionKey, tp)
return backupDataAll(ctx, cli, namespace, ps, container, backupArtifactPrefix, includePath, encryptionKey, insecureTLS, tp)
}

func (*backupDataAllFunc) RequiredArgs() []string {
Expand All @@ -144,18 +148,20 @@ func (*backupDataAllFunc) Arguments() []string {
BackupDataAllBackupArtifactPrefixArg,
BackupDataAllPodsArg,
BackupDataAllEncryptionKeyArg,
InsecureTLS,
}
}

func backupDataAll(ctx context.Context, cli kubernetes.Interface, namespace string, ps []string, container string, backupArtifactPrefix, includePath, encryptionKey string, tp param.TemplateParams) (map[string]interface{}, error) {
func backupDataAll(ctx context.Context, cli kubernetes.Interface, namespace string, ps []string, container string, backupArtifactPrefix, includePath, encryptionKey string,
insecureTLS bool, tp param.TemplateParams) (map[string]interface{}, error) {
errChan := make(chan error, len(ps))
outChan := make(chan BackupInfo, len(ps))
Output := make(map[string]BackupInfo)
// Run the command
for _, pod := range ps {
go func(pod string, container string) {
ctx = field.Context(ctx, consts.PodNameKey, pod)
backupOutputs, err := backupData(ctx, cli, namespace, pod, container, fmt.Sprintf("%s/%s", backupArtifactPrefix, pod), includePath, encryptionKey, tp)
backupOutputs, err := backupData(ctx, cli, namespace, pod, container, fmt.Sprintf("%s/%s", backupArtifactPrefix, pod), includePath, encryptionKey, insecureTLS, tp)
errChan <- errors.Wrapf(err, "Failed to backup data for pod %s", pod)
outChan <- BackupInfo{PodName: pod, BackupID: backupOutputs.backupID, BackupTag: backupOutputs.backupTag}
}(pod, container)
Expand Down
14 changes: 11 additions & 3 deletions pkg/function/checkRepository.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ func (*CheckRepositoryFunc) Name() string {
return CheckRepositoryFuncName
}

func CheckRepository(ctx context.Context, cli kubernetes.Interface, tp param.TemplateParams, encryptionKey, targetPaths, jobPrefix string, podOverride crv1alpha1.JSONMap) (map[string]interface{}, error) {
func CheckRepository(ctx context.Context, cli kubernetes.Interface, tp param.TemplateParams, encryptionKey, targetPaths, jobPrefix string, insecureTLS bool, podOverride crv1alpha1.JSONMap) (map[string]interface{}, error) {
namespace, err := kube.GetControllerNamespace()
if err != nil {
return nil, errors.Wrapf(err, "Failed to get controller namespace")
Expand All @@ -59,7 +59,7 @@ func CheckRepository(ctx context.Context, cli kubernetes.Interface, tp param.Tem
PodOverride: podOverride,
}
pr := kube.NewPodRunner(cli, options)
podFunc := CheckRepositoryPodFunc(cli, tp, encryptionKey, targetPaths)
podFunc := CheckRepositoryPodFunc(cli, tp, encryptionKey, targetPaths, insecureTLS)
return pr.Run(ctx, podFunc)
}

Expand All @@ -68,6 +68,7 @@ func CheckRepositoryPodFunc(
tp param.TemplateParams,
encryptionKey,
targetPath string,
insecureTLS bool,
) func(ctx context.Context, pc kube.PodController) (map[string]interface{}, error) {
return func(ctx context.Context, pc kube.PodController) (map[string]interface{}, error) {
pod := pc.Pod()
Expand All @@ -89,6 +90,7 @@ func CheckRepositoryPodFunc(
tp.Profile,
targetPath,
encryptionKey,
insecureTLS,
cli,
pod.Namespace,
pod.Name,
Expand Down Expand Up @@ -126,12 +128,17 @@ func (c *CheckRepositoryFunc) Exec(ctx context.Context, tp param.TemplateParams,
defer func() { c.progressPercent = progress.CompletedPercent }()

var checkRepositoryArtifactPrefix, encryptionKey string
var insecureTLS bool
if err := Arg(args, CheckRepositoryArtifactPrefixArg, &checkRepositoryArtifactPrefix); err != nil {
return nil, err
}
if err := OptArg(args, CheckRepositoryEncryptionKeyArg, &encryptionKey, restic.GeneratePassword()); err != nil {
return nil, err
}
if err := OptArg(args, InsecureTLS, &insecureTLS, false); err != nil {
return nil, err
}

podOverride, err := GetPodSpecOverride(tp, args, CheckRepositoryPodOverrideArg)
if err != nil {
return nil, err
Expand All @@ -147,7 +154,7 @@ func (c *CheckRepositoryFunc) Exec(ctx context.Context, tp param.TemplateParams,
if err != nil {
return nil, errors.Wrapf(err, "Failed to create Kubernetes client")
}
return CheckRepository(ctx, cli, tp, encryptionKey, checkRepositoryArtifactPrefix, CheckRepositoryJobPrefix, podOverride)
return CheckRepository(ctx, cli, tp, encryptionKey, checkRepositoryArtifactPrefix, CheckRepositoryJobPrefix, insecureTLS, podOverride)
}

func (*CheckRepositoryFunc) RequiredArgs() []string {
Expand All @@ -158,6 +165,7 @@ func (*CheckRepositoryFunc) Arguments() []string {
return []string{
CheckRepositoryArtifactPrefixArg,
CheckRepositoryEncryptionKeyArg,
InsecureTLS,
}
}
func (c *CheckRepositoryFunc) ExecutionProgress() (crv1alpha1.PhaseProgress, error) {
Expand Down
14 changes: 11 additions & 3 deletions pkg/function/copy_volume_data.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ func copyVolumeData(
pvcName,
targetPath,
encryptionKey string,
insecureTLS bool,
podOverride map[string]interface{},
) (map[string]interface{}, error) {
// Validate PVC exists
Expand All @@ -99,7 +100,7 @@ func copyVolumeData(
PodOverride: podOverride,
}
pr := kube.NewPodRunner(cli, options)
podFunc := copyVolumeDataPodFunc(cli, tp, mountPoint, targetPath, encryptionKey)
podFunc := copyVolumeDataPodFunc(cli, tp, mountPoint, targetPath, encryptionKey, insecureTLS)
return pr.Run(ctx, podFunc)
}

Expand All @@ -109,6 +110,7 @@ func copyVolumeDataPodFunc(
mountPoint,
targetPath,
encryptionKey string,
insecureTLS bool,
) func(ctx context.Context, pc kube.PodController) (map[string]interface{}, error) {
return func(ctx context.Context, pc kube.PodController) (map[string]interface{}, error) {
// Wait for pod to reach running state
Expand All @@ -133,13 +135,14 @@ func copyVolumeDataPodFunc(
pod.Spec.Containers[0].Name,
targetPath,
encryptionKey,
insecureTLS,
tp.Profile,
); err != nil {
return nil, err
}
// Copy data to object store
backupTag := rand.String(10)
cmd, err := restic.BackupCommandByTag(tp.Profile, targetPath, backupTag, mountPoint, encryptionKey)
cmd, err := restic.BackupCommandByTag(tp.Profile, targetPath, backupTag, mountPoint, encryptionKey, insecureTLS)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -184,6 +187,7 @@ func (c *copyVolumeDataFunc) Exec(ctx context.Context, tp param.TemplateParams,

var namespace, vol, targetPath, encryptionKey string
var err error
var insecureTLS bool
if err = Arg(args, CopyVolumeDataNamespaceArg, &namespace); err != nil {
return nil, err
}
Expand All @@ -196,6 +200,9 @@ func (c *copyVolumeDataFunc) Exec(ctx context.Context, tp param.TemplateParams,
if err = OptArg(args, CopyVolumeDataEncryptionKeyArg, &encryptionKey, restic.GeneratePassword()); err != nil {
return nil, err
}
if err = OptArg(args, InsecureTLS, &insecureTLS, false); err != nil {
return nil, err
}
podOverride, err := GetPodSpecOverride(tp, args, CopyVolumeDataPodOverrideArg)
if err != nil {
return nil, err
Expand All @@ -211,7 +218,7 @@ func (c *copyVolumeDataFunc) Exec(ctx context.Context, tp param.TemplateParams,
if err != nil {
return nil, errors.Wrapf(err, "Failed to create Kubernetes client")
}
return copyVolumeData(ctx, cli, tp, namespace, vol, targetPath, encryptionKey, podOverride)
return copyVolumeData(ctx, cli, tp, namespace, vol, targetPath, encryptionKey, insecureTLS, podOverride)
}

func (*copyVolumeDataFunc) RequiredArgs() []string {
Expand All @@ -228,6 +235,7 @@ func (*copyVolumeDataFunc) Arguments() []string {
CopyVolumeDataVolumeArg,
CopyVolumeDataArtifactPrefixArg,
CopyVolumeDataEncryptionKeyArg,
InsecureTLS,
}
}

Expand Down
Loading

0 comments on commit 3c41939

Please sign in to comment.