Skip to content

Commit

Permalink
RepositoryServer support in kando command line (#1951)
Browse files Browse the repository at this point in the history
* RepositoryServer support in kando

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* RepositoryServer support in kando

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* RepositoryServer support in kando using Interface

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update Documentation

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Interface Implementation

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add Comments

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Address Review

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Address Review

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Fix Issues

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Address Review

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add License

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Fix Lint Error

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update pkg/kando/location.go

Co-authored-by: Vivek Singh <vsingh.ggits.2010@gmail.com>

* Update pkg/kando/location.go

Co-authored-by: Vivek Singh <vsingh.ggits.2010@gmail.com>

* Address Comments

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Sort Imports

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* context parameter to methods in the Datamover interface

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update pkg/datamover/repository_server.go

Co-authored-by: Prasad Ghangal <prasad.ghangal@gmail.com>

* Address Comments

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* return datamover.DataMover from dataMoverFromCMD

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Refactor

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update utils.go

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add ContentCacheMB and MetadataCacheMB in RepositoryServer templating params

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Change Type from string to DataMoverType

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Address Comments

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Address Comments

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Make connectToKopiaRepositoryServer as a part of receivers

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Make connectToKopiaRepositoryServer as a part of receivers

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add compliance checks

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Lint

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Common Functions in repository_server.go

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Address Comments

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Fix Lint Issue

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Move common code

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* revert param.go

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Make profile and repositoryServer unexported

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Modularize functions

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Simplify Interface implementation

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Restructure code

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Restructure code

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update pkg/datamover/repository_server.go

Co-authored-by: Prasad Ghangal <prasad.ghangal@gmail.com>

* Update pkg/datamover/utils.go

Co-authored-by: Prasad Ghangal <prasad.ghangal@gmail.com>

* Update pkg/datamover/profile.go

Co-authored-by: Prasad Ghangal <prasad.ghangal@gmail.com>

* Update pkg/kando/location.go

Co-authored-by: Prasad Ghangal <prasad.ghangal@gmail.com>

* Update pkg/datamover/repository_server.go

Co-authored-by: Prasad Ghangal <prasad.ghangal@gmail.com>

* Revert changes for client.go

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Implement functions dataMoverForKopiaSnapshotFlag and dataMoverForOutputNameFlag

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Lint

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add common context for kando command line tool

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Revert changes of kando.go

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add repositoryServerUserHostname Flag

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Rename repositoryServerUserHostname const

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update repositoryServerUserHostnameFlagName value

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add method to check if hostname provided exists in user access map

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* remove shorthand from repositoryServerUserHostnameFlagName

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update Flag Name for providing client hostname

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Update Flag Name for providing repository server user hostname

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Enable Return snapInfo in kopiaLocationPush (#2166)

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Check if Kopia Repository Server is in Ready State

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Fix Lint Issue - Added Error Handling

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Remove check for kopia repository server in ready state before executing kando command

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add Tests for updated Interface based kando command line (#2168)

* Add Testing Utilities

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add Tests for Repository Server Tests for Kando

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add Tests for Profile for Kando

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add Parent Tests for datamover

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Remove newTestClient helper function

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Refactor Function Names

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Refactor Helper Function Names

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add c *C in Profile Test Suite

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Refactor Code to reuse already available helper functions

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Minor Refactoring - Use rss.repoServer.Username instead of rss.user

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Refactor Variable Name

* Export S3Compliant Test Related consts

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Use Test S3Compliant Values from Env instead of Hard-coding it.

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Use Random Port from NodePort range instead of 31325

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* use kanister-tools v9.99.9-dev instead of v0.93.0

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Removed unnecessary consts

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Remove unused import

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Refactor GetDefaultS3StorageCreds according to latest changes

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Remove TearDownSuite from profile_test.go

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Move Repo Server Setup to a separate method

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add Validation for data deletion

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add TODO comment for Data verification after kando pull

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Add helper function cleanupTestSecrets

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

* Lint TODO comment

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

---------

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

---------

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>
Co-authored-by: Vivek Singh <vsingh.ggits.2010@gmail.com>
Co-authored-by: Prasad Ghangal <prasad.ghangal@gmail.com>
  • Loading branch information
3 people committed Aug 1, 2023
1 parent f6e755a commit 0dc4385
Show file tree
Hide file tree
Showing 13 changed files with 1,133 additions and 174 deletions.
29 changes: 29 additions & 0 deletions pkg/datamover/datamover.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
// Copyright 2023 The Kanister Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package datamover

import "context"

type DataMover interface {
// Pull is used to download the data from object storage
// using the preferred data-mover
Pull(ctx context.Context, sourcePath, destinationPath string) error
// Push is used to upload the data to object storage
// using the preferred data-mover
Push(ctx context.Context, sourcePath, destinationPath string) error
// Delete is used to delete the data from object storage
// using the preferred data-mover
Delete(ctx context.Context, destinationPath string) error
}
26 changes: 26 additions & 0 deletions pkg/datamover/datamover_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Copyright 2023 The Kanister Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package datamover

import (
"testing"

. "gopkg.in/check.v1"
)

// Hook up gocheck into the "go test" runner.
func Test(t *testing.T) {
TestingT(t)
}
117 changes: 117 additions & 0 deletions pkg/datamover/profile.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
// Copyright 2023 The Kanister Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package datamover

import (
"context"

"github.com/pkg/errors"

crv1alpha1 "github.com/kanisterio/kanister/pkg/apis/cr/v1alpha1"
"github.com/kanisterio/kanister/pkg/kopia"
"github.com/kanisterio/kanister/pkg/kopia/repository"
"github.com/kanisterio/kanister/pkg/kopia/snapshot"
"github.com/kanisterio/kanister/pkg/param"
)

// Check that Profile implements DataMover interface
var _ DataMover = (*profile)(nil)

type profile struct {
outputName string
profile *param.Profile
snapJSON string
}

func (p *profile) Pull(ctx context.Context, sourcePath, destinationPath string) error {
if p.profile.Location.Type == crv1alpha1.LocationTypeKopia {
kopiaSnap, err := p.unmarshalKopiaSnapshot(ctx)
if err != nil {
return err
}
if err := p.connectToKopiaRepositoryServer(ctx); err != nil {
return err
}
return kopiaLocationPull(ctx, kopiaSnap.ID, destinationPath, sourcePath, p.profile.Credential.KopiaServerSecret.Password)
}
target, err := targetWriter(sourcePath)
if err != nil {
return err
}
return locationPull(ctx, p.profile, destinationPath, target)
}

func (p *profile) Push(ctx context.Context, sourcePath, destinationPath string) error {
if p.profile.Location.Type == crv1alpha1.LocationTypeKopia {
if err := p.connectToKopiaRepositoryServer(ctx); err != nil {
return err
}
_, err := kopiaLocationPush(ctx, destinationPath, p.outputName, sourcePath, p.profile.Credential.KopiaServerSecret.Password)
return err
}
source, err := sourceReader(sourcePath)
if err != nil {
return err
}
return locationPush(ctx, p.profile, destinationPath, source)
}

func (p *profile) Delete(ctx context.Context, destinationPath string) error {
if p.profile.Location.Type == crv1alpha1.LocationTypeKopia {
kopiaSnap, err := p.unmarshalKopiaSnapshot(ctx)
if err != nil {
return err
}
if err := p.connectToKopiaRepositoryServer(ctx); err != nil {
return err
}
return kopiaLocationDelete(ctx, kopiaSnap.ID, destinationPath, p.profile.Credential.KopiaServerSecret.Password)
}
return locationDelete(ctx, p.profile, destinationPath)
}

func (p *profile) connectToKopiaRepositoryServer(ctx context.Context) error {
contentCacheSize := kopia.GetDataStoreGeneralContentCacheSize(p.profile.Credential.KopiaServerSecret.ConnectOptions)
metadataCacheSize := kopia.GetDataStoreGeneralMetadataCacheSize(p.profile.Credential.KopiaServerSecret.ConnectOptions)
return repository.ConnectToAPIServer(
ctx,
p.profile.Credential.KopiaServerSecret.Cert,
p.profile.Credential.KopiaServerSecret.Password,
p.profile.Credential.KopiaServerSecret.Hostname,
p.profile.Location.Endpoint,
p.profile.Credential.KopiaServerSecret.Username,
contentCacheSize,
metadataCacheSize,
)
}

func (p *profile) unmarshalKopiaSnapshot(ctx context.Context) (*snapshot.SnapshotInfo, error) {
if p.snapJSON == "" {
return nil, errors.New("kopia snapshot information is required to manage data using kopia")
}
kopiaSnap, err := snapshot.UnmarshalKopiaSnapshot(p.snapJSON)
if err != nil {
return nil, err
}
return &kopiaSnap, nil
}

func NewProfileDataMover(prof *param.Profile, outputName, snapJson string) *profile {
return &profile{
outputName: outputName,
profile: prof,
snapJSON: snapJson,
}
}
80 changes: 80 additions & 0 deletions pkg/datamover/profile_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
// Copyright 2023 The Kanister Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package datamover

import (
"bytes"
"context"
"path/filepath"

. "gopkg.in/check.v1"

crv1alpha1 "github.com/kanisterio/kanister/pkg/apis/cr/v1alpha1"
"github.com/kanisterio/kanister/pkg/objectstore"
"github.com/kanisterio/kanister/pkg/testutil"
)

type ProfileSuite struct {
ctx context.Context
location *crv1alpha1.Location
}

var _ = Suite(&ProfileSuite{})

const testContent = "test-content"

func (ps *ProfileSuite) SetUpSuite(c *C) {
// Set Context as Background
ps.ctx = context.Background()

// Set Location
ps.location = &crv1alpha1.Location{
Type: crv1alpha1.LocationTypeS3Compliant,
Bucket: testutil.TestS3BucketName,
}
}

func (ps *ProfileSuite) TestLocationOperationsForProfileDataMover(c *C) {
p := testutil.ObjectStoreProfileOrSkip(c, objectstore.ProviderTypeS3, *ps.location)
dir := c.MkDir()
path := filepath.Join(dir, "test-object1.txt")

source := bytes.NewBufferString(testContent)
err := locationPush(ps.ctx, p, path, source)
c.Assert(err, IsNil)

target := bytes.NewBuffer(nil)
err = locationPull(ps.ctx, p, path, target)
c.Assert(err, IsNil)
c.Assert(target.String(), Equals, testContent)

// test deleting single artifact
err = locationDelete(ps.ctx, p, path)
c.Assert(err, IsNil)

//test deleting dir with multiple artifacts
source = bytes.NewBufferString(testContent)
err = locationPush(ps.ctx, p, path, source)
c.Assert(err, IsNil)

path = filepath.Join(dir, "test-object2.txt")

source = bytes.NewBufferString(testContent)
err = locationPush(ps.ctx, p, path, source)
c.Assert(err, IsNil)

err = locationDelete(ps.ctx, p, dir)
c.Assert(err, IsNil)
}
141 changes: 141 additions & 0 deletions pkg/datamover/repository_server.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
// Copyright 2023 The Kanister Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package datamover

import (
"context"

"github.com/pkg/errors"

"github.com/kanisterio/kanister/pkg/kopia"
"github.com/kanisterio/kanister/pkg/kopia/repository"
"github.com/kanisterio/kanister/pkg/kopia/snapshot"
"github.com/kanisterio/kanister/pkg/param"
)

// Check that RepositoryServer implements DataMover interface
var _ DataMover = (*repositoryServer)(nil)

type repositoryServer struct {
outputName string
repositoryServer *param.RepositoryServer
snapJSON string
hostName string
}

func (rs *repositoryServer) Pull(ctx context.Context, sourcePath, destinationPath string) error {
kopiaSnap, err := rs.unmarshalKopiaSnapshot()
if err != nil {
return err
}
password, err := rs.connectToKopiaRepositoryServer(ctx)
if err != nil {
return err
}
return kopiaLocationPull(ctx, kopiaSnap.ID, destinationPath, sourcePath, password)
}

func (rs *repositoryServer) Push(ctx context.Context, sourcePath, destinationPath string) error {
password, err := rs.connectToKopiaRepositoryServer(ctx)
if err != nil {
return err
}
_, err = kopiaLocationPush(ctx, destinationPath, rs.outputName, sourcePath, password)
return err
}

func (rs *repositoryServer) Delete(ctx context.Context, destinationPath string) error {
kopiaSnap, err := rs.unmarshalKopiaSnapshot()
if err != nil {
return err
}
password, err := rs.connectToKopiaRepositoryServer(ctx)
if err != nil {
return err
}
return kopiaLocationDelete(ctx, kopiaSnap.ID, destinationPath, password)
}

func (rs *repositoryServer) connectToKopiaRepositoryServer(ctx context.Context) (string, error) {
hostname, userPassphrase, err := rs.hostnameAndUserPassphrase()
if err != nil {
return "", errors.Wrap(err, "Error Retrieving Hostname and User Passphrase from Repository Server")
}

return userPassphrase, repository.ConnectToAPIServer(
ctx,
string(rs.repositoryServer.Credentials.ServerTLS.Data[kopia.TLSCertificateKey]),
userPassphrase,
hostname,
rs.repositoryServer.Address,
rs.repositoryServer.Username,
rs.repositoryServer.ContentCacheMB,
rs.repositoryServer.MetadataCacheMB,
)
}

func (rs *repositoryServer) unmarshalKopiaSnapshot() (*snapshot.SnapshotInfo, error) {
if rs.snapJSON == "" {
return nil, errors.New("kopia snapshot information is required to manage data using kopia")
}
kopiaSnap, err := snapshot.UnmarshalKopiaSnapshot(rs.snapJSON)
if err != nil {
return nil, err
}
return &kopiaSnap, nil
}

func (rs *repositoryServer) hostnameAndUserPassphrase() (string, string, error) {
var hostname, userPassphrase string
userAccessMap := make(map[string]string)
for key, value := range rs.repositoryServer.Credentials.ServerUserAccess.Data {
userAccessMap[key] = string(value)
}

// if hostname is not provided, use the random hostname in the map as default
for key, val := range userAccessMap {
hostname = key
userPassphrase = val
break
}
// check if hostname is provided in the repository server
if rs.hostName != "" {
err := rs.checkHostnameExistsInUserAccessMap(userAccessMap)
if err != nil {
return "", "", err
}
hostname = rs.hostName
userPassphrase = userAccessMap[hostname]
}

return hostname, string(userPassphrase), nil
}

func (rs *repositoryServer) checkHostnameExistsInUserAccessMap(userAccessMap map[string]string) error {
// check if hostname is provided in the repository server exists in the user access map
if _, ok := userAccessMap[rs.hostName]; !ok {
return errors.New("hostname provided in the repository server does not exist in the user access map")
}
return nil
}

func NewRepositoryServerDataMover(repoServer *param.RepositoryServer, outputName, snapJson, userHostname string) *repositoryServer {
return &repositoryServer{
outputName: outputName,
repositoryServer: repoServer,
snapJSON: snapJson,
hostName: userHostname,
}
}
Loading

0 comments on commit 0dc4385

Please sign in to comment.