Add repositoryServerUserHostname to the kopia repository based functions #2177

r4rajat · 2023-07-14T08:06:39Z

Change Overview

Add repositoryServerUserHostname as an Optional Argument to the Kopia Repository Based functions.

It enables to users to pass the hostname from the user access map which they want to use while performing kopia repository server based operations

Pull request type

Please check the type of change your PR introduces:

Issues

fixes #issue-number

Test Plan

💪 Manual
⚡ Unit test
💚 E2E

Manual Testing Steps

1) Create Images for Kanister and Repo Server controller

git tag -fa v21-repo-server-rajat -m "Testing"

bash build/gorelease.sh

docker tag ghcr.io/kanisterio/controller:v21-repo-server-rajat r4rajat/controller:v21-repo-server-rajat

docker tag ghcr.io/kanisterio/repo-server-controller:v21-repo-server-rajat r4rajat/repo-server-controller:v21-repo-server-rajat

docker push r4rajat/controller:v21-repo-server-rajat && docker push r4rajat/repo-server-controller:v21-repo-server-rajat

2) Install Kanister

helm install kanister ./helm/kanister-operator \
--namespace kanister \
--set image.repository=r4rajat/controller \
--set image.tag=v21-repo-server-rajat \
--set repositoryServerImage.repository=r4rajat/repo-server-controller \
--set repositoryServerImage.tag=v21-repo-server-rajat \ 
--set controller.parallelism=10  \
--create-namespace

3) Apply Repo Server CRD

kubectl apply -f pkg/customresource/repositoryserver.yaml -n kanister

4) Create Test Application [Time Logger]

kubectl create namespace time-logger

kubectl create -f ./examples/time-log/time-logger-deployment.yaml -n time-logger

5) Create OpenSSL Certificate

openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem

6) Create S3 Location and Location Secret Config Files

S3 Location Secret

vi s3_location_creds.yaml

apiVersion: v1
kind: Secret
metadata:
   name: s3-creds
   namespace: kanister
   labels:
      repo.kanister.io/target-namespace: monitoring
type: secrets.kanister.io/aws
data:
   # required: base64 encoded value for key with proper permissions for the bucket
   aws_access_key_id: <base64 encoded access key>
   # required: base64 encoded value for the secret corresponding to the key above
   aws_secret_access_key: <base64 encoded secret key>

S3 Location

vi s3_location.yaml

apiVersion: v1
kind: Secret
metadata:
   name: s3-location
   namespace: kanister
   labels:
      repo.kanister.io/target-namespace: monitoring
type: Opaque
data:
   # required: specify the type of the store
   # supported values are s3, gcs, azure, and file-store
   type: czM=
   bucket: cmFqYXQtaW5mcmFjbG91ZA==
   # optional: used as a sub path in the bucket for all backups
   path: L3JlcG8tY29udHJvbGxlci8=
   # required, if supported by the provider
   region: dXMtZWFzdC0x
   # optional: if set to true, do not verify SSL cert.
   # Default, when omitted, is false
   #skipSSLVerify: false
   # required: if type is `file-store`
   # optional, otherwise
   #claimName: store-pvc

7) Apply Secrets

kubectl create secret tls repository-server-tls-cert --cert=certificate.pem --key=key.pem -n kanister

kubectl create secret generic repository-server-user-access -n kanister --from-literal=localhost=test1234 --from-literal=localhost2=test12345 --from-literal=localhost3=test123456

kubectl create secret generic repository-admin-user -n kanister --from-literal=username=admin@testpod1 --from-literal=password=test1234

kubectl create secret generic repo-pass -n kanister --from-literal=repo-password=test1234

kubectl apply -f s3_location_creds.yaml -n kanister

kubectl apply -f s3_location.yaml -n kanister

8) Create Repository

kopia --log-level=error --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-cache repository create --no-check-for-updates --cache-directory=/tmp/cache.dir --content-cache-size-mb=0 --metadata-cache-size-mb=500 --override-hostname=mysql.app --override-username=kanisterAdmin s3 --bucket=rajat-infracloud --prefix=/repo-controller/ --region=us-east-1 --access-key=<ACCESS_KEY> --secret-access-key=<SECRET_ACCESS_KEY>

9) Create Repository Server CR

vi repo-server-cr.yaml

apiVersion: cr.kanister.io/v1alpha1
kind: RepositoryServer
metadata:
  labels:
    app.kubernetes.io/name: repositoryserver
    app.kubernetes.io/instance: repositoryserver-sample
    app.kubernetes.io/part-of: kanister
    app.kuberentes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kanister
  name: kopia-repo-server-1
  namespace: kanister
spec:
  storage:
    secretRef:
      name: s3-location
      namespace: kanister
    credentialSecretRef:
      name: s3-creds
      namespace: kanister
  repository:
    rootPath: /repo-controller/
    passwordSecretRef:
      name: repo-pass
      namespace: kanister
    username: kanisterAdmin
    hostname: mysql.app
  server:
    adminSecretRef:
      name: repository-admin-user
      namespace: kanister
    tlsSecretRef:
      name: repository-server-tls-cert
      namespace: kanister
    userAccess:
      userAccessSecretRef:
        name: repository-server-user-access
        namespace: kanister
      username: kanisteruser

kubectl apply -f repo-server-cr.yaml -n kanister

Wait till the status of Repository Server CR gets to ServerReady , You could check it by running following command

kubectl describe -n kanister repositoryserver.cr.kanister.io/kopia-repo-server-1

10) Create Blueprint

vi test-blueprint.yaml

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: backupdate-bp
  namespace: kanister
actions:
  backup:
    outputArtifacts:
      timeLog:
        keyValue:
          path: '/repo-controller/time-logger/'
      backupIdentifier:
        keyValue:
          id: "{{ .Phases.backupToS3.Output.backupID }}"
    phases:
    - func: BackupDataUsingKopiaServer
      name: backupToS3
      args:
        namespace: "{{ .Deployment.Namespace }}"
        pod: "{{ index .Deployment.Pods 0 }}"
        container: test-container
        includePath: /var/log
        repositoryServerUserHostname: localhost2

kubectl create -f test-blueprint.yaml -n kanister

11) Build kanctl with latest changes

go build -o kanctl cmd/kanctl/main.go

12) Take Backup of the Application

./kanctl create actionset --action backup --namespace kanister --blueprint backupdate-bp --deployment time-logger/time-logger --repository-server=kopia-repo-server-1

actionset backup-7m5lh created

Check Status of the actionset

kubectl describe actionsets -n kanister backup-7m5lh

Events:
  Type    Reason           Age   From                 Message
  ----    ------           ----  ----                 -------
  Normal  Started Action   9s    Kanister Controller  Executing action backup
  Normal  Started Phase    9s    Kanister Controller  Executing phase backupToS3
  Normal  Ended Phase      3s    Kanister Controller  Completed phase backupToS3
  Normal  Update Complete  3s    Kanister Controller  Updated ActionSet 'backup-7m5lh' Status->complete

13) Update Blueprint with userHostname with value not in user access map

vi test-blueprint.yaml

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: backupdate-bp
  namespace: kanister
actions:
  backup:
    outputArtifacts:
      timeLog:
        keyValue:
          path: '/repo-controller/time-logger/'
      backupIdentifier:
        keyValue:
          id: "{{ .Phases.backupToS3.Output.backupID }}"
    phases:
    - func: BackupDataUsingKopiaServer
      name: backupToS3
      args:
        namespace: "{{ .Deployment.Namespace }}"
        pod: "{{ index .Deployment.Pods 0 }}"
        container: test-container
        includePath: /var/log
        repositoryServerUserHostname: localhost20

kubectl apply -f test-blueprint.yaml -n kanister

14) Take Backup of the Application

./kanctl create actionset --action backup --namespace kanister --blueprint backupdate-bp --deployment time-logger/time-logger --repository-server=kopia-repo-server-1

actionset backup-w85qd created

Check Status of the actionset

kubectl describe actionsets -n kanister backup-w85qd

Events:
  Type     Reason                          Age   From                 Message
  ----     ------                          ----  ----                 -------
  Normal   Started Action                  9s    Kanister Controller  Executing action backup
  Normal   Started Phase                   9s    Kanister Controller  Executing phase backupToS3
  Warning  ActionSetFailed Action: backup  9s    Kanister Controller  Failed to execute phase: v1alpha1.Phase{Name:"backupToS3", State:"pending", Output:map[string]interface {}(nil)}: Failed to fetch Hostname/User Passphrase from Secret: Failed to find Hostname in User Access Map: hostname provided in the repository server does not exist in the user access map

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

github-actions · 2023-07-14T08:06:53Z

Thanks for submitting this pull request 🎉. The team will review it soon and get back to you.

If you haven't already, please take a moment to review our project contributing guideline and Code of Conduct document.

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

…nctions

kale-amruta · 2023-07-18T07:47:00Z

docs/functions.rst

@@ -1607,6 +1607,7 @@ Arguments:
   `container`, Yes, `string`, name of the kanister sidecar container
   `includePath`, Yes, `string`, path of the data to be backed up
   `snapshotTags`, No, `string`, custom tags to be provided to the kopia snapshots
+   `userHostname`, No, `string`, user's hostname from user access map to be provided to the kopia repository server for performing backup


We should change it to RepositoryServerUserHostname

What is user access map? We will have to simplify this for users.

kale-amruta

We should rename userHostname

viveksinghggits

LGTM

viveksinghggits · 2023-07-18T08:05:39Z

docs/functions.rst

@@ -1607,6 +1607,7 @@ Arguments:
   `container`, Yes, `string`, name of the kanister sidecar container
   `includePath`, Yes, `string`, path of the data to be backed up
   `snapshotTags`, No, `string`, custom tags to be provided to the kopia snapshots
+   `userHostname`, No, `string`, user's hostname from user access map to be provided to the kopia repository server for performing backup


What is user access map? We will have to simplify this for users.

…nctions

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

viveksinghggits · 2023-07-18T12:51:20Z

docs/functions.rst

@@ -1607,6 +1607,7 @@ Arguments:
   `container`, Yes, `string`, name of the kanister sidecar container
   `includePath`, Yes, `string`, path of the data to be backed up
   `snapshotTags`, No, `string`, custom tags to be provided to the kopia snapshots
+   `repositoryServerUserHostname`, No, `string`, user's hostname to access the kopia repository server. Hostname would be available in the user access credential secret


does second sentence mean, user should figure out the hostname by looking at the user access credential secret?
What does it mean?

Yes ! User could provide the hostname which he/she wants to use by looking at user access credentials secret. If it's not provided we'll get the last entry from the user access map which is present inside the secret by default.

viveksinghggits · 2023-07-18T12:52:51Z

pkg/function/backup_data_using_kopia_server.go

+		if err != nil {
+			return "", "", errors.Wrap(err, "Failed to find Hostname in User Access Map")
+		}
+		hostName = hostname


do we really need this assignment?

Yes Vivek, so if we look at this function, we're taking hostname as an argument. So what we're doing is we're initially assigning the last value of hostname from user access map to hostName. That is the default case.

Later on we're checking if user has provided any hostname and if it exists in the user access map. If it's present, we're updating the hostName's value with the hostname provided by the user and returning hostName

aah, I got confused between the casing of the variables, didn't see difference between hostName and hostname.
I don't think we need hostName right? Even if we just remove hostName I think we can work with hostname easily. Let me know if you have questions about this.

Removed the hostName variable

viveksinghggits · 2023-07-18T12:53:35Z

pkg/function/backup_data_using_kopia_server.go

@@ -254,3 +267,11 @@ func userCredentialsAndServerTLS(tp *param.TemplateParams) (string, string, erro
 	}
 	return string(userCredJSON), string(certJSON), nil
 }
+
+func checkHostnameExistsInUserAccessMap(userAccessMap map[string]string, hostname string) error {
+	// check if hostname is provided in the repository server exists in the user access map


this is the hostname is provided or it exists in the user access map?

multiple mapping of hostnames and passwords could be present in User Access Map inside Repository Server User Access Credentials Secret, So the end user has to provide one of the hostname which he/she wants to use for connecting with kopia repository server

pkg/function/backup_data_using_kopia_server.go

…nctions

Co-authored-by: Vivek Singh <vsingh.ggits.2010@gmail.com>

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

pkg/function/backup_data_using_kopia_server.go

…nctions

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Co-authored-by: Vivek Singh <vsingh.ggits.2010@gmail.com>

…nctions

viveksinghggits · 2023-07-24T08:34:18Z

pkg/function/backup_data_using_kopia_server.go

@@ -223,24 +229,36 @@ func backupDataUsingKopiaServer(
 	return kopiacmd.ParseSnapshotCreateOutput(stdout, stderr)
 }

-func hostNameAndUserPassPhraseFromRepoServer(userCreds string) (string, string, error) {
+func hostNameAndUserPassPhraseFromRepoServer(userCreds, hostname string) (string, string, error) {


Suggested change

func hostNameAndUserPassPhraseFromRepoServer(userCreds, hostname string) (string, string, error) {

func hostNameAndUserPassPhraseFromRepoServer(userCreds, hostName string) (string, string, error) {

should hostName be preferred?

I see hostname is used as directly or as substring in other var name. I think we should change them to hostName or HostName. Whatever applicable.

Yeah, seems good. Will make the changes !

…nctions

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

…nctions

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

viveksinghggits · 2023-07-25T15:34:37Z

Looks good to me, let's 🚢 it.

…nctions

r4rajat added 2 commits July 14, 2023 13:31

Add userHostname to the kopia repository based functions

384a075

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

refactored functions.rst - removed unnecessary line

81b4ed3

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

infraq added this to In Progress in Kanister Jul 14, 2023

r4rajat marked this pull request as draft July 14, 2023 08:08

r4rajat and others added 2 commits July 17, 2023 09:58

Refactor username - remove unwanted variable

dd0ba4b

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

0a37b2b

…nctions

r4rajat marked this pull request as ready for review July 17, 2023 05:10

r4rajat requested review from viveksinghggits, kale-amruta and ankitjain235 July 17, 2023 05:10

r4rajat assigned viveksinghggits Jul 17, 2023

r4rajat added 2 commits July 18, 2023 10:29

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

97cc57b

…nctions

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

2179439

…nctions

kale-amruta reviewed Jul 18, 2023

View reviewed changes

viveksinghggits reviewed Jul 18, 2023

View reviewed changes

r4rajat and others added 4 commits July 18, 2023 13:55

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

a3babd4

…nctions

Change KopiaRepositoryServerUserHostname Argument Value

e5d1d56

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Update docs with repositoryServerUserHostname argument

716e5fd

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Update docs with repositoryServerUserHostname argument's description

5c397a6

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

r4rajat requested review from kale-amruta and viveksinghggits July 18, 2023 12:07

viveksinghggits reviewed Jul 18, 2023

View reviewed changes

r4rajat and others added 3 commits July 19, 2023 11:17

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

9d3701b

…nctions

Update pkg/function/backup_data_using_kopia_server.go

8cdc03b

Co-authored-by: Vivek Singh <vsingh.ggits.2010@gmail.com>

Update comment

9d68115

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

r4rajat requested a review from viveksinghggits July 19, 2023 06:13

viveksinghggits reviewed Jul 19, 2023

View reviewed changes

pkg/function/backup_data_using_kopia_server.go Outdated Show resolved Hide resolved

r4rajat and others added 5 commits July 19, 2023 19:04

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

b7e381a

…nctions

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

c502748

…nctions

remove hostName variable dependency

6821550

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Add comment

345c4a5

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Update pkg/function/backup_data_using_kopia_server.go

79bd293

Co-authored-by: Vivek Singh <vsingh.ggits.2010@gmail.com>

r4rajat requested a review from viveksinghggits July 20, 2023 06:54

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

a57695c

…nctions

viveksinghggits approved these changes Jul 24, 2023

View reviewed changes

Kanister automation moved this from In Progress to Reviewer approved Jul 24, 2023

r4rajat and others added 5 commits July 24, 2023 19:20

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

ced0ba7

…nctions

Rename Argument from hostname -> hostName

07f5c23

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Update Comments

31cbaef

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Update argument hostname

be2460d

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

4bc81bd

…nctions

r4rajat changed the title ~~Add userHostname to the kopia repository based functions~~ Add repositoryServerUserHostname to the kopia repository based functions Jul 25, 2023

Refactored variable userPassPhrase -> userPassphrase

5fb1bd0

Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com>

Merge branch 'master' into enable_custom_user_hostname_in_kanister_fu…

92344ff

…nctions

r4rajat added the kueue label Jul 26, 2023

mergify bot merged commit 74a0a31 into master Jul 26, 2023
16 checks passed

mergify bot deleted the enable_custom_user_hostname_in_kanister_functions branch July 26, 2023 08:39

Kanister automation moved this from Reviewer approved to Done Jul 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add repositoryServerUserHostname to the kopia repository based functions #2177

Add repositoryServerUserHostname to the kopia repository based functions #2177

r4rajat commented Jul 14, 2023 •

edited

Loading

github-actions bot commented Jul 14, 2023

kale-amruta Jul 18, 2023 •

edited

Loading

viveksinghggits Jul 18, 2023

r4rajat Jul 18, 2023

kale-amruta left a comment

viveksinghggits left a comment

viveksinghggits Jul 18, 2023

viveksinghggits Jul 18, 2023

r4rajat Jul 19, 2023

viveksinghggits Jul 18, 2023

r4rajat Jul 19, 2023

viveksinghggits Jul 19, 2023

r4rajat Jul 20, 2023

viveksinghggits Jul 18, 2023

r4rajat Jul 19, 2023

viveksinghggits Jul 24, 2023

viveksinghggits Jul 24, 2023

r4rajat Jul 24, 2023

r4rajat Jul 24, 2023

viveksinghggits commented Jul 25, 2023

	func hostNameAndUserPassPhraseFromRepoServer(userCreds, hostname string) (string, string, error) {
	func hostNameAndUserPassPhraseFromRepoServer(userCreds, hostName string) (string, string, error) {

Add repositoryServerUserHostname to the kopia repository based functions #2177

Add repositoryServerUserHostname to the kopia repository based functions #2177

Conversation

r4rajat commented Jul 14, 2023 • edited Loading

Change Overview

Pull request type

Issues

Test Plan

Manual Testing Steps

1) Create Images for Kanister and Repo Server controller

2) Install Kanister

3) Apply Repo Server CRD

4) Create Test Application [Time Logger]

5) Create OpenSSL Certificate

6) Create S3 Location and Location Secret Config Files

7) Apply Secrets

8) Create Repository

9) Create Repository Server CR

10) Create Blueprint

11) Build kanctl with latest changes

12) Take Backup of the Application

13) Update Blueprint with userHostname with value not in user access map

14) Take Backup of the Application

github-actions bot commented Jul 14, 2023

kale-amruta Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kale-amruta left a comment

Choose a reason for hiding this comment

viveksinghggits left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viveksinghggits commented Jul 25, 2023

r4rajat commented Jul 14, 2023 •

edited

Loading

kale-amruta Jul 18, 2023 •

edited

Loading