add validation for kopia repository server controller secrets #1940

kale-amruta · 2023-03-02T08:22:17Z

Change Overview

This PR adds validation of the schema for the secrets that are required as a part of configuration of repository server CR. The design PR has the details for it - #1880

Pull request type

Please check the type of change your PR introduces:

Issues

fixes #issue-number

Test Plan

💪 Manual
⚡ Unit test
💚 E2E

Test Plan:

Build kanctl binary

go build -o kanctl cmd/kanctl/main.go

Create the secret that need to be validated

k create secret generic repo-pass-invalid --type='secrets.kanister.io/kopia-repository/password' -n kanister --from-literal=repo-test-password=test1234

Validate the invalid created secret

./kanctl validate repository-server-secrets --name repo-pass-invalid --resource-namespace kanister
Error: repo-password field is required in the kopia repository password secret repo-pass-invalid: validation Failed

File based validation, before creating secret

Contents of the file:

cat ~/config/samples/s3_location.yaml                            
apiVersion: v1
kind: Secret
metadata:
   name: s3-location
   namespace: kanister
   labels:
      repo.kanister.io/target-namespace: monitoring
type: secrets.kanister.io/storage-location
data:
   # required: specify the type of the store
   # supported values are s3, gcs, azure, and file-store
   type: czM=
   bucket: YW1ydXRhLXRlc3QtazEw
   # optional: used as a sub path in the bucket for all backups
   path: a2FuaXN0ZXIvYmFja3Vwcw==
   # required, if supported by the provider
   region: dXMtZWFzdC0x

Validate the file

./kanctl validate repository-server-secrets -f ~/config/samples/s3_location.yaml
Error: secret 'kanister:s3-location' does not have required field secrets.kanister.io/storage-location

github-actions · 2023-03-02T08:22:31Z

Thanks for submitting this pull request 🎉. The team will review it soon and get back to you.

If you haven't already, please take a moment to review our project contributing guideline and Code of Conduct document.

ankitjain235

Added some comments, nit-picks but overall, design looks good to me.

pkg/kanctl/secrets.go

pkg/secrets/secrets.go

pkg/secrets/repositoryserver/azure_location.go

pkg/secrets/repositoryserver/gcp_location.go

pkg/secrets/repositoryserver/repository_password.go

pkg/secrets/repositoryserver/aws_location.go

pkg/secrets/repositoryserver/secrets_interface.go

pkg/secrets/secrets.go

pkg/kanctl/secrets.go

pkg/secrets/secrets.go

…equired

pkg/secrets/gcp.go

PrasadG193

LGTM, left few minor comments

pkg/kanctl/secrets.go

pkg/secrets/filestore.go

pkg/secrets/gcp.go

pkg/secrets/secrets.go

…o/kanister into validateKopiaRepoServerSecrets

pkg/secrets/secrets.go

PrasadG193 · 2023-06-20T07:23:22Z

pkg/kanctl/validate.go

@@ -66,6 +66,8 @@ func performValidation(cmd *cobra.Command, args []string) error {
 		return performProfileValidation(p)
 	case "blueprint":
 		return performBlueprintValidation(p)
+	case "repository-server-secrets":
+		return performRepoServerSecretsValidation(p)


@kale-amruta this is not resolved yet

…o/kanister into validateKopiaRepoServerSecrets

pkg/kanctl/validate.go

…o/kanister into validateKopiaRepoServerSecrets

PrasadG193

LGTM

* add skeleton for validation repoitory server secrets * fix bugs * add licence headers * address review comments * change the secret type to v1.secrettype * initialize k8s client instead of all the clients since they are not required * move the consts to a secrets package so that they can be reused * move the consts to a secrets package so that they can be reused * refactor to use common constants * move the consts to a secrets package so that they can be reused * refactor to use the constants from secrets folder * address review comments * add headers * address review comments * move location key constants to secrets package * move location key constants to repositoryserver package * refactor use the constants from repositoryserver package * resolve merge conflicts * address review comments * add support for filestore secret * make lint happy * address comments * make lint happy * add licence headers * add comments on the constants * consistent error messages * Change name of constants for better readability * fix typo * simplify validation logic * change error message * remove unwanted comments * add comment in the validate repository password function * change function name for getLocationType to getLocationSecret * add nil checks for secrets * fix build issues * add empty secret checks * rename interface name to secret for readability * address review comments * address review comments * use cmd.Context

* add skeleton for validation repoitory server secrets * fix bugs * add licence headers * address review comments * change the secret type to v1.secrettype * initialize k8s client instead of all the clients since they are not required * move the consts to a secrets package so that they can be reused * move the consts to a secrets package so that they can be reused * refactor to use common constants * move the consts to a secrets package so that they can be reused * refactor to use the constants from secrets folder * address review comments * add headers * address review comments * move location key constants to secrets package * move location key constants to repositoryserver package * refactor use the constants from repositoryserver package * resolve merge conflicts * address review comments * add support for filestore secret * make lint happy * address comments * make lint happy * add licence headers * add comments on the constants * consistent error messages * Change name of constants for better readability * fix typo * simplify validation logic * change error message * remove unwanted comments * add comment in the validate repository password function * change function name for getLocationType to getLocationSecret * add nil checks for secrets * fix build issues * add empty secret checks * rename interface name to secret for readability * address review comments * address review comments * use cmd.Context * Add Cache Size Support in Templating Params in RepositoryServer (#2111) * Update param.go Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com> * Address Comments Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com> * Error Handling Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com> --------- Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com> * Make RepositoryServer struct consistent with RepositoryServer CRD (#2122) * Remove optional Markers, Add omitempty to fields in RepositoryServer struct as specified in repositoryserver.yaml Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Fix the indentation of repositoryserver CRD(repositoryserver.yaml) Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> --------- Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * add unit tests for repository server location credentials * make lint happy * add validation for kopia repository server controller secrets (#1940) * add skeleton for validation repoitory server secrets * fix bugs * add licence headers * address review comments * change the secret type to v1.secrettype * initialize k8s client instead of all the clients since they are not required * move the consts to a secrets package so that they can be reused * move the consts to a secrets package so that they can be reused * refactor to use common constants * move the consts to a secrets package so that they can be reused * refactor to use the constants from secrets folder * address review comments * add headers * address review comments * move location key constants to secrets package * move location key constants to repositoryserver package * refactor use the constants from repositoryserver package * resolve merge conflicts * address review comments * add support for filestore secret * make lint happy * address comments * make lint happy * add licence headers * add comments on the constants * consistent error messages * Change name of constants for better readability * fix typo * simplify validation logic * change error message * remove unwanted comments * add comment in the validate repository password function * change function name for getLocationType to getLocationSecret * add nil checks for secrets * fix build issues * add empty secret checks * rename interface name to secret for readability * address review comments * address review comments * use cmd.Context * Update MAINTAINERS.md (#2127) * support projectid in gcp provider (#2129) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * fix delete action (#2131) * Import the general client/auth package (#2134) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Fix pkg/config to use standard kubeconfig machinery (#2133) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Fix pkg/kube to use standard kubeconfig machinery (#2132) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Update ubi-minimal base image to ubi-minimal:9.2-691 (#2130) Co-authored-by: Kasten Production <infra@kasten.io> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * add tests for validating repository password and repository server admin credentials * fix tests * fix lint * fix e2e volume snapshot test (#2141) * Remove unused dirs to increase free space in GH runner (#2138) * Refactoring push_images.sh to consume a single source of truth containing valid images (#2147) * Refactoring push_images.sh to take in the single source of truth Made change * Added single source of truth file * Fixing issue with image list * Update RepositoryServer CR Status field to add `metav1.Condition` (#2135) * Update RepositoryServer CR Status fields, add metav1.conditions Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Update RepositoryServer CRD Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Run 'make codegen' Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Fix Lint Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Refactor name, set ClientInitialized to ClientUserInitialized Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> --------- Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add: ROADMAP.md (#2126) * Add: ROADMAP.md Signed-off-by: Mark <mlavi@users.noreply.github.com> * Update ROADMAP.md Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> * Update ROADMAP.md Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> * Update ROADMAP.md Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> * Update ROADMAP.md Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> * Update ROADMAP.md Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> * Update ROADMAP.md Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> * Update ROADMAP.md Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> * Update ROADMAP.md Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> --------- Signed-off-by: Mark <mlavi@users.noreply.github.com> Co-authored-by: Mark <mlavi@users.noreply.github.com> Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Kanister docs update to version 0.93.0 (#2149) Co-authored-by: Kasten Production <infra@kasten.io> * Refactor RepositoryServer controller (#2136) * Update RepositoryServer CR Status fields, add metav1.conditions Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Update RepositoryServer CRD Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Run 'make codegen' Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Fix Lint Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Refactor name, set ClientInitialized to ClientUserInitialized Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Refactor RepositoryServer Controller Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Run gofmt Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * add error handling in refreshServer function Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> * Add function to update Progress in RepositoryServerStatus Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> --------- Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> Co-authored-by: kale-amruta <41624751+kale-amruta@users.noreply.github.com> * Revert "Refactor RepositoryServer controller (#2136)" (#2150) This reverts commit fb4bd07. * Add base unit test suite for repository server controller(#PR1) (#2100) * add base suite for repository server controller * use constants for location secret keys * make the constants local to the package * add licence headers * fix licence headers * fix lint issues * remove unused functions and move to next PR * fix build issues * move secret creation utils under test suite * add base suite for repository server controller * use constants for location secret keys * make the constants local to the package * add licence headers * fix licence headers * fix lint issues * remove unused functions and move to next PR * fix build issues * move secret creation utils under test suite * address review comments * move test utilities to pkg/testutil/testutil.go * rename kopia repository path constant name * move constants to const.go * move constants to const.go * initialize reconciler in a better way * Run PostgreSQL integration tests on 12.6.0 chart (#2156) Signed-off-by: Prasad Ghangal <prasad.ghangal@gmail.com> * Adjust ref apps for OCP 4.13 (#2152) Kanister test automation config for OCP 4.13 version Co-authored-by: Vladislav Naumov <vladislav.naumov@PRG10457-MacBook-Pro.local> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * deps(go): bump google.golang.org/api from 0.120.0 to 0.130.0 (#2157) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.120.0 to 0.130.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.120.0...v0.130.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * deps(go): bump golang.org/x/oauth2 from 0.7.0 to 0.10.0 (#2158) Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.7.0 to 0.10.0. - [Commits](golang/oauth2@v0.7.0...v0.10.0) --- updated-dependencies: - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove unwanted declaration of context variable --------- Signed-off-by: Rajat Gupta <rajat.gupta@veeam.com> Signed-off-by: Akanksha Kumari <akankshakumari393@gmail.com> Signed-off-by: Mark <mlavi@users.noreply.github.com> Signed-off-by: Prasad Ghangal <prasad.ghangal@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Rajat Gupta <37516416+r4rajat@users.noreply.github.com> Co-authored-by: Akanksha kumari <akankshakumari393@gmail.com> Co-authored-by: Pavan Navarathna <6504783+pavannd1@users.noreply.github.com> Co-authored-by: alexy.bee <alexey.blokhin@kasten.io> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Mark Severson <mark@kasten.io> Co-authored-by: Infra Bot <63741182+infraq@users.noreply.github.com> Co-authored-by: Kasten Production <infra@kasten.io> Co-authored-by: Nishant Ravi Shankar <23146332+mellon-collie@users.noreply.github.com> Co-authored-by: Mark Lavi <5642075+mlavi@users.noreply.github.com> Co-authored-by: Mark <mlavi@users.noreply.github.com> Co-authored-by: Prasad Ghangal <prasad.ghangal@gmail.com> Co-authored-by: Vladislav Naumov <vladislav.naumov@kasten.io> Co-authored-by: Vladislav Naumov <vladislav.naumov@PRG10457-MacBook-Pro.local> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

infraq added this to In Progress in Kanister Mar 2, 2023

kale-amruta force-pushed the validateKopiaRepoServerSecrets branch 2 times, most recently from b1f47ae to 6f57b17 Compare March 2, 2023 10:21

kale-amruta changed the title ~~add skeleton for validation repoitory server secrets~~ add validation for kopia repository server controller secrets Mar 3, 2023

kale-amruta force-pushed the validateKopiaRepoServerSecrets branch from d4e2cf6 to fd48abe Compare March 3, 2023 09:25

kale-amruta marked this pull request as ready for review March 3, 2023 09:26

kale-amruta force-pushed the validateKopiaRepoServerSecrets branch 2 times, most recently from a620ab1 to e80a32e Compare March 3, 2023 09:40

add skeleton for validation repoitory server secrets

b64aa69

kale-amruta force-pushed the validateKopiaRepoServerSecrets branch from e80a32e to b64aa69 Compare March 3, 2023 09:57

fix bugs

26e56c1

kale-amruta force-pushed the validateKopiaRepoServerSecrets branch from e0b1825 to 26e56c1 Compare April 3, 2023 06:15

Merge branch 'master' into validateKopiaRepoServerSecrets

d6337d4

kale-amruta requested review from viveksinghggits, PrasadG193, ankitjain235 and pavannd1 April 3, 2023 06:16

ankitjain235 reviewed Apr 25, 2023

View reviewed changes

Merge branch 'master' into validateKopiaRepoServerSecrets

0998a34

PrasadG193 reviewed May 10, 2023

View reviewed changes

pkg/kanctl/secrets.go Outdated Show resolved Hide resolved

pkg/kanctl/secrets.go Show resolved Hide resolved

kale-amruta and others added 3 commits May 10, 2023 14:53

Merge branch 'master' into validateKopiaRepoServerSecrets

68e998b

add licence headers

dbd4438

address review comments

ed22616

viveksinghggits reviewed May 10, 2023

View reviewed changes

pkg/secrets/secrets.go Outdated Show resolved Hide resolved

kale-amruta added 2 commits May 11, 2023 14:30

change the secret type to v1.secrettype

11db2ee

initialize k8s client instead of all the clients since they are not r…

f29c26c

…equired

kale-amruta mentioned this pull request May 11, 2023

Move kopia storage consts to a secrets package #2059

Merged

10 tasks

kale-amruta changed the base branch from master to refactoringKopiaConstants May 11, 2023 09:44

kale-amruta force-pushed the refactoringKopiaConstants branch from d2a3d67 to 63e856b Compare May 11, 2023 10:17

kale-amruta requested a review from pavannd1 June 13, 2023 12:51

pavannd1 reviewed Jun 14, 2023

View reviewed changes

pkg/secrets/gcp.go Outdated Show resolved Hide resolved

address review comments

824bae1

kale-amruta force-pushed the validateKopiaRepoServerSecrets branch from 42fd38b to 824bae1 Compare June 14, 2023 06:10

kale-amruta added 2 commits June 14, 2023 11:40

Merge branch 'master' into validateKopiaRepoServerSecrets

464697b

Merge branch 'master' into validateKopiaRepoServerSecrets

6b2f76b

PrasadG193 reviewed Jun 16, 2023

View reviewed changes

pkg/kanctl/secrets.go Outdated Show resolved Hide resolved

pkg/secrets/filestore.go Outdated Show resolved Hide resolved

pkg/secrets/gcp.go Outdated Show resolved Hide resolved

pkg/secrets/secrets.go Outdated Show resolved Hide resolved

kale-amruta and others added 3 commits June 20, 2023 12:38

address review comments

0fa6443

Merge branch 'validateKopiaRepoServerSecrets' of github.com:kanisteri…

0961d6c

…o/kanister into validateKopiaRepoServerSecrets

Merge branch 'master' into validateKopiaRepoServerSecrets

018f02a

kale-amruta requested review from PrasadG193 and pavannd1 June 20, 2023 07:11

PrasadG193 reviewed Jun 20, 2023

View reviewed changes

kale-amruta added 4 commits June 20, 2023 15:16

use cmd.Context

0943b12

Merge branch 'validateKopiaRepoServerSecrets' of github.com:kanisteri…

994680c

…o/kanister into validateKopiaRepoServerSecrets

Merge branch 'validateKopiaRepoServerSecrets' of github.com:kanisteri…

9d198b6

…o/kanister into validateKopiaRepoServerSecrets

Merge branch 'validateKopiaRepoServerSecrets' of github.com:kanisteri…

1908806

…o/kanister into validateKopiaRepoServerSecrets

PrasadG193 reviewed Jun 20, 2023

View reviewed changes

pkg/kanctl/validate.go Outdated Show resolved Hide resolved

kale-amruta added 2 commits June 20, 2023 16:14

Merge branch 'validateKopiaRepoServerSecrets' of github.com:kanisteri…

9b027b2

…o/kanister into validateKopiaRepoServerSecrets

Merge branch 'validateKopiaRepoServerSecrets' of github.com:kanisteri…

43d11f6

…o/kanister into validateKopiaRepoServerSecrets

PrasadG193 approved these changes Jun 20, 2023

View reviewed changes

kale-amruta added the kueue label Jun 20, 2023

mergify bot merged commit 7c81e70 into master Jun 20, 2023
14 checks passed

Kanister automation moved this from Reviewer approved to Done Jun 20, 2023

mergify bot deleted the validateKopiaRepoServerSecrets branch June 20, 2023 13:26

kale-amruta mentioned this pull request Jul 6, 2023

Add Type S3Compliant to Kopia Repository Storage Args #2159

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add validation for kopia repository server controller secrets #1940

add validation for kopia repository server controller secrets #1940

kale-amruta commented Mar 2, 2023 •

edited

Loading

github-actions bot commented Mar 2, 2023

ankitjain235 left a comment

PrasadG193 left a comment

PrasadG193 Jun 20, 2023

PrasadG193 left a comment

add validation for kopia repository server controller secrets #1940

add validation for kopia repository server controller secrets #1940

Conversation

kale-amruta commented Mar 2, 2023 • edited Loading

Change Overview

Pull request type

Issues

Test Plan

github-actions bot commented Mar 2, 2023

ankitjain235 left a comment

Choose a reason for hiding this comment

PrasadG193 left a comment

Choose a reason for hiding this comment

PrasadG193 Jun 20, 2023

Choose a reason for hiding this comment

PrasadG193 left a comment

Choose a reason for hiding this comment

kale-amruta commented Mar 2, 2023 •

edited

Loading