Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rbd: create token and use it for vault SA #3174

Merged
merged 2 commits into from
Jun 17, 2022

Conversation

Rakshith-R
Copy link
Contributor

@Rakshith-R Rakshith-R commented Jun 9, 2022

create the token if kubernetes version is 1.24+ and use it for vault sa.

Signed-off-by: Madhu Rajanna madhupr007@gmail.com
Signed-off-by: Rakshith R rar@redhat.com

Resolves: #3135

@mergify mergify bot added the component/rbd Issues related to RBD label Jun 9, 2022
@Rakshith-R Rakshith-R force-pushed the madhu/service-account branch 2 times, most recently from 29fc8ef to 078238d Compare June 9, 2022 10:53
@Rakshith-R Rakshith-R marked this pull request as ready for review June 9, 2022 10:53
@Rakshith-R Rakshith-R requested review from a team June 9, 2022 10:54
@Rakshith-R
Copy link
Contributor Author

/retest all

@Rakshith-R Rakshith-R marked this pull request as draft June 9, 2022 10:54
@@ -31,4 +31,7 @@ rules:
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["list", "get"]
- apiGroups: ["authentication.k8s.io"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same change is required in Rook, dont forget to send patch to Rook

Copy link
Collaborator

@Madhu-1 Madhu-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit, changes LGTM

@@ -303,6 +307,27 @@ func (kms *vaultTenantSA) getToken() (string, error) {
}
}

version, err := c.ServerVersion()
if err != nil {
return "", fmt.Errorf("can not get ServiceVersion %w", err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return "", fmt.Errorf("can not get ServiceVersion %w", err)
return "", fmt.Errorf("failed to get ServiceVersion %w", err)

}
// from kubernetes v1.24+, secret for service account tokens are not
// automatically created. Hence, use the create token api to fetch it.
if version.Major == kubeMajorVersion && version.Minor >= kubeMinorVersion {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add kubernetes changelog link here for future reference?

@@ -35,6 +36,9 @@ const (
// should be available in the Tenants namespace. This ServiceAccount
// will be used to connect to Hashicorp Vault.
vaultTenantSAName = "ceph-csi-vault-sa"
// Kubernetes version to get token from the ServiceAccount.
kubeMajorVersion = "1"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the name of these constants do not say anything about the feature that Kubernetes introduced. Please use a better name.

@@ -303,6 +307,27 @@ func (kms *vaultTenantSA) getToken() (string, error) {
}
}

version, err := c.ServerVersion()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move this to a helper function, one that can always be called. If the version does not need to create a token, just return no error.

@@ -303,6 +307,27 @@ func (kms *vaultTenantSA) getToken() (string, error) {
}
}

version, err := c.ServerVersion()
if err != nil {
return "", fmt.Errorf("can not get ServiceVersion %w", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the standard format "<message>: %w", with :

@Rakshith-R
Copy link
Contributor Author

https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/mini-e2e-helm_k8s-1.24/detail/mini-e2e-helm_k8s-1.24/18/pipeline/
https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/mini-e2e_k8s-1.24/detail/mini-e2e_k8s-1.24/19/pipeline

@pkalever , rbd nbd tests are failing


Jun  9 13:22:53.324: INFO: ExecWithOptions: execute(POST https://192.168.39.43:8443/api/v1/namespaces/cephcsi-e2e-4834c72e07c2/pods/csi-rbdplugin-787bh/exec?command=%2Fbin%2Fsh&command=-c&command=pstree+--arguments+%7C+grep+%5Br%5Dbd-nbd&container=csi-rbdplugin&container=csi-rbdplugin&stderr=true&stdout=true)
Jun  9 13:22:53.640: INFO: rbd-nbd process is not running yet: command terminated with exit code 1
Jun  9 13:22:53.640: FAIL: timed out waiting for the rbd-nbd process: rbd-nbd process is not running yet: command terminated with exit code 1

@Rakshith-R Rakshith-R force-pushed the madhu/service-account branch 2 times, most recently from 90e3127 to 30a6338 Compare June 13, 2022 10:22
@Rakshith-R
Copy link
Contributor Author

Rakshith-R commented Jun 13, 2022

I've tested it manually on k8s 1.24 and it works
PTAL
thanks

logs

I0613 05:36:22.167407 1 utils.go:193] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 GRPC call: /csi.v1.Controller/CreateVolume
I0613 05:36:22.167803 1 utils.go:197] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 GRPC request: {“capacity_range”:{“required_bytes”:1073741824},“name”:“pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70”,“parameters”:{“clusterID”:“rook-ceph”,“[csi.storage.k8s.io/pv/name":“pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70”,“csi.storage.k8s.io/pvc/name”:“rbd-pvc-enc”,“csi.storage.k8s.io/pvc/namespace”:“rook-ceph”,“encrypted”:“true”,“encryptionKMSID”:“vault-tenant-sa-test”,“imageFeatures”:“layering”,“imageFormat”:“2”,“pool”:“replicapool”},“secrets”:"](http://csi.storage.k8s.io/pv/name%22:%22pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70%22,%22csi.storage.k8s.io/pvc/name%22:%22rbd-pvc-enc%22,%22csi.storage.k8s.io/pvc/namespace%22:%22rook-ceph%22,%22encrypted%22:%22true%22,%22encryptionKMSID%22:%22vault-tenant-sa-test%22,%22imageFeatures%22:%22layering%22,%22imageFormat%22:%222%22,%22pool%22:%22replicapool%22%7D,%22secrets%22:%22)stripped”,“volume_capabilities”:[{“AccessType”:{“Mount”:{“fs_type”:“ext4”}},“access_mode”:{“mode”:1}}]}
I0613 05:36:22.168053 1 rbd_util.go:1275] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 setting disableInUseChecks: false image features: [layering] mounter: rbd
I0613 05:36:22.205620 1 omap.go:88] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 got omap values: (pool=“replicapool”, namespace=“”, name=“csi.volumes.default”): map[]
I0613 05:36:22.220023 1 omap.go:158] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 set omap keys (pool=“replicapool”, namespace=“”, name=“csi.volumes.default”): map[csi.volume.pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70:c21c71f9-eada-11ec-8cff-0242ac110005])
I0613 05:36:22.226953 1 omap.go:158] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 set omap keys (pool=“replicapool”, namespace=“”, name=“csi.volume.c21c71f9-eada-11ec-8cff-0242ac110005”): map[csi.imagename:csi-vol-c21c71f9-eada-11ec-8cff-0242ac110005 csi.volname:pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 csi.volume.encryptKMS:vault-tenant-sa-test csi.volume.owner:rook-ceph])
I0613 05:36:22.226976 1 rbd_journal.go:487] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 generated Volume ID (0001-0009-rook-ceph-0000000000000001-c21c71f9-eada-11ec-8cff-0242ac110005) and image name (csi-vol-c21c71f9-eada-11ec-8cff-0242ac110005) for request name (pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70)
I0613 05:36:22.227082 1 rbd_util.go:421] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 rbd: create replicapool/csi-vol-c21c71f9-eada-11ec-8cff-0242ac110005 size 1024M (features: [layering]) using mon 192.168.39.60:6789
I0613 05:36:22.285214 1 controllerserver.go:675] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 created image replicapool/csi-vol-c21c71f9-eada-11ec-8cff-0242ac110005 backed for request name pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70
I0613 05:36:22.314157 1 omap.go:158] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 set omap keys (pool=“replicapool”, namespace=“”, name=“csi.volume.c21c71f9-eada-11ec-8cff-0242ac110005”): map[csi.imageid:12ac7331b29a])
I0613 05:36:22.405549 1 utils.go:204] ID: 91 Req-ID: pvc-87c88c0f-340f-4931-a38c-1dbb4a189b70 GRPC response: {“volume”:{“capacity_bytes”:1073741824,“volume_context”:{“clusterID”:“rook-ceph”,“encrypted”:“true”,“encryptionKMSID”:“vault-tenant-sa-test”,“imageFeatures”:“layering”,“imageFormat”:“2”,“imageName”:“csi-vol-c21c71f9-eada-11ec-8cff-0242ac110005”,“journalPool”:“replicapool”,“pool”:“replicapool”},“volume_id”:“0001-0009-rook-ceph-0000000000000001-c21c71f9-eada-11ec-8cff-0242ac110005”}}

Madhu-1
Madhu-1 previously approved these changes Jun 14, 2022
@Rakshith-R Rakshith-R requested a review from a team June 14, 2022 04:41
Rakshith-R added a commit to Rakshith-R/rook that referenced this pull request Jun 14, 2022
This rbac is required to fetch serviceaccount
token for vault tenant sa encryption type on k8s 1.24+.
refer: ceph/ceph-csi#3174

Signed-off-by: Rakshith R <rar@redhat.com>
Rakshith-R added a commit to Rakshith-R/rook that referenced this pull request Jun 14, 2022
This rbac is required to fetch serviceaccount
token for vault tenant sa encryption type on k8s 1.24+.
refer: ceph/ceph-csi#3174

Signed-off-by: Rakshith R <rar@redhat.com>
yati1998
yati1998 previously approved these changes Jun 14, 2022
@Rakshith-R
Copy link
Contributor Author

/retest ci/centos/mini-e2e-helm/k8s-1.22

@Rakshith-R
Copy link
Contributor Author

/retest ci/centos/mini-e2e-helm/k8s-1.23

@ceph-csi-bot
Copy link
Collaborator

@Mergifyio requeue

@mergify
Copy link
Contributor

mergify bot commented Jun 17, 2022

requeue

☑️ This pull request is already queued

@ceph-csi-bot
Copy link
Collaborator

/retest ci/centos/k8s-e2e-external-storage/1.22

@ceph-csi-bot
Copy link
Collaborator

@Rakshith-R "ci/centos/k8s-e2e-external-storage/1.22" test failed. Logs are available at location for debugging

@ceph-csi-bot
Copy link
Collaborator

@Mergifyio requeue

@mergify
Copy link
Contributor

mergify bot commented Jun 17, 2022

requeue

☑️ This pull request is already queued

@ceph-csi-bot
Copy link
Collaborator

/retest ci/centos/mini-e2e-helm/k8s-1.21

@ceph-csi-bot
Copy link
Collaborator

@Rakshith-R "ci/centos/mini-e2e-helm/k8s-1.21" test failed. Logs are available at location for debugging

@ceph-csi-bot
Copy link
Collaborator

/retest ci/centos/k8s-e2e-external-storage/1.22

@ceph-csi-bot
Copy link
Collaborator

@Rakshith-R "ci/centos/k8s-e2e-external-storage/1.22" test failed. Logs are available at location for debugging

@ceph-csi-bot
Copy link
Collaborator

/retest ci/centos/k8s-e2e-external-storage/1.21

@ceph-csi-bot
Copy link
Collaborator

@Rakshith-R "ci/centos/k8s-e2e-external-storage/1.21" test failed. Logs are available at location for debugging

@ceph-csi-bot
Copy link
Collaborator

@Mergifyio requeue

@mergify
Copy link
Contributor

mergify bot commented Jun 17, 2022

requeue

☑️ This pull request is already queued

@ceph-csi-bot
Copy link
Collaborator

/retest ci/centos/k8s-e2e-external-storage/1.22

@ceph-csi-bot
Copy link
Collaborator

@Rakshith-R "ci/centos/k8s-e2e-external-storage/1.22" test failed. Logs are available at location for debugging

@ceph-csi-bot
Copy link
Collaborator

@Mergifyio requeue

@mergify
Copy link
Contributor

mergify bot commented Jun 17, 2022

requeue

☑️ This pull request is already queued

@ceph-csi-bot
Copy link
Collaborator

/retest ci/centos/mini-e2e-helm/k8s-1.21

@ceph-csi-bot
Copy link
Collaborator

@Rakshith-R "ci/centos/mini-e2e-helm/k8s-1.21" test failed. Logs are available at location for debugging

@ceph-csi-bot
Copy link
Collaborator

@Mergifyio requeue

@mergify
Copy link
Contributor

mergify bot commented Jun 17, 2022

requeue

☑️ This pull request is already queued

@mergify mergify bot merged commit 7a00736 into ceph:devel Jun 17, 2022
mergify bot pushed a commit to rook/rook that referenced this pull request Jun 19, 2022
This rbac is required to fetch serviceaccount
token for vault tenant sa encryption type on k8s 1.24+.
refer: ceph/ceph-csi#3174

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 623c515)
Rakshith-R added a commit to Rakshith-R/rook that referenced this pull request Jun 20, 2022
This rbac is required to fetch serviceaccount
token for vault tenant sa encryption type on k8s 1.24+.
refer: ceph/ceph-csi#3174

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 623c515)
(cherry picked from commit 54bf464)
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/rook that referenced this pull request Jun 21, 2022
This rbac is required to fetch serviceaccount
token for vault tenant sa encryption type on k8s 1.24+.
refer: ceph/ceph-csi#3174

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 623c515)
(cherry picked from commit 54bf464)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci/retry/e2e Label to retry e2e retesting on approved PR's component/rbd Issues related to RBD
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rbd: k8s 1.24 changes wrt to sa secrets may affect vault tenant sa encryption.
6 participants