fix: Nutanix CSI credentials Secret creation

[dkoshkin]

Depends on dkoshkin/build-add-nutanix-csi-to-examples

The existing code created a ClusterResourceSet with the user provided Secret. However, that won't work unless that Secret has an embedded Secret in it. Keeping it simple and just creating the Secret using the remote client. This way we can bubble up errors to the Cluster object.

---

Used the example cluster files and was able to create a Pod with a PV (spec from konvoy e2e files)

kubectl get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                  STORAGECLASS     REASON   AGE
pvc-f0369ac2-ce64-49ce-a472-6e895b1f423a   10Gi       RWO            Delete           Bound    e2e-volume-pvc-bound-to-pv/pv-volume-volume-pvc-bound-to-pv-tester-0   nutanix-volume            10m

[faiq]

However, that won't work unless that Secret has an embedded Secret in it.

Can you explain what this means? I don't think I quite understand. Can you show what a secret would look like?

[dkoshkin]

However, that won't work unless that Secret has an embedded Secret in it.

Can you explain what this means? I don't think I quite understand. Can you show what a secret would look like?

Yes @faiq , it would be something like this.

---
apiVersion: v1
kind: Secret
metadata:
  name: ${CLUSTER_NAME}-pe-creds-for-csi # this is the Secret that's on the bootstrap/management cluster
  labels:
    cluster.x-k8s.io/provider: nutanix
type: addons.cluster.x-k8s.io/resource-set
stringData:
  secret: |
    apiVersion: v1
    kind: Secret
      name: nutanix-csi-credentials # name that CSI driver expects
      namespace: ntnx-system # namespace of where the driver is installed
    stringData:
      key: ${NUTANIX_PRISM_ELEMENT_ENDPOINT}:${NUTANIX_PORT}:${NUTANIX_USER}:${NUTANIX_PASSWORD}

[faiq]

We ended up going with the CRS approach with @jimmidyson recommendation

#7 (comment)

I think it makes sense to keep a consistent of delivering things to the workload cluster.

However I do like a lot of your other changes

[dkoshkin]

We ended up going with the CRS approach with @jimmidyson recommendation

#7 (comment)

I think it makes sense to keep a consistent of delivering things to the workload cluster.

However I do like a lot of your other changes

I guess we could do the same with this Secret and create an embedded Secret on user's behalf, but I would like to discuss the overall approach with you and @jimmidyson again please.

I'm not sure if we gain anything much with CRS for these, it ties us closer to CRS (even for addons that have a CAAPH strategy) and makes it more difficult to debug failures since we aren't able to push events to the Cluster object on failed CRS applies.

[faiq]

Great changes!

[supershal]

LGTM. nice improvement. Cleanup of the secret should be handled by CAREN in delete cluster hook in addition to uninstalling helm release.