[BUG]: X_CSI_AUTH_TYPE cannot be set in CSM Operator

[lohbe]

Bug Description

adding X_CSI_ISI_AUTH_TYPE=1 to match PowerScale's default session-type authentication in storage_csm_powerscale_v280.yaml has no effect. This is not really documented, so I've set it in multiple locations:

spec.driver.common.envs[]
spec.driver.controller.envs[]
spec.driver.node.envs[]

specifically,

envs:
  - name: X_CSI_ISI_AUTH_TYPE
    value: "1"

This is likely a CSM issue - the csi-powerscale helm chart deploys with no issue after setting isiAuthType: 1 in values.yaml.

Logs

$ kubectl logs -n isilon pod/isilon-node-q4flp 
Defaulted container "driver" out of: driver, registrar
csi-powerscale logger initiated. This should be called only once.
time="2023-09-28T06:43:44Z" level=debug   msg="check if sock file '/var/lib/kubelet/plugins/csi-isilon/csi_sock' has already been created" file="/go/src/common/utils/utils.go:100"
time="2023-09-28T06:43:44Z" level=debug   msg="sock file '/var/lib/kubelet/plugins/csi-isilon/csi_sock' does not exist yet, move along" file="/go/src/common/utils/utils.go:121"
time="2023-09-28T06:43:44Z" level=warning msg="env var not found: CSI_RETRIEVER_ENDPOINT"
time="2023-09-28T06:43:44Z" level=debug msg="enabled context injector"
time="2023-09-28T06:43:44Z" level=debug msg="init req & rep validation" withSpec=false
time="2023-09-28T06:43:44Z" level=debug msg="init implicit req validation" withSpecReq=false
time="2023-09-28T06:43:44Z" level=debug msg="init req validation" withSpecReq=true
time="2023-09-28T06:43:44Z" level=debug msg="enabled request ID injector"
time="2023-09-28T06:43:44Z" level=debug msg="enabled request logging"
time="2023-09-28T06:43:44Z" level=debug msg="enabled response logging"
time="2023-09-28T06:43:44Z" level=debug msg="enabled spec validator opt: request validation"
time="2023-09-28T06:43:44Z" level=debug msg="enabled serial volume access"
time="2023-09-28T06:43:44Z" level=info msg="Configured 'csi-isilon.dellemc.com'" accesspoint=System autoprobe=true mode=node path=/ifs/data/csi quotaenabled=true skipCertificateValidation=true
time="2023-09-28T06:43:44Z" level=debug   msg="X_CSI_ISI_NO_PROBE_ON_START is false, set noProbeOnStart to false " file="/go/src/service/service.go:362"
time="2023-09-28T06:43:44Z" level=info   msg="log level set to 'debug'" file="/go/src/service/service.go:818"
time="2023-09-28T06:43:44Z" level=info   msg="************* Synchronizing Isilon Clusters' config **************" file="/go/src/service/service.go:610"
time="2023-09-28T06:43:44Z" level=debug   msg="Current isilon configs:" file="/go/src/service/service.go:620"
time="2023-09-28T06:43:44Z" level=info   msg="reading secret file to validate cluster config details" file="/go/src/service/service.go:665"
time="2023-09-28T06:43:44Z" level=debug   msg="parsing config details for cluster charon" file="/go/src/service/service.go:683"
time="2023-09-28T06:43:44Z" level=warning   msg="using default as EndpointPort not provided for cluster charon in secret at index [0]" file="/go/src/service/service.go:704"
time="2023-09-28T06:43:44Z" level=warning   msg="using default as IsiPath not provided for cluster charon in secret at index [0]" file="/go/src/service/service.go:713"
time="2023-09-28T06:43:44Z" level=warning   msg="using default as IsiVolumePathPermissions not provided for cluster charon in secret at index [0]" file="/go/src/service/service.go:718"
[DEBUG] opts.Insecure : 'true'
[DEBUG] 
    -------------------------- GOISILON HTTP REQUEST -------------------------
    GET /platform/latest/ HTTP/1.1
    Host: isilon.lan:8080
    Authorization: root:******
    

time="2023-09-28T06:43:44Z" level=error   msg="init client failed for isilon cluster 'charon': 'invalid character '<' looking for beginning of value'" file="/go/src/service/service.go:454"
[DEBUG] 
    -------------------------- GOISILON HTTP RESPONSE -------------------------
    HTTP/1.1 401 Unauthorized
    Content-Length: 486
    Accept-Ranges: bytes
    Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data:; script-src 'self' 'unsafe-eval'; style-src 'unsafe-inline' 'self';
    Content-Type: text/html
    Date: Thu, 28 Sep 2023 06:44:18 GMT
    Etag: "1e6-5efe861188000"
    Last-Modified: Fri, 16 Dec 2022 02:03:44 GMT
    Server: Apache
    Strict-Transport-Security: max-age=31536000;
    X-Content-Type-Options: nosniff
    X-Frame-Options: sameorigin
    X-Xss-Protection: 1; mode=block
    
time="2023-09-28T06:43:44Z" level=error   msg="failed to get isi client for  cluster charon, error: invalid character '<' looking for beginning of value" file="/go/src/service/service.go:731"
time="2023-09-28T06:43:44Z" level=info msg="new config details set for cluster charon" ClusterName=charon Endpoint=isilon.lan EndpointPort=8080 IgnoreUnresolvableHosts=false IsDefault=true IsiPath=/ifs/data/csi IsiVolumePathPermissions=0777 Password="*******" SkipCertificateValidation=true Username=root
time="2023-09-28T06:43:44Z" level=debug   msg="New isilon configs:" file="/go/src/service/service.go:636"
time="2023-09-28T06:43:44Z" level=debug   msg="ClusterName: charon, Endpoint: isilon.lan, EndpointPort: 8080, EndpointURL: https://isilon.lan:8080, User: root, SkipCertificateValidation: true, IsiPath: /ifs/data/csi, IsiVolumePathPermissions: 0777, IsDefault: true, IgnoreUnresolvableHosts: false, AccessZone: , isiSvc: <nil>" file="/go/src/service/service.go:780"
time="2023-09-28T06:43:44Z" level=debug   msg="calling probe for cluster 'charon'" file="/go/src/service/service.go:315"
[DEBUG] opts.Insecure : 'true'
[DEBUG] 
    -------------------------- GOISILON HTTP REQUEST -------------------------
    GET /platform/latest/ HTTP/1.1
    Host: isilon.lan:8080
    Authorization: root:******
    

time="2023-09-28T06:43:44Z" level=info   msg="Updating cluster config details" file="/go/src/service/service.go:553"
time="2023-09-28T06:43:44Z" level=debug   msg="Config folder: /isilon-configs" file="/go/src/service/service.go:558"
time="2023-09-28T06:43:44Z" level=info msg="podmon is not enabled"
[DEBUG] 
    -------------------------- GOISILON HTTP RESPONSE -------------------------
    HTTP/1.1 401 Unauthorized
    Content-Length: 486
    Accept-Ranges: bytes
    Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data:; script-src 'self' 'unsafe-eval'; style-src 'unsafe-inline' 'self';
    Content-Type: text/html
    Date: Thu, 28 Sep 2023 06:44:18 GMT
    Etag: "1e6-5efe861188000"
    Last-Modified: Fri, 16 Dec 2022 02:03:44 GMT
    Server: Apache
    Strict-Transport-Security: max-age=31536000;
    X-Content-Type-Options: nosniff
    X-Frame-Options: sameorigin
time="2023-09-28T06:43:44Z" level=error   msg="init client failed for isilon cluster 'charon': 'invalid character '<' looking for beginning of value'" file="/go/src/service/service.go:454"
time="2023-09-28T06:43:44Z" level=debug   msg="Probe failed for isilon cluster 'charon' error:'clusterConfig.isiSvc (type isiService) is nil, probe failed'" file="/go/src/service/service.go:301"
    X-Xss-Protection: 1; mode=block
    
time="2023-09-28T06:43:44Z" level=info msg="removed sock file" path=/var/lib/kubelet/plugins/csi-isilon/csi_sock
time="2023-09-28T06:43:44Z" level=fatal msg="grpc failed" error="probe of all isilon clusters failed"

Screenshots

No response

Additional Environment Information

OneFS version: 9.5.0.5
CSM operator version: 1.3.0
CSI version: 2.8.0

Kubernetes: v1.26.9+rke2r1
Node: Rocky 8.8

Steps to Reproduce

1. Install OneFS simulator 9.5.0.0
2. Patch 9.5.0.0 to 9.5.0.5 & reboot nodes simultaneously
3. Install RKE2-server via quickstart on 1 x Rocky 8.8 node
4. Install Dell CSM operator 1.3.0
5. Followed documentation (edit & apply storage_csm_powerscale_v280.yaml)

Expected Behavior

Installation of CSI via CSM must be successful for PowerScale session-type authentication. The helm chart method is successful.

CSM Driver(s)

CSI Driver for PowerScale 2.8

Installation Type

Operator 1.3.0

Container Storage Modules Enabled

None

Container Orchestrator

Kubernetes 1.26.9

Operating System

Rocky 8.8

[hoppea2]

/sync

[jooseppi-luna]

Hi @lohbe! Thanks for filing this issue -- we have a variable in the sample file, ISICLIENT_AUTH_TYPE, that sets the authentication type. The default value is 1, which selects basic authentication -- if you want to use session-based authentication, you can set that to 0 instead.

[jooseppi-luna]

My apologies, on further investigation, this parameter is only available in the metrics-powerscale package for observability. @bharathsreekanth since this is an option available in helm that doesn't appear to be in csm-operator, do we want to prioritize adding it?

For further clarification, @lohbe, I don't believe we support adding env vars to the sample file -- if you add additional env vars to modify the driver installation, there is no guarantee on the resulting behavior.

[ybrock]

Hello,
I confirm the issue. We're having exactly this using OneFS 9.4.0.13 and CSM version 1.2.0 on Openshift 4.11

Exactly same behaviour.
We tried to add X_CSI_ISI_AUTH_TYPE to spec.driver.common.envs[], spec.driver.controller.envs[] and spec.driver.node.envs[]
without luck.
It's working on the controller deployment, but not on the node daemonset.

We had to patch the daemonset to add an environment var to the nodes to make it work.

[csmbot]

link: 18646

[jooseppi-luna]

This issue was fixed in this PR and will be part of CSM 1.9. Thanks for reporting and if there is anything else needed to bridge the gap until CSM 1.9, please let us know!