[BUG]: Host definitions not being created after adding new appliance to secret

[lukeatdell]

Bug Description

When adding a new appliance configuration to an existing secret, host definitions are not being created on the new appliance.
Based on documentation at https://dell.github.io/csm-docs/docs/deployment/helm/drivers/installation/powerstore/#dynamically-update-the-powerstore-secrets

Logs

Single array in secret

Read CSI_LOG_FORMAT from log configuration file, format: json
Read CSI_LOG_LEVEL from log configuration file, level: debug
{"level":"debug","msg":"set podmon API port to :8083","time":"2024-09-25T11:18:18.039812155Z"}
{"level":"info","msg":"updating array info","time":"2024-09-25T11:18:18.039869427Z"}
2024/09/25 11:18:18 Session management is enabled.
{"level":"info","msg":"https://x.x.x.38/api/rest,PS<id1>,admin,openshift-nas,true,true,NVMETCP,x.x.x.38","time":"2024-09-25T11:18:18.127194234Z"}
START: ISCSIConnector.GetInitiatorName
{"level":"info","msg":"get initiator name","time":"2024-09-25T11:18:18.127648797Z"}
END: ISCSIConnector.GetInitiatorName
START: FCConnector.GetInitiatorPorts
START: FCConnector.getFCHBASInfo
{"level":"info","msg":"initiator name is: [iqn.1994-05.com.redhat:<iqn>]","time":"2024-09-25T11:18:18.128518907Z"}
{"level":"debug","msg":"iscsi initiators found on node","time":"2024-09-25T11:18:18.128547995Z"}
{"level":"info","msg":"get initiator name","time":"2024-09-25T11:18:18.128559745Z"}
{"level":"info","msg":"get FC hbas info","time":"2024-09-25T11:18:18.128568074Z"}
{"level":"info","msg":"check is FC supported","time":"2024-09-25T11:18:18.128575495Z"}
START: FCConnector.isFCSupported
{"level":"info","msg":"FC is not supported for this host","time":"2024-09-25T11:18:18.128610295Z"}
END: FCConnector.isFCSupported
END: FCConnector.getFCHBASInfo
END: FCConnector.GetInitiatorPorts
START: NVMeConnector.GetInitiatorName
{"level":"error","msg":"failed to read initiator ports names: FC is not supported for this host","time":"2024-09-25T11:18:18.128622615Z"}
{"level":"info","msg":"initiator FC ports names are: []","time":"2024-09-25T11:18:18.128628778Z"}
{"level":"info","msg":"FC initiators found: []","time":"2024-09-25T11:18:18.128637586Z"}
{"level":"error","msg":"FC was not found or filtered with FCPortsFilterFile","time":"2024-09-25T11:18:18.128643084Z"}
{"level":"info","msg":"get initiator name","time":"2024-09-25T11:18:18.128650803Z"}
END: NVMeConnector.GetInitiatorName
{"level":"info","msg":"initiator name is: [nqn.2014-08.org.nvmexpress:uuid:<uuid>]","time":"2024-09-25T11:18:18.129039519Z"}
{"level":"debug","msg":"NVMe initiators found on node","time":"2024-09-25T11:18:18.129059522Z"}
{"level":"info","msg":"NVMeTCP Protocol is requested","time":"2024-09-25T11:18:18.129068518Z"}
{"level":"info","msg":"setting up host on x.x.x.38","time":"2024-09-25T11:18:18.129073761Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/host?name=eq.csi-node-<machine-id>-x.x.x.141\u0026select=%2A HTTP/1.1 Host: x.x.x.38 Application-Type: CSI Driver for Dell EMC PowerStore/2.11.0+dirty ","time":"2024-09-25T11:18:18.129178091Z"}
{"level":"debug","msg":"acquire a lock","time":"2024-09-25T11:18:18.129200894Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 2"
{"level":"debug","msg":"release a lock","time":"2024-09-25T11:18:18.134814116Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/host?limit=1000\u0026offset=0\u0026order=name\u0026select=%2A HTTP/1.1 Host: x.x.x.38 Application-Type: CSI Driver for Dell EMC PowerStore/2.11.0+dirty ","time":"2024-09-25T11:18:18.134880597Z"}
{"level":"debug","msg":"acquire a lock","time":"2024-09-25T11:18:18.134955951Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 910"
{"level":"debug","msg":"release a lock","time":"2024-09-25T11:18:18.151832486Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/software_installed?limit=1000\u0026offset=0\u0026order=id\u0026select=id%2Cis_cluster%2Crelease_version%2Cbuild_version%2Cbuild_id HTTP/1.1 Host: x.x.x.38 Application-Type: CSI Driver for Dell EMC PowerStore/2.11.0+dirty ","time":"2024-09-25T11:18:18.151986204Z"}
{"level":"debug","msg":"acquire a lock","time":"2024-09-25T11:18:18.152012881Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 283"
{"level":"debug","msg":"release a lock","time":"2024-09-25T11:18:18.156187961Z"}
{"level":"debug","msg":"REQUEST: POST /api/rest/host HTTP/1.1 Host: x.x.x.38 Application-Type: CSI Driver for Dell EMC PowerStore/2.11.0+dirty ..."
{"level":"debug","msg":"acquire a lock","time":"2024-09-25T11:18:18.156364942Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 201 Created"
{"level":"debug","msg":"release a lock","time":"2024-09-25T11:18:18.600804469Z"}
{"level":"info","msg":"finished setting up host on x.x.x.38","time":"2024-09-25T11:18:18.600818588Z"}

After updating the secret

{"level":"info","msg":"updating array info","time":"2024-09-25T11:23:04.460313404Z"}
2024/09/25 11:23:04 Session management is enabled.
2024/09/25 11:23:04 REQUEST: GET /api/rest/login_session HTTP/1.1 Host: x.x.x.38 
2024/09/25 11:23:04 acquire a lock
2024/09/25 11:23:04 release a lock
{"level":"info","msg":"https://x.x.x.38/api/rest,PS<id1>,admin,openshift-nas,true,false,NVMETCP,x.x.x.38","time":"2024-09-25T11:23:04.537467256Z"}
2024/09/25 11:23:04 Session management is enabled.
2024/09/25 11:23:04 REQUEST: GET /api/rest/login_session HTTP/1.1 Host: x.x.x.185
2024/09/25 11:23:04 acquire a lock
2024/09/25 11:23:04 release a lock
{"level":"info","msg":"https://x.x.x.185/api/rest,PS<id2>,admin,openshift-nas,true,true,ISCSI,x.x.x.185","time":"2024-09-25T11:23:04.621881985Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetCapabilities: REQ 0019: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-09-25T11:23:56.849935199Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetCapabilities: REP 0019: Capabilities=[rpc:\u003ctype:STAGE_UNSTAGE_VOLUME \u003e  rpc:\u003ctype:EXPAND_VOLUME \u003e  rpc:\u003ctype:SINGLE_NODE_MULTI_WRITER \u003e ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-09-25T11:23:56.850092105Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetCapabilities: REQ 0020: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-09-25T11:24:58.467256203Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetCapabilities: REP 0020: Capabilities=[rpc:\u003ctype:STAGE_UNSTAGE_VOLUME \u003e  rpc:\u003ctype:EXPAND_VOLUME \u003e  rpc:\u003ctype:SINGLE_NODE_MULTI_WRITER \u003e ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-09-25T11:24:58.467414075Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetCapabilities: REQ 0021: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-09-25T11:26:56.892872353Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetCapabilities: REP 0021: Capabilities=[rpc:\u003ctype:STAGE_UNSTAGE_VOLUME \u003e  rpc:\u003ctype:EXPAND_VOLUME \u003e  rpc:\u003ctype:SINGLE_NODE_MULTI_WRITER \u003e ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-09-25T11:26:56.89304707Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetCapabilities: REQ 0022: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-09-25T11:28:17.482330429Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetCapabilities: REP 0022: Capabilities=[rpc:\u003ctype:STAGE_UNSTAGE_VOLUME \u003e  rpc:\u003ctype:EXPAND_VOLUME \u003e  rpc:\u003ctype:SINGLE_NODE_MULTI_WRITER \u003e ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-09-25T11:28:17.482546074Z"}

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

1. Environment preparation

• No running pods, pvc, pv on appliances
• Removed host defintions for worker nodes on both appliances

2. Update secret with only one powerstore appliance and NVMeTCP protocol (13:18 CET)
   oc create secret generic powerstore-config -n csi-powerstore --from-file=config=config.single.yaml -o yaml --dry-run=client | oc replace -f -

3. Delete csi-powerstore pods in order to create host definitions on appliance (13:18 CET)
   mypods=oc get pod -n csi-powerstore| grep node | cut -f 1 -d " "
   for pod in $mypods; do oc delete pod $pod -n csi-powerstore; done

Wait 60 seconds and verify host definitions have been created

4. Create workload which uses NVMeTCP storage class (13:19 CET)
   oc create -f workload_nvmetcp.yaml

5. Verify pods, pvc and pv are created
   oc get pod,pvc,pv -n fio

6. Remove pods, pvc and pv
   oc delete -f workload_nvmetcp.yaml
   oc delete pvc fio-data-fio-0 fio-data-fio-1 fio-data-fio-2 -n fio

all the pv should be removed automatically - no separate command required

7. Update secret with second appliance, use iSCSI protocol, set it to default (13:22 CET)
   oc create secret generic powerstore-config -n csi-powerstore --from-file=config=config.two.yaml -o yaml --dry-run=client | oc replace -f -

8. Verify host definition has been created on appliance
   result: host definition is not created

Expected Behavior

Expected hosts to be created in PowerStore array without restarting the driver.

CSM Driver(s)

csi-powerstore v2.11.1

Installation Type

No response

Container Storage Modules Enabled

No response

Container Orchestrator

OpenShift 4.15.25

Operating System

unknown

[lukeatdell]

Driver working as intended.
Documentation is updated to clarify which scope of dynamic secret update for CSI PowerStore.