[K8S][EKS] 'MountVolume.SetUp failed for volume "volumes-dir" : hostPath type check failed: /mnt/persistent-volumes is not a directory' error installing local csi driver

[vponomaryov]

Issue description

• This issue is a regression.
• It is unknown if this issue is a regression.

Local CSI driver failed with this:

'MountVolume.SetUp failed for volume "volumes-dir" : hostPath type check failed:
  /mnt/persistent-volumes is not a directory'

The /mnt/persistent-volumes must have been created by the NodeConfig.
It has following config:

    localDiskSetup:
      filesystems:
      - device: /dev/md/nvmes
        type: xfs
      mounts:
      - device: /dev/md/nvmes
        mountPoint: /mnt/persistent-volumes
        unsupportedOptions:
        - prjquota
      raids:
      - RAID0:
          devices:
            modelRegex: Amazon EC2 NVMe Instance Storage
            nameRegex: /dev/nvme\d+n\d+$
        name: nvmes
        type: RAID0

List of disks on one of the K8S Scylla node:

+ lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1       259:0    0    30G  0 disk 
|-nvme0n1p1   259:2    0    30G  0 part /mnt/hostfs
`-nvme0n1p128 259:3    0     1M  0 part 
nvme1n1       259:1    0 872.6G  0 disk 

So, probably we should update the regex for the Scylla disks the way it doesn't touch OS-related disk if matches to it.

Impact

Scylla-operator cannot be tested on EKS backend using SCT.

How frequently does it reproduce?

100%. All our EKS deployments fail this way.

Installation details

Cluster size: 4 nodes (i4i.4xlarge)

Scylla Nodes used in this run:
No resources left at the end of the run

OS / Image: `` (k8s-eks: eu-north-1, `eu-west-1`)

Test: longevity-scylla-operator-multidc-12h-eks
Test id: 51927961-58a8-4e79-bace-60475111bb39
Test name: scylla-operator/operator-master/eks/longevity-scylla-operator-multidc-12h-eks
Test config file(s):

• longevity-scylla-operator-multidc-12h.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor 51927961-58a8-4e79-bace-60475111bb39
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs 51927961-58a8-4e79-bace-60475111bb39

Logs:

• kubernetes-51927961.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/51927961-58a8-4e79-bace-60475111bb39/20240114_004027/kubernetes-51927961.tar.gz
• kubernetes-must-gather-51927961.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/51927961-58a8-4e79-bace-60475111bb39/20240114_004027/kubernetes-must-gather-51927961.tar.gz
• sct-runner-events-51927961.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/51927961-58a8-4e79-bace-60475111bb39/20240114_004027/sct-runner-events-51927961.tar.gz
• sct-51927961.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/51927961-58a8-4e79-bace-60475111bb39/20240114_004027/sct-51927961.log.tar.gz

Jenkins job URL
Argus

[vponomaryov]

The real root cause is here: scylladb/scylla-operator#1690