NFS Mount paths don't mount unless the directories exist

[meathead23]

Issue:

When running MQ with the following properties:

  mq:
    version: 9.1.3.0
    # Set to True if running MQ in HA mode
    useConnectionNameList: true
    tlsSecretName: 'spm-dev01-mq-secret'
    queueManager:
      name: 'QM1'
      secret:
        # name is the secret that contains the 'admin' user password and the 'app' user password to use for messaging
        name: ''
        # adminPasswordKey is the secret key that contains the 'admin' user password
        adminPasswordKey: 'adminPasswordKey'
        # appPasswordKey is the secret key that contains the 'admin' user password
        appPasswordKey: 'appPasswordKey'
    metrics:
      enabled: false
    resources: {}
    multiInstance:
      cephEnabled: false
      storageClassName: 'nfs'
      nfsEnabled: true
      nfsIP: 'fs-xxxxxxxx.efs.eu-west-2.amazonaws.com'
      nfsFolder: 'spm-dev01'
      nfsMountOptions:
        - "nfsvers=4.1"
        - "rsize=1048576"
        - "wsize=1048576"
        - "hard"
        - "timeo=600"
        - "retrans=2"
        - "noresvport"

When the curam-mq and rest-mq pods start they connect mount the AWS EFS file system, and Kubernetes(EKS) returns the following error:

  Warning  FailedMount  2m31s  kubelet, ip-100-64-18-180.eu-west-2.compute.internal  MountVolume.SetUp failed for volume "spm-dev01-curam-pv-qm" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/11e52fd5-fb8c-40b6-9cf7-b252f1f4e1ac/volumes/kubernetes.io~nfs/spm-dev01-curam-pv-qm --scope -- mount -t nfs -o hard,nfsvers=4.1,noresvport,retrans=2,rsize=1048576,timeo=600,wsize=1048576 fs-1c2e6eed.efs.eu-west-2.amazonaws.com:/spm-dev01/curam /var/lib/kubelet/pods/11e52fd5-fb8c-40b6-9cf7-b252f1f4e1ac/volumes/kubernetes.io~nfs/spm-dev01-curam-pv-qm
Output: Running scope as unit run-12552.scope.
mount.nfs: mounting fs-xxxxxx.efs.eu-west-2.amazonaws.com:/spm-dev01/curam failed, reason given by server: No such file or directory

Solution:

The only solution I've found for this is to the manually mount the EFS filesystem to one of the worker nodes and make the directories using:

sudo mount -t nfs -o hard,nfsvers=4.1,noresvport,retrans=2,rsize=1048576,timeo=600,wsize=1048576 fs-xxxxx.efs.eu-west-2.amazonaws.com:/ /mnt/
sudo mkdir -p /mnt/spm-dev01/curam/logs
sudo mkdir -p /mnt/spm-dev01/curam/data
sudo mkdir -p /mnt/spm-dev01/rest/data
sudo mkdir -p /mnt/spm-dev01/rest/logs

I'm not sure if there's an easier why do this using IKS but this seems to be the only way for AWS EFS, other suggestions include starting a init-container(see here) to start, mount the filesystem and then create the paths.

Couldn't the file system just about mounted as / and any paths created by the pods themselves at runtime?

[cnrmrph]

Documentation has been added in the December release that clarifies that the directories must be present prior to deploying MQ with NFS.

[niambhscullion12]

Folks, we'll re-open this based on the call today is there anything friendlier we can do here.
N..

[andreyzher]

Hi @meathead23

We have a candidate fix for this, but could use your assistance in verifying before it is merged, as we don't have access to any AWS environments.

The code is on the nfs-paths branch.

I have verified on a standalone NFS server (using the pre-defined PersistentVolumes) and using IBM Cloud File Storage using the dynamic provisioning.

To facilitate this change, there are 2 new properties being added to the MQ configuration:

• global.mq.security.initVolumeAsRoot (boolean) - required to be able to initialise the directory structure, if it doesn't already exist.
• global.mq.multiInstance.useDynamicProvisioning (boolean) - required to use dynamically provisioner volumes using the storage class defined by global.mq.multiInstance.storageClassName

To use the dynamic provisioning capabilities in a multi-zone cluster, you may need to create a custom storage class with the volumeBindingMode attribute set to WaitForFirstConsumer to make sure that the storage is provisioned in the same zone as the node on which the corresponding pod has been scheduled.

[meathead23]

Hi @andreyzher that's excellent, I'll see if i can gt someone to have a look at this next week, thanks a lot!

[tidridgej]

Hi @andreyzher I have checked these changes on our OpenShift cluster hosted in AWS.
Dynamic provisioning for nfs worked without any issues, so this should be good to merge

[andreyzher]

Hi @meathead23, @tidridgej

The above changes have been included in the 21.5.0 release.

Thank you for your support. If you have any further issues on this topic, feel free to reopen this issue or create a new one.

Regards

andreyzher