Lagoon not compatible with AWS EFS CSI Driver

[dwoods]

Describe the bug

Lagoon deployments fail when using the AWS EFS CSI Driver for shared storage.

To Reproduce

Steps to reproduce the behavior:

1. Install the EFS-CSI driver into the Lagoon Remote cluster

   helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver/
   helm repo update
   helm upgrade --install aws-efs-csi-driver --namespace kube-system aws-efs-csi-driver/aws-efs-csi-driver

2. Create a EFS Filesystem in AWS, and setup the required permissions (I was using this in EKS)

3. Create the storage class like the following

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     annotations:
       storageclass.kubernetes.io/is-default-class: "false"
     name: bulk
   mountOptions:
   - tls
   parameters:
     basePath: /
     directoryPerms: "775"
     fileSystemId: fs-091b9dcd849000000
     provisioningMode: efs-ap
   provisioner: efs.csi.aws.com
   reclaimPolicy: Delete
   volumeBindingMode: Immediate

4. Deploy a Drupal project environment, using Rootless configuration.

5. The deployment will fail on the initContainer with an error indicating that a chown command failed.

6. Deploy a Drupal project environment using standard (non-rootless) config

7. It will deploy (since it's not chowning anything), but Drupal will complain that it cannot create any directories in /app/web/sites/default/files/

Expected behavior

The Drupal site deploys normally and shared directory is writable.

Additional context

The main cause of the error appears to be in the nginx-php-persistent helm chart in the kubectl-build-deploy-dind image, at this line.

The AWS EFS CSI driver creates an Access Point to connect to EFS for each dynamically provisioned volume, with a dynamic UID and GID for that part of the filesystem. Filesystem requests are automatically mapped to the AP uid+gid on the server-side, so there is no need to manage ownership on the client (chmod command do work though).

If I remove the chmod command, rebuild the image, and specify my custom kubectl-build-deploy-dind image in the overrideBuildDeployImage option of the lagoon-remote helm chart, everything works as expected when running rootless.

I'm happy to create a pull request for this, but not sure how/where to create a config option, or if there's a better way to make this work without creating a special case.

More info:

• Related issue
• EFS Access Point info

[tobybellwood]

THanks @dwoods - I'll dig into this with some of our infra peeps - we're still using the deprecated EFS provisioner (:ugh:) - but let's see if there's a way we can tackle both methods in one. The update to the rootless migration logic happened in #3051 (and @smlx created a test repo at https://github.com/amazeeio/rootless-migration-tests, but that doesn't cover the EFS CSI use case...yet)

[smlx]

Maybe the solution here could be to check if the directory is empty and if it is to skip the chmod/chown?