rootless initscript changes file permission of persistent storage sometimes to wrong users

[Schnitzel]

Describe the bug

So far we found this only on AWS EFS RWX PVCs:
If an environment that was running as root-full and was rsynced without the --omit-dir-times --no-perms --no-group --no-owner --chmod=ugo=rwX flags as suggested in https://docs.lagoon.sh/drupal/drush-9/#drush-rsync-from-local-to-remote-environments it was possible that the file permissions of the source were synced to the destination files.
As at that point the containers are running as root which had full access to everything even if the files are not owned by root or a group root, the issue was never detected.
Now that customers are switching to rootless the permission become an issue, an example here:

files directory owned by 501:20

[xxx]master@cli-drupal:/app/sites/default$ ll
total 92
-rw-rw-r--  1 root root      657 Feb 17 12:24 all.settings.php
-rw-rw-r--  1 root root    26250 Feb 17 12:24 default.settings.php
-rw-rw-r--  1 root root      773 Feb 17 12:24 development.settings.php
-rw-rw-r--  1 root root      633 Feb 17 12:24 drushrc.php
-rw-rw-r--  1 root root    26233 Feb 17 12:24 example.settings.php
drwxr-xr-x 15  501 dialout 14336 Feb 17 12:27 files
-rw-rw-r--  1 root root      863 Feb 17 12:24 production.settings.php
-rw-rw-r--  1 root root        0 Feb 17 12:24 settings.common.php
-rw-rw-r--  1 root root     4832 Feb 17 12:24 settings.php

Files inside files directory are also owed by 501:20

[xxx]master@cli-drupal:/app/sites/default/files$ ll
total 377872
-rw-rw-r-- 1  501 dialout   365333 Jan  6 14:07  077_AH_Dog_03.jpg
-rwxrwxr-x 1  501 dialout  2685121 Jan  6 14:07  0815_1.jpg
-rwxrwxr-x 1  501 dialout   132049 Jan  6 14:07  0815_3.jpg
-rw-rw-r-- 1  501 dialout  6798851 Jan  6 14:07  2019_06_03_Parigi

The container is running neither as the user or group that has these files.

[xxx]master@cli-drupal:/tmp$ id
uid=10000(user) gid=0(root) groups=0(root),10001,40023

this causes the container not being able to write into the files directory:

[xxx]master@cli-drupal:/app/sites/default/files$ touch test
touch: cannot touch 'test': Permission denied

After some research we found the issue being this:

lagoon/images/kubectl-build-deploy-dind/helmcharts/nginx-php-persistent/templates/deployment.yaml

Line 49 in 36d0891

find /storage -mindepth 1 -exec chgrp $(stat -c "%g" /storage) {} +

the stat -c "%g" /storage gets the group of the files directory (in our case 20 (dailout)) and applies the permission to all files.

We where able to fix this for this cluster with running:

/persistentvolumes # chown -R 10000:0 .

directly in the EFS provisioner pod. This changes ALL files and directories of the EFS filesystem to user 10000 and group 0. This can be edited and written by root-full and rootless containers.
It's though not a sustainable solution:

• if a customer rsyncs again it will be broken again
• it takes a long time