Rootless NFS Volume Permissions: What am I doing wrong with my Nextcloud/MaraiDB/Redis pod?

[Shadow8472]

This is a cross post from the Podman Discord server. Last night, I had this question:

/mnt/PhotoTrunk/nextcloudPhotoVault is a directory hosted on an NFS-share volume
The user has permission to view it through Bash.
Debian 11
Podman 3.0.1

$ podman volume create --opt=device=/mnt/PhotoTrunk/nextcloudPhotoVault nextcloud-capacity
$ podman run --name BusyBox --volume nextcloud-capacity:/root/disk -it --rm busybox
Error: error mounting volume nextcloud-capacity for container e429831e9bf91abc1bda8f268c6d45b8e9c2a5d8d784efca37c83543e40f9ff7: cannot mount volumes without root privileges: operation requires root privileges

In contrast:

$ podman run --name BusyBox --volume /mnt/PhotoTrunk/nextcloudPhotoVault:/root/disk -it --rm busybox
# /

I'm trying to understand what the difference is here.
I can't create and mount a volume, but mounting directories directly is cool.

selckin recommended over a relay:

its a bind mount, you need something like (from google) docker volume create --driver local --opt 'type=none' --opt 'device=/tmp/SourceMountDocker' --opt 'o=bind' TestVolumeDocker

But when I tried that, I got the same error:

$ podman volume create --driver local --opt 'type=none' --opt 'device=/mnt/PhotoTrunk/nextcloudPhotoVault' --opt 'o=bind' nextcloud-capacity
$ podman run --name BusyBox --volume nextcloud-capacity:/root/disk -it --rm busybox
Error: error mounting volume nextcloud-capacity for container 71ac98b6faeed20a584b45ddc8a34265fbab385d10eaa4805838df8a64e195f0: cannot mount volumes without root privileges: operation requires root privileges

Today, I configured my Nextcloud/MariaDB/Redis pod script to mount directories over NFS, but both Nextcloud and MariaDB's logs filled with permissions errors regarding chown:

(MariaDB)

2023-10-27 22:20:17+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
chown: changing ownership of '/var/lib/mysql/': Operation not permitted

(Nextcloud)

rsync: [receiver] chown "/var/www/html/" failed: Operation not permitted (1)

Am I doing something obviously wrong, or does Podman/NFS just not work that way and I'll have to investigate making a virtual drive or something? Maybe something can be arranged with SSH?

Please be patient if I don't respond right away as I take Sabbath off (sundown Friday sundown Saturday, Pacific time) from my computer projects.

EDIT: I got into Synology to enable root squashing (to guest, or so it says), but that didn't seem to help.

EDIT 2: It's been almost a year with no breakthrough. I'm working on it again, but I'm starting with a fresh slate in a new discussion.

[rhatdan]

In the volume case rootless podman is attempting to do a mount --t=nfs which the kernel does not allow for non root users, (Even if they are in a User Namespace).

In the -v /mnt ... case podman is doing a mount -t bind which is allowed to rootless users. The mount of the NFS was done by a root process (A process with CAP_SYS_ADMIN).

There is nothing Podman can do about this until the kernel figures a way to safely allow a rootless user to mount an NFS share.

[rhatdan]

I think there is a fuse file system support for NFS mounts, but I have never used it. @giuseppe thoughts?

[Shadow8472]

Sounds promising. Unfortunately:

$ uname -srm
Linux 5.10.0-26-amd64 x86_64

https://www.redhat.com/sysadmin/podman-rootless-overlay

Podman can use native overlay file system with the Linux kernel versions 5.13. Up until now, we have been using fuse-overlayfs. The kernel gained rootless support in the 5.11 kernel, but a bug prevented SELinux use with the file system; this bug was fixed in 5.13.

If I am to understand this correctly, my kernel version is too old. That and/or I just stumbled across a much better method for when I either upgrade or work on my Rocky 8 machine (for now, I'm on Debian 11).

[eriksjolund]

This does not answer your question about named volumes, but if you would like to use bind-mounts
you might find this information useful. As a test I ran a number of popular container images and then checked
which ownership newly created files and directories had. From the information a podman run option in the form of
--userns=keep-id:uid=$uid,gid=$gid could deduced.

The shell scripts and results:
https://github.com/eriksjolund/podman-detect-option#result-from-some-popular-container-images

From the results:

image	podman command-line option
docker.io/library/mariadb	--userns=keep-id:uid=999,gid=999
docker.io/library/nextcloud	--userns=keep-id:uid=33,gid=33
docker.io/library/redis	--userns=keep-id:uid=999,gid=999

One test you could do is adding such --userns options and see if it starts to works.

Edit 1

If you would like to run the containers with different UID/GID mappings, the containers can't be running in the same pod.
Probably it's easier to get it to work by not using a pod.

[Shadow8472]

--publish omitted on purpose because I'm working within the context of Caddy.

echo stop and remove containers
podman rm MariaDBContainer Redis Nextcloud

subuidSize=$(( $(podman info \
        --format "{{ range.Host.IDMappings.UIDMap }}+ \
                {{.Size }}{{end }}" ) - 1 ))

subgidSize=$(( $(podman info \
        --format "{{ range.Host.IDMappings.GIDMap }}+ \
                {{.Size }}{{end }}" ) - 1 ))

echo podman run MariaDB
uid=999
gid=999
podman run \
        --name MariaDBContainer \
        --network podman \
        --uidmap 0:1:$uid \
        --uidmap $uid:0:1 \
        --uidmap $(($uid+1)):$(($uid+1)):$(($subuidSize-$uid)) \
        --gidmap 0:1:$gid \
        --gidmap $gid:0:1 \
        --gidmap $(($gid+1)):$(($gid+1)):$(($subgidSize-$gid)) \
        -v /mnt/usbShare/nextcloud/MariaDBVolume:/var/lib/mysql \
        -e MARIADB_ROOT_PASSWORD="EBh&Rh3eWuEa0uX..." \
        -e MARIADB_DATABASE=NextcloudDB \
        -e MARIADB_USER=nextcloudDbUser \
        -e MARIADB_PASSWORD="*W9$MZOcj@nLz%TH..." \
        --restart on-failure \
        docker.io/library/mariadb:latest

echo podman run Redis
uid=999
gid=999
podman run \
        --name RedisContainer \
        --network podman \
        --uidmap 0:1:$uid \
        --uidmap $uid:0:1 \
        --uidmap $(($uid+1)):$(($uid+1)):$(($subuidSize-$uid)) \
        --gidmap 0:1:$gid \
        --gidmap $gid:0:1 \
        --gidmap $(($gid+1)):$(($gid+1)):$(($subgidSize-$gid)) \
        --restart on-failure \
        docker.io/library/redis:latest \
        --requirepass "RedisPass"

echo podman run Nextcloud
uid=33
gid=33
podman run \
        --name NextcloudContainer \
        --network podman \
        --uidmap 0:1:$uid \
        --uidmap $uid:0:1 \
        --uidmap $(($uid+1)):$(($uid+1)):$(($subuidSize-$uid)) \
        --gidmap 0:1:$gid \
        --gidmap $gid:0:1 \
        --gidmap $(($gid+1)):$(($gid+1)):$(($subgidSize-$gid)) \
        -v /mnt/usbShare/nextcloud/fastvolume:/var/www/html \
        -e REDIS_HOST="RedisContainer" \
        -e REDIS_PASSWORD="RedisPass" \
        -e MYSQL_DATABASE=NextcloudDB \
        -e MYSQL_USER=nextcloudDbUser \
        -e MYSQL_PASSWORD="*W9$MZOcj@nLz%TH..." \
        -e MYSQL_HOST=MariaDBContainer \
        --restart on-failure \
        docker.io/library/nextcloud:latest

I'm guessing I'm bumping into slirp4netns issues?

stop and remove containers
Error: no container with name or ID Redis found: no such container
podman run MariaDB
Error: CNI networks not supported with user namespaces
podman run Redis
Error: CNI networks not supported with user namespaces
podman run Nextcloud
Error: CNI networks not supported with user namespaces

I'm starting to suspect this is a dry end on this angle. That network is how Caddy is getting to Nextcloud. It's also why I had --ip in there earlier.

[eriksjolund]

And is there anything special about uid/gid's 999 and 33, or they just likely to be open?

The container images are different. As a test I ran these container images and checked which ownership
(UID/GID) that newly created directories and files had. The numbers 999 and 33 were detected by this test.

See the result list for some popular container images:
https://github.com/eriksjolund/podman-detect-option/tree/main#result-from-some-popular-container-images

I wrote an stackoverflow answer that describes how --uidmap and --gidmap works:

https://stackoverflow.com/questions/70770437/mapping-of-user-ids/70774211#70774211

This github discussion might also be interesting:

• After mount a host directory can't write there. #podman #wordpress #mariadb #podman-unshare #16053 (comment)

The situation in that example is pretty similar to your use case (i.e. multiple containers that need to have different UID/GID mappings)

[eriksjolund]

Regarding

Error: no container with name or ID Redis found: no such container

replace

podman rm MariaDBContainer Redis Nextcloud

with

podman rm MariaDBContainer  RedisContainer NextcloudContainer

---

You could also start using --rm, then the containers are automatically removed when the containers stop.

---

Regarding

Error: CNI networks not supported with user namespaces

try a newer Podman version or maybe try using Unix sockets.

Or maybe someone else knows?

[Shadow8472]

Debian 11. I'm thoroughly stuck with Podman 3.0.1 unless I update the whole system, and I'm wanting to wait for Debian 13. In the meantime, I did check the backports repository and found it empty. I'm not up to compiling Podman.

Thank you for your time in exploring this route. If nothing else, I learned a thing or two about how mapping UID's and GID's works. I'll call this a win, even if it's not the big one I've been chasing for the past few years. What is another week or two on that scale, anyway?

[eriksjolund]

Debian 11. I'm thoroughly stuck with Podman 3.0.1 unless I update the whole system, and I'm wanting to wait for Debian 13.

@Shadow8472 if you upgrade, maybe you could try this method

1. mount the NFS directory (i.e. sudo mount --t=nfs ...)
2. verify that your regular user on the host is able to write files in that directory
3. follow the example https://github.com/eriksjolund/nextcloud-podman but replace the directory paths ~/mariadb_data and ~/shared_html with paths somewhere under the NFS directory

I'm curious if it would work. My theory is that the risk of seeing permission denied problems can be reduced by using different UID/GID-mappings for the containers. The container user writing the files can be mapped to the regular user on the host.

[Shadow8472]

I've not made much progress on this, but I've set aside the remainder of this week and next for working on it. I did some background research of fuse-overlayfs, and I'm stumped on what to do or if it will help at all.

Do I like overlay an empty local directory with one hosted on the NFS or something? And if I get that working, can the overlay be loaded from a counterpart Podman -v on another server if needed?

[Shadow8472]

Here's what I have:

fuse-overlayfs \
        -o upperdir=/mnt/PhotoTrunk/nextcloudPhotoVault \
        /home/nextclouduser-redlaptop/nextcloudBase/photoVault

And here's what I get (running as nextclouduser-redlaptop):

$ ./startFuse.sh 
fuse-overlayfs: cannot read lower dirs: No such file or directory

[Shadow8472]

I looked up some examples, and tried it with an account with sudo privileges, and got them switched. Arch Wiki helped me unmount. https://man.archlinux.org/man/fuse-overlayfs.1.en

I've tried a few things, and here is my best guess so far:

$ sudo fuse-overlayfs \
    -o allow_other \
    -o lowerdir=/home/nextclouduser-redlaptop/nextcloudBase/photoVault/ \
    /mnt/PhotoTrunk/nextcloudPhotoVault/

nextclouduser-redlaptop@Shadow8472-Laptop:~/nextcloudBase/photoVault$ touch helloWorld
touch: cannot touch 'helloWorld': Read-only file system

I also got this result with lowerdir=/mnt/... and the mountpoint at /home/...

[Shadow8472]

I'm making progress.

#startfuse.sh 
fuse-overlayfs \
        -d \
        -o allow_other \
        -o lowerdir=/home/nextclouduser-redlaptop/nextcloudBase/fuseLower/ \
        -o upperdir=/mnt/PhotoTrunk/nextcloudPhotoVault/ \
        -o workdir=/home/nextclouduser-redlaptop/nextcloudBase/fuseWorkdir \
        /home/nextclouduser-redlaptop/nextcloudBase/photoMountpoint

$ sudo ./startfuse.sh 
uid=unchanged
gid=unchanged
upperdir=/mnt/PhotoTrunk/nextcloudPhotoVault
workdir=/home/nextclouduser-redlaptop/nextcloudBase/fuseWorkdir
lowerdir=/home/nextclouduser-redlaptop/nextcloudBase/fuseLower/
mountpoint=/home/nextclouduser-redlaptop/nextcloudBase/photoMountpoint
plugins=<none>
fsync=enabled
FUSE library version: 3.10.3
unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.32
flags=0x03fffffb
max_readahead=0x00020000
   INIT: 7.31
   flags=0x0041f069
   max_readahead=0x00020000
   max_write=0x00100000
   max_background=0
   congestion_threshold=0
   time_gran=1
   unique: 2, success, outsize: 80
unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 744988
ovl_getattr(ino=1)
   unique: 4, success, outsize: 120
unique: 6, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 744989
ovl_lookup(parent=1, name=helloWorld)
   unique: 6, success, outsize: 144
unique: 8, opcode: CREATE (35), nodeid: 1, insize: 67, pid: 744989
ovl_create(parent=1, name=helloWorld)
   unique: 8, error: -18 (Invalid cross-device link), outsize: 16

nextclouduser-redlaptop@Shadow8472-Laptop:~/nextcloudBase/photoMountpoint$ touch helloWorld
touch: cannot touch 'helloWorld': Invalid cross-device link

[Shadow8472]

So, I found this.
containers/fuse-overlayfs#33 (comment)

I've moved my workdir to the NFS and got a new error.

$ sudo ./startfuse.sh 
uid=unchanged
gid=unchanged
upperdir=/mnt/PhotoTrunk/nextcloudPhotoVault
workdir=/mnt/PhotoTrunk/.fuseWorkdir
lowerdir=/home/nextclouduser-redlaptop/nextcloudBase/fuseLower/
mountpoint=/home/nextclouduser-redlaptop/nextcloudBase/photoMountpoint
plugins=<none>
fsync=enabled
FUSE library version: 3.10.3
unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.32
flags=0x03fffffb
max_readahead=0x00020000
   INIT: 7.31
   flags=0x0041f069
   max_readahead=0x00020000
   max_write=0x00100000
   max_background=0
   congestion_threshold=0
   time_gran=1
   unique: 2, success, outsize: 80
unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 738278
ovl_getattr(ino=1)
   unique: 4, success, outsize: 120
unique: 6, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 748152
ovl_lookup(parent=1, name=helloWorld)
   unique: 6, success, outsize: 144
unique: 8, opcode: CREATE (35), nodeid: 1, insize: 67, pid: 748152
ovl_create(parent=1, name=helloWorld)
   unique: 8, error: -1 (Operation not permitted), outsize: 16

nextclouduser-redlaptop@Shadow8472-Laptop:~/nextcloudBase/photoMountpoint$ touch helloWorld
touch: setting times of 'helloWorld': No such file or directory

EDIT: I've tried a couple suggestions online, but my issue is getting narrow enough that half my search results are in Chinese.

• Rebooting NFS client machine did not help.
• Though we recently reverted from Daylight Savings Time, the problem persisted once the clocks matched (eyeballed to the minute).
• Rebooting the NFS server did not help.

[eriksjolund]

I did a test by using quadlets and nginx as a proxy (with socket activation).

Something is working. I've only checked that the output from curl contains the text Nextcloud:

$ curl -s localhost:8080  | grep title
                <title>
                        Nextcloud               </title>

For data storage two directories were used

• ~/shared_html/
• ~/mariadb_data/

After running the demo I can see that the directories only contain directories and files belonging to the regular user on the host.

For more info see https://github.com/eriksjolund/nextcloud-podman

[Shadow8472]

I've built and installed quadlet. This would be the correct one, right? https://github.com/containers/quadlet.

[eriksjolund]

No, that's old code. The quadlet code has been integrated into https://github.com/containers/podman

[Shadow8472]

As noted above, Podman doesn't show up in any Debian backports, so I'm locked into working with Podman 3.0.1 -- before quadlet was integrated in Podman 4.4. I'd need to upgrade out of Debain 11, and I've been meaning to wait for Debian 13 before I do that.

Edit: I just checked out of curiosity, and Debian 12 runs Podman 4.3, so I'd still be stuck before the merge.

[Shadow8472]

Just an update, but after around a day of consideration, I decided to go ahead and rebuild on a Fedora base. I'm presently trying to find the right version for my needs. Fedora CoreOS is nice and small, but will take a while to configure, and the server image I tried is UEFI only (I'm on BIOS).

[Shadow8472]

Another update, in case anyone ever sees this:

Rocky Linux 9, Podman 4.6.1
I'm working with the --userns=keep-id flag as part of pod definition, but UID/GID don't match my user on hidden files, and I have to clean things out with root between attempts.