Running Singularity in a user namespace created by an unprivileged user doesn't work

[stxue1]

Version of Singularity
What version of Singularity are you using? Run:

singularity-ce version 4.1.1-jammy

Describe the bug
When making a namespace as an unprivileged user, either the UID in the namespace is mapped to 0 or something else. When the UID is mapped to 0 singularity thinks it is root, and tries to perform operations that it may not be permitted to do even if it's not technically root. If the UID is mapped to something else, Singularity says that it's not installed correctly as setuid means it expects to be root but isn't getting UID 0.

To Reproduce
Steps to reproduce the behavior:
To reproduce when UID is mapped to 0, the container can't be built:

unshare -r singularity -v -d run docker://ubuntu:latest
...
...
...
FATAL   [U=0,P=165134]     replaceURIWithImage()         Unable to handle docker://ubuntu:latest uri: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: etc/gshadow: apply hdr metadata: restore chown metadata: /tmp/build-temp-1529656932/rootfs/etc/gshadow: lchown /tmp/build-temp-1529656932/rootfs/etc/gshadow: invalid argument

Full logs here

To reproduce when UID is mapped to something else. The container is able to be built/unpacked but it can't execute:

unshare -c singularity -v -d run docker://ubuntu:latest
...
...
...
ERROR   [U=1000,P=162828]  init()                        No setuid installation found, for unprivileged installation use: ./mconfig --without-suid

Full logs here

Expected behavior
Singularity should be able to run the container.

OS / Linux Distribution
Which Linux distribution are you using?

NAME="Pop!_OS"
VERSION="22.04 LTS"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 22.04 LTS"
VERSION_ID="22.04"
HOME_URL="https://pop.system76.com"
SUPPORT_URL="https://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=jammy
UBUNTU_CODENAME=jammy
LOGO=distributor-logo-pop-os

Installation Method
Installed through the .deb file on the Github releases page.

Additional context
I'm trying get Singularity with FUSE mounts to run inside an unprivileged Docker container. In order to use FUSE inside an unprivileged docker container, I need to be inside a user namespace according to this issue, but Singularity doesn't work inside an unprivileged user namespace.

[dtrudg]

Please could you give a bit more context:

• Are the failing commands run inside a Docker container, or are they run on the host at this point? If in a Docker container, do they work instead on the host?
• What is the overall goal here... i.e. why do you need to use Singularity with FUSE mounts insides an unprivileged Docker container? Can you explain why singularity must be run inside an unprivileged Docker container, and what FUSE mounts are needed... do you mean Singularity's FUSE mounts or external FUSE mounts?

[dtrudg]

Did some quick digging, and may be able to solve this without additional context.

For the unshare -r case the following patch should solve the issue, and will be included in the next release:

diff --git a/internal/pkg/ociimage/unpack.go b/internal/pkg/ociimage/unpack.go
index fdbc132d7..55cbd0736 100644
--- a/internal/pkg/ociimage/unpack.go
+++ b/internal/pkg/ociimage/unpack.go
@@ -16,7 +16,9 @@ import (
        umocilayer "github.com/opencontainers/umoci/oci/layer"
        "github.com/opencontainers/umoci/pkg/idtools"
        "github.com/sylabs/singularity/v4/internal/pkg/util/fs"
+       "github.com/sylabs/singularity/v4/pkg/util/namespaces"
        "github.com/sylabs/singularity/v4/pkg/sylog"
+
 )
 
 // isExtractable checks if we have extractable layers in the image. Shouldn't be
@@ -69,7 +71,8 @@ func UnpackRootfs(_ context.Context, srcImage v1.Image, destDir string) (err err
        }
 
        // Allow unpacking as non-root
-       if os.Geteuid() != 0 {
+       insideUserNs, _ := namespaces.IsInsideUserNamespace(os.Getpid())
+       if os.Geteuid() != 0 || insideUserNs {
                mapOptions.Rootless = true
 
                uidMap, err := idtools.ParseMapping(fmt.Sprintf("0:%d:1", os.Geteuid()))

[dtrudg]

The unshare -c case is a separate issue.

With unshare -c you will need to use the -u flag for singularity - this is required to tell a setuid installation of singularity to run in usernamespace mode, instead of setuid mode.

unshare -c singularity run -u alpine_latest.sif

At present this is not working, as a wrapped extraction of an image is failing to honour the -u flag. I'll track this as a separate issue.

You can workaround this issue, at present, via creating a sandbox dir and then running with -u, e.g.:

$ singularity build -s test-sandbox docker://ubuntu:latest

$ unshare -c singularity run -u test-sandbox/
Singularity> echo "HELLO"
HELLO
Singularity> 

[stxue1]

Hi, thanks for the quick response!

* Are the failing commands run inside a Docker container, or are they run on the host at this point? If in a Docker container, do they work instead on the host?

I first encountered this issue when running the commands inside a Docker container, but was able to reproduce the exact same issue on a host machine. Neither the container or the host work, but the host seemed more easily reproducible so I included that as the bug report instead.

* What is the overall goal here... i.e. why do you need to use Singularity with FUSE mounts insides an unprivileged Docker container? Can you explain why singularity must be run inside an unprivileged Docker container, and what FUSE mounts are needed... do you mean Singularity's FUSE mounts or external FUSE mounts?

I'm trying to create a Docker container with an application preinstalled. The app is a workflow engine and uses Singularity as one of it's container backends to run tasks. I noticed that sandboxing images could take up a small amount of time for each task, so I wanted to see how much of a difference having Singularity mount with FUSE would be. In theory the container could be set up as privileged, which would allow it to work, but it would be better if that weren't the case.

I applied the patch and unshare -r and unshare -c work properly on my host machine and within the Docker container. Making Singularity use FUSE with unshare -c --keep-caps -m singularity -v -d run -u docker://hello-world does return an error on cleanup:

DEBUG   [U=1000,P=508312]  UnmountWithFuseLazy()         Executing FUSE unmount command: /usr/bin/fusermount3 -z -u /tmp/rootfs-3641030937/root
/usr/bin/fusermount3: failed to unmount /tmp/rootfs-3641030937/root: Invalid argument
ERROR   [U=1000,P=508312]  CleanupHost()                 While running host cleanup tasks: while unmounting fuse directory: /tmp/rootfs-3641030937/root: exit status 1
DEBUG   [U=1000,P=508312]  CleanupHost()                 Exiting CleanupHost - Cleanup failure
ERROR   [U=1000,P=508199]  Master()                      Unprivileged host cleanup failed: host cleanup failed

My guess is because Singularity tries to cleanup twice as the same unmount command appears above:

DEBUG   [U=1000,P=508311]  UnmountWithFuseLazy()         Executing FUSE unmount command: /usr/bin/fusermount3 -z -u /tmp/rootfs-3641030937/root

Log

This is probably a separate issue, but it doesn't break Singularity's functionality since the /tmp/rootfs-* folder is properly cleaned up. But everything looks to be working properly now, thanks!

[dtrudg]

Thanks for the feedback. The fusermount error is indeed something that will need to be addressed differently / separately. I'll open a separate issue for that, and we can tackle it a bit later on.