Any plans for supporting user namespaces?

[LCaparelli]

Security docs say it's currently not supported: https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_GUIDANCE.md#do-not-run-containers-as-uid-0

Bottlerocket does not currently support user namespaces.

Moving forward, it seems like a good security improvement for containerized workloads, and I suppose it would be good for Bottlerocket to support it.

One example of new stuff that relies on user namespaces is Crossplane's default implementation of its functions runtime, XFN: crossplane/crossplane#3899

Unfortunately, this does not currently work on Bottlerocket and it's making us consider a different distro just for these nodes that will run Crossplane.

[bcressey]

The security guidance is a bit confusing on account of "user namespace support" being somewhat overloaded and confused with "rootless" containers which also require user namespaces.

For the kernel feature specifically, user namespaces are supported today but disabled by default, meaning that user.max_user_namespaces is set to 0. This can be changed via user data at launch; if you change it, and all you need is the ability to create new user namespaces within a container, then that will work. The default limit of zero is motivated by the security risks of user namespaces: they imply the ability to create child processes with capabilities that the parent process doesn't possess, which can expose a lot more kernel surface area to otherwise untrusted processes.

In the specific case of Crossplane, it looks like increasing user.max_user_namespaces would work fine and is fully supported.

For "rootless" containers, where the goal is usually something along the lines of running another container manager that will also run containers - such as running docker-in-docker or podman in a Kubernetes pod - user namespaces aren't supported. That would require the pod spec to set hostUsers: false (i.e. run the container in a new user namespace), and then kubelet would communicate that to containerd (via CRI), which would set it in the OCI spec passed to runc, which would handle setting up UID/GID mappings.

The major blocker to this is the "alpha" state of the feature in Kubernetes; as that feature moves toward GA, we'll add support to Bottlerocket. That's a longer road since we need a 6.3+ kernel (for idmapped mounts), updated versions of containerd and runc with idmapped mount support, and some additional host tools for runc.

[LCaparelli]

@bcressey thank you so much for the detailed response. I'll give the config you mentioned a go and see where that gets us.

The security guidance is a bit confusing on account of "user namespace support" being somewhat overloaded and confused with "rootless" containers which also require user namespaces.

Do you think we could maybe elaborate on this section a bit? The wording in it seems very final, not something configurable at all. If time to change this is a problem, I'm willing to contribute, though I'll definitely need some help making statements accurate as I have very little Bottlerocket-specific knowledge (and the little kernel knowledge I have is very rusty).

The major blocker to this is the "alpha" state of the feature in Kubernetes; as that feature moves toward GA, we'll add support to Bottlerocket. That's a longer road since we need a 6.3+ kernel (for idmapped mounts), updated versions of containerd and runc with idmapped mount support, and some additional host tools for runc.

Yeah, that makes perfect sense.

[LCaparelli]

The default limit of zero is motivated by the security risks of user namespaces: they imply the ability to create child processes with capabilities that the parent process doesn't possess, which can expose a lot more kernel surface area to otherwise untrusted processes.

I was not aware of that at all, btw. I just assumed more segregation == more security hahahaha

Any good references with which I can dive a little deeper? 🙏

[bcressey]

Do you think we could maybe elaborate on this section a bit?

For sure - I opened #3328 to track it.

Any good references with which I can dive a little deeper? 🙏

A security-module hook for user-namespace creation on LWN covers a somewhat recent controversy on the topic. 😀

[LCaparelli]

You rock @bcressey, thanks again 😄