host-containers: allow mount propagations from privileged containers #1601

arnaldo2792 · 2021-06-03T00:46:04Z

Issue number:
#1209

Description of changes:

e8936fc5 host-containers: allow mount propagations from privileged containers

This commit adds support to propagate mount points created in bootstrap and superpowered containers, across mount peer groups.

The root filesystem of bootstrap and superpowered containers is setup with the rshared configuration to allow mounts propagations across peer groups. All mount points attached to the containers are configured as rprivate (except for the mnt mount). This prevents bootstrap and superpowered containers from remounting directories in the host's root filesystem.

The /.bottlerocket/rootfs/mnt mount point was added to bootstrap and superpowered containers. This mount point is a bind mount that points to /mnt in the host, which itself is a bind mount of /local/mnt. This is required to let users create mount points underneath /mnt. This mount point is setup with the rshared configuration to allow propagations across peer groups. This is the only mount point from which propagations are allowed across peer groups.

With this change, bootstrap containers now have access to all the devices in the host. Also, they now have the CAP_SYS_ADMIN capability to let users manage ephemeral disks. The logic to build the container specs was refactored to provide a better understanding of what options are set for the containers' spec.

Testing done:

Admin and control containers are still working as expected and can make calls using the apiclient
Non-superpowered containers don't have access to /dev, /.bottlerocket/rootfs/mnt or the host's root filesystem
I launched a c5d.2xlarge instance, which has one ephemeral disk attached to it. I created a bootstrap container with the following settings:

[settings.bootstrap-containers.ephemeral]
mode = "once"
source = "<SOURCE>"
essential = false

Where setup-ephemeral-disk is defined as

FROM alpine
RUN apk add e2fsprogs bash parted
ADD script ./
RUN chmod +x ./script
ENTRYPOINT ["sh", "script"]

And with script as:

#!/usr/bin/env bash
set -ex

DISK=/.bottlerocket/rootfs/dev/nvme2n1
PERSISTENT_DIR=/.bottlerocket/bootstrap-containers/current
PARTITIONS_CREATED=/.bottlerocket/bootstrap-containers/current/created
BASE_MOUNT_POINT=/.bottlerocket/rootfs/mnt

if [ ! -f $PARTITIONS_CREATED ]; then
  parted -s $DISK mklabel gpt 1>/dev/null
  parted -s $DISK mkpart primary ext4 0% 50% 1>/dev/null
  parted -s $DISK mkpart primary ext4 50% 100% 1>/dev/null
  mkfs.ext4 -F ${DISK}p1
  mkfs.ext4 -F ${DISK}p2
  touch $PARTITIONS_CREATED
fi

mkdir -p $BASE_MOUNT_POINT/part1
mkdir -p $BASE_MOUNT_POINT/part2

mount ${DISK}p1 $BASE_MOUNT_POINT/part1
mount ${DISK}p2 $BASE_MOUNT_POINT/part2

I confirmed that the partitions were mounted and propagated:

# From the admin container
[ec2-user@ip-172-31-5-108 ~]$ lsblk
# ...
├─nvme2n1p1  259:14   0  93.1G  0 part /.bottlerocket/rootfs/mnt/part1
└─nvme2n1p2  259:15   0  93.1G  0 part /.bottlerocket/rootfs/mnt/part2

# From the host
bash-5.0# lsblk
# ...
|-nvme2n1p1  259:14   0  93.1G  0 part /local/mnt/part1
|-nvme2n1p2  259:15   0  93.1G  0 part /local/mnt/part2

From the admin container I ran:

[ec2-user@ip-172-31-12-31 ~]$ sudo su
bash-4.2# umount /.bottlerocket/rootfs/mnt/part1
bash-4.2# mount /dev/nvme2n1p1 /.bottlerocket/rootfs/etc/
bash-4.2# ls /.bottlerocket/rootfs/etc/
lost+found

And confirmed that the mount point didn't propagate:

# From the host
bash-5.0# lsblk
# ...
nvme2n1      259:0    0 186.3G  0 disk
|-nvme2n1p1  259:14   0  93.1G  0 part
|-nvme2n1p2  259:15   0  93.1G  0 part /local/mnt/part2

bash-5.0# ls /etc/
bootstrap-containers  ...

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

README.md

packages/release/mnt.mount

sources/host-ctr/cmd/host-ctr/main.go

arnaldo2792 · 2021-06-04T21:42:16Z

Forced push includes:

Fixes in main README
Refactor how container specs are created as suggested in feedback
Mount /.bottlerocket/rootfs/mnt instead of /mnt inside privileged containers

arnaldo2792 · 2021-06-04T21:44:24Z

Forced push to fix commit message

samuelkarp

The reorganized code is significantly easier to read; thanks for doing that.

samuelkarp · 2021-06-04T23:08:18Z

README.md

@@ -493,7 +493,10 @@ Bootstrap containers are host containers that can be used to "bootstrap" the hos

 Bootstrap containers are very similar to normal host containers; they come with persistent storage and with optional user data.
 Unlike normal host containers, bootstrap containers can't be treated as `superpowered` containers.
-However, these containers have access to the underlying root filesystem on `/.bottlerocket/rootfs`.
+However, bootstrap containers do have additional permissions that normal host containers do not have.
+Bootstrap containers have access to the underlying root filesystem on `/.bottlerocket/rootfs` as well as to all the devices in the host, and they are setup with the `CAP_SYS_ADMIN` capability.


nit:

Suggested change

Bootstrap containers have access to the underlying root filesystem on `/.bottlerocket/rootfs` as well as to all the devices in the host, and they are setup with the `CAP_SYS_ADMIN` capability.

Bootstrap containers have access to the underlying root filesystem on `/.bottlerocket/rootfs` as well as to all the devices in the host, and they are set up with the `CAP_SYS_ADMIN` capability.

samuelkarp · 2021-06-04T23:08:58Z

README.md

+##### Mount propagations in bootstrap and superpowered containers
+Both bootstrap and superpowered host containers are configured with the `/.bottlerocket/rootfs/mnt` bind mount that points to `/mnt` in the host, which itself is a bind mount of `/local/mnt`.
+This bind mount is set up with shared propagations, so any new mount point created underneath `/.bottlerocket/rootfs/mnt` in any bootstrap or superpowered host container will propagate across mount namespaces.
+You can use this feature to configure ephemeral disks attached to your hosts, that you may want to use on your workloads.


nit:

Suggested change

You can use this feature to configure ephemeral disks attached to your hosts, that you may want to use on your workloads.

You can use this feature to configure ephemeral disks attached to your hosts that you may want to use on your workloads.

sources/host-ctr/cmd/host-ctr/main.go

arnaldo2792 · 2021-06-05T00:44:18Z

Forced push includes fixes for feedback in the main README

arnaldo2792 · 2021-06-08T21:38:31Z

Forced push includes:

All mounts, except for /.bottlerocket/rootfs/mnt, are rprivate to prevent propagations from them
Removed /dev from mounts since the host container already has access to it from /.bottlerocket/rootfs/dev

packages/release/mnt.mount

samuelkarp · 2021-06-11T00:03:35Z

sources/host-ctr/cmd/host-ctr/main.go

+	var hasPropagation = false
+	// Propagations can be shared, rshared, private, rprivate, slave, rslave
+	re := regexp.MustCompile(`r?(shared|private|slave)`)
+
+	for _, option := range mount.Options {
+		hasPropagation = re.FindString(option) != ""
+
+		if hasPropagation {
+			break
+		}
+	}
+
+	return hasPropagation


I'd prefer to avoid the use of a regular expression here. There are only six values we need to test for.

Suggested change

var hasPropagation = false

// Propagations can be shared, rshared, private, rprivate, slave, rslave

re := regexp.MustCompile(`r?(shared|private|slave)`)

for _, option := range mount.Options {

hasPropagation = re.FindString(option) != ""

if hasPropagation {

break

}

}

return hasPropagation

// Propagations can be shared, rshared, private, rprivate, slave, rslave

for _, option := range mount.Options {

switch option {

case "shared", "rshared", "private", "rprivate", "slave", "rslave":

return true

}

}

return false

arnaldo2792 · 2021-06-11T00:56:50Z

Forced push includes refactor in hasPropagations method as suggested by @samuelkarp

arnaldo2792 · 2021-06-14T23:22:13Z

Forced push due to conflicts in rebase

This commit adds support to propagate mount points created in bootstrap and superpowered containers, across mount peer groups. The root filesystem of bootstrap and superpowered containers is setup with the `rshared` configuration to allow mounts propagations across peer groups. All mount points attached to the containers are configured as `rprivate` (except for the `mnt` mount). This prevents bootstrap and superpowered containers from remounting directories in the host's root filesystem. The `/.bottlerocket/rootfs/mnt` mount point was added to bootstrap and superpowered containers. This mount point is a bind mount that points to `/mnt` in the host, which itself is a bind mount of `/local/mnt`. This is required to let users create mount points underneath `/mnt`. This mount point is setup with the `rshared` configuration to allow propagations across peer groups. This is the only mount point from which propagations are allowed across peer groups. With this change, bootstrap containers now have access to all the devices in the host. Also, they now have the `CAP_SYS_ADMIN` capability to let users manage ephemeral disks. The logic to build the container specs was refactored to provide a better understanding of what options are set for the containers' spec. Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

arnaldo2792 · 2021-06-15T00:07:09Z

Forced push includes file missing in previous commit to fix rebase conflicts

arnaldo2792 requested review from bcressey and samuelkarp June 3, 2021 00:46

samuelkarp reviewed Jun 3, 2021

View reviewed changes

arnaldo2792 force-pushed the ephimeral-disks branch from 63b3a71 to 69684e0 Compare June 4, 2021 21:35

arnaldo2792 force-pushed the ephimeral-disks branch from 69684e0 to cac5a18 Compare June 4, 2021 21:43

samuelkarp reviewed Jun 4, 2021

View reviewed changes

arnaldo2792 force-pushed the ephimeral-disks branch from cac5a18 to 9e6d96d Compare June 5, 2021 00:43

arnaldo2792 requested a review from samuelkarp June 5, 2021 00:45

jhaynes requested a review from tjkirch June 7, 2021 21:18

arnaldo2792 force-pushed the ephimeral-disks branch from 9e6d96d to ea9ca19 Compare June 8, 2021 21:36

bcressey reviewed Jun 9, 2021

View reviewed changes

packages/release/mnt.mount Show resolved Hide resolved

samuelkarp reviewed Jun 11, 2021

View reviewed changes

arnaldo2792 force-pushed the ephimeral-disks branch from ea9ca19 to 3d79d2c Compare June 11, 2021 00:54

arnaldo2792 requested review from samuelkarp and bcressey June 11, 2021 00:56

bcressey approved these changes Jun 11, 2021

View reviewed changes

samuelkarp approved these changes Jun 14, 2021

View reviewed changes

arnaldo2792 force-pushed the ephimeral-disks branch from 3d79d2c to 8858a6d Compare June 14, 2021 23:21

arnaldo2792 force-pushed the ephimeral-disks branch from 8858a6d to 08092c1 Compare June 15, 2021 00:06

arnaldo2792 requested review from samuelkarp and bcressey June 15, 2021 00:41

bcressey approved these changes Jun 15, 2021

View reviewed changes

samuelkarp approved these changes Jun 15, 2021

View reviewed changes

arnaldo2792 merged commit 357c2a8 into bottlerocket-os:develop Jun 16, 2021

arnaldo2792 mentioned this pull request Jun 16, 2021

Provide settings to change containerd's persistent storage #1617

Closed

arnaldo2792 deleted the ephimeral-disks branch June 16, 2021 22:48

arnaldo2792 mentioned this pull request Jun 16, 2021

provide settings for managing ephemeral disks #1209

Closed

mjgp2 mentioned this pull request Jan 18, 2022

Support Swap #1911

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

host-containers: allow mount propagations from privileged containers #1601

host-containers: allow mount propagations from privileged containers #1601

arnaldo2792 commented Jun 3, 2021 •

edited

Loading

arnaldo2792 commented Jun 4, 2021

arnaldo2792 commented Jun 4, 2021

samuelkarp left a comment

samuelkarp Jun 4, 2021

samuelkarp Jun 4, 2021

arnaldo2792 commented Jun 5, 2021

arnaldo2792 commented Jun 8, 2021

samuelkarp Jun 11, 2021

arnaldo2792 commented Jun 11, 2021

arnaldo2792 commented Jun 14, 2021

arnaldo2792 commented Jun 15, 2021

	Bootstrap containers have access to the underlying root filesystem on `/.bottlerocket/rootfs` as well as to all the devices in the host, and they are setup with the `CAP_SYS_ADMIN` capability.
	Bootstrap containers have access to the underlying root filesystem on `/.bottlerocket/rootfs` as well as to all the devices in the host, and they are set up with the `CAP_SYS_ADMIN` capability.

	You can use this feature to configure ephemeral disks attached to your hosts, that you may want to use on your workloads.
	You can use this feature to configure ephemeral disks attached to your hosts that you may want to use on your workloads.

host-containers: allow mount propagations from privileged containers #1601

host-containers: allow mount propagations from privileged containers #1601

Conversation

arnaldo2792 commented Jun 3, 2021 • edited Loading

arnaldo2792 commented Jun 4, 2021

arnaldo2792 commented Jun 4, 2021

samuelkarp left a comment

Choose a reason for hiding this comment

samuelkarp Jun 4, 2021

Choose a reason for hiding this comment

samuelkarp Jun 4, 2021

Choose a reason for hiding this comment

arnaldo2792 commented Jun 5, 2021

arnaldo2792 commented Jun 8, 2021

samuelkarp Jun 11, 2021

Choose a reason for hiding this comment

arnaldo2792 commented Jun 11, 2021

arnaldo2792 commented Jun 14, 2021

arnaldo2792 commented Jun 15, 2021

arnaldo2792 commented Jun 3, 2021 •

edited

Loading