provide settings for creating paths on the host

[bcressey]

What I'd like:
I'd like the ability to create paths on the host, with specific permissions and an SELinux label. This will help scenarios where a single container needs to be able to write to a directory while many containers need to read from it.

Any alternatives you've considered:
This can be worked around in general with a daemonset pod that mounts host volumes, creates the required directories, sets the permissions, and then sleeps forever. But this is somewhat clunky compared to the alternative on other distros, where that can be handled via cloud-init as a one-time step on startup.

[bcressey]

Here's the design I have in mind for this.

Create /local/host-paths/example1 with specific permissions and label:

[settings.host-paths.example1]
uid = 0
gid = 0
mode = 755
label = "system_u:object_r:local_t:s0"

Create /local/host-paths/example2 with default permissions and specific label:

[settings.host-paths.example2]
label = "system_u:object_r:any_t:s0"

Create /local/host-paths/example3 with default permissions and label:

[settings.host-paths.example3]

The main constraint is that the parent path is fixed, and the directories can't be nested. This simplifies the implementation since we won't end up with intermediate paths with ambiguous permissions, and we won't need to worry about the paths that come in via settings conflicting with paths that system services use, which may have other restrictions for security reasons.

[bcressey]

This could intersect with #1209 by also allowing host-paths to indicate a storage device:

[settings.host-paths.example4]
storage-device = "ephemeral-0"

The above snippet would create the directory and a symlink:

/local/ephemeral-disks/by-index/0/host-paths/example1
/local/host-paths/example1 -> /local/ephemeral-disks/by-index/0/host-paths/example1

[bcressey]

We decided not to allow creating host paths on ephemeral storage devices. It introduces a lot of complexity since we have to account for the device being added or removed out of sync with directory creation. It's also inherently ambiguous since we don't control enumeration order for devices, and can't guarantee where the path will be created.

[jhaynes]

Bootstrap containers can be used to accomplish this as of Bottlerocket version 1.0.8. We've documented an example in the discussions here.