provide settings for managing ephemeral disks

[bcressey]

What I'd like:
I'd like the ability to opt-in to "managed" ephemeral disks, where the OS handles partitioning, formatting, and mounting the devices. I'd also like to be able to use this storage for the images pulled by the container runtime.

Any alternatives you've considered:
It's possible to handle some of this via orchestrated containers, since ephemerals now show up in /dev/disk/ephemeral, but that requires a daemonset or daemon task to handle something that could be done through cloud-init elsewhere.

However, it's not possible to use this approach to configure storage for container images, since it has to be ready before containers run.

[bcressey]

Here's one potential approach, where we would extend ghostdog to optionally manage devices that show up in /dev/disk/ephemeral.

[settings.ephemeral-disks]
# Whether ephemeral disks will be partitioned and formatted.
managed = true 

# Whether ephemeral disks will be mounted.
mounted = true

# Which filesystem to use for formatting.
filesystem-format = [ "ext4" | "xfs" ]

On an EC2 instance with two ephemerals, this would create the following paths:

/local/ephemeral-disks/by-index/0 -> ../by-uuid/0000-....
/local/ephemeral-disks/by-index/1 -> ../by-uuid/1111-....
/local/ephemeral-disks/by-uuid/0000-....
/local/ephemeral-disks/by-uuid/1111-....

The index path would be a symlink to the UUID path. ghostdog would create the partition and filesystem if they didn't exist, and otherwise leave the device alone.

It would maintain a local mapping of the device index to the random UUID; that means some level of locking to ensure the index was only assigned once, but otherwise the paths wouldn't be predictable enough to be used by other applications. I'd thought about encoding the index in the partition label or UUID somehow, but we want to support the case where a disk is moved between systems, where the index might already be in use. The mapping could just be the symlink in the by-index directory.

This assumes that all the ephemeral disks are interchangeable - same size, same device type. When we run into the case of heterogenous disks, we could extend the model with settings.ephemeral-disks.overrides to allow for matching by properties ("size > 20G") and selectively changing the other properties - whether to mount it, what format to use, etc.

Originally I'd thought to extend the proposal in #1208 to allow different host paths to be specified, and different security options to be applied to the directory, but I'm not sure it's worth it because of how it complicates the next feature.

The other piece of this is configuring the container runtime storage.

[settings.container-runtime]
storage-device = [ "default" | "ephemeral-N" ]

If default is specified, or if the setting is left alone, we'd use these paths:

/var/lib/containerd
/var/lib/host-containerd

If ephemeral-0 is specified, we'd use these:

/local/ephemeral-disks/by-index/0/var/lib/containerd
/local/ephemeral-disks/by-index/0/var/lib/host-containerd

We'd need a new helper in schnauzer to handle the transformation, and then update the templates for containerd and host-containerd to use it.

Regarding the host paths note, there are two problems. If the SELinux label for the directory is changed, we might not follow the file transition rules we rely on to protect the cache layers. And if the location is changed, then we might inadvertently bypass filtering done at a higher level, such as pod security policies.

[bcressey]

This also could be extended to support persistent disks, such as EBS volumes, by replacing "ephemeral" with "persistent" in the above. The key question is whether that would be sufficiently flexible to use with platforms where CSI is not currently supported, such as ECS.

[samuelkarp]

For completeness: container storage should also influence the /var/lib/docker path on variants that use Docker (aws-dev, aws-ecs-1).

The key question is whether that would be sufficiently flexible to use with platforms where CSI is not currently supported, such as ECS.

For the use-case where a path is specified in the ECS task definition, I think this plus #1208 is a good starting point.

[diranged]

Just checking in - is this on the roadmap for completion?

[bcressey]

Yes it is; @arnaldo2792 is planning to pick this up after #1208.

[arnaldo2792]

With #1601, we added the ability to partition, format and mount ephemeral disks using bootstrap containers. #1618 is a github discussion that shows an example on how to partition, format and mount an ephemeral disk.

We still have to work on a mechanism to allow users to change the storage that containerd uses to store persistent data (like container images), this last part will be tracked in #1617.