Usable path for shared hostPath volume

[z0rc]

I'm looking into using ReadWrite hostPath volume, that will be used to share cache data between multiple pods.

I'm running GitLab CI Runners in EKS with Bottlerocket OS nodes. With volume that can be shared across multiple gitlab-ci-runner pods on the same node, I can share a /build path which is used to hold fetched git repos. Idea is that by default GitLab CI Runners in kubernetes always start empty and must clone a repo as first step. But if the /build path is mounted and it holds already checked out repos, they can be reused, reducing time for runner doing prep work. Additional details at https://docs.gitlab.com/runner/executors/kubernetes.html#custom-builds-directory-mount.

I'm looking specifically into hostPath volumes because:

• emptyDir volumes aren't usable to store and share data between pods.
• EBS volumes via aws-ebs-csi-driver are bound to AZ and can be mounted to single node only, this will result into all CI jobs running on single node, which is not what I want.
• EFS volumes via aws-efs-csi-driver might work, but shenanigans around chown/chgrp and file ownership can be really painful to debug. Also speed would be a problem, as NFS doesn't handle well many small files, which git repos are. I don't think it's worth it.

All in all having /build as hostPath volume is just local disk cache, GitLab Runners can handle when it's empty and populate it or reuse it when it has needed data (git repo clones), so it can exists with node and doesn't need to be persisted when node is terminated.

So here are my questions:

• Does Bottlerocket OS allow to have ReadWrite hostPath volume that can be shared between pods on the same node?
• If yes, what would be the path on the host? I assume root volume is mounted read-only, so /build (as seen in gitlab-ci-runner pod filesystem) isn't usable. Maybe there is some mount point/path that allows read-write access? Right now I run Bottlerocket instances with single EBS volume, no ephemeral volumes or secondary EBS volumes available.
• If there is such path, what would be security implications around sharing hostPath volume between pods? For example some security policies must be set on containers.

[jpmcb]

Typically, if you want to have cross-pod volumes in Kubernetes, you need to utilize a persistent volume type that is ReadWriteMany: https://kubernetes.io/docs/concepts/storage/persistent-volumes

Or in other-words, some volume that supports reading and writing at the same time. The AWSElasticBlockStore persistent volume type doesn't support ReadWriteMany unfortunately. Are there any other from the above list that would work for you?

But if the /build path is mounted and it holds already checked out repos, they can be reused, reducing time for runner doing prep work.

Question: does the GitLab mechanism support this type of abstraction? If you are reading / writing to a single volume across many different pods (which may exist across many different nodes in the cluster), aren't you inviting a race-condition?

Does Bottlerocket OS allow to have ReadWrite hostPath volume that can be shared between pods on the same node?

While I haven't tried it, my guess is yes. This is not so much a Bottlerocket abstraction as it is a kubernetes one since a specific volume type needs to enable ReadWriteMany on the cluster, typically coordinated through the kubernetes api server.

It may depend on which volume type you use and where it gets mounted in order to deduce the path. Does the GitLab runner mechanism enable you to set the path to this /build directory?

If there is such path, what would be security implications around sharing hostPath volume between pods? For example some security policies must be set on containers.

Most certainly there would be security implications. Our default SELinux policy likely will block read/write access to a single volume from across many different PID namespaces (this has the benefit of preventing cross container poisoning on the filesystem). I'd also worry about making your cache a one-stop-shop for potential malicious actors: if you're using a single volume for all your workloads, this could be a single point of failure that a sole compromised container could take advantage of.

[arnaldo2792]

Hello @z0rc, I think you can accomplish what you need with bootstrap containers and propagated mounts.

From a bootstrap container, you can mount an ephemeral local disk on /.bottlerocket/mnt/build and the mount will be propagated to the root filesystem at /mnt, so in your hostPath mount you should be able to use /mnt/build.

Regarding the SELinux concern, we do enable MCS in SELinux to prevent pods from writing to each others' filesystems. However, according to this discussion, hostPath mounts won't be relabeled with the pod's label and will remain writable by all containers:

On Kubernetes, volume types like emptyDir and Amazon EBS volumes are automatically relabeled with the MCS pair when the pod is brought up, so containers in that pod can always read and write to them. Volume types like hostPath and Amazon EFS volumes are not relabeled, which allows reads and writes from all containers.

[z0rc]

@arnaldo2792 It there a way to accomplish this without bootstrap containers? I'm a bit constrained here, as I spin up nodes using EKS Managed Node Groups, and it allows to configure only size of the root volume. Modification (and switching to self-management) of launch template's userdata is technically possible, but I would consider it too complex solution for this task.

I found https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_FEATURES.md#stateless-tmpfs-for-etc, which states:

All variants will include a secondary filesystem for local storage. It will be mounted at /local with bind mounts for /var and /opt. Modifications to this area will survive an OS update or a reboot.

It this still true? So I can use something like /local/var/... for hostPath volumes, right?

[z0rc]

As I'm reading #2240, probably I should use path under /var mount, maybe something like /var/cache/build or /var/tmp/build.

What is /.bottlerocket mount point? Should it be used instead of / for hostPath? Something like /.bottlerocket/var/tmp/build in my case.

[arnaldo2792]

So I can use something like /local/var/... for hostPath volumes, right?

You could use /var, yes (even without the prefix /.bottlerocket, that prefix is only used in host and bootstrap containers). But, there are implications of using /var and you should be careful about what you let your containers see from that directory (even though the SELinux policy will protect sensible directories).

[z0rc]

But, there are implications of using /var and you should be careful about what you let your containers see from that directory

Noted, thanks.