[ECS] Permission denied when trying to mount volume from another container within same task

[fquffio]

Hello everyone,

I'm having an issue with this scenario: ECS task definition with two containers:

• Container A: runs in non-privileged mode, the container user is non-root, exposes a volume /my-volume/my-file.txt
• Container B: runs in non-privileged mode, the container user is non-root, mounts volume /my-volume/my-file.txt from container A using volumesFrom in the task definition and tries to read its content

When this task is run on an ECS Container Instance launched from the latest Amazon Linux 2 ECS-optimized AMI, everything works like a charm.
However, when using the latest Bottlerocket OS ECS-optimized, I get a permission error: container B gets permission denied when it tries to read the file in the mounted volume.

After a few tries and searches, I've made the hypothesis issue is caused by the volume bind not being mounted with Docker's z flag, which causes the volume not being correctly re-labeled to be accessed by multiple containers.

Can anyone tell me if I'm pointing in the right direction?
If yes, is there any known solution or workaround to make this work?
Is there some way I can test out my hypothesis?

Thanks in advance for your help,
Paolo

[bcressey]

I suspect your hypothesis is correct, and that it's a labeling issue that shows up because of the SELinux related changes in the 1.3.0 release. You can confirm by enabling the admin container on one of the nodes and checking dmesg for denial messages.

AFAIK this is undocumented, but ECS allows you to pass the relabel flag (z or Z) as part of the MountPoint property in the task defintion:

"mountPoints": [
  {
    "sourceVolume": "my-volume",
    "containerPath": "/my-volume"
  },
  {
    "sourceVolume": "my-shared-volume",
    "containerPath": "/my-shared-volume:z"
  },
  {
    "sourceVolume": "my-private-volume",
    "containerPath": "/my-private-volume:Z"
  }
]

[fquffio]

Wow, thank you so much for your very helpful response. 🙌🏼

For the time being, I've reverted to Amazon Linux 2, but I'll try to do what you suggested as soon as I can.

[omarformswift]

Hi @fquffio , I have been struggling with mounting issue and wanted to ask if you could share how you expose the volume on Container A to be picked up by Container B. Is this done within your docker file, or do you do it within the task definition?

[fquffio]

Hi @omarformswift, actually after banging our heads on this, we've changed approach.

We've changed our pipelines to make sure the static assets are present in both container images A and B, so that we don't need sharing a volume between the two containers at runtime.

Of course, this approach only works because we're dealing with static assets, however I assume that when the contents of the container are not static, you might use an EBS volume or EFS filesystem mounted into both containers rather than a volatile volume to share data between the two containers.

In any case, having a documented way to share even volatile data between two containers even with SELinux enforced on the host would be great.