ECS Container Metadata File not readable in container on RHEL by default

[vail130]

Summary

ECS_CONTAINER_METADATA_FILE not readable from within container on RHEL unless chcon -Rt svirt_sandbox_file_t /var/lib/ecs/data is run on the host

Description

This blog post explains the problem better than I can: http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/

By default, on RHEL, the /var directory is not readable within a container, even if mounted, unless it has been labeled appropriately. Simply running chcon -Rt svirt_sandbox_file_t /var/lib/ecs/data on the EC2 instance fixed the problem. The ECS agent should take care of this.

Expected Behavior

root@system-09a0f119c3fd27eb1:/# echo $ECS_CONTAINER_METADATA_FILE
/opt/ecs/metadata/15726a86-938f-49b1-9521-80e6334e6e6b/ecs-container-metadata.json
root@system-09a0f119c3fd27eb1:/# ls -l $ECS_CONTAINER_METADATA_FILE
-rw-r--r--. 1 root root 779 Nov 27 12:10 ecs-container-metadata.json

Observed Behavior

root@system-09a0f119c3fd27eb1:/# echo $ECS_CONTAINER_METADATA_FILE
/opt/ecs/metadata/15726a86-938f-49b1-9521-80e6334e6e6b/ecs-container-metadata.json
root@system-09a0f119c3fd27eb1:/# ls -l $ECS_CONTAINER_METADATA_FILE
ls: cannot access '/opt/ecs/metadata/15726a86-938f-49b1-9521-80e6334e6e6b/ecs-container-metadata.json': Permission denied
/opt/ecs/metadata/15726a86-938f-49b1-9521-80e6334e6e6b/ecs-container-metadata.json
root@system-09a0f119c3fd27eb1:/# ls -l /opt/ecs/metadata/15726a86-938f-49b1-9521-80e6334e6e6b/
ls: cannot access '/opt/ecs/metadata/15726a86-938f-49b1-9521-80e6334e6e6b/ecs-container-metadata.json': Permission denied
-????????? ? ? ? ?            ? ecs-container-metadata.json

Environment Details

Docker command to run ECS agent container:

/usr/bin/docker run \
  '--env=ECS_AVAILABLE_LOGGING_DRIVERS=["awslogs"]' \
  --env=ECS_CLUSTER=${CLUSTER_NAME} \
  --env=ECS_DATADIR=/data \
  --env=ECS_ENABLE_CONTAINER_METADATA=true \
  --env=ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true \
  --env=ECS_LOGFILE=/log/ecs-agent.log \
  --env=ECS_LOGLEVEL=info \
  --env=HTTP_PROXY=${HTTP_PROXY} \
  --env=HTTPS_PROXY=${HTTPS_PROXY} \
  --env=NO_PROXY=${NO_PROXY} \
  --name ecs-agent \
  --privileged \
  --publish=127.0.0.1:51678:51678 \
  --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
  --volume=/var/lib/ecs/data:/data:z \
  --volume=/var/log/ecs/:/log:z \
  --volume=/var/run/docker.sock:/var/run/docker.sock \
  --volume=/var/run/docker/execdriver/native:/var/lib/docker/execdriver/native:ro \
  amazon/amazon-ecs-agent:v1.15.2

Mounts and Binds from docker inspect ... command:

"Binds": [
                "/var/lib/ecs//data/metadata/a14c8968-0dbf-4d34-9957-7327f27731c3/statements:/opt/ecs/metadata/15726a86-938f-49b1-9521-80e6334e6e6b"
            ],
"Mounts": [
            {
                "Source": "/var/lib/ecs/data/metadata/a14c8968-0dbf-4d34-9957-7327f27731c3/statements",
                "Destination": "/opt/ecs/metadata/15726a86-938f-49b1-9521-80e6334e6e6b",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
]

[petderek]

Hi,

I'm think theres a few options available that could make this more straightforward for future use cases.

If we put this into agent, we could do something like this:

1. Check metadata is enabled.
2. If enabled, check if selinux is enforced and if the permissions are set correctly.
3. If required, set the permissions.

An alternative would be to fail fast, indicating that there is a problem with the host setup. This would :

1. Check if metadata is enabled.
2. If enabled, check that the data directory is readable.
3. If not readable, fail to start agent.

I'm hesitant to do the first option, because we typically delegate host setup to ecs-init. Agent already expects the parameters its given to be usable.

[vail130]

Failing fast would be perfectly acceptable to me, ideally with an error message pointing very specifically at how to solve the problem.

[mythri-garaga]

Failing fast is not an option here as ecs-agent runs in privileged mode and already has permission to access /data directory.

While we are working on an actual fix, here is a workaround that you can use to access /data directory.

• Run the task in privileged mode.

Actual fix could be like:
If both ECS_SELINUX_CAPABLE and ECS_ENABLE_CONTAINER_METADATA are set to true on the ECS Agent, we mount the metadata directody in Z mode.

[mythri-garaga]

Fixed in #2273.