[ECS] [Volumes]: Ability to create config "volume" and mount it into container as file

[taraspos]

Tell us about your request
Would be very nice to be able to mount strings (secrets/configurations) defined in the task definition into the container as a file.

Which service(s) is this request for?
ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Some apps need to be configured via config file on the file system and in order to put it there, we need to build/store a custom image. This is very uncomfortable, especially in the case when the same container can be reused with just as small configuration difference. Being able to define a config file as a "volume" and mount into the container would solve the issue, actually, k8s can do that.

Additional context
This could be used for secrets as well (#8).
Kubernetes does it via ConfigMaps

[FernandoMiguel]

Sounds like you need the next version of Docker engine, that will have that from buildkit

[taraspos]

You are talking about the experimental features? That's nice, but I would like to be able to do it during the start of the container, not during the build of the image, or I understood what it does wrong.

[FernandoMiguel]

@trane9991 oh right 🤭

[mumoshu]

This is affecting the Fargate provider for virtual-kubelet as well.

https://github.com/virtual-kubelet/virtual-kubelet/issues/185#issuecomment-452542691

To maintain feature parity with Kubernetes configmap/secret, an ECS API that updates
the "config" object, with the ability to stream config updates to the container volume would be ideal.

[jpalomaki]

Now that ECS Fargate seems to have native support for passing in AWS Secrets Manager secrets via environment variables, one possible workaround, for being able to mount secrets as files into an ECS task (app container):

Run a secrets-holding sidecar container in the ECS task, and pass Base64-encoded Secrets Manager secrets to it via environment variables, and have that sidecar Base64-decode and save those secrets in its file system, in e.g. /etc/secrets.d, root-owned, but readable by application user. Then we could use the volumesFrom (with sourceContainer pointing to sidecar) ECS container definition stanza to mount that directory (with the secrets) to the app container, read-only?

Pros:

• No environment variables (with secrets) are exposed to the main (app) container?
• Secrets Manager access is limited to Task exec IAM role (task IAM role don't need it)?
• No need to do special secrets resolution in the app container image (e.g. CMD or ENTRYPOINT script)
• Does this avoid Secrets manager API throttling, as the secrets are injected in task exec phase and not pulled in at app container startup?

Cons:

• Uses a volume (not tmpfs I presume) to hold secrets, dunno about the security implications of that? Are Fargate volumes EBS-encrypted? How about inter-task/container isolation?
• Some complexity in setting this up in the task definition, though also fairly easily ejectable when we get the native support for this?

Any thoughts on this workaround and, the pros and cons, and in particular, its security implications in the ECS Fargate environment?

[philandstuff]

We have a similar workaround to @jpalomaki, except without the sidecar. It relies on having access to a shell (sh or bash) in the underlying container.

The trick is: you set your file contents as base64-encoded environment variables, then override entrypoint to sh -c, and insert a shell snippet that base64-decodes the envvars into the appropriate files on disk, then starts the binary at the original entrypoint. (We base64-encode them to make it easier to include in the task definition JSON file; in principle, base64 is not required; especially for environment variables populated from Parameter Store.)

Pros:

• no sidecar container
• no race condition (a sidecar runs concurrently with the primary container, so there's a race to write the config file before the primary container reads it)

Cons:

• ugly 😞
• doesn't work with containers that don't have a shell included

Here's an example task definition JSON doing this trick for prometheus. It's an almighty hack, but until a feature like this lands, it's better than nothing.

[magrossm]

@philandstuff
I have the same problem with an awx installation on Fargate. I had the same idea as you, since the idea of @jpalomaki seemed a bit too much storage-wise (costs), but it the container doesnt come because it tells me Permission Denied since its in /etc/. I will try it with PArameter Store but still that wouldnt solve the permission issue. any advice?

[matthewcummings]

I like this idea a lot, would be helpful for me with ECS and EKS. I deployed a RabbitMQ image the other day - being able to mount a pre-canned config volume would be convenient.

[jpalomaki]

@philandstuff Looks like the sidecar startup race condition might be avoidable: see dependsOn at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html

[zhangalex]

I'm trying to put the config/secret file on S3 then read it from the application. The idea comes from Aliyu Container Service, maybe ECS can internally support it.

[jpalomaki]

@abby-fuller Any updates on the progress on this one?

[s-i-l-k-e]

How is this progressing? This is absolutely a must have to avoid having to bake in (minor) config changes to public images.

[cmutzel]

+5

[TapTap21]

Any news on this? Adding side-cars or customized entrypoints to achieve this isn't great.

[Vlaaaaaaad]

An option to mount a file would also help a lot for larger config files which are now limited by the 8Kb limit for SSM Advanced Parameters and 10Kb limit for Secrets Manager. Creating files from multiple parts is... particularly terrible.

[slmg]

We solved this issue with the following process:
Push config to Github -> CD push to S3 bucket -> lambda function copy config from S3 to an EFS share -> EFS gets mounted as a volume in containers.

[JohnPreston]

I actually find the side-car option rather convenient, especially using Secret Manager as the source of secrets or SSM for config.

Pros:

1. If your application requires a specific file format for the configuration, your side-car can then insure that the right format is
   respected. For example if you want to declare variables in PHP, that won't be the same format(or formatting anyway) as if your application has a well known config format (for example, [app.db]).

2. you could define a shared volume between your side car and your application, have the app mount in read-only the file so it cannot modify the values itself. The Execution role gets the grant to fetch the secret but not the apps. If the app is compromised, you can (fix your app first) rotate the secret, re-deploy, and that gets all the things updated.

services:
  app:
    volumes:
      - shared:/path/to/mount:ro
   deploy:
     labels:
       ecs.task.family: app01
    depends_on:
      - secret-fetcher

  secret-fetcher:
    secrets: [secret-01]
   volumes:
     - shared:/path/to/mount:rw
   deploy:
     labels:
       ecs.task.family: app01
       ecs.depends.condition: SUCCESS
     
volumes:
  shared: {}
  
secrets:
  secret-01:
    Name: /path/to/secrets/manager

3. Using the ECS dependency check, you can ensure that the side-car has successfully completed its task prior to starting your app. There is very little point in starting your app if something went wrong in setting the configuration.
   That could be a con of its own as that means that your side car code has to work ..

4. very versatile, you never have to hardcode anything in the docker image, you can have a volume bind locally to use your local config and not make a change to the definition of your service

Cons.
you need to write your own sidecar..

note: using ECS CLI v1 or v2 you would get two separate services etc.
To make use of the labels above to group your docker service together in AWS ECS you'd need to use Compose-X to group them in the same task definition and implement the dependency.

[RichiCoder1]

It's not a fix, but for CDK users I'd created a proof-of-concept extension for ECS that basically codifies @dw's example: https://constructs.dev/packages/@richicoder/cdk-ecs-s3volume/v/0.2.1?lang=typescript

[larstobi]

I've used @dw's example to make a Terraform module implementation for our ECS services with CODE_DEPLOY.

But, istead of using aws-cli to fetch config from S3, in this example the config will be encoded into the task definition by Terraform, by using bash and base64.

Relevant excerpt:
https://gist.github.com/larstobi/59db98fb75d28f85fb7118fd0c207f17#file-taskdef-tf-L105

[diogonicoleti]

Hi all! Any news about this feature request?

It'll be really helpful to have a first class configuration file solution instead of using alternative solutions like sidecars or modifying the container image to download and create the configuration file

[dgreda]

This feature would be really helpful. I'm surprised this is actually not present for AWS ECS.

[rgoltz]

I guess for some use-cases this newly released option for ECS on EC2 and ECS on Fargate might help:

https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ecs-fargate-integrate-ebs/?nc1=h_ls

To use EBS volumes with your Amazon ECS tasks, simply configure the path you want the EBS volume to be mounted on in your task definition, and pass desired EBS volume attributes (e.g. size, type, IOPS, throughput), Amazon Key Management Service key, and snapshot-id (if you want the volume to be initialized from an existing EBS snapshot) in the RunTask, CreateService, or UpdateService API request. When you configure EBS volumes for your Amazon ECS tasks or services, Amazon ECS provisions an equal number of EBS volumes as the number of tasks and mounts one EBS volume to each task. By default, Amazon ECS automatically deletes the attached EBS volume when a task exits. This integration gives you access to all EBS features including configurable volume types and performance, snapshots, DataLifecycleManager, and encryption for your applications deployed with Amazon ECS.

https://aws.amazon.com/de/blogs/aws/amazon-ecs-supports-a-native-integration-with-amazon-ebs-volumes-for-data-intensive-workloads/

[rupertbg]

+1

[ksitnik-tc]

+1
This should really be a part of core ECS functionality... not everything is configured with environment vars

[avpjanm]

+1

[RomanIzvozchikov]

I need this feature too. AWS, any updates?

[henrycpainter]

• 1
  There are many application container images out there which take a configuration file (yaml/json) where environment variables would become too cumbersome. The ability to easily mount a file in ECS Fargate would negate having to create custom images for a frequent use case.

Files up to some sensible limit could be mounted via a dedicated ECS file upload endpoint or refer to an S3 object location I guess?

[nalgenewaterbottle]

Would love to see this as well.

[Genebson]

Any news on this?

[tomas-bareikis]

I need this feature as well.

[bradleybarton-glitch]

I need this feature as well. Using a 3rd party docker image from Snowflake for real-time postgres data ingestion, and they're wanting a couple of config files, but if I mount the whole volume from EFS, I find that they have their own files in the directory that then can't be read.

[jmundey-bls]

much needed feature

[nalgenewaterbottle]

We get around this using this hack btw (not affiliated w/ HoneyBadger, but found their implementation useful) - https://www.honeybadger.io/blog/configure-docker-on-ecs/

[ComiskeyJC]

We get around this using this hack btw (not affiliated w/ HoneyBadger, but found their implementation useful) - https://www.honeybadger.io/blog/configure-docker-on-ecs/

Would love to get your feedback on ecs files composer

Let me know

[hkhelif]

@vibhav-ag this issue has been requested 2,017 days ago, I see that you have moved it to "Researching" wondering what is the current prioritization of this issue, and when can we expect to get an ETA? Despite being open for a while it does have a significant amount of traffic and people are still requesting this feature.

[ryanrolds]

+1

[marian-wang-arc-boats]

+1

[marsteel]

+1

[jacob-ian]

+1