ECS Secrets Management #8

abby-fuller · 2018-11-28T17:18:44Z

Enable customers to vend sensitive information from AWS Systems Manager Parameter Store and AWS Secrets Manager in the form of environment variables when a service on ECS or Fargate is started. The various components of this issue are:

ECS support for AWS Systems Manager Parameter Store [Live!]
ECS support for AWS Secrets Manager [Live!]
Fargate support for AWS Systems Manager Parameter Store [Live in Platform Version 1.3]
Fargate support for AWS Secrets Manager [Live in Platform Version 1.3]

whithajess · 2018-12-11T21:12:02Z

Will this only be AWS Secrets Manager and not AWS Systems Manager Parameter Store?

Bensign · 2018-12-12T21:07:32Z

@whithajess ECS supports AWS Systems Manager Parameter Store as of 11/15/18: https://aws.amazon.com/about-aws/whats-new/2018/11/aws-launches-secrets-support-for-amazon-elastic-container-servic/

whithajess · 2018-12-12T21:40:13Z

@Bensign Thanks for the info I guess I missed that - Unfortunately still lacking support for Fargate at the moment so will have to stick to: https://github.com/segmentio/chamber

Bensign · 2018-12-12T21:50:04Z

@whithajess We are actively working on Fargate support for both options: AWS Systems Manager Parameter Store & AWS Secrets Manager. I'll update back here on this thread as things go live.

whithajess · 2018-12-12T21:53:17Z

@Bensign Awesome - Thanks I really like this new public roadmap and hope it continues, has already given me a lot of useful info.

I follow a RSS feed of whats new in AWS but its pretty easy to miss relevant stuff given the amount of new things daily.

maartenvanderhoef · 2018-12-12T22:15:33Z

First of all. It's great this roadmap is shared, kudos for that!

The new Task definition parameter secrets is a good start for variables stored in SSM/KMS. But it's not a replacement for Chamber unfortunately. It would be great to have a parameter with which one can expose all variables of a certain path inside SSM as Environment Variables like chamber does.

Next to that SSM has been problematic when it comes to API limits. It would help if AWS would come clear regarding the API limits so that it's possible to use SSM more extensively without the risk of hitting API limits.

garrardkitchen · 2018-12-13T09:39:37Z

@maartenvanderhoef This is such a great idea...I would request for Secrets and Fargate too.

We've also been hit by SSM API throttling (via Lambda) :(

taraspos · 2018-12-13T11:31:29Z

What about being able to mount secret into the container as the file, as Kubernetes does?

tabern · 2018-12-17T06:28:24Z

@Bensign Awesome - Thanks I really like this new public roadmap and hope it continues, has already given me a lot of useful info.

I follow a RSS feed of whats new in AWS but its pretty easy to miss relevant stuff given the amount of new things daily.

You can also check out https://aws.amazon.com/containers/new – we keep this regularly updated with the latest containers related release postings and blog links.

Bensign · 2018-12-18T00:11:12Z

Hi all,

A few updates here:

Fargate Platform Version 1.3 now supports AWS Systems Manager Parameter Store: https://aws.amazon.com/about-aws/whats-new/2018/12/aws-fargate-platform-version-1-3-adds-secrets-support/
ECS now supports direct AWS Secrets Manager integration. With the latest version of the agent you may now specify AWS Secrets Manager ARNs. This will be coming to Fargate in the not so distant future as well so hang tight for that.

Volume mounts for secrets rather than environment variables has been something we've discussed in the past. Thanks for opening the issue on that. I think I'll also separately create an issue for vending an entire path as that has popped up a few times as well.

Thanks everyone for the continued feedback on this!

sj26 · 2018-12-18T10:00:32Z

vending an entire path

Yes please! This would be very useful for mounting in config files for community built images of non-cloud-native software without having to re-bake images or juggle sidecar containers.

rlafferty · 2018-12-19T21:45:43Z

Would this potentially change the # of api calls that are made to the SSM service at container start? Currently, if you have a custom startup script to get N number of secrets from SSM? We've found it fairly cost prohibitive if you start/stop containers very frequently and have 10+ secrets that get fetched/decrypted at container start time in terms of the SSM costs.

Nr18 · 2018-12-20T20:47:22Z

Will this also contain using the secrets from the AWS::ECS::TaskDefinition resource in CloudFormation?

cervantek · 2019-01-13T15:40:20Z

Will this also contain using the secrets from the AWS::ECS::TaskDefinition resource in CloudFormation?

Just want to second this. You (Amazon) would know better than us, but I think many of your customers manage their TaskDefinitions in CFN. Also, looks like it is a separate issue to track this #97 .

ronkorving · 2019-01-17T09:05:40Z

Is it me, or does https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html not yet reflect the new Secrets property that is mentioned here?

hluchej · 2019-01-28T13:32:25Z

What about similar support in EKS? We have run into it here: kubernetes-sigs/aws-load-balancer-controller#754 (comment)

Zanndorin · 2019-03-29T10:00:36Z

Hi,

Is there an ETA for

Fargate support for AWS Secrets Manager [In Progress]

Has been in Coming Soon for quite a while now :)

Zanndorin · 2019-04-04T11:39:41Z

If anyone missed it:
Beginning on April 2, 2019, any new Fargate task launched supports injecting sensitive data into your containers by storing your sensitive data in either AWS Secrets Manager secrets or AWS Systems Manager Parameter Store.

sarojgharat · 2019-04-04T11:41:24Z

But this still is not supported in cloudformation

…

On Thu, Apr 4, 2019, 13:39 Max Scholz ***@***.***> wrote: If anyone missed it: Beginning on April 2, 2019, any new Fargate task launched supports injecting sensitive data into your containers by storing your sensitive data in either AWS Secrets Manager secrets or AWS Systems Manager Parameter Store. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ab5nJA7NHJMdgASdbKnnuoH404n9iaXxks5vdeSFgaJpZM4Y4Ej4> .

Zanndorin · 2019-04-04T11:43:03Z

@sarojgharat We have started doing a workaround creating the container_definitions outside of CFN and then letting CFN referencing it when creating the task. Which works just fine for our needs..

sarojgharat · 2019-04-04T11:51:59Z

Thanks. Can you give me some hints on how to achieve this ?

…

On Thu, Apr 4, 2019, 13:43 Max Scholz ***@***.***> wrote: @sarojgharat <https://github.com/sarojgharat> We have started doing a workaround creating the container_definitions outside of CFN and then letting CFN referencing it when creating the task. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ab5nJDPW_NTkx_rrrfDZVa2biX85CcA6ks5vdeVRgaJpZM4Y4Ej4> .

Zanndorin · 2019-04-04T12:00:05Z

Well

{
   "containerDefinitions": [ 
      { 
         "name": "Service",
         "image": "XYZ:1.0.37",
         "environment": [
          {
            "name": "ENV_NAME",
            "value": "prod"
          },
        "secrets": [
          {
            "name": "XYZ_PASSWORD",
            "valueFrom": "arn:aws:ssm:region:accountid:parameter/secretname"
          }],
        "logConfiguration": { 
          "logDriver": "awslogs",
            "options": { 
              "awslogs-group" : "",
              "awslogs-region": "",
              "awslogs-stream-prefix": "XYZService"
            }
          }
      },

Then we have a script that checks for changes on this file and pushes it. We then change the Revision number we want to deploy by hand actually (organizational process) but you could for example let your CFN always run the latest (by not specifying a number) but then I think CFN has problems finishing the creation of the stack...

cornerman · 2019-06-11T22:11:27Z

Is it somehow possible to specify the Version of a systems manager parameter?

I tried parameter NAME:VERSION but then I get an error:

The Systems Manager parameter name specified for secret NAME is invalid. The parameter name can be up to 2048 characters and include the following letters and symbols: a-zA-Z0-9_.-,
--

kiros-sc · 2019-11-22T19:08:36Z

Hello I believe I might have an issues with this integration?
I'm setting up FireLens using Fargate and Http container as a test, sending the httpd log to datadog. When I tried to define the "apiKey" using a secret in secretsmanager the containers starts but the connection to datadog returns 400. When I pass the "api-key" literally, I see it on the conf of the container, and the connection works.

I check if I was able to get the value of the secret using the aws CLI, and yes, doe it brings more than just the value of the secret. I'm wondering if maybe there might be and issue with this when using the secret on ECS?

fenxiong · 2019-11-22T19:21:05Z

@kiros-sc can you send your task definition (the one that failed) to ecs-agent-external AT amazon.com so we can take a look? thanks.

kiros-sc · 2019-11-22T19:22:19Z

Ok, mail sent with the TaskDef on it.

kiros-sc · 2019-11-26T10:42:34Z

I hope this doesn't bother anyone but I found the solution to my issue. It was more related to how I was creating the "secret" in the secret manager.
At first I thought the issue was related to having a diferent "secret name" and "Name" in the "secretOptions" (task definition), yesterday that seem to be solving the issue, but, looking into two differents documents from amazon I change the way I created the "secret", instead of using the "key/value" pair, I created it as "plaintext", replacing any existing text with the "value" of my datadog apiKey, and when require to set the "secret Name" set the work apikey (just in case). After that the issue was gone.

This morning I used another secret which was created the same way, using "plain text" ()not the key/value, but as "secret Name" it was used the word "datadog", updated my task configuration with the new ARN and carefull here, setting the "name" in the "secretOptions" to "ApiKey" (just in case try different way to set that up >> apikey, Apikey, apiKey and ApiKey) and the issue was gone aswell.

So resuming all of this... I believe my issue was resolved when creating the secre using the "plaintext" setting.
Hope this helps someone.

abby-fuller added Fargate AWS Fargate ECS Amazon Elastic Container Service labels Nov 28, 2018

Bensign changed the title ~~Secrets Management v2~~ ECS Secrets Management v2 Dec 5, 2018

taraspos mentioned this issue Dec 13, 2018

[ECS] [Volumes]: Ability to create config "volume" and mount it into container as file #56

Open

Bensign changed the title ~~ECS Secrets Management v2~~ ECS Secrets Management Dec 18, 2018

jritsema mentioned this issue Jan 18, 2019

Support for Fargate secretes turnerlabs/fargate#37

Closed

abby-fuller closed this as completed Jun 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECS Secrets Management #8

ECS Secrets Management #8

abby-fuller commented Nov 28, 2018 •

edited by Bensign

Loading

whithajess commented Dec 11, 2018

Bensign commented Dec 12, 2018

whithajess commented Dec 12, 2018 •

edited

Loading

Bensign commented Dec 12, 2018

whithajess commented Dec 12, 2018

maartenvanderhoef commented Dec 12, 2018

garrardkitchen commented Dec 13, 2018

taraspos commented Dec 13, 2018

tabern commented Dec 17, 2018

Bensign commented Dec 18, 2018

sj26 commented Dec 18, 2018

rlafferty commented Dec 19, 2018

Nr18 commented Dec 20, 2018

cervantek commented Jan 13, 2019 •

edited

Loading

ronkorving commented Jan 17, 2019 •

edited

Loading

hluchej commented Jan 28, 2019

Zanndorin commented Mar 29, 2019 •

edited

Loading

Zanndorin commented Apr 4, 2019

sarojgharat commented Apr 4, 2019 via email

Zanndorin commented Apr 4, 2019 •

edited

Loading

sarojgharat commented Apr 4, 2019 via email

Zanndorin commented Apr 4, 2019

cornerman commented Jun 11, 2019

kiros-sc commented Nov 22, 2019

fenxiong commented Nov 22, 2019

kiros-sc commented Nov 22, 2019 •

edited

Loading

kiros-sc commented Nov 26, 2019

ECS Secrets Management #8

ECS Secrets Management #8

Comments

abby-fuller commented Nov 28, 2018 • edited by Bensign Loading

whithajess commented Dec 11, 2018

Bensign commented Dec 12, 2018

whithajess commented Dec 12, 2018 • edited Loading

Bensign commented Dec 12, 2018

whithajess commented Dec 12, 2018

maartenvanderhoef commented Dec 12, 2018

garrardkitchen commented Dec 13, 2018

taraspos commented Dec 13, 2018

tabern commented Dec 17, 2018

Bensign commented Dec 18, 2018

sj26 commented Dec 18, 2018

rlafferty commented Dec 19, 2018

Nr18 commented Dec 20, 2018

cervantek commented Jan 13, 2019 • edited Loading

ronkorving commented Jan 17, 2019 • edited Loading

hluchej commented Jan 28, 2019

Zanndorin commented Mar 29, 2019 • edited Loading

Zanndorin commented Apr 4, 2019

sarojgharat commented Apr 4, 2019 via email

Zanndorin commented Apr 4, 2019 • edited Loading

sarojgharat commented Apr 4, 2019 via email

Zanndorin commented Apr 4, 2019

cornerman commented Jun 11, 2019

kiros-sc commented Nov 22, 2019

fenxiong commented Nov 22, 2019

kiros-sc commented Nov 22, 2019 • edited Loading

kiros-sc commented Nov 26, 2019

abby-fuller commented Nov 28, 2018 •

edited by Bensign

Loading

whithajess commented Dec 12, 2018 •

edited

Loading

cervantek commented Jan 13, 2019 •

edited

Loading

ronkorving commented Jan 17, 2019 •

edited

Loading

Zanndorin commented Mar 29, 2019 •

edited

Loading

Zanndorin commented Apr 4, 2019 •

edited

Loading

kiros-sc commented Nov 22, 2019 •

edited

Loading