[ECS] [Metrics]: Container Insights for ECS

[bradenschmidt]

Tell us about your request
Task level CPU and memory metrics in CloudWatch

Which service(s) is this request for?
ECS, Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Having CPU/memory metrics per task allows us to debug task issues such as a hot task. We're able to isolate a task that might be in trouble and kill it before it introduces an issue. The cluster level metrics aren't helpful in many other cases.

Are you currently working around this issue?
This is impossible in Fargate without introducing a buddy container but is possible in ECS by using an agent like cAdvisor on the ECS instance. Adding a buddy container, like DataDogs (https://docs.datadoghq.com/integrations/ecs_fargate/#installation), to each and every task adds significant overhead in a microservice architecture.

Additional context
From what we've gathered there is indeed metrics in the metadata of each task but it's not possible to collect it for Fargate externally.

[abby-fuller]

this should also resolve #106

[podon]

This would be awesome. We use Cloudwatch Event Rules (cron) to schedule jobs that run Tasks on our ECS Cluster. If a job is delayed, we have no visibility into the resource utilisation of the task.

[dersteve]

Is this a limitation for fargte only? We have a EC2 launch type cluster and also only get Service level metrics. But I would like to see more broken down metrics for each Task that a Service spawned, because I want to identify tasks that reach their memory limit.

[effyteva]

+1

[rganesan-gsngames]

+1

[bgkelly]

+1

[Mehul1313]

+1

[coultn]

This feature will leverage the new "CloudWatch Container Insights" that was just announced: https://aws.amazon.com/about-aws/whats-new/2019/05/cloudwatch-container-insights-for-eks-and-kubernetes-preview/

[bgkelly]

@coultn - to confirm - this will allow us to see cloud watch metrics on regular ECS Fargate tasks?

[coultn]

@bgkelly We will have more details when it launches, but in a nutshell, this will give you the capability to (1) see cloudwatch metrics (and set alarms) at the cluster/service/task definition-level, including stats such as max, min, average, and percentiles (2) Using Cloud Watch Logs Insights (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html), you will be able to visualize virtually any ad-hoc metric constructed via queries of your structured log data.

[iarlyy]

Is it possible that recent release broke ecs cluster/services metrics?

And it seems that not only me is experiencing same issue:

https://forums.aws.amazon.com/thread.jspa?threadID=305369

• My env config

amzn2-ami-ecs-hvm-2.0.20190614-x86_64-ebs (ami-04a084a6d17d9816e)

$ cat /etc/ecs/ecs.config 
ECS_CLUSTER=dojo-platform-staging
ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs","syslog","gelf"]
ECS_ENABLE_TASK_IAM_ROLE=true
ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true
ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=10m
ECS_IMAGE_CLEANUP_INTERVAL=1h
ECS_IMAGE_MINIMUM_CLEANUP_AGE=10m
ECS_NUM_IMAGES_DELETE_PER_CYCLE=10

[petderek]

@iarlyy we're looking into this and will keep you posted. Thanks much for the details!

[akshayram-wolverine]

Hi everyone,

This feature is now in preview: https://aws.amazon.com/about-aws/whats-new/2019/07/introducing-container-insights-for-ecs-and-aws-fargate-in-preview/

Look forward to feedback!

[jeffmarcinko]

Hi All-

I tried to enable container insights in one of our accounts this morning using the cli in the documentation and it failed. Any chance I'm missing a step here?

aws ecs put-account-setting --name "containerInsights" --value "enabled"

An error occurred (InvalidParameterException) when calling the PutAccountSetting operation: containerInsights should be one of [serviceLongArnFormat,taskLongArnFormat,containerInstanceLongArnFormat,awsvpcTrunking]

I tried the account settings in the ECS console and do not see insights as an option.

Thanks,

Jeff

[Mehul1313]

The same issue encountered as @jeffmarcinko .
Does the new feature require the new cluster to be created? It can't be enabled to the existing cluster?

[guitar80ep]

@jeffmarcinko what region are you running these calls in? At the moment, Container Insights is only available in Ohio, Frankfurt and Tokyo according to the AWS Public Docs.

@Mehul1313 Container Insights can only be enabled on new clusters. "During this public preview, only new clusters can be enabled for Container Insights." from the AWS Public Docs.

[jeffmarcinko]

@jeffmarcinko what region are you running these calls in? At the moment, Container Insights is only available in Ohio, Frankfurt and Tokyo according to the AWS Public Docs.

@Mehul1313 Container Insights can only be enabled on new clusters. "During this public preview, only new clusters can be enabled for Container Insights." from the AWS Public Docs.

Totally glossed over the region availability, thanks for pointing that out. I will be looking forward to the GA release.

[redzoya]

@jeffmarcinko
They have been enabled in a lot more region yesterday (7/11/2019), including us-east-1

[redzoya]

Is there any way we can enable Container Insights on existing clusters?

Even if I delete and re-create cluster with the same name and container insights option checked the cluster will be re-created without container insights enabled.

In more details what is going:
1.I delete existing cluster from the aws console
2.It is no longer showing in the list of cluster in the aws console
3.However I can still see it if I am using CLI ecs describe-clusters It is just marked with status INACTIVE
4.I proceed to re-create it via aws console with the same name, I check container insights to be enabled.
5.The cluster gets created but Container Insights are not enabled.

[daya]

Is there any way we can enable Container Insights on existing clusters?

Even if I delete and re-create cluster with the same name and container insights option checked the cluster will be re-created without container insights enabled.

In more details what is going:
1.I delete existing cluster from the aws console
2.It is no longer showing in the list of cluster in the aws console
3.However I can still see it if I am using CLI ecs describe-clusters It is just marked with status INACTIVE
4.I proceed to re-create it via aws console with the same name, I check container insights to be enabled.
5.The cluster gets created but Container Insights are not enabled.

I am experiencing the exact same issue

[hscheib]

I am also experiencing the issue @redzoya and @daya have. Would be awesome to be able to recreate with the same cluster name OR enable Container Insights on existing clusters.

[adl2db]

Would be interested in seeing this available in govcloud (us-gov-west-1)

[akshayram-wolverine]

Shipped! More info here: https://aws.amazon.com/about-aws/whats-new/2019/08/container-monitoring-for-amazon-ecs-eks-and-kubernetes-is-now-available-in-amazon-cloudwatch/

Also @redzoya updating existing clusters now supported. Learn more here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-ECS.html

[redzoya]

Thank you!

[daya]

@akshayram-wolverine Thanks

[redzoya]

@Manoj426 I think you need to upgrade your AWS cli to latest version.

[mkcgem]

erm what about cloudformation?

[bendavies]

I can't see that this is working.
"ECS Tasks" is showing the exact same filters as "ECS Services" for us, even though we have many more tasks than services.
anyone else?

[frankitox]

I don't see any difference either @bendavies. Also my CPU Utilization metric is not logging anything. I expected to get a per task resource utilization view from this Containers Insights feature.

[adiii717]

I can see memory and all listed insight metrics from the insight dashboard except CPU per task, I am using two task-linking, can somebody explain why it not showing CPU stats?

[frankitox]

Hey @adiii717, in my case the problem was that I needed to set the cpu parameter in the task definition. Here's Amazon documentation on that

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html

And what AWS support told me when I asked them

In summary CPU Utilization graph shows no value because "CpuUtilized" metric is collected only for tasks that have a defined CPU reservation but cpu unit in task definition is set as 0. So I request you to update the task definitions by setting CPU unit greater than 0. Please refer to the attached screenshot for the cpu unit updation.

Good luck!

[KishoGit]

I need the Task level metrics like CPU & Memory utilization using CLI command. I I use get-metric-statistic comman, can get at TaskDefinitionFamily level but not at Task level which I can see in the console. Not sure which command to use to get the summary of the CPUutilization at each Task/container level