[ECS] [request]: Control which containers are terminated on scale in #125
Comments
|
What we do is using StepScaling (instead of SimpleScaling), because once the ASG triggers the termination process, it is not blocking any further scaling activities. And in addition we've a lifecycle hook which sets the instance to draining (in ECS) and waits until all tasks are gone (or the timeout). It's based on this blog post: https://aws.amazon.com/blogs/compute/how-to-automate-container-instance-draining-in-amazon-ecs/ |
|
Thanks @pgarbe, I'm not sure how that helps! I'm talking about scaling in ECS tasks when a ECS service gets a decreased |
|
What I understood is, that you want to keep the EC2 hosts running as long as some tasks run on it, right? Even when this instance is marked to be terminated by the AutoScaling Group. Actually, you can't really control which EC2 instance gets terminated. But, with the lifecycle hook I mentioned above, you can delay the termination until all tasks are gone. |
|
Apologies if I've done a bad job of explaining myself @pgarbe, that is not at all what I mean. The autoscaling I am talking about is the autoscaling of ECS Tasks in an ECS Service, not the EC2 instances underneath them. As you say, there are a heap of tools for controlling the scale in and out of the underlying instances, but what I'm after are similar mechanisms for the ECS services. Imagine you have a 100 "jobs" that need processing, and you run "agents" to process those jobs as ECS tasks in an ECS service which is controlled by auto-scaling the DesiredCount. The specific problem I am trying to solve is how to intelligently scale in the ECS tasks that aren't running jobs. Currently setting DesiredCount on the ECS Service seems to basically pick Tasks at random to kill. I would like some control (like lifecycle hooks provides for ECS) to make sure that Tasks finish their work before being randomly terminated. |
|
Ok, got it. Unfortunately, in that case, I can't help you much. |
|
I have this same issue. I am using Target Tracking as my scaling policy and is tracking CPU Utillization. So whenever it does a scale-in, it kills the tasks for that service even if there are clients connected to it. I would love to know if there's a way to implement some kind of a lifecycle hook or a draining status so it will only kill the task when all connections are drained. |
|
I think there are two things in ECS which can help for connection/job draining before the ECS task stopped.
Are they able to handle your case? @lox @travis-south |
|
Thanks @wbingli, Is there an option for ELB connection draining for ALBs? I can't seem to find it. As for the stopTimeout, I'll try this and will give feedback. Thanks. |
|
Yeah, |
|
I don't think stopTimeout is in CloudFormation already, or am I missing something? 😃 |
|
I certainly hadn't heard of it before! |
|
@lox @travis-south The documentation says
|
|
I see, well, I think I'll resort to |
|
@travis-south I think here is the document to configure ELB connection draining, ELB Deregistration Delay. There is no need for a configuration on ECS service side, it will always respect ELB target draining and stop the task once target draining completed. The As for the availability, it should be available to all regions if using EC2 launch type, agent version 1.26.0+ required. The document is kind of misleading to say "This parameter is available for tasks using the Fargate launch type in the Ohio (us-east-2) region only", it actually means "For tasks using Fargate launch type, it's only available in Ohio (us-east-2) region only and requires platform version 1.3.0 or later". |
|
@wbingli thanks for the explanation. I'll try this. At the moment, my deregistration delay is 10 secs, i'll try to increase this and see what happens. |
|
Hi everyone, the We are in the process of updating ECS/Fargate and CloudFormation docs to reflect the fact that these features are available in all regions where those services are available. |
|
How would one disable SIGKILL entirely @coultn? Sometimes tasks might take 30+ minutes to finish. |
|
You can't disable SIGKILL entirely, but you can set the value to a very large number (on ECS; there is a 2 minute limit on Fargate). |
|
I tried increasing my deregistration delay to 100 secs and it made thing worst for my case. I receive a lot of 5xx errors during deployments. |
|
update: the stop timeout parameter is now documented in CloudFormation (see release history here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html). However, the original issue is about controlling which tasks get selected for termination when a service is scaling in due to a scaling policy action. Per-container stop timeouts can help with that but won't provide a complete solution. |
|
This basically brings things up to parity with lifecycle hooks on EC2, so I'd say this pretty much addresses my original issue. Happy to close this out, thanks for your help @coultn. |
|
The stop timeout does provide an ECS equivalent for EC2's termination lifecycle hook. However ECS is still missing an equivalent of EC2's instance protection, which would allow solving exactly the problem in this issue's title. Using EC2 instance protection, you can mark some instances in an autoscaling group as protected from scale in. When scaling in, EC2 will only consider instances without scaling protection enabled for termination. By manipulating the instance protection flag, an application can control exactly which EC2 instances are terminated during scale-in. If ECS would add an equivalent "task protection" flag for ECS tasks, problems like the one @lox described would have a straightforward solution. You'd simply set the protection flag to on for tasks that are busy, and turn it off when a task is idle. When an ECS service was told to scale in, it would only be allowed to kill tasks with protection turned off. I've been wrestling with a similar problem recently, and it would be very helpful if AWS would add a "task protection" feature. |
|
is there a maximum value for stopTimeout ? |
|
In my case I preffered stopTimeout to be 0 so so the container will be killed immeadetly, but apperently the minimum value is 2 seconds. What is the reason of not allowing less than 2 seconds values? |
|
This would a great value addition since ECS-EC2 tasks are usually run for processes that need to be always up and running. And in scenarios where the process cannot be stopped for hour(s) due to tasks running from the time of SIGTERM being called, this can mean incompleted tasks. Would have been wonderful to see a managed solution to this, instead of us having to build an entire architecture around this and maintain the lifecycle of the process ourselves. |
|
@ajenkins-cargometrics I 100% agree with your suggestion, ECS tasks for a job should able to enable "task protection". |
|
Is there any solution how to work around it? |
|
Thanks for the updates, I checked some more and added approach and doubt in this stackoverflow question - ECS choose idle tasks while scale in & deployment checkpointing for batch processing, could someone please have a look? |
|
>the container will kill the task it belongs to via the ECS API, and as part of the ShutdownHook/SIGTERM (Scala application for us), it will reduce the DesiredCount.
We were trying to keep the containers themselves Cloud-agnostics, so right now they could be executed anywhere.
It will be nice if AWS will have a generic solution such that when a container gracefully exits (however that is determined) the DesiredCount will go down as we don't need that service anymore (but as you say, AWS still has to obey the minimum_desired_count) .
…________________________________
From: francoislagier ***@***.***>
Sent: Wednesday, August 24, 2022 11:57 AM
To: aws/containers-roadmap ***@***.***>
Cc: Carmi Gazit ***@***.***>; Comment ***@***.***>
Subject: Re: [aws/containers-roadmap] [ECS] [request]: Control which containers are terminated on scale in (#125)
We ran into this issue at [cloudforecast.io]CloudForecast but was able to find a workaround. Our approach to this was to manage the scale down via the container. The cluster will scale up based on various CloudWatch alarms, but the entire logic to scale down is managed inside the container.
For context, our containers are processing messages from various queues. A message could take a couple of hours to be processed, and we don't want ECS to kill tasks that were busy processing a message. Also, one task == one container for us.
Each container can shutdown itself based on custom criteria (i.e., is the task processing anything at the moment?). Once the criteria are met, the container will kill the task it belongs to via the ECS API, and as part of the ShutdownHook/SIGTERM (Scala application for us), it will reduce the DesiredCount. Since the task is already killed (at least in the process of being killed), once we decrease the DesiredCount the ECS service doesn't need to stop another task.
To make sure it's working as expected, we:
* Check if the container was processing a message in the last X minutes, knowing that we will check the queues multiple times in that interval.
* Implemented a minimum_desired_count to ensure we are not down to 0 tasks at any given point. We also implemented a maximum_desired_count to keep costs under control.
* Implemented a delay for a task to shutdown (i.e., let's not shut down the task in the first 5 minutes)
* Turned on the instance protection, so the CapacityProvider ASG won't kill an instance with tasks running on it
* Implemented a custom CloudWatch alarm to decrease the ASG node count faster (based on our application, we didn't want to wait 15 minutes).
There is a potential race condition problem since multiple tasks could shut down simultaneously, but none of them will be busy (the task is shutting themselves down), and worst case, we will start a new task due to minimum_desired_count. It's not perfect, but it works well for us.
This approach allowed us to never shut down a task that was processing something important and long (1+ hours) since the container is responsible for shutting down. The ASG configuration helped us save money, but it's also specific to our workload.
Reach out via email or Twitter if I can help answer any questions.
—
Reply to this email directly, view it on GitHub<#125 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AF3VEP3GUWS2IMU6RRASBZ3V2ZPA3ANCNFSM4GRH32BQ>.
You are receiving this because you commented.Message ID: ***@***.***>
|
I agree, it works but it's not ideal since it brings AWS-specific code to my container. I'll be happy to remove the AWS-specific code once ECS supports a similar feature. |
|
I agree with you all.
…On Wed, Aug 24, 2022, 5:14 PM francoislagier ***@***.***> wrote:
We were trying to keep the containers themselves Cloud-agnostics
I agree, it works but it's not ideal since it brings AWS-specific code to
my container. I'll be happy to remove the AWS-specific code once ECS
supports a similar feature.
—
Reply to this email directly, view it on GitHub
<#125 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A2HRHUH4GQ3G5UBI2LVL5GTV22NERANCNFSM4GRH32BQ>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
We ran into same issue. Can we expect this feature "termination protection flag" for ECS task to be available before this year end? |
|
Hi All, Amazon ECS team is actively working on delivering this feature, and we appreciate your patience. I wanted to share a little more detail about the solution for your feedback - As mentioned previously, the way it would work is that you could set a scale-in protection flag on tasks that require protection along with an optional expiration time for the protection. The scale-in protection flag would prevent your tasks from being terminated during scale-in events from service autoscaling or deployments. You would be able to modify the flag using new Amazon ECS APIs (Get/UpdateTaskProtection) or through a new endpoint from within the container (similar to the task metadata endpoint https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html). These options provide you the flexibility to modify the flag from outside the boundary of a container or from within a container - to mark its task as protected - respectively. The scale-in protection flag will have a default expiration of 8 hours, which you can optionally set to up to 48 hours. The expiration helps ensure that your tasks are not inadvertently left in protected status when something goes awry and are cleaned up automatically. Note that invoking the API again will always reset/extend the protection expiration. Let us know if this sounds good, or if you have any feedback. We're especially curious to learn about the time duration requirement for keeping your tasks in the protected state. Edit: The protection expiration will be modifiable to any value up to 48 hours. It'll be an integer - expiresInMinutes - with max value as 2880 (48h) and default as 480 (8h) |
|
The expiration time functionality would work great for our use-case. Generally we have tasks that start in an idle state, receive some work and process it for a bounded period of time, and then self-terminate. We'd like to be able to have the pool of tasks scale up and down depending on the volume of pending work, but need to be able to do the scale-down without interrupting the tasks that are busy. |
|
It would be great if the expiration time can be modified down less than 8 hours - I'd rather have our tasks set a short timeout (maybe 5 minutes) and renew it if needed. |
Please see the edit. You can set it to 5 minutes and renew it repeatedly. |
|
This is a great way to go about it, covers all ends. How soon should we expect this? @AbhishekNautiyal |
|
@AbhishekNautiyal that solution is great for my use case. I am running queue workers (single thread of workers per container) that pick up jobs with peaky load. The jobs can take around 40 minutes with no practical way of being broken down or having their state saved. I wanted to scale up and down based on queue depth, but without this there was no way to do it. If I had this feature I would expect to protect the container for around an hour each time a job is picked up and release it when the job is complete. |
|
@AbhishekNautiyal This solution might help our use-case as well |
|
I want to have a clarification. You said:
> You would be able to modify the flag using new Amazon ECS APIs (Get/UpdateTaskProtection) or through a new endpoint from within >>the container (similar to the task metadata endpoint
I want to stay generic as possible, and therefore would like to avoid AWS APIs in my container.
The use of a new endpoint from within the container would be ideal.
However, reading the provided URL, it seems those stats are read-only.
So, to verify my understanding. ,There will be some ENV variable inside the container , let's call it : Scale-inProtectionFlag.
Initially, when the container starts, I want my job to run, so I will have it set (default) Scale-inProtectionFlag=true
When my job is done, I'll set it to: Scale-inProtectionFlag=off , and now this container is a fair game to be killed.
If my job was not killed (it was idle and just listening on a queue), then if it starts processing again, then it will set Scale-inProtectionFlag=true and now it's safe again (and when done will do: Scale-inProtectionFlag=false).
This is a very simplified scenario.
Did I get it right ?
Thanks,
Carmi
…________________________________
From: AbhishekNautiyal ***@***.***>
Sent: Thursday, September 15, 2022 11:14 AM
To: aws/containers-roadmap ***@***.***>
Cc: Carmi Gazit ***@***.***>; Comment ***@***.***>
Subject: Re: [aws/containers-roadmap] [ECS] [request]: Control which containers are terminated on scale in (#125)
It would be great if the expiration time can be modified down less than 8 hours - I'd rather have our tasks set a short timeout (maybe 5 minutes) and renew it if needed.
Please see the edit. You can set it to 5 minutes and renew it repeatedly.
—
Reply to this email directly, view it on GitHub<#125 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AF3VEPY4T35JMBJTKOKS4BLV6NKP3ANCNFSM4GRH32BQ>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
@AbhishekNautiyal you told us it was being actively built 10 months ago. Is it actually being built? |
@cgazit Your understanding is correct. We will be releasing a new environment variable for mutating task protection status (the current task metadata URL is and will continue to be read-only). @jimmymic Appreciate your patience. While we cannot commit to a launch date, I can confirm that this feature is in the works to be released fairly soon and have updated the status likewise. Please feel free to reach out to me at nautiya[at]amazon[dot]com for more details. |
|
Hi Abhishek, Thanks for the update and letting me know via email that this is a few weeks out from release. As requested, here is our use case. We have a Fargate service that scales up/down based on the number of messages in a SQS Queue. However, each task can take several minutes to process (and potentially longer in the future), so we didn't want to go the Lambda approach. Our use case is that we don't want a task that is actively running a job to die in the middle of processing it, because we'd have to restart the job from scratch, which is not ideal. What we'd like to do is when our ECS task gets a new message from SQS, we'd set the termination protection flag to true, and then back to false when it's done processing. Thanks again. |
|
I would like to add my use case here as well for posterity. We use ECS tasks to run multiplayer game server instances, and scale up the service according to the number of players looking to join a game. We would like to control which containers are terminated on scale-in to prevent tasks which are currently hosting an active game from being killed (and hence unceremoniously ending the game for connected players). |
|
I have almost the exact same type of use case. Presently, scaling in has a high chance of ECS deciding to kill the queue workers that have long-running tasks instead of the ones that are idle. |
|
I am very much looking forward to this feature. @AbhishekNautiyal do you have any rough estimate when we might see it? If it's soon I could potentially avoid a side quest to work around :) |
|
Looks like this'll be coming soon: https://github.com/aws-containers/ecs-task-protection-examples was made public about 12 hours ago 👀 Reading through the example code, I have to say I love the implementation of this ❤️ |
|
We're excited to announce launch of Amazon ECS task scale-in protection! Please see What's new and blog posts for details. Appreciate the engagement, feedback, and patience from everyone on this thread. We look forward to any additional comments/feedback. |
AbhishekNautiyal
added
Shipped
|
I love you |
|
Closing this issue with the aforementioned launch. Please feel free to open a new issue if you have any further feature requests/feedback. |
|
@AbhishekNautiyal thanks a lot for this feature, I'm using it and it's able to protect the ECS tasks from getting terminated as promised. However, is there a way I can view which tasks are protected at any given time? Specifically, I want to monitor for a given time range how many scale-in events were bypassed by protecting the ECS tasks. |
and others
We use ECS for auto-scaling build agents for buildkite.com. We are using custom metrics and several Lambdas for scaling the ECS service that runs our agent based on pending CI jobs. Presently when a we scale in the DesiredCount on the service, it seems like it's random which running containers get killed. It would be great to have more control over this, either a customizable timeframe to wait for containers to stop after being signaled or something similar to EC2 Lifecycle Hooks.
We're presently working around this by handling termination as gracefully as possible, but it often means cancelling an in-flight CI build, which we'd prefer not to do if other idle containers could be selected.
The text was updated successfully, but these errors were encountered: