-
Notifications
You must be signed in to change notification settings - Fork 618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow unbounded ram tasks on Windows #2142
Conversation
6fe4f45
to
eafb065
Compare
eafb065
to
5d04c87
Compare
109a8f1
to
cd42477
Compare
Hey, Sorry for the delay in responding here. We're discussing the implications of this internally. An issue with this is that it effectively disables memory limits. While it would definitely unblock your use case, it breaks the ability to bin pack a host effectively and can lead to unpredictable memory contention issues if gone unchecked. |
No problem. Thanks for the answer @jocgir can chime it but we have been using this in production since this issue was opened (without the env flag, hardcoded). Our Windows cluster has been very stable for a long while now. |
In fact, it does not break the ability to bin pack based on RAM. This is exactly what we are doing. The only side effect is that the jobs are not actually limited in RAM. But without this option, we have no way to specify a soft limit since it is not supported on Windows and that results in an error when trying to launch tasks on which we only specify a soft limit. So, the only option left is to impose a hard memory limit (since ECS requires to specify either a soft or a hard limit). This PR allows us to pack host according to our estimated memory requirements and if a task takes more memory than anticipated, the worse case would be that the host will swap ram for a while until the job is completed. Which works perfectly well for more than a year in our case. This is an opt-in feature, without explicitly specifying ECS_ENABLE_RAM_UNBOUNDED_WINDOWS_WORKAROUND, there is no change in the current behaviour. What is the difference with ECS_ENABLE_CPU_UNBOUNDED_WINDOWS_WORKAROUND? |
The big difference here is that memory is a finite resource with absolute limits, whereas CPU is limited by schedule. When you specify "arbitrarlily low" or "zero" cpu, what you're effectively requesting is CPU time but only if no other process with higher priority needs it. This metaphor doesn't translate very well over to memory, which makes it a risk. I think the way you are using this makes a lot of sense-- but I'd be hesitant to encourage folks to make heavy use of it. Soft limits on linux still get taken into account by the OS, but in this case we are ignoring it. |
This is why we make it an opt-in feature. I fully understand that it would be better to have the OS handling it. But while docker does not support soft memory limit on Windows, disabling it on container launch is the closest we can get to soft memory limit configuration. As I previously said, Windows memory management does the job pretty well and that allows us to launch containers that may sometime exceed the required memory specified. As soon as the task ends, the memory is released and available to future tasks. Maybe we can change the documentation to make it clear that this options actually disable memory control. As said before, without this option, specifying soft memory limit on task actually results into a bug since the task will never be launched. Do you have any suggestion to improve the documentation? Or to achieve the desired result? |
Agreed with your points. The documentation is something our team can sort out -- we'll need to update the docs website too. |
Hello, do you have any update on this? We'd very much like to get back to the official ECS agent version. It would simplify our release process a lot |
@julienduchesne, @jocgir: closing this in favor of carrying your commits here #2239 |
Fixes #1144
Summary
It adds an environment variable
ECS_ENABLE_RAM_UNBOUNDED_WINDOWS_WORKAROUND
which allows using containers with no hard limit on Windows hostsImplementation details
MemoryReservation is an invalid attribute in a windows host config. Therefore, if the env variable is set, and the container has no hard limit, we remove the given soft limit and the container will run unbounded.
New tests cover the changes: yes
Description for the changelog
Allow Windows containers to run without a memory limit using the
ECS_ENABLE_RAM_UNBOUNDED_WINDOWS_WORKAROUND
environment variableLicensing
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.