Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow unbounded ram tasks on Windows #2142

Closed

Conversation

julienduchesne
Copy link

Fixes #1144

Summary

It adds an environment variable ECS_ENABLE_RAM_UNBOUNDED_WINDOWS_WORKAROUND which allows using containers with no hard limit on Windows hosts

Implementation details

MemoryReservation is an invalid attribute in a windows host config. Therefore, if the env variable is set, and the container has no hard limit, we remove the given soft limit and the container will run unbounded.

New tests cover the changes: yes

Description for the changelog

Allow Windows containers to run without a memory limit using the ECS_ENABLE_RAM_UNBOUNDED_WINDOWS_WORKAROUND environment variable

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@julienduchesne julienduchesne force-pushed the allow-unbounded-ram-tasks-windows branch from 6fe4f45 to eafb065 Compare July 26, 2019 14:32
@julienduchesne julienduchesne force-pushed the allow-unbounded-ram-tasks-windows branch from eafb065 to 5d04c87 Compare July 26, 2019 14:34
@julienduchesne julienduchesne force-pushed the allow-unbounded-ram-tasks-windows branch from 109a8f1 to cd42477 Compare July 26, 2019 15:01
@petderek
Copy link
Contributor

petderek commented Aug 1, 2019

Hey,

Sorry for the delay in responding here. We're discussing the implications of this internally. An issue with this is that it effectively disables memory limits. While it would definitely unblock your use case, it breaks the ability to bin pack a host effectively and can lead to unpredictable memory contention issues if gone unchecked.

@julienduchesne
Copy link
Author

julienduchesne commented Aug 1, 2019

No problem. Thanks for the answer

@jocgir can chime it but we have been using this in production since this issue was opened (without the env flag, hardcoded). Our Windows cluster has been very stable for a long while now.

@jocgir
Copy link

jocgir commented Sep 13, 2019

In fact, it does not break the ability to bin pack based on RAM. This is exactly what we are doing. The only side effect is that the jobs are not actually limited in RAM. But without this option, we have no way to specify a soft limit since it is not supported on Windows and that results in an error when trying to launch tasks on which we only specify a soft limit.

So, the only option left is to impose a hard memory limit (since ECS requires to specify either a soft or a hard limit). This PR allows us to pack host according to our estimated memory requirements and if a task takes more memory than anticipated, the worse case would be that the host will swap ram for a while until the job is completed. Which works perfectly well for more than a year in our case.

This is an opt-in feature, without explicitly specifying ECS_ENABLE_RAM_UNBOUNDED_WINDOWS_WORKAROUND, there is no change in the current behaviour. What is the difference with ECS_ENABLE_CPU_UNBOUNDED_WINDOWS_WORKAROUND?

@petderek
Copy link
Contributor

What is the difference with ECS_ENABLE_CPU_UNBOUNDED_WINDOWS_WORKAROUND?

The big difference here is that memory is a finite resource with absolute limits, whereas CPU is limited by schedule.

When you specify "arbitrarlily low" or "zero" cpu, what you're effectively requesting is CPU time but only if no other process with higher priority needs it. This metaphor doesn't translate very well over to memory, which makes it a risk.

I think the way you are using this makes a lot of sense-- but I'd be hesitant to encourage folks to make heavy use of it. Soft limits on linux still get taken into account by the OS, but in this case we are ignoring it.

@jocgir
Copy link

jocgir commented Sep 24, 2019

This is why we make it an opt-in feature. I fully understand that it would be better to have the OS handling it. But while docker does not support soft memory limit on Windows, disabling it on container launch is the closest we can get to soft memory limit configuration. As I previously said, Windows memory management does the job pretty well and that allows us to launch containers that may sometime exceed the required memory specified. As soon as the task ends, the memory is released and available to future tasks.

Maybe we can change the documentation to make it clear that this options actually disable memory control. As said before, without this option, specifying soft memory limit on task actually results into a bug since the task will never be launched. Do you have any suggestion to improve the documentation? Or to achieve the desired result?

@petderek
Copy link
Contributor

Agreed with your points. The documentation is something our team can sort out -- we'll need to update the docs website too.

@julienduchesne
Copy link
Author

Hello, do you have any update on this? We'd very much like to get back to the official ECS agent version. It would simplify our release process a lot

@adnxn
Copy link
Contributor

adnxn commented Oct 16, 2019

@julienduchesne, @jocgir: closing this in favor of carrying your commits here #2239

@adnxn adnxn closed this Oct 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants