Soft limit should not pass the memory reservation parameter on Windows

[julienduchesne]

Summary

Since the new ECS version (1.16), the memory reservation parameter is passed to docker when setting a soft memory limit. However, on Windows, docker sends the following error:
docker: Error response from daemon: invalid option: Windows does not support MemoryReservation.

Description

This is a breaking change for our use case since we are now forced to set a hard limit which is something we not want to do.

Environment Details

ECS agent v1.16
Docker 17.06.2-ee-5
Windows

[samuelkarp]

Soft memory limits work on Linux through memory reservations. Since Windows does not support that, soft limits are not currently supported on Windows.

[julienduchesne]

The problem is that we are forced to set either a hard limit or a soft limit. Here's our use case:

We want to deploy 10 task on a machine with 16GB memory. If we set the hard limit at 1.5GB, our tasks will get killed when they go over. If we set the hard limit above 1.5GB, there are less than 10 tasks deployed per machine.

Previously, we had a soft limit at 1.5GB and it worked fine. Tasks did not consume 1.5GB all the time so other tasks could go over 1.5GB without crashing.

I hope this clarifies why this is a problem for us.

[jocgir]

Hi @samuelkarp. It is really breaking for us. Do you have any idea when this could be solved?

[samuelkarp]

Apologies for the delayed response here. We don't currently have a mechanism to safely implement soft memory limits on Windows right now, as Docker on Windows only implements hard limits through JOB_OBJECT_LIMIT_JOB_MEMORY.

Previously, we had a soft limit at 1.5GB and it worked fine.

Prior to the 1.16.0 release of the ECS agent (and Windows support being generally available), no memory limits were enforced at all. This was a functionality gap tracked in #616.

We understand that this is an important use-case and that this represents a gap in functionality as compared to features supported on Linux. I've marked this as a feature request and we'll continue to track the capability to implement soft memory limits on Windows here.

[jocgir]

Hi,

I understood that there was a «bug» before that no memory limit were enforced at all.

But that bug make it works on ECS for Windows Beta.

Isn't possible for ECS to just not pass the soft memory limit to docker on Windows until this is supported? That would let ECS calculate the task placement according to available memory, but not avoid the task to start.

I understand that there will be no real soft memory management on docker and this is not under your control. But it is far less worse that not being able to launch tasks without specifying hard limit IMHO.

Thank you for your answer.

[julienduchesne]

Hi @samuelkarp @richardpen.
Following my PR #1480. We thought about this and would redoing the PR #1238 and getting the setting from an env variable be something that would get merged?
Similar to ECS_ENABLE_CPU_UNBOUNDED_WINDOWS_WORKAROUND
We could have env variable ECS_ENABLE_RAM_UNBOUNDED_WINDOWS_WORKAROUND that would be stored alongside CpuUnbounded in the PlatformFields

[Drewster727]

CannotCreateContainerError: API error (400): invalid option: Windows does not support MemoryReservation Do we have a fix for this? I'm seeing this on the latest ECS optimized windows 2016 AMI. I am forced to set a soft or hard memory limit in the settings.

[DanielLaberge]

Any update or guidance on this?

We're considering whether to adopt ECS and issue is a bit of a turn off. Not passing the soft limit to docker would also satisfy our use cases (as it was prior to the 1.16.0 release).

[CloudViking86]

So if I understand this correctly when this release is released when updating the ECS-Agent on your EC2's running in your cluster you should no longer need to set a "hard limit"?

Ran into this same problem when trying to launch a Windows container (defined a "soft limit" and the task refused to start and read about the errors in the ECS-log).

[adnxn]

@CloudViking86: that is exactly correct

[CloudViking86]

@adnxn Thank you so much for your fast reply. Any idea on when this will be released?

[adnxn]

we don't have a public date. however, expect it to be in step with the ecs windows ami release cadence.