Update readme for unstable reserved memory value reported when ECS_POLL_METRICS is enabled and ECS_POLLING_METRICS_WAIT_DURATION is set to a high value

[Realmonia]

Summary

Update readme for unstable reserved memory value reported when ECS_POLL_METRICS is enabled and ECS_POLLING_METRICS_WAIT_DURATION is set to a high value

Background: During experiment (instance type m5.4xlarge, latest ECS optimized AMI), AL2023 do not see an issue with ECS_POLLING_METRICS_WAIT_DURATION = 20s, while AL2 see reserved memory "vibrates" when ECS_POLLING_METRICS_WAIT_DURATION equals 19s or 20s. The issue only appears when high number of tasks running on instance (in the experiment 500 tasks, 950 containers), in the case that lower tasks/containers count, reserved memory will not be impacted. This is likely due to docker metrics response latency when overloaded, and therefore causing metrics to miss reporting window (20s).

Considering the different behavior among different compute type and workload, and we have a solid default value (10s), I decided to not change specific agent handling code, but instead update readme to raise user's caution around this issue under specific circumstance.

Implementation details

Testing

New tests cover the changes: N/A

Description for the changelog

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[prateekchaudhry]

where reserved memory of ECS cluster becomes unstable due to missing metrics sample at metric collection time

I am little unclear here, does this relate to ECS_RESERVED_MEMORY/config.ReservedMemory? If that is a config, how does that fluctuate with polling metrics duration?

[Realmonia]

where reserved memory of ECS cluster becomes unstable due to missing metrics sample at metric collection time

I am little unclear here, does this relate to ECS_RESERVED_MEMORY/config.ReservedMemory? If that is a config, how does that fluctuate with polling metrics duration?

No it's a different config in task definition that is the memory limit of task/container. This data is collected in metrics to show the theoretical limit of utilization.

[prateekchaudhry]

I see, is it the task level memory limit? And total memory reserved using it, or something similar to that? In that case, I wonder if this could be rephrased to make it distinct from the 'other' Reserved Memory? (non blocking)

of ECS cluster

In hindsight this does makes it clearer

[Realmonia]

I see, is it the task level memory limit? And total memory reserved using it, or something similar to that? In that case, I wonder if this could be rephrased to make it distinct from the 'other' Reserved Memory? (non blocking)

of ECS cluster

In hindsight this does makes it clearer

Total memory does not use it; ECS_RESERVED_MEMORY is a value we subtract when we calculate total available memory during RCI call, it is the memory that is projected to be used by agent managed processes. This reserved memory is for scaling purpose, so if certain task has reserved memory, ECS scheduling procedure will guarantee those tasks can use this amount of memory when needed.

[Realmonia]

I will change all "memory reserved" to "memory reservation value in metrics" to avoid confusion. That SGTY?

[Realmonia]

Not able to trigger gpu integ tests. Force pushed.