Change default docker metric gathering behavior #2452
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sparrc
force-pushed
the
change-default-metric-gathering
branch
from
f02e7c6
to
73bc2f0
Compare
sparrc
changed the title
shubham2892
reviewed
May 13, 2020
shubham2892
reviewed
May 13, 2020
shubham2892
approved these changes
May 13, 2020
yhlee-aws
approved these changes
May 14, 2020
sparrc
force-pushed
the
change-default-metric-gathering
branch
from
800ca4f
to
5704544
Compare
sparrc
commented
May 14, 2020
| @@ -1369,56 +1369,56 @@ func (dg *dockerGoClient) Stats(ctx context.Context, id string, inactivityTimeou | |||
| }() | |||
There was a problem hiding this comment.
The changes here are because functional tests caught that polling metrics was taking longer to populate the task stats API versus streaming stats. This is because polling stats attempts to jitter the initial stats poll in order to avoid hammering the docker stats API on startup.
So change the behavior of polling metrics to poll for docker stats immediately on startup. This is not ideal from a load perspective but is a necessity to avoid unintentional behavior changes.
1. Change default docker metric gathering behavior from streaming metrics to polling. 2. Change the default polling interval to half of the TACS publishing interval (currently 20s), so that every publish interval we have two docker metrics. 3. Change the minimum polling interval to 5s to prevent customers from configuring polling to be just as resource-intensive as streaming metrics. These changes are being made because we have found that docker streaming stats consumes considerable resources from the agent, dockerd daemon, and containerd daemon.
to avoid changing behavior from streaming stats, we need to populate the stats endpoint immediately when the stats engine starts. So instead of jittering the first stats gather we need to just do it immediately.
sparrc
force-pushed
the
change-default-metric-gathering
branch
from
5704544
to
c4871aa
Compare
sparrc
force-pushed
the
change-default-metric-gathering
branch
from
544d296
to
8bb1a6b
Compare
sparrc
force-pushed
the
change-default-metric-gathering
branch
from
8bb1a6b
to
1d59927
Compare
5/12 release broke our tests
sparrc
force-pushed
the
change-default-metric-gathering
branch
from
1d59927
to
28ba4aa
Compare
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
metrics to polling.
interval of 20s), so that every publish interval we have two docker
metrics (previously 15s).
configuring resource-intensive polling (previously 1s).
log a warning and set the interval to the minimum (previously set to default).
log a warning and set the interval to the maximum (previously set to default).
These changes are being made because we have found that docker streaming
stats consumes considerable resources from the agent, dockerd daemon, and
containerd daemon.
The graph below shows the improvement in cpu utilization on a single-instance cluster with an m5.large with 120 containers. On agent 1.39.0 with default settings our cluster utilization maxes out at 86% because agent/dockerd/containerd is using ~14% of the instance's cpu resources. With this change (1.40.0) we max out at 94.5%. This means we see a 60% reduction in resources consumed by ECS daemons and 10% improvement in overall cluster utilization.
NOTE: higher cpu utilization in these graphs is a good thing, as it means the customer's containers/tasks are utilizing more of the cluster's cpu, and not daemons required for ECS.
Improvement gets more dramatic with more containers:
and not as dramatic but still substantial with fewer:
Testing
New tests cover the changes: no
Description for the changelog
Agent's default stats gathering is changing from docker streaming stats to polling. This should not affect the metrics that customers ultimately see in cloudwatch, but it does affect how the agent gathers the underlying metrics from docker. This change was made for considerable performance gains. Customers with high CPU loads may see their cluster utilization increase; this is a good thing because it means the containers are utilizing more of the cluster, and agent/dockerd/containerd are utilizing less.
Licensing
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.