Add more validation of stats received from Docker #4295
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danehlim
force-pushed
the
filter-out-invalid-stats
branch
from
691449e to
c979c34
Compare
danehlim
force-pushed
the
filter-out-invalid-stats
branch
from
c979c34 to
f2db481
Compare
danehlim
force-pushed
the
filter-out-invalid-stats
branch
from
f2db481 to
dec4150
Compare
danehlim
force-pushed
the
filter-out-invalid-stats
branch
from
dec4150 to
699b4f2
Compare
danehlim
force-pushed
the
filter-out-invalid-stats
branch
from
699b4f2 to
57f030f
Compare
danehlim
force-pushed
the
filter-out-invalid-stats
branch
from
57f030f to
a8cfc6a
Compare
yinyic
approved these changes
Aug 17, 2024
sparrc
reviewed
Aug 17, 2024
| logger.Debug(fmt.Sprintf( | ||
| "Error processing stats stream of container, backing off %s before reopening", d), logger.Fields{ | ||
| loggerfield.DockerId: dockerID, | ||
| loggerfield.Error: err, |
sparrc
approved these changes
Aug 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add additional validation of container stats received from Docker such that they are considered invalid if they have a
readtime of0001-01-01T00:00:00Z(i.e., zero value of typetime.Time) the associated container has a container restart policy is enabled.This is so that if Docker does send ECS Agent a stat with a
readtime of0001-01-01T00:00:00Zfor a container with a container restart policy enabled, then ECS Agent drops/filters it out.Implementation details
validateDockerStatsfunction to consider stats withreadtime of0001-01-01T00:00:00Zas invalid for containers with a container restart policy enabledgetAggregatedDockerStatAcrossRestartsfunction to set an aggregated stat'sprereadtime to be the read time of the last stat stored in the stats queue, in case ECS Agent does receive from Docker a stat with areadtime of0001-01-01T00:00:00Zdirectly prior to the current stat being processedTesting
Automated PR tests.
Also test via internal functional test that container metadata aggregation across restarts logic from #4206 works as expected even when invalid stats from Docker are received on the latest AL2 AMD64 and AL2023 AMD64 ECS-Optimized AMIs.
New tests cover the changes: yes
Description for the changelog
Add more validation of stats received from Docker
Additional Information
Does this PR include breaking model changes? If so, Have you added transformation functions?
No
Does this PR include the addition of new environment variables in the README?
No
Licensing
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.