container stats: ignore context.Canceled errors

[sparrc]

we often receive context.Canceled errors when a container exits during
docker stats collection.

there is a benign race condition where if we process the error first
before processing that the context is "Done", then we will log this
canceled error as a warning message here:

amazon-ecs-agent/agent/stats/container.go

Lines 118 to 122 in 5be7aa0

	case <-container.ctx.Done():
	return nil
	case err := <-errC:
	seelog.Warnf("Error encountered processing metrics stream from docker, this may affect cloudwatch metric accuracy: %s", err)
	returnError = true

this change ignores these context.Canceled errors so that we don't log
them. This will eliminate log messages that look like this when a
container exits:

level=warn time=2020-12-25T07:51:33Z msg="Error encountered processing metrics stream from docker, this may affect cloudwatch metric accuracy: DockerGoClient: Unable to decode stats for container REDACTED: context canceled" module=container.go

Testing

Tested by running a large number of tasks with lots of container and task state changes and exits. Before this change I was seeing the log message around 20% of the time that containers exited. After the change I do not see this message.

New tests cover the changes: no

Description for the changelog

enhancement: eliminate benign docker stats "context canceled" warning messages from logs

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[angelcar]

I don't think the client is the right place to ignore this error.
Can we still return it and ignore it where this method is invoked instead?

[sparrc]

i guess we can have the client wrapping this error with %w....and then we could call this at a higher level, I will try it

[angelcar]

That works. Alternatively you could just use the plain error when it's context.Canceled

if errors.Is(err, context.Canceled) {
    errC <- err
    return
}

[sharanyad]