Fix a issue that agent does not clean task execution credentials from credential manager when stopping a task

[Realmonia]

Summary

Fix a issue that agent does not clean task execution credentials from credential manager when stopping a task

Implementation details

Introducing task execution credential removal to cleanupTask method

Testing

make test

Tested with an al2 instance, stop task with taskExecutionRole seems working fine.

Verify if this is related to memory leak issue: tried with a service of which the task definition makes task exits immediately. (run exit in command) and set desired task count to be 10 and run for about 48h. Both instances start with about the same memory usage (13MiB/31.15GiB)

Result

Using dev branch

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
e92dee603223        ecs-agent           6.92%               468.1MiB / 31.15GiB   1.47%               0B / 0B             0B / 7.63GB         17

Using branch with fix

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
ba0db2aaf082        ecs-agent           7.79%               450.2MiB / 31.15GiB   1.41%               0B / 0B             0B / 7.59GB         18

delta is about 18MiB.

The result shows that this issue is a very minor contributing factor to agent memory leak issue. More research on the memory leak issue will be done based on #3001

New tests cover the changes: yes

Description for the changelog

Fix a issue that agent does not clean task execution credentials from credential manager when stopping a task

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[yinyic]

It's definitely the right thing to do to clean up execution role creds. I'm curious how confident we are that this is the issue contributing to the memory leak? The structure used for storing creds contains string fields only (and short ones too it seems), which usually won't take megabytes of memory I feel like.

There could be something else i'm missing. I think it would be great if we could run the experiment a few more times and observe the consistency.

[sharanyad]

from the testing, it looks like some amount of memory leak is mitigated with this change, but not fully. is that right?

[mythri-garaga]

May be this questions is out of this PR scope but I am wondering if we are currently only cleaning up the latest task and execution credentials received from ACS credentials refresh payload at the time of task deletion. The previous expired credentials still live in the credential manager?

[Realmonia]

May be this questions is out of this PR scope but I am wondering if we are currently only cleaning up the latest task and execution credentials received from ACS credentials refresh payload at the time of task deletion. The previous expired credentials still live in the credential manager?

Seems so, we are not doing credentials clean up based on credential expiration time.

[sharanyad]

minor comment, changes lgtm

[Realmonia]

Wait for the release is done to merge --- this is not in the planned release content.