Watchdog failed to retrieve AWS security credentials after 1 hour

[xian13]

I'm using EFS as volume at ECS Task Definition https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-efsvolumeconfiguration.html
The task definition is deployed at EC2 Instances

My task definition setup

"efsVolumeConfiguration": {
    "fileSystemId": "xxx",
    "authorizationConfig": {
        "iam": "ENABLED",
        "accessPointId": "xxxx"
    },
    "transitEncryption": "ENABLED",
    "rootDirectory": "/"
}

At first everything is fine, but after an hour, there will be many error printed in /var/log/amazon/efs/mount-watchdog.log

021-02-10 16:23:04,814 - INFO - amazon-efs-mount-watchdog, version 1.28.2, is enabled and started
2021-02-10 17:23:05,069 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/xxx
2021-02-10 17:23:05,069 - ERROR - Cannot recreate self-signed certificate
2021-02-10 17:23:06,122 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/xxx
2021-02-10 17:23:06,123 - ERROR - Cannot recreate self-signed certificate
2021-02-10 17:23:07,174 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/xxx
2021-02-10 17:23:07,174 - ERROR - Cannot recreate self-signed certificate
2021-02-10 17:23:08,269 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/xxx
2021-02-10 17:23:08,269 - ERROR - Cannot recreate self-signed certificate
2021-02-10 17:23:09,322 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/xxx
2021-02-10 17:23:09,322 - ERROR - Cannot recreate self-signed certificate
2021-02-10 17:23:10,370 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/xxx
2021-02-10 17:23:10,370 - ERROR - Cannot recreate self-signed certificate
2021-02-10 17:23:11,433 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/xxx
2021-02-10 17:23:11,433 - ERROR - Cannot recreate self-signed certificate

The already mounted EFS can still be accessed both read and write
If new container created, I still not sure whether the new container can access EFS with no error or not, but I think it should be OK because if after an hour new container can't access EFS, there will be already many error on my production apps

The current problem is high cpu load because I think it keeps printing to the log
So all my t3 cpu credits being used to print this log

[wangnyue]

Thanks for the feedback. We’ll have someone take a look

[Cappuccinuo]

Hey @xian13 ,

Thanks for the feedback. Can you provide the following info?

1. The result of sudo df on the host instance, we can tell how many file systems are mounted.
2. The result of sudo docker ps -a on the host instance, we can tell how many dockers are running. In theory, if you have N dockers Up and in the task definition you have EFS defined as volume, then there will be N file system mounted on the host instance.
3. The result of sudo ls /var/run/efs on the host instance, also the result of ps aux | grep stunnel on the host instance, we can tell how many stunnel process and stunnel state files are there.
4. The mountPoints part in the task definition ContainerDefinition.
5. Did you kill the task during this period?

Thanks.

[xian13]

Hi @Cappuccinuo , here are the informations needed

1. sudo df

Filesystem            1K-blocks    Used        Available Use% Mounted on
devtmpfs                 990180       0           990180   0% /dev
tmpfs                   1000984       0          1000984   0% /dev/shm
tmpfs                   1000984    4232           996752   1% /run
tmpfs                   1000984       0          1000984   0% /sys/fs/cgroup
/dev/nvme0n1p1         30830568 5291060         25208572  18% /
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/a2e94ed6f92fa6c595d34062689dd0c1d327fc75ecbf0c1bf5beef7f7c22ce67/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/943727c8214465546e877c6760bf0e7aeaa836be5a11b6c5535c9f161c226d09/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/227a96bde6b94164698e7d343620d4ba16ef1cc527a3c71cedca4333537ffbe2/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/5bb85db70c7a8dfcccd1543c9e30169b2d90773cb1781327939fafea696a3e68/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/73fb7e36c04c31851d3540c5f0da37aa2da5fc3baf9573553d699b5a54fee693/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/72b7e429f0913932b5f540803dec697f7ecf4185852e2a6132ccaafc0f2a8c1c/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/9f32d6102a3c0e0ae9a4606e9d3fbf3bc5af02b32a278bc5d937cf08e61c5827/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/7cd1b4a31ee07274963c9e1e2afe8ebc9e534749c62e8fc33335a941dc9c5c30/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/f3aba2c80be52a2c058d9d7f0c796d6a8a059b017147265fa92f65147f865a61/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/acc825f17c21f885107ea18c41c7b27022a47c4e90f8ce3b124e4e25c3f5c533/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/3452ce888ce228a6eee63056806445f1da6fdf2bda257ac7f75e63f5934e3601/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/1df4dfc1bd20fcedcf1624fc79226ac3f93ff67bfc9d5c2b7ac45c46e71c1659/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/807115cb52190b2471c973b40f9123786f0e23e209a8a9c7dfc80481a4aa569e/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/a81ea60e34a14439511d845cde809dfbdc6b53b340c9da133e4aad711bdd0bbe/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/f2c47ffa5e0480e095236ede3c629ec34229f098da8f9cb708cb755d79f0b700/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/023123f145bddca3d15693bbd5dbff109187b190282f28946c31520b15aee17f/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/43db1b01c5ccf39905f9b00c8bcc462837dd47c93bb07427bf38dd6372737c6c/merged
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/13c76f6895dcf39c3a3c67d9e25c5e9e0191428654328bc6bd0e4ec414de7db9/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/20323267939f7ee32292be1fc2f131bc0ae84b9fa9bc406fb3558dfb3251919b/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/03e0293695d89d2951eb282ce10b2b3302ca026f4057f21717bfec9968ef8378/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/28db6864ed43a9f9500162c913e593bdad7478687bf3431844c74e5f162dd26d/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/b59497507206a6276341f59e99dce21bc98ae9852bbeaaf8bcab17a63bed38df/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/feeb5d8286140f121f1a4a43610385708a470125771f546793bb14c1b8113e99/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/dc10029163bfa878b04ececb5312191a980e8ac3386e7be1171c47b5dfb1a2b0/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/260f52ea23d8851d376974032c904b6ac3895bf50a72e6cd4794f4739960e10d/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/538a752807d7ca80f235a62c196b8132965dddcd5c299d1c55339def1bdee47e/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/4b4f3b0f8f4d8531c5f54b3b963885586131d3bf1abf421df7ecdfb8e2fa1a3f/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/aa4d4a897d434b7ac58f30d47060ffc65524eef58c7711deab7ce0402af28be7/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/fe249c5e61a413427d25d2e61173502070a3eb71c0f64583b253ca11f8f27d8c/merged
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/0e9432fb62467f3fee4d82b38f3a3f0ac0f52e9be93d3076d6b2b98971ecc2ed/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/c8c9b4f3415581b9e8acb562c842de157bf8479ada898a0350f8a4d9184d320f/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/7bcadeb2df3d74abd4f36513fb0d9b0ead3f53765053217256cc0c800f45e36f/mounts/shm
overlay                30830568 5291060         25208572  18% /var/lib/docker/overlay2/ff7171e306632a6f98151f2142e046a8531bcdc32fa34d69a072085e7d85e4f6/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/25bd5232ba121f107005df048ba9f6c84e7c0f14b4fb6ef93346d7e07ba4816e/mounts/shm
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-web-958-JaknotWWWEFSVolume-e8d188d3f186f3bbb701
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-consume-jakpay-message-99-JaknotWWWEFSVolume-d8be9ee6e389f1e0f601
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-finance-auto-approve-555-JaknotWWWEFSVolume-e6dcecbdf891ebaa1f00
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-purchasing-omzet-item-573-JaknotWWWEFSVolume-9ad1dbe0fbf5c5fd5300
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-invoice-473-JaknotWWWEFSVolume-f6d4cfc38fc9ac9b6a00
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-default-556-JaknotWWWEFSVolume-fcfca7aa87dad7a93400
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/f75b1d057e9fb419296a9b86b536e9c831e657e3eb9d9a4fcfc8302f30035cf9/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/722c7483ceb36d9cb58392d8ba16c7cb33871eed5f7869f48ce0b305ac09dfaf/mounts/shm
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-web-959-JaknotWWWEFSVolume-d0c4aafae9bdee84f501
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/aad2d7fd42ef6e2ae18db8c19967744d9ebb71c8ea0746460192944b1ad2818d/merged
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/75ceaf22d7f18c48f5875a6b77d169aaf556483a9e291500debaa7a371ae9ca7/merged
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/3e364e40fc11965dcd64e7fe4e5629cd96c95aac554e0d216bad2d02db0f0d10/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/62b8e5e000a6dde84a91fae6fe43c28101f96a2096cad08e1945f66d08a4f60d/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/be417ec7050707287097bf98456c355c0e9f36fbf31718c257c818cf272b95fb/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/52b62eb72717d49405d4c8ccb77ff80f369c077ae4a8fcd5894cd10e450c694f/mounts/shm
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/0ba068a501b3f4c3224229cf92a73d30c5d5dce43d06286510a2f66bc9ce6a4f/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/d528451ad577989b46d2b03c283ebb6c88db5daf8ab3c9aaa1eedbbe5fdfc49a/mounts/shm
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/5f905fac03240a13ae1baff7ab2a985a3dec374f4d380040d7a8c3810b98b31a/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/9cb81ccf97cb112523696f6179e6ff9bc5c93d06abd39b627c79b3dfdc5c741f/mounts/shm
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-mail-priority-802-JaknotWWWEFSVolume-bef884d3bb8fa1cf4000
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-finance-auto-approve-556-JaknotWWWEFSVolume-8efdbdfea8a795d7a001
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-mail-898-JaknotWWWEFSVolume-f8b0dfbc8e888dafa501
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-ae91a6c9abc49ecd4500
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-purchasing-omzet-item-574-JaknotWWWEFSVolume-fa9ce7d4f1f8c58a1700
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-mail-898-JaknotWWWEFSVolume-d8a4f1848bffa9dd0400
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-caf1d39a99a1b2e8fb01
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/231cc50e6fa2733fea405cb1268320293d771053bcfdee90b47579d1a5c1f41c/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/e4088ba132f5555f97819b7ba81b57e98804a9cb0df6ac804a52ff56db4eacb1/mounts/shm
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/1ac045f7e816faba9bfe1fe3be5372248e3c6403e04d1f83f1b097ecad5ee4a6/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/f7d9916e4e673e092dc38237ec087875203bcfec173f1396830e3b9af259647c/mounts/shm
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/c0dfdaf2a5a84b46ccb51c16bae86d6d55b4a25635ef9a11dd3b9a6b6b30ba42/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/fcd23b004d75ba09b354410b18457b67895125345c8c368dfc2a7c1bfe37e2bf/mounts/shm
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/01a508886c3ab27262a4c97b18ad18973263a9b12d62978ec7f01f68b90ca3f6/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/f24cc93b2ddfea5e185fbd34307d60ee73014591d1a359e20fcb5a363b13370b/mounts/shm
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-queue-restart-33-JaknotWWWEFSVolume-a8a4b2a6ffd0d3a7e201
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-c4b8a286c4c582c42e00
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-mail-898-JaknotWWWEFSVolume-ae80f2bbbff9aad15200
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-default-556-JaknotWWWEFSVolume-f2dfb980fdddb4c17800
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/e91596f65effd662dcb98ec3fc64bfc5732e55486f77740734896c9b0690dfa1/merged
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/7df07b64605cf5132108ca3edd53a5bfc06ea7db2316cdc47df1c82cb63b00ec/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/614fde4856fba699cedcaadc62d390646f1f5bb1d169d024a667b48203531134/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/898b814ea320022c92130e532b90ffe0b9d95cdd4f8ebb6b8d26c263955f0398/mounts/shm
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/36025469fdf9dbcbbe4335cab53d1d7b3a81019c4cf3c601eec0b054f82e32eb/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/11379bc4c60feb7bb064c0a48ef7e3e317769b10b68d34b70238a165c992560d/mounts/shm
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/f06b2de66a0c6e141b3afddac19d8b63f39759df3387f052c3e466f1206a4010/merged
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/93ddd3a42f9a1e932ae6c5f51f57c1a8ea1039993c8f67599ea11c5f4b43a7e0/merged
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/bbeadfe8d91e25806a11408d4a31dc7c491c0392130977ce9a9c28f71859b228/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/c00f9b85bc41ef757a370691af4cd0ef4dea46cf1bffaf5d5f2ad055fe4fc0bf/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/2be9644d0aeff4dd769665b9a9fd816322abf4b63c78855c052a88c4f3196e60/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/944bc8cc6fa155af480e6d4893378228a49f7c6984818946fbfeac3b41d73cf5/mounts/shm
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/35c3c37234dfc005d99bf88d0c340b78bf14e392ef8b325538731ae12fba7052/merged
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/a98f430e7a67d6f72fffcf70397e7120ec6723098dabad476ac576a209faa652/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/a60f4b5a5c8b8dfd4c4bb22ed078adaf871689fa94ba7a0519d990cd32e4795c/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/ebef2c20f9d369b80d8d4b1b14dc15083b969e755a74b31b2562fbc5d0dc8fbf/mounts/shm
127.0.0.1:/    9007199254739968 1916928 9007199252823040   1% /var/lib/ecs/volumes/ecs-www-worker-purchasing-omzet-item-574-JaknotWWWEFSVolume-f0bad692dccfb7970500
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/e24eabf03b964a97f9f7010c0d0ea5d675a513d428610f9c555ea520bc0e20d0/merged
overlay                30830568 5291064         25208568  18% /var/lib/docker/overlay2/e86259b5712ad4745251cdf1b39655a4c9c00e7236632973e3fb85a54d523e5d/merged
shm                       65536       0            65536   0% /var/lib/docker/containers/1b66a7a6b96a39d20cfeb60ce1fbfba1eed255ce428b740769c703c43a735767/mounts/shm
shm                       65536       0            65536   0% /var/lib/docker/containers/bf6283560b4b3d39d7360b6a4f97a4396a5972db0a026508f71bb4f78313fa13/mounts/shm
tmpfs                    200200       0           200200   0% /run/user/0

2. sudo docker ps -a

CONTAINER ID        IMAGE                                                                                                           COMMAND                  CREATED             STATUS                         PORTS                     NAMES
1b66a7a6b96a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-purchasing-omzet-item-574-php-cli-fef3a9cacdeae6ffb801
bf6283560b4b        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-purchasing-omzet-item-574-log-cloudwatch-f0d3e5b9d3b9b8f59201
6a6fbd11d730        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   5 minutes ago       Exited (0) 5 minutes ago                                   ecs-www-worker-purchasing-omzet-item-574-file-config-setup-d48783e5d993b5c50b00
ff3babfb42bf        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   5 minutes ago       Exited (0) 5 minutes ago                                   ecs-www-worker-purchasing-omzet-item-574-log-volume-setup-d4ede9abd28fd7e29501
a60f4b5a5c8b        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-mail-898-log-cloudwatch-a0abebae8486b8896c00
ebef2c20f9d3        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-mail-898-php-cli-9eecb791b1bfebf87300
2be9644d0aef        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-consume-jakpay-message-100-jakpay-cloudwatch-faa8ece49eb7eda4dc01
c00f9b85bc41        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-consume-jakpay-message-100-log-cloudwatch-f2cbc0d3ecf2ddb5a901
944bc8cc6fa1        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-consume-jakpay-message-100-php-cli-96a1b9a3f3889ad53d00
11379bc4c60f        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-default-556-php-cli-8c8981bff4cae9839201
614fde4856fb        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-default-556-jakpay-cloudwatch-c2adc2c0c7f893f90500
898b814ea320        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   5 minutes ago       Up 5 minutes                                               ecs-www-worker-default-556-log-cloudwatch-f4a09acebd94fbfff501
41a60c27402c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   6 minutes ago       Exited (0) 5 minutes ago                                   ecs-www-worker-default-556-file-config-setup-fcbdb4de8b81d1b0af01
2f1708adf700        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   6 minutes ago       Exited (0) 5 minutes ago                                   ecs-www-worker-mail-898-log-volume-setup-ea8e83ab8a9f87cecc01
4f251fd0634f        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   6 minutes ago       Exited (0) 5 minutes ago                                   ecs-www-worker-mail-898-file-config-setup-c699bae7ebf5f7cc3400
74a2af2a32aa        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   6 minutes ago       Exited (0) 5 minutes ago                                   ecs-www-worker-default-556-log-volume-setup-b495feeeba8fdadc1000
eb3b7ec464bd        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   6 minutes ago       Exited (0) 6 minutes ago                                   ecs-www-worker-consume-jakpay-message-100-file-config-setup-98baa3b5d89886894b00
7accabbe32a0        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   6 minutes ago       Exited (0) 6 minutes ago                                   ecs-www-worker-consume-jakpay-message-100-log-volume-setup-94bbfcc5f6e889f52d00
e516544af1a6        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release                                    "app-entrypoint php …"   6 minutes ago       Exited (0) 6 minutes ago                                   ecs-www-queue-restart-33-php-cli-e8f69998ecede4a12700
ff1fe350bbb5        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   6 minutes ago       Exited (137) 6 minutes ago                                 ecs-www-queue-restart-33-log-cloudwatch-96fc8ca6a7a389a15200
3975dde3410d        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   6 minutes ago       Exited (0) 6 minutes ago                                   ecs-www-queue-restart-33-file-config-setup-c4c9f9f1ff8bb7f0a401
2c0e9f0cb0c5        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   6 minutes ago       Exited (0) 6 minutes ago                                   ecs-www-queue-restart-33-log-volume-setup-f2d6f9d3cbb2f7b2c801
f24cc93b2ddf        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint php …"   13 minutes ago      Up 13 minutes                                              ecs-api-internal-worker-migration-199-php-cli-b4d2d7cead95adc27a00
fcd23b004d75        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   13 minutes ago      Up 13 minutes                                              ecs-api-internal-worker-migration-199-log-cloudwatch-a6adb7bcdbccb9c46900
f0d7e23620c3        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   13 minutes ago      Exited (0) 13 minutes ago                                  ecs-api-internal-worker-migration-199-file-config-setup-82daeea3afe58abdcf01
351647dff696        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   13 minutes ago      Exited (0) 13 minutes ago                                  ecs-api-internal-worker-migration-199-log-volume-setup-b6a8aea1f3c08cff9d01
f7d9916e4e67        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint php …"   13 minutes ago      Up 13 minutes                                              ecs-api-internal-worker-service-message-default-192-php-cli-9c818de6feb68fafc501
e4088ba132f5        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   13 minutes ago      Up 13 minutes                                              ecs-api-internal-worker-service-message-default-192-log-cloudwatch-c0c6f780afe29ce10900
0989d256ae4a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   13 minutes ago      Exited (0) 13 minutes ago                                  ecs-api-internal-worker-service-message-default-192-file-config-setup-80dcf6aba4aa94e3cd01
f55c354dbc9e        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   13 minutes ago      Exited (0) 13 minutes ago                                  ecs-api-internal-worker-service-message-default-192-log-volume-setup-98ef8ff0cd97d1814600
1fe503d2c38c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   About an hour ago   Exited (137) 6 minutes ago                                 ecs-www-worker-consume-jakpay-message-100-jakpay-cloudwatch-9e9efd87ebe1af936400
d25c53153740        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   About an hour ago   Exited (0) 6 minutes ago                                   ecs-www-worker-consume-jakpay-message-100-php-cli-c2a8a9898df6e0af6800
655181db5a82        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   About an hour ago   Exited (137) 6 minutes ago                                 ecs-www-worker-consume-jakpay-message-100-log-cloudwatch-bee99383fbf0f1c83f00
a46b1ac57264        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   About an hour ago   Exited (0) About an hour ago                               ecs-www-worker-consume-jakpay-message-100-file-config-setup-88cdcde38bd0f9c0eb01
d4769f7fe80d        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   About an hour ago   Exited (0) About an hour ago                               ecs-www-worker-consume-jakpay-message-100-log-volume-setup-e4b3a3c4c9e8d3801900
a9c5be328619        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   About an hour ago   Exited (0) 6 minutes ago                                   ecs-www-worker-mail-898-php-cli-e49ef2a98ad1f5c81f00
67b3d068c4da        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   About an hour ago   Exited (137) 6 minutes ago                                 ecs-www-worker-mail-898-log-cloudwatch-fefe9af2ba9abde6d001
6552b3f3f53a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   About an hour ago   Exited (0) About an hour ago                               ecs-www-worker-mail-898-file-config-setup-f4cea98e85efe6fd7100
c9a599111e3b        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   About an hour ago   Exited (0) About an hour ago                               ecs-www-worker-mail-898-log-volume-setup-9087c0c49782a2a89001
0c7b6df9df25        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint php …"   About an hour ago   Exited (0) 14 minutes ago                                  ecs-api-internal-worker-default-200-php-cli-b697aeb3b5999de58501
4e420f549bd4        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   About an hour ago   Exited (137) 14 minutes ago                                ecs-api-internal-worker-default-200-log-cloudwatch-8085d6edd5c6bbc6de01
690bc473cb1c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint php …"   About an hour ago   Exited (0) 14 minutes ago                                  ecs-api-internal-worker-migration-199-php-cli-acaafec7b7e1eaa8f901
1b4a604ecf4c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   About an hour ago   Exited (137) 14 minutes ago                                ecs-api-internal-worker-migration-199-log-cloudwatch-bad9cdcce7b5f4cb6e00
4cd606ae8e46        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   About an hour ago   Exited (0) About an hour ago                               ecs-api-internal-worker-default-200-file-config-setup-daf7bfe8a2d5e29b2f00
df390f7ffbde        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   About an hour ago   Exited (0) About an hour ago                               ecs-api-internal-worker-migration-199-log-volume-setup-8e8dbaabe5dca3fa7e00
6108a6191a75        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   About an hour ago   Exited (0) About an hour ago                               ecs-api-internal-worker-default-200-log-volume-setup-fac6c1d4b3dae9bed301
f71eda2899a8        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   About an hour ago   Exited (0) About an hour ago                               ecs-api-internal-worker-migration-199-file-config-setup-f2dbdec0d1e8aca76600
a176652ee022        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint php …"   About an hour ago   Exited (0) 14 minutes ago                                  ecs-api-internal-worker-service-message-default-192-php-cli-f4e39dd4e18fffb6a001
0f19bdc7d995        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   About an hour ago   Exited (137) 14 minutes ago                                ecs-api-internal-worker-service-message-default-192-log-cloudwatch-dca592a8e78e8fe68b01
a5ca6d2d4605        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   About an hour ago   Exited (0) About an hour ago                               ecs-api-internal-worker-service-message-default-192-file-config-setup-d689fd86b889aafa9c01
b396265e7abb        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   About an hour ago   Exited (0) About an hour ago                               ecs-api-internal-worker-service-message-default-192-log-volume-setup-fcdbafccc3cbc5d07b00
e76700749025        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www-cron/php-cli/release:4.0797f81-40.bddabd4          "app-entrypoint php …"   2 hours ago         Exited (0) 39 minutes ago                                  ecs-www-cron-worker-default-65-php-cli-9ec3fdefd3a2f2cc4a00
4c3e27a46394        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) 39 minutes ago                                ecs-www-cron-worker-default-65-log-cloudwatch-ae96b8cfc8f28abdf901
47f54291641c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-cron-worker-default-65-log-volume-setup-e8a4f0cca1fea6842200
620034ce35c6        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-www-worker-purchasing-omzet-item-574-log-cloudwatch-968adea3cd86b1a83000
80f2b7883b82        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   2 hours ago         Exited (0) About an hour ago                               ecs-www-worker-purchasing-omzet-item-574-php-cli-c0fcf6a88096acb95c00
f60f566a02dc        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   2 hours ago         Exited (0) About an hour ago                               ecs-www-worker-mail-898-php-cli-d4f890d6b9dfc38f0600
1c9bba9550fa        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   2 hours ago         Exited (0) About an hour ago                               ecs-www-worker-consume-jakpay-message-100-php-cli-aee9dbaee6db9794be01
176b2109214f        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-www-worker-consume-jakpay-message-100-jakpay-cloudwatch-b286989ec6e7919d4900
ac26774ab23a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-www-worker-consume-jakpay-message-100-log-cloudwatch-96baadb4c4bd90c62900
e6043a35d7eb        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-www-worker-mail-898-log-cloudwatch-c2949af6cb97e3da1100
f7d5e1ffb5b2        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   2 hours ago         Exited (0) About an hour ago                               ecs-www-worker-finance-auto-approve-556-php-cli-c4829dfaa1e7aff3be01
28df6b2b8574        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-www-worker-finance-auto-approve-556-jakpay-cloudwatch-9cb9e5a09ceff383c301
61baaf934a3c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-www-worker-finance-auto-approve-556-log-cloudwatch-9cf7eba1aca3c9dd4d00
05e30c96a249        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-purchasing-omzet-item-574-file-config-setup-f69b93eccac8fbb8d101
a63e693cec5c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-purchasing-omzet-item-574-log-volume-setup-e2dbbef2f894c7ffe701
fa72bb315af9        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-consume-jakpay-message-100-file-config-setup-f49ae98ac0d0c0c08701
72d4acf6ec5b        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-mail-898-file-config-setup-98c7d1b792bfada8e101
1ca4e60c167d        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-consume-jakpay-message-100-log-volume-setup-e4c89ec5bbf0e9900700
858aee7f6b9f        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-mail-898-log-volume-setup-b4fd9d9fa3d8e5cfd901
56526aacfb7a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   2 hours ago         Exited (0) About an hour ago                               ecs-www-worker-mail-priority-802-php-cli-80f8aef796a2df8b5000
789aaa4e3352        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-www-worker-mail-priority-802-log-cloudwatch-a29b9fa189a9f7ed2500
ffcb2be4453d        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-finance-auto-approve-556-file-config-setup-fce582a6ecbdcf878501
a0876a2540ec        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-mail-priority-802-file-config-setup-8685d7c697c5e8d5f901
70fc93b88cdb        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-finance-auto-approve-556-log-volume-setup-968bd189f0e787f33600
1d315f958403        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-mail-priority-802-log-volume-setup-beb5a8d8cae8aac02400
9cb81ccf97cb        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/nginx/release:20.52150a5-818.e3f8a27               "app-entrypoint ngin…"   2 hours ago         Up 2 hours                       0.0.0.0:32783->80/tcp     ecs-www-web-959-nginx-c4ad858afba4c186cc01
d528451ad577        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-fpm/release:9.db2dfaa-15.70778a2-818.e3f8a27   "app-entrypoint php-…"   2 hours ago         Up 2 hours                       9000/tcp                  ecs-www-web-959-php-fpm-9e9af680b9d4af9ccc01
52b62eb72717        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Up 2 hours                                                 ecs-www-web-959-jakpay-cloudwatch-80e4fe9ffcafb8ed9701
be417ec70507        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Up 2 hours                                                 ecs-www-web-959-log-cloudwatch-e8e9c9c5c28be6eb7f00
62b8e5e000a6        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Up 2 hours                                                 ecs-www-web-959-slow-cloudwatch-aae0d58fa388f1f84a00
722c7483ceb3        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/newrelic/php-daemon:9.11.0                                    "sh /usr/bin/docker-…"   2 hours ago         Up 2 hours                       31339/tcp                 ecs-www-web-959-newrelic-agent-a284e280cdc7deafa701
ccd0f793b4a2        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-web-959-log-volume-setup-86b2c38994baa5865000
a24d34c9e96c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-web-959-file-config-setup-b4d9afe69def8efc5b00
36d8d5ff7a66        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-818.e3f8a27              "app-entrypoint ./ci…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-default-556-php-cli-ac9fdad8da8bdac16d00
1094cc599a32        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-default-556-jakpay-cloudwatch-d2e0f5eaa8b0c5bd2400
3095d34d9865        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-default-556-log-cloudwatch-fa86affcd5eb93bcea01
7b7d742b704f        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-default-556-file-config-setup-f8e5fa81caf088b18e01
5006c4cbae39        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-default-556-log-volume-setup-96a1f3cdaa98fad32c00
29ccce1da103        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint php …"   2 hours ago         Exited (0) About an hour ago                               ecs-api-internal-worker-migration-199-php-cli-d691ecb282f9d187dc01
982078364bdc        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-api-internal-worker-migration-199-log-cloudwatch-a4b3ffe5d68fc2b90d00
a28f300cd8f9        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 hours ago         Exited (137) About an hour ago                             ecs-api-internal-worker-service-message-default-192-log-cloudwatch-f6c6a6d4a6c3e2d37f00
3e2043cdc129        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint php …"   2 hours ago         Exited (0) About an hour ago                               ecs-api-internal-worker-service-message-default-192-php-cli-84b4d6b99783c7f8c901
6eb1d9331c98        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-api-internal-worker-migration-199-log-volume-setup-94f9b49da9cedad28c01
a0c902ec482e        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-api-internal-worker-migration-199-file-config-setup-f6978bc4b7fc808fee01
3b2d7215462a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-api-internal-worker-service-message-default-192-log-volume-setup-8aa9b68c81d1b1f71900
ede49dce7137        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   2 hours ago         Exited (0) 2 hours ago                                     ecs-api-internal-worker-service-message-default-192-file-config-setup-b8d2d998c08efbcc2900
dea45b058d6c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-817.4de33d1              "app-entrypoint ./ci…"   3 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-invoice-473-php-cli-cedae98a8fbcc59a7100
ccf55d320274        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   3 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-invoice-473-log-cloudwatch-cab5eead97e891fa4d00
781b6b2c5838        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   3 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-invoice-473-jakpay-cloudwatch-8ae6a1bbedb095d60b00
a44dad8dcf60        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-817.4de33d1              "app-entrypoint ./ci…"   3 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-purchasing-omzet-item-573-php-cli-94bd8eb2f9e49898b201
adade2a7c414        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   3 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-purchasing-omzet-item-573-log-cloudwatch-f6baa7d0f2fec095d901
223eeff1044d        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-817.4de33d1              "app-entrypoint ./ci…"   3 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-finance-auto-approve-555-php-cli-d6fc9db7b6ac9fba6f00
5d9ae404f602        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   3 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-finance-auto-approve-555-jakpay-cloudwatch-80968d9dfac88ca30c00
b5803e8e654f        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   3 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-finance-auto-approve-555-log-cloudwatch-c2d398fbb1bcefffb701
f9cdb9981d4f        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-www-worker-invoice-473-file-config-setup-90edd59fe8f4e6c96000
b06e164bf295        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-www-worker-invoice-473-log-volume-setup-8689bee387a4e8b8c201
bf63047878cd        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-cli/release:8.28e758d-817.4de33d1              "app-entrypoint ./ci…"   3 hours ago         Exited (0) 2 hours ago                                     ecs-www-worker-consume-jakpay-message-99-php-cli-e2efdbcfd493f6eff701
65c24fc35d07        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   3 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-consume-jakpay-message-99-log-cloudwatch-c8dda4d592e8ebc1fe01
ddd7f5dba0ea        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   3 hours ago         Exited (137) 2 hours ago                                   ecs-www-worker-consume-jakpay-message-99-jakpay-cloudwatch-8ae48b8d95e7eaa45100
ee5ee82de23e        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-www-worker-purchasing-omzet-item-573-file-config-setup-82d6f79dd39d86e57500
f630a59d460c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-www-worker-purchasing-omzet-item-573-log-volume-setup-f2f5c6afb2cbcba6d801
2f998dbcfede        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-www-worker-finance-auto-approve-555-log-volume-setup-b0a8df93c0e9c8910100
8697b60c9405        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-www-worker-finance-auto-approve-555-file-config-setup-acef84bff6edf7c3aa01
a365a44a362c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-www-worker-consume-jakpay-message-99-log-volume-setup-b4daee9abdd6c5944900
69dcf64d41ce        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-www-worker-consume-jakpay-message-99-file-config-setup-90cae8a38ffda7e1dd01
75352ab1702c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   3 hours ago         Exited (137) 2 hours ago                                   ecs-api-internal-worker-migration-199-log-cloudwatch-caefc89cd5b4eac41a00
c8351d1ade1a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint php …"   3 hours ago         Exited (0) 2 hours ago                                     ecs-api-internal-worker-migration-199-php-cli-f6daf9e1f9819392c301
d2d82b1ee97b        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-api-internal-worker-migration-199-log-volume-setup-9cb4c9b8c9f99acff801
e876e1e66b94        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   3 hours ago         Exited (0) 3 hours ago                                     ecs-api-internal-worker-migration-199-file-config-setup-808ff5fdfac0de9b0b00
7d6607af2cb7        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www-cron/php-cli/release:4.0797f81-40.bddabd4          "app-entrypoint php …"   4 hours ago         Exited (0) 3 hours ago                                     ecs-www-cron-worker-default-65-php-cli-92bb93b9d792bfa82500
e176efc14e34        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   4 hours ago         Exited (137) 3 hours ago                                   ecs-www-cron-worker-default-65-log-cloudwatch-d2f0dfe7d9d9eb880f00
73d95bb59d47        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   4 hours ago         Exited (0) 4 hours ago                                     ecs-www-cron-worker-default-65-log-volume-setup-96d8d6aeddbcf6fc9c01
06dd7257f453        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/nginx/release:20.52150a5-817.4de33d1               "app-entrypoint ngin…"   2 days ago          Exited (0) 2 hours ago                                     ecs-www-web-958-nginx-80bee3a0b8aac38a1100
40e54b995c68        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www/php-fpm/release:9.db2dfaa-15.70778a2-817.4de33d1   "app-entrypoint php-…"   2 days ago          Exited (0) 2 hours ago                                     ecs-www-web-958-php-fpm-a4eac6e9ba9ebab16500
d8569aa9d51a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 days ago          Exited (137) 2 hours ago                                   ecs-www-web-958-log-cloudwatch-90819aa5e4fed6feb101
09799505464b        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 days ago          Exited (137) 2 hours ago                                   ecs-www-web-958-slow-cloudwatch-fe9cd9dda0c5d2aee601
554bdd016c79        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   2 days ago          Exited (137) 2 hours ago                                   ecs-www-web-958-jakpay-cloudwatch-b481b2fbbee09ac9a001
2c9fd84b0db0        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/newrelic/php-daemon:9.11.0                                    "sh /usr/bin/docker-…"   2 days ago          Exited (0) 2 hours ago                                     ecs-www-web-958-newrelic-agent-b8a5c5a6faf2c2f97d00
70408fd029fb        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $A…"   2 days ago          Exited (0) 2 days ago                                      ecs-www-web-958-file-config-setup-cad98599a39fe8a5f601
bddcd53ba72a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   2 days ago          Exited (0) 2 days ago                                      ecs-www-web-958-log-volume-setup-e0978fe3dfe9f7c01800
25bd5232ba12        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot-ui/next/release:16.5b97c45                             "docker-entrypoint.s…"   3 days ago          Up 3 days                        0.0.0.0:32780->3000/tcp   ecs-jaknot-ui_web_next-6-next-b0949ba7bbd1e8bd7d00
7bcadeb2df3d        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   4 days ago          Up 4 days                                                  ecs-jakpay-worker-default-43-log-cloudwatch-8ace81a4a0849db70a00
c8c9b4f34155        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/jakpay/php-cli/release                                 "app-entrypoint php …"   4 days ago          Up 4 days                                                  ecs-jakpay-worker-default-43-php-cli-f2899bebd1c597ff5f00
93660622fbd0        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   4 days ago          Exited (0) 4 days ago                                      ecs-jakpay-worker-default-43-log-volume-setup-8298d5acf0da82bff801
aa4d4a897d43        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www-node/nginx/base                                    "app-entrypoint ngin…"   4 days ago          Up 4 days                        0.0.0.0:32771->80/tcp     ecs-www-node-49-nginx-868387c5e49fa9890800
538a752807d7        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/www-node/node-socket-io/release                        "docker-entrypoint.s…"   4 days ago          Up 4 days                                                  ecs-www-node-49-node-8ab9b6edb98eb0e49d01
dc10029163bf        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/api-internal/php-cli/release:6.e22c54a-339.ed33c27     "app-entrypoint /bin…"   4 days ago          Up 4 days                                                  ecs-api-internal-scheduler-359-php-cli-d4ecdbbdd5c5ed8ea501
b59497507206        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   4 days ago          Up 4 days                                                  ecs-api-internal-scheduler-359-log-cloudwatch-e8b282d8afe8db8fea01
65ec9cc4a9de        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   4 days ago          Exited (0) 4 days ago                                      ecs-api-internal-scheduler-359-log-volume-setup-ded2bed2e5e5b6ea6800
7173b029bd1c        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $D…"   4 days ago          Exited (0) 4 days ago                                      ecs-api-internal-scheduler-359-file-config-setup-9e98dbbda8d982850b00
20323267939f        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/grabber/php-cli/release                                "app-entrypoint php …"   4 days ago          Up 4 days                                                  ecs-grabber-worker-service-message-default-10-php-cli-88848e81a5a2f2810600
03e0293695d8        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/jakpay/php-cli/release                                 "app-entrypoint php …"   4 days ago          Up 4 days                                                  ecs-jakpay-worker-www-message-handler-40-php-cli-c0f6d3d187a1ccbe3a00
023123f145bd        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/jakpay/nginx/base                                      "/docker-entrypoint.…"   4 days ago          Up 4 days                        0.0.0.0:32770->80/tcp     ecs-jakpay-web-41-nginx-9c9dcfaa83ed8696dc01
a81ea60e34a1        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/pickup-point/node/release                              "docker-entrypoint.s…"   4 days ago          Up 4 days                        0.0.0.0:32769->8080/tcp   ecs-pickup-point-socket-io-16-node-e49aceacb3aadc975c00
1df4dfc1bd20        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/jaknot/jakpay/php-fpm/release                                 "app-entrypoint php-…"   4 days ago          Up 4 days                        9000/tcp                  ecs-jakpay-web-41-php-fpm-90c6b482bca89beec501
acc825f17c21        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   4 days ago          Up 4 days                                                  ecs-grabber-worker-service-message-default-10-log-cloudwatch-aad88785d8a2e3cf2000
7cd1b4a31ee0        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   4 days ago          Up 4 days                                                  ecs-jakpay-worker-www-message-handler-40-log-cloudwatch-b0ccaf86dae2ecd42b00
d7a4fd2d36f6        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'echo $F…"   4 days ago          Exited (0) 4 days ago                                      ecs-grabber-worker-service-message-default-10-fortuna-public-key-setup-f6c0b6c98580db854000
d567117af892        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   4 days ago          Exited (0) 4 days ago                                      ecs-grabber-worker-service-message-default-10-log-volume-setup-b4a89de8febd85f72c00
3de5f9b3bac2        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   4 days ago          Exited (0) 4 days ago                                      ecs-jakpay-worker-www-message-handler-40-log-volume-setup-e8bcbca8bceb8af56e00
72b7e429f091        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "tail -f -n+1 /var/l…"   4 days ago          Up 4 days                                                  ecs-jakpay-web-41-log-cloudwatch-8e84c8b0fcca80db9e01
0f8296428aa1        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/busybox:1.31.1                                                "/bin/sh -c 'touch /…"   4 days ago          Exited (0) 4 days ago                                      ecs-jakpay-web-41-log-volume-setup-b89bc7f5f9f281cb3300
5bb85db70c7a        598080467453.dkr.ecr.ap-southeast-1.amazonaws.com/newrelic/php-daemon:9.11.0                                    "sh /usr/bin/docker-…"   4 days ago          Up 4 days                        31339/tcp                 ecs-jakpay-web-41-newrelic-agent-a2c082d4a6f1bad50500
943727c82144        amazon/amazon-ecs-agent:latest                                                                                  "/agent"                 4 days ago          Up 4 days (healthy)                                        ecs-agent

3. sudo ls /var/run/efs

s-b27ee4f3.var.lib.ecs.volumes.ecs-www-queue-restart-33-JaknotWWWEFSVolume-a8a4b2a6ffd0d3a7e201.20337
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-queue-restart-33-JaknotWWWEFSVolume-a8a4b2a6ffd0d3a7e201.20337+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-web-958-JaknotWWWEFSVolume-e8d188d3f186f3bbb701.20371
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-web-958-JaknotWWWEFSVolume-e8d188d3f186f3bbb701.20371+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-web-959-JaknotWWWEFSVolume-d0c4aafae9bdee84f501.20224
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-web-959-JaknotWWWEFSVolume-d0c4aafae9bdee84f501.20224+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-ae91a6c9abc49ecd4500.20258
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-ae91a6c9abc49ecd4500.20258+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-c4b8a286c4c582c42e00.20394
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-c4b8a286c4c582c42e00.20394+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-caf1d39a99a1b2e8fb01.20119
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-caf1d39a99a1b2e8fb01.20119+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-99-JaknotWWWEFSVolume-d8be9ee6e389f1e0f601.20161
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-99-JaknotWWWEFSVolume-d8be9ee6e389f1e0f601.20161+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-default-556-JaknotWWWEFSVolume-f2dfb980fdddb4c17800.20276
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-default-556-JaknotWWWEFSVolume-f2dfb980fdddb4c17800.20276+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-default-556-JaknotWWWEFSVolume-fcfca7aa87dad7a93400.20336
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-default-556-JaknotWWWEFSVolume-fcfca7aa87dad7a93400.20336+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-finance-auto-approve-555-JaknotWWWEFSVolume-e6dcecbdf891ebaa1f00.20245
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-finance-auto-approve-555-JaknotWWWEFSVolume-e6dcecbdf891ebaa1f00.20245+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-finance-auto-approve-556-JaknotWWWEFSVolume-8efdbdfea8a795d7a001.20172
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-finance-auto-approve-556-JaknotWWWEFSVolume-8efdbdfea8a795d7a001.20172+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-invoice-473-JaknotWWWEFSVolume-f6d4cfc38fc9ac9b6a00.20246
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-invoice-473-JaknotWWWEFSVolume-f6d4cfc38fc9ac9b6a00.20246+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-ae80f2bbbff9aad15200.20395
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-ae80f2bbbff9aad15200.20395+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-d8a4f1848bffa9dd0400.20332
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-d8a4f1848bffa9dd0400.20332+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-f8b0dfbc8e888dafa501.20356
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-f8b0dfbc8e888dafa501.20356+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-priority-802-JaknotWWWEFSVolume-bef884d3bb8fa1cf4000.20146
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-priority-802-JaknotWWWEFSVolume-bef884d3bb8fa1cf4000.20146+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-573-JaknotWWWEFSVolume-9ad1dbe0fbf5c5fd5300.20208
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-573-JaknotWWWEFSVolume-9ad1dbe0fbf5c5fd5300.20208+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-574-JaknotWWWEFSVolume-f0bad692dccfb7970500.20061
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-574-JaknotWWWEFSVolume-f0bad692dccfb7970500.20061+
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-574-JaknotWWWEFSVolume-fa9ce7d4f1f8c58a1700.20439
fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-574-JaknotWWWEFSVolume-fa9ce7d4f1f8c58a1700.20439+
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-queue-restart-33-JaknotWWWEFSVolume-a8a4b2a6ffd0d3a7e201.20337
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-web-958-JaknotWWWEFSVolume-e8d188d3f186f3bbb701.20371
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-web-959-JaknotWWWEFSVolume-d0c4aafae9bdee84f501.20224
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-ae91a6c9abc49ecd4500.20258
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-c4b8a286c4c582c42e00.20394
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-100-JaknotWWWEFSVolume-caf1d39a99a1b2e8fb01.20119
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-consume-jakpay-message-99-JaknotWWWEFSVolume-d8be9ee6e389f1e0f601.20161
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-default-556-JaknotWWWEFSVolume-f2dfb980fdddb4c17800.20276
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-default-556-JaknotWWWEFSVolume-fcfca7aa87dad7a93400.20336
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-finance-auto-approve-555-JaknotWWWEFSVolume-e6dcecbdf891ebaa1f00.20245
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-finance-auto-approve-556-JaknotWWWEFSVolume-8efdbdfea8a795d7a001.20172
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-invoice-473-JaknotWWWEFSVolume-f6d4cfc38fc9ac9b6a00.20246
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-ae80f2bbbff9aad15200.20395
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-d8a4f1848bffa9dd0400.20332
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-898-JaknotWWWEFSVolume-f8b0dfbc8e888dafa501.20356
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-mail-priority-802-JaknotWWWEFSVolume-bef884d3bb8fa1cf4000.20146
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-573-JaknotWWWEFSVolume-9ad1dbe0fbf5c5fd5300.20208
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-574-JaknotWWWEFSVolume-f0bad692dccfb7970500.20061
stunnel-config.fs-b27ee4f3.var.lib.ecs.volumes.ecs-www-worker-purchasing-omzet-item-574-JaknotWWWEFSVolume-fa9ce7d4f1f8c58a1700.20439

4. mountPoints

      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/mnt/efs",
          "sourceVolume": "JaknotWWWEFSVolume"
        }

5. I'm not sure the error only happened if there's killed container or not, but from result no 2. yes there's some container killed because of regularly restart scheduler or new pipeline deployment

[Cappuccinuo]

Thanks for all the info. I opened a issue on ecs agent side for further investigation/fix.

Currently the root cause is that The EFS volume is not umounted after the task & container is killed, and due to the iam role credentials is fetched in the container ENV itself, after the task is killed, the credentials link is invalid which cause the credentials retrieval failed. The expected behaviour should be after the task / container is killed, the volume resource should be cleaned up.

The temporary workaround now will be manually umount the volumes on host, so that the EFS watchdog can cleanup the unused volumes stunnels.