ECS agent should handle EFS volume umount after a task container is killed #2810
Labels
Comments
|
hello, please run the ecs logs collector: https://github.com/aws/amazon-ecs-logs-collector, you can email the bundle to ecs-agent-external at amazon.com Also to clarify, what do you mean by "the task container is killed"? are you stopping the task via the ECS API? or killing the container directly on the instance via |
|
From the ECS console, I believe that is through ECS API right. Before I collect more logs, can you confirm that EFS volume should be umounted after the task is stopped ? |
|
to follow up, it appears that they ecs agent does unmount, but not until the task is cleaned up. The waiting time between a task stopping and being cleaned up is controlled by the env var |
|
Hey @sparrc , Thanks for the suggestion, could you please confirm that my ENV setting is correct? (I am following https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html) $ cat /etc/ecs/ecs.config
ECS_CLUSTER=Github83
ECS_BACKEND_HOST=
ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=1mDo I need to restart the agent to make this effective? Seems after this config, nothing changed, after the task is stopped on the ECS console, the issue is still persist after 1min due to the file system is not umounted. # The time when I trigger task stop
level=info time=2021-02-17T02:09:44Z msg="Managed task has reached stopped; waiting for container cleanup" taskARN="arn:aws:ecs:us-east-1:617566850839:task/Github83/8fee1df8feb7433dba149dd6dd702cf0"
# After 1m+
level=error time=2021-02-17T02:13:19Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
# The file system is still mounted
$ date -u
Wed Feb 17 02:15:35 UTC 2021
$ sudo df
Filesystem 1K-blocks Used Available Use% Mounted on
...
127.0.0.1:/ 9007199254739968 4194304 9007199250545664 1% /var/lib/ecs/volumes/ecs-Github83-2-EFS-8cf0e0a597df80c6a101 |
|
yes you will need you restart the ecs agent to pickup config changes: |
|
Thanks, confirmed that after restart, the file system is umounted and this issue is not happening anymore. level=info time=2021-02-17T04:55:26Z msg="Container change is redundant" taskARN="arn:aws:ecs:us-east-1:617566850839:task/Github83/8a8df65a0a204f7e82f7a45d36635efe" container="nginx" runtimeID="16b1aea51f66d47099369f65d96aea668e7cb96735b6b70e4f96185246d5b870" status="STOPPED" knownStatus="STOPPED"
level=info time=2021-02-17T04:55:26Z msg="Waiting for task event" taskARN="arn:aws:ecs:us-east-1:617566850839:task/Github83/8a8df65a0a204f7e82f7a45d36635efe"
level=error time=2021-02-17T04:56:09Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=warn time=2021-02-17T04:56:09Z msg="Unknown eventType: GetCredentialsInvalidRoleType" module=entry_types.go
...
level=info time=2021-02-17T04:56:26Z msg="Waiting for task event" taskARN="arn:aws:ecs:us-east-1:617566850839:task/Github83/8a8df65a0a204f7e82f7a45d36635efe"
level=info time=2021-02-17T04:56:26Z msg="Cleaning up task's containers and data" taskARN="arn:aws:ecs:us-east-1:617566850839:task/Github83/8a8df65a0a204f7e82f7a45d36635efe"
level=info time=2021-02-17T04:56:26Z msg="Task engine [arn:aws:ecs:us-east-1:617566850839:task/Github83/8a8df65a0a204f7e82f7a45d36635efe]: removing container: nginx" module=docker_task_engine.go
level=info time=2021-02-17T04:56:26Z msg="Removing Container Reference: nginx from Image State- sha256:298ec0e28760b8eb1aad79711dc29c19041c61d7cf342dd1f445e91f30500549" module=types.go
level=info time=2021-02-17T04:56:26Z msg="Task engine [arn:aws:ecs:us-east-1:617566850839:task/Github83/8a8df65a0a204f7e82f7a45d36635efe]: resource cgroup cleanup complete" module=docker_task_engine.go
level=info time=2021-02-17T04:56:26Z msg="Task engine [arn:aws:ecs:us-east-1:617566850839:task/Github83/8a8df65a0a204f7e82f7a45d36635efe]: resource EFS cleanup complete" module=docker_task_engine.go
level=info time=2021-02-17T04:56:26Z msg="Task engine [arn:aws:ecs:us-east-1:617566850839:task/Github83/8a8df65a0a204f7e82f7a45d36635efe]: finished removing task data, removing task from managed tasks" module=docker_task_engine.goAre there any method else that we can config the agent instead of manually edit the config file? e.g. Can we config this in the task definition or cluster definition? |
|
You can set the environment variable using userdata for the EC2 instance, you can find more info here -- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bootstrap_container_instance.html#bootstrap_container_agent |
|
Hi @Cappuccinuo @shubham2892 thanks for updating the issue Is setting up I see many errors printed and also at |
|
@xian13 I think setting |
|
@Cappuccinuo in the previous log I describe setting Lets say we put the value to 30m, there's a window of 30m error will be printed to the log Shouldn't either ECS / EFS aware that if the volume is no longer use, then no need to get the credentials? |
|
@xian13 , can you share your efs-utils config file mount-watchdog part and confirm that the renewal interval min is larger than [mount-watchdog]
...
# Set client auth/access point certificate renewal rate. Minimum value is 1 minute.
tls_cert_renewal_interval_min = 60If you set the On EFS side, we tell whether the volume is |
|
Hi @Cappuccinuo here is the config file I'm using the AWS ECS optimized AMI and not changing anything for the efs utils |
|
@xian13 Did you restart the ecs agent? And when the task is started and killed? |
|
@Cappuccinuo I insert So technically it's a new instance and no need to restart it manually |
|
@xian13 , if the agent is start before the cloud-init, then you still need to restart the agent right? Can you confirm that the config is updated before ecs agent running? Say at 00:00 you launch a new instance with the new user data -> at 00:05 the task is running and the file system is mounted -> at 00:10 the task is terminated -> at 01:10 the watchdog is printing the error log, this should be the clean up config is not effective. |
|
@Cappuccinuo Because I'm using AWS ECS Optimized Instance, isn't the flow which one cloud-ini and agent started should be already taken care of by AWS team? https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bootstrap_container_instance.html |
|
@xian13 , are you saying that after applying the ecs agent config, the issue is not mitigated at all? Or there are only errors during the 1m period? How many tasks are seeing such errors? Based on your efs-utils config file, efs-utils watchdog only refresh the tls certificate after 60 min. From my tests on a ECS agent that is restarted after the config, what I see from the log is # Triggering task termination
2021-02-23 08:52:14,659 - DEBUG - Current local NFS mounts: [Mount(server='127.0.0.1:/', mountpoint='/var/lib/ecs/volumes/ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00', type='nfs4', options='rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,port=20067,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1', freq='0', passno='0')]
2021-02-23 08:52:14,659 - DEBUG - Translating "fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067" into mount point and port "var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067"
2021-02-23 08:52:14,659 - DEBUG - Current state files in "/var/run/efs": ['fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067']
2021-02-23 08:52:14,659 - DEBUG - TLS tunnel for fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067 is running
2021-02-23 08:52:15,660 - DEBUG - Current local NFS mounts: []
2021-02-23 08:52:15,661 - DEBUG - Translating "fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067" into mount point and port "var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067"
2021-02-23 08:52:15,661 - DEBUG - Current state files in "/var/run/efs": ['fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067']
2021-02-23 08:52:15,661 - INFO - No mount found for "fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067"
2021-02-23 08:52:15,661 - DEBUG - Marking fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067 as unmounted at 1614070335
...
2021-02-23 08:52:45,715 - INFO - Unmount grace period expired for fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067
2021-02-23 08:52:45,715 - INFO - Terminating running TLS tunnel - PID: 16086, group ID: 16086
2021-02-23 08:52:45,716 - INFO - TLS tunnel: 16086 is still running, will retry termination
2021-02-23 08:52:46,717 - DEBUG - Current local NFS mounts: []
2021-02-23 08:52:46,717 - DEBUG - Translating "fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067" into mount point and port "var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067"
2021-02-23 08:52:46,717 - DEBUG - Current state files in "/var/run/efs": ['fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067']
2021-02-23 08:52:46,717 - INFO - Unmount grace period expired for fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067
2021-02-23 08:52:46,717 - INFO - TLS tunnel: 16086 is no longer running, cleaning up state
2021-02-23 08:52:46,717 - DEBUG - Deleting /var/run/efs/stunnel-config.fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067
2021-02-23 08:52:46,718 - DEBUG - Deleted /var/run/efs/stunnel-config.fs-*.var.lib.ecs.volumes.ecs-Github83-2-EFS-f081fa8a8af3bcdc2f00.20067
2021-02-23 08:52:47,719 - DEBUG - Current local NFS mounts: []
2021-02-23 08:52:47,719 - DEBUG - Current state files in "/var/run/efs": [] |
|
Since @xian13 said the errors occurred for about a minute, it does sound like the ECS cleanup wait duration is taking effect. I'm curious if the polling intervals configured in the watchdog have any effect on still seeing the errors. In particular Im curious about poll_interval_sec since you appear to be getting the errors about once every second til the task is cleaned up: |
|
Sorry for late reply @Cappuccinuo Yes, I'm seeing the error in 1 minute window Here is the example again: The same error also printed in @sparrc is your suggestion to change the value of |
|
@xian13 Can you please provide time of the task start and stop? As @Cappuccinuo mentioned earlier the certs renewal should not happen for 60 min in this case. |
|
Hi @mythri-garaga can you guide me how to get the task start and stop time from the given error log on |
|
@xian13 Can you share the logs at ecs-agent-external@amazon.com. |
|
Hi @shubham2892 , can you give me more specific which logs to be shared? Last request by @mythri-garaga is the start and stop time of the task, |
|
I dont think we need any more logs for this case at the moment, but I believe shubham2892@ was asking for you to run the ecs logs collector @xian13 yes please try adjusting |
|
@sparrc is it like increasing |
Yes, so that we can observe logs when mount-watchdog does not request credentials during that time frame. |
|
@xian13 any updates? |
|
Hi @sparrc sorry for late response Today I just update the I run this commands on the instances cat /var/log/amazon/efs/mount-watchdog.log | grep "Failed to retrieve AWS security"
cat /var/log/ecs/ecs-agent.log | grep "Credentials not found"Until now there is no error log printed, will update this again later after more hours So for now this is the only solution? As from my point of view, it try to removing the log by trading it with other consequences
|
|
Latest update I still see some errors in |
|
It turns out the error still printed on Here is the example sh-4.2$ cat /var/log/amazon/efs/mount-watchdog.log | grep "Failed to retrieve AWS security"
2021-05-18 07:38:42,298 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/ceefc713-abe8-4197-bce9-0779f1cf562b
2021-05-18 10:38:50,118 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/fb43239b-f3f2-4e06-8263-b1384a4dc848
2021-05-18 11:38:53,006 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/63b8ec75-8749-4ce5-bec3-c8f9e52cdb29
2021-05-18 11:38:53,014 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/73070725-4678-414a-baf1-e64363c7edaa
2021-05-18 12:38:55,575 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/d648ffa0-0ec6-4894-8ad1-a9ccfe8b9805
2021-05-18 12:38:55,581 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/ad954f82-0f36-4bfd-8d67-d1fe2f265756
2021-05-18 12:38:55,588 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/583ba59a-8124-4862-bcb7-4cb8350da462
2021-05-18 13:38:58,278 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/0b0a8308-f9a9-4f3c-83c7-f89651c27e5d
2021-05-18 13:38:58,287 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/7883ee3e-ffc1-4c5c-ad2f-d7fd0fee61b3
2021-05-18 13:38:58,294 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/161d8a16-27b6-44c8-8247-8cff17543c98
2021-05-18 13:38:58,300 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/a8c0ab8d-294c-4115-8d1d-cd1f4b4a7d17
2021-05-18 14:39:01,244 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/23223832-c891-4422-b407-587cdef8de37
2021-05-18 14:39:01,298 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/b651a019-92ae-49bc-81e1-479b1f026671
2021-05-18 14:39:01,344 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/d48fda30-6497-4aba-875b-79e7b0ad5262
2021-05-18 15:39:04,177 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/d5556b82-552f-4b52-888e-e37d561ab86f
2021-05-18 15:39:04,183 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/df39b653-0477-420a-8e32-92fe11273b61
2021-05-18 15:39:04,190 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/c0aeea59-ade2-4e00-bbc9-3ce4aa8a02e6
2021-05-18 15:39:04,197 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/69ff6cfb-3d12-4312-af03-89508d093ba1
2021-05-18 15:39:04,204 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/723b0139-a525-4251-b75e-3ca15d5e83d1
2021-05-18 16:39:06,873 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/61d4cc26-caf1-45a0-a6c6-fd58a41764ba
2021-05-18 16:39:06,880 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/a5dc254e-4a0b-4cfd-ae59-6c08cb3a3f1b
2021-05-18 16:39:06,888 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/193d10bb-86ca-4928-a181-9d81833d5fd3
2021-05-18 16:39:06,895 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/26956789-39c9-4c21-8f1f-a33e5151d9c9
2021-05-18 17:39:09,546 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/18480f9c-2ca4-4be7-92e7-47070e78358e
2021-05-18 17:39:09,553 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/c4d5b6e4-b51d-4ca9-bb5f-c58aaba912bb
2021-05-18 17:39:09,559 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/2b13eb0c-4635-4236-87bf-ce804ca92d00
2021-05-18 17:39:09,566 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/754d1697-8df6-42a1-a4d7-4db354b70462
2021-05-18 18:39:12,348 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/d9ecde53-a762-4e05-a736-02721bcc60c1
2021-05-18 18:39:12,356 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/ba23c9d2-eac2-4a98-aef5-47e3ba8d9bc4
2021-05-18 18:39:12,363 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/1c11c32a-272b-4d30-a97f-c636123a2922
2021-05-18 18:39:12,370 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/1c234860-3e55-498f-86d1-d1f06210e105
2021-05-18 18:39:12,378 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/218a612d-a4d5-445d-84ee-077476d91d16
2021-05-18 18:39:12,385 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/3a01b365-246f-402e-9422-9c7df0d5d919
2021-05-18 19:39:15,027 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/02b8fb75-707a-4b83-b13b-b80d8d167d4f
2021-05-18 19:39:15,042 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/8c203cef-c16b-4156-b102-ed0281092da6
2021-05-18 20:39:17,863 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/522b788f-b4fa-4bd7-9319-58c490bf0afe
2021-05-18 20:39:17,869 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/9d8c8713-df44-4999-b1e7-2b7e3c8789cb
2021-05-18 21:39:21,011 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/97790514-2a6a-4acb-bc45-985684c351a0
2021-05-18 21:39:21,017 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/8c405277-852a-414f-b42b-1c14c39f8197
2021-05-18 21:39:21,025 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/798b069f-45ed-452b-a7d1-e7dba713901e
2021-05-18 21:39:21,031 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/a6b6a1c6-a66d-4632-828a-c993db563143
2021-05-18 22:39:23,729 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/77308e31-8580-49f2-9346-52bbe826be7b
2021-05-18 22:39:23,735 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/899bc67b-91ff-4fdf-a837-871c89447285
2021-05-18 22:39:23,743 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/d9e70065-458c-4f3e-9936-3898a9d09af2
2021-05-18 22:39:23,749 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/9e02f601-3de7-4f96-953a-e3694998c98e
2021-05-18 23:39:26,462 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/02622010-08cc-45c7-b051-09c1d6497e32
2021-05-18 23:39:26,470 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/eadf6375-07ae-4bd5-893e-da5a945c65bf
2021-05-18 23:39:26,477 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/f84b6d97-4931-43c9-a678-e3ee7859801b
2021-05-18 23:39:26,484 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/bc3bec21-a507-4b76-a087-1060fa33da9b
2021-05-18 23:39:26,492 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/bf24d4fb-094b-4be5-839a-f6c30d42cafe
2021-05-18 23:39:26,499 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/ef97e0a3-f787-4f0c-bb8d-719a58445c88
2021-05-19 00:39:29,252 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/b521a3e3-a7aa-413a-8758-9cd114b22cf9
2021-05-19 00:39:29,259 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/94254fac-f8b8-402e-bc46-bfaf3a162609
2021-05-19 00:39:29,267 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/4cc557f8-4502-46d2-96db-2bdd5e9becac
2021-05-19 00:39:29,274 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/7077eb2e-d9ff-494f-954d-2eba7ecfea7c
2021-05-19 01:39:31,955 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/6ba031e5-64ed-4047-a263-0072095d0a67
2021-05-19 01:39:31,961 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/524935aa-cc7e-4846-bf00-921f09be0a04
2021-05-19 01:39:31,968 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/a330342d-17f0-4b34-8218-bf6d96292eb7
2021-05-19 01:39:31,974 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/34c3c76b-56f9-4b00-b92b-1a974df444ef
sh-4.2$ cat /var/log/ecs/ecs-agent.log* | grep "Credentials not found"
level=error time=2021-05-18T07:38:42Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T07:38:42Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T10:38:50Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T10:38:50Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T11:38:53Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T11:38:53Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T11:38:53Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T11:38:53Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T12:38:55Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T12:38:55Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T12:38:55Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T12:38:55Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T12:38:55Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T12:38:55Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T13:38:58Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T13:38:58Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T13:38:58Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T13:38:58Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T13:38:58Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T13:38:58Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T13:38:58Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T13:38:58Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T14:39:01Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T14:39:01Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T14:39:01Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T14:39:01Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T14:39:01Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T14:39:01Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T15:39:04Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T15:39:04Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T15:39:04Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T15:39:04Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T15:39:04Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T15:39:04Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T15:39:04Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T15:39:04Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T15:39:04Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T15:39:04Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T16:39:06Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T16:39:06Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T16:39:06Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T16:39:06Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T16:39:06Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T16:39:06Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T16:39:06Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T16:39:06Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T17:39:09Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T17:39:09Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T17:39:09Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T17:39:09Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T17:39:09Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T17:39:09Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T17:39:09Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T17:39:09Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T18:39:12Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T18:39:12Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T18:39:12Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T18:39:12Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T18:39:12Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T18:39:12Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T18:39:12Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T18:39:12Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T18:39:12Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T18:39:12Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T18:39:12Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T18:39:12Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T19:39:15Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T19:39:15Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T19:39:15Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T19:39:15Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T20:39:17Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T20:39:17Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T20:39:17Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T20:39:17Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T21:39:21Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T21:39:21Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T21:39:21Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T21:39:21Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T21:39:21Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T21:39:21Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T21:39:21Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T21:39:21Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T22:39:23Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T22:39:23Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T22:39:23Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T22:39:23Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T22:39:23Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T22:39:23Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T22:39:23Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T22:39:23Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T23:39:26Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T23:39:26Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T23:39:26Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T23:39:26Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T23:39:26Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T23:39:26Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T23:39:26Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T23:39:26Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T23:39:26Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T23:39:26Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-18T23:39:26Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-18T23:39:26Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-19T00:39:29Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-19T00:39:29Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-19T00:39:29Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-19T00:39:29Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-19T00:39:29Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-19T00:39:29Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-19T00:39:29Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-19T00:39:29Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-19T01:39:31Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-19T01:39:31Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-19T01:39:31Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-19T01:39:31Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-19T01:39:31Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-19T01:39:31Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.go
level=error time=2021-05-19T01:39:31Z msg="Error processing credential request: CredentialsV2Request: Credentials not found" module=credentials_handler.go
level=error time=2021-05-19T01:39:31Z msg="HTTP response status code is '400', request type is: credentials, and response in JSON is {\"code\":\"InvalidIdInRequest\",\"message\":\"CredentialsV2Request: Credentials not found\",\"HTTPErrorCode\":400}" module=helpers.goI will try to update |
|
Even after updating the |
|
sorry for the misdirection, but according to comment from @Cappuccinuo here: #2810 (comment) the correct config variable to adjust is So Im curious what exactly is the timeline of your issue? Is it possible that this is a timing issue where you are killing tasks right around the time that they decide they need to renew their tls cert? Could you also try increaing |
|
@sparrc The EFS is used for application deployed as multiple tasks that has 2 types:
Both of this types will stop and created new when there is new deployment, For web server there's an autoscaling based on traffic From here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html The I still believe setting |
|
Hi @xian13 Correct me if I am wrong, so the task that runs the “long running queue consumer” application is being stopped here every 60 minutes due to new deployment? That would confirm that task is being stopped around the time of tls cert being renewed. |
|
@mythri-garaga No, there are 2 conditions that the container exit
Like my explanation earlier, if container created at minute 25, and hourly restart at minute 30, then container is restarted with 5 minutes after it's created |
|
To further investigate this issue, could you please run the logs collector - https://github.com/aws/amazon-ecs-logs-collector and share them at at ecs-agent-external@amazon.com? Thank you. |
|
I've send the log to ecs-agent-external@amazon.com with subject Thanks |
|
@Cappuccinuo are there implications of setting |
|
Changing If the mount is umounted properly after the container is killed, the issue won't be happening anymore. Is that possible to immediately force umount the file system once the task is killed? |
|
Following up on this. I will post an update when the change is released so you can test. |
|
Not sure if this is the same issue, if not I will make a new one, let me know. ECS leak a bunch of stunnel processes & kthreads, possibly only for tasks that die and/or do not start. One of my instance has a load of 250 now because of all the leaking threads. A few example processes: As mentioned, this seems to happen when I define a service for which the image does not start or dies quickly. It might just be all tasks, but since the usually don't restart that often it is unnoticable. |
|
Hi @holstvoogd, would you confirm whether the EFS volumes associated with the stopped task were successfully unmounted? ECS Agent should output If the volume was unmounted, the actual process that reaps |
|
Hi @yinyic, I can see they are being unmounted, but with a different message: On closer inspection, the stunnels are reaped properly aswell. I just end up with tons of '[127.0.0.1-manag]' processes cluttering up my servers & cause a very high load without actually doing anything. |
|
Hi @holstvoogd, it is hard to tell for sure from the given information what the issue is related to. The processes are stuck in Could you run the https://github.com/aws/amazon-ecs-logs-collector and share the logs, steps to reproduce the issue and the task def that you are using? You can email the information to ecs-agent-external@amazon.com |
|
Closing due to inactivity. Please feel free to reopen if necessary. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
aws/efs-utils#83
The EFS volume attached to a task is not umounted after the task is killed. Umount is client side behaviour, so once the task & container is exited, the volume attached should be umounted.
Description
After the task is running, everything works fine and volume is attached.
After the task is killed, the volume is still mounted, the amazon-efs-watchdog cannot kill the stunnel since the volume is not umounted, since the iam credentials are fetched from
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, the watchdog is complaining2021-02-15 03:35:03,419 - ERROR - Cannot recreate self-signed certificate 2021-02-15 03:35:04,448 - ERROR - Failed to retrieve AWS security credentials using lookup method: ecs:/v2/credentials/****Expected Behavior
The volume resource is umounted after the task & container is killed.
Observed Behavior
The volume resource is still attached to the host instance after the task & container is killed.
You can see that there are two volume attached, the volume mounted on
/var/lib/ecs/volumes/ecs-Github83-1-EFS-d8d9dfe0a1d9b4918a01cannot retrieve credentials properly[root@ip-172-31-64-80 efs]# df Filesystem 1K-blocks Used Available Use% Mounted on ... 127.0.0.1:/ 9007199254739968 4194304 9007199250545664 1% /var/lib/ecs/volumes/ecs-Github83-1-EFS-d8d9dfe0a1d9b4918a01 ... 127.0.0.1:/ 9007199254739968 4194304 9007199250545664 1% /var/lib/ecs/volumes/ecs-Github83-2-EFS-e083d4bdace2f4908b01 ...While the container
0c3c32e57096is already killedAfter umount the file system on the instance, everything works fine
Environment Details
Task definition:
{ ... "mountPoints": [ { "readOnly": null, "containerPath": "/efs", "sourceVolume": "EFS" } ], ... "image": "nginx", ... "name": "nginx" } ], ... "volumes": [ { ... "efsVolumeConfiguration": { "transitEncryptionPort": null, "fileSystemId": "fs-12345678", "authorizationConfig": { "iam": "ENABLED", "accessPointId": null }, "transitEncryption": "ENABLED", "rootDirectory": "/" }, "name": "EFS", "host": null, "dockerVolumeConfiguration": null } ] }Supporting Log Snippets
N/A, if needed I will collect. Otherwise I think the investigation should be around how ECS agent handle the resource clean up (EFS volume umount).
The text was updated successfully, but these errors were encountered: