Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFDK_X509Generator resource in DELETE_FAILED state after rollback #409

Closed
aws-painec opened this issue May 4, 2021 · 4 comments · Fixed by #476
Closed

RFDK_X509Generator resource in DELETE_FAILED state after rollback #409

aws-painec opened this issue May 4, 2021 · 4 comments · Fixed by #476
Assignees
@ddneilson
Labels
bug This issue is a bug.

Comments

@aws-painec
Copy link
Contributor

aws-painec commented May 4, 2021 •
edited
Loading

This was a test deployment of Nimble StudioBuilder in the YUL region (prod)
The studio's EFS lambda failed to execute causing a rollback, on which the X509 custom resource failed to delete:

Reproduction Steps

Rare failure state deploying with Nimble StudioBuilder

Error Log

30/101 | 12:40:56 AM | DELETE_FAILED        | Custom::RFDK_X509Generator                  | RootCA/Default/Default (RootCA) Received response status [FAILED] from custom resource. Message returned: DeleteItem '7cf316a0cf4633ad5448d9a8ee86f56490667411edbb678039fd1196e9009d4e' 'Private Key:" UnrecognizedClientException -- The security token included in the request is invalid.

Environment

Amazon Nimble production environment
StudioBuilder 1.0.8
Region ca-central-1

  • CDK CLI Version : 1.99.0
  • CDK Framework Version:
  • RFDK Version: 0.30.0
  • Deadline Version:
  • Node.js Version:
  • OS :
  • Language (Version):

Other

Link to the execution log the StudioBuilder deployment that failed:
https://tiny.amazon.com/felhxvd4/IsenLink

This is 🐛 Bug Report
@aws-painec aws-painec added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 4, 2021
@aws-painec
Copy link
Contributor Author

aws-painec commented Jun 8, 2021 •
edited
Loading

One of the StudioBuilder tests encountered another failure deleting the cert custom resource. We investigated the lambda execution that timed out and found these logs (private info has been redacted):

    START RequestId: 9d854ea2-4989-45c8-91fa-ebfd827b5708 Version: $LATEST
    2021-06-08T00:33:25.466Z	9d854ea2-4989-45c8-91fa-ebfd827b5708	INFO	Using DynamoDB Table for recording state: studiobuilderreleaseintegrationprodService-RenderQueueAcmCertTableE23A135F-17WVEJEN3PQJ
    2021-06-08T00:33:25.503Z	9d854ea2-4989-45c8-91fa-ebfd827b5708	INFO	Handling event: 
    {
        "RequestType": "Delete",
        "ServiceToken": "arn:aws:lambda:us-west-2:99999999999:function:studiobuilderreleaseinteg-SingletonLambda2d20d8f27-HUO8Fy7tW1TT",
        "ResponseURL": "",
        "StackId": "arn:aws:cloudformation:us-west-2:99999999999:stack/studiobuilderreleaseintegrationprodService/260db490-c7e6-11eb-aab2-0ae16cbbc953",
        "RequestId": "65972ad4-6daf-4da4-9e03-0fafff48f2b9",
        "LogicalResourceId": "RenderQueueAcmCert1A501BFD",
        "PhysicalResourceId": "242fde9441ad4e53c3e3b817290ab2f5fdcd0007f0fefee0257cb328ed4d6799",
        "ResourceType": "Custom::RFDK_AcmImportedCertificate",
        "ResourceProperties": {
            "ServiceToken": "arn:aws:lambda:us-west-2:99999999999:function:studiobuilderreleaseinteg-SingletonLambda2d20d8f27-HUO8Fy7tW1TT",
            "X509CertificatePem": {
                "CertChain": "arn:aws:secretsmanager:us-west-2:99999999999:secret:studiobuilderreleaseintegrationprodService/RQCert-X.509-CertificateChain-2021-5-7-1623107809719-UazLsy",
                "Passphrase": "arn:aws:secretsmanager:us-west-2:99999999999:secret:RQCertPassphrase00C8D549-SCKbNMC4BTIQ-lbcj7A",
                "Cert": "arn:aws:secretsmanager:us-west-2:99999999999:secret:studiobuilderreleaseintegrationprodService/RQCert-X.509-Certificate-2021-5-7-1623107809719-zCAWvh",
                "Key": "arn:aws:secretsmanager:us-west-2:99999999999:secret:studiobuilderreleaseintegrationprodService/RQCert-X.509-PrivateKey-2021-5-7-1623107809719-LDjXcN"
            },
            "Tags": [
                {
                    "Value": "7f42837ef4ed1b15625c5c24520c70f5",
                    "Key": "AcmCertImport-7F42837E"
                },
                {
                    "Value": "7f42837ef4ed1b15625c5c24520c70f5",
                    "Key": "Name"
                }
            ]
        }
    }
     
    2021-06-08T00:33:52.672Z	9d854ea2-4989-45c8-91fa-ebfd827b5708	INFO	CfnResponse: 
    {
        "Status": "FAILED",
        "Reason": "Timeout error\nError: Timeout error\n    at AcmCertificateImporter.handler (/var/task/lib/custom-resource/simple-resource.js:58:136)\n    at Runtime.importCert [as handler] (/var/task/x509-certificate/acm-handlers.js:166:26)\n    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)",
        "PhysicalResourceId": "242fde9441ad4e53c3e3b817290ab2f5fdcd0007f0fefee0257cb328ed4d6799",
        "StackId": "arn:aws:cloudformation:us-west-2:99999999999:stack/studiobuilderreleaseintegrationprodService/260db490-c7e6-11eb-aab2-0ae16cbbc953",
        "RequestId": "65972ad4-6daf-4da4-9e03-0fafff48f2b9",
        "LogicalResourceId": "RenderQueueAcmCert1A501BFD"
    }
     
    2021-06-08T00:33:52.760Z	9d854ea2-4989-45c8-91fa-ebfd827b5708	INFO	Result: FAILEDTimeout error
    Error: Timeout error
        at AcmCertificateImporter.handler (/var/task/lib/custom-resource/simple-resource.js:58:136)
        at Runtime.importCert [as handler] (/var/task/x509-certificate/acm-handlers.js:166:26)
        at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)
    END RequestId: 9d854ea2-4989-45c8-91fa-ebfd827b5708
    REPORT RequestId: 9d854ea2-4989-45c8-91fa-ebfd827b5708	Duration: 27453.55 ms	Billed Duration: 27454 ms	Memory Size: 128 MB	Max Memory Used: 91 MB	Init Duration: 434.09 ms

The event log from the failed stack:

    2021-06-08 11:03:59 UTC-0700	studiobuilderreleaseintegrationprodService	
    DELETE_COMPLETE	-
    2021-06-08 11:03:57 UTC-0700	RenderQueueAcmCert1A501BFD	
    DELETE_COMPLETE	-
    2021-06-08 11:03:57 UTC-0700	RenderQueueAcmCert1A501BFD	
    DELETE_IN_PROGRESS	-
    2021-06-08 11:03:54 UTC-0700	studiobuilderreleaseintegrationprodService	
    DELETE_IN_PROGRESS	User Initiated
    2021-06-07 17:39:10 UTC-0700	studiobuilderreleaseintegrationprodService	
    DELETE_FAILED	The following resource(s) failed to delete: [RenderQueueAcmCert1A501BFD].
    2021-06-07 17:39:09 UTC-0700	LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aServiceRole9741ECFB	
    DELETE_COMPLETE	-
    2021-06-07 17:39:08 UTC-0700	LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aServiceRole9741ECFB	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:39:07 UTC-0700	LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aServiceRoleDefaultPolicyADDA7DEB	
    DELETE_COMPLETE	-
    2021-06-07 17:39:06 UTC-0700	LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aServiceRoleDefaultPolicyADDA7DEB	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:39:05 UTC-0700	LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aFD4BFC8A	
    DELETE_COMPLETE	-
    2021-06-07 17:38:58 UTC-0700	LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aFD4BFC8A	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:38:57 UTC-0700	RenderQueueLogGroupWrapperC747BBF5	
    DELETE_COMPLETE	-
    2021-06-07 17:38:56 UTC-0700	RenderQueueCluster479052D0	
    DELETE_COMPLETE	-
    2021-06-07 17:38:56 UTC-0700	RenderQueueClusterRCSCapacityInstanceRole2FFFAE5A	
    DELETE_COMPLETE	-
    2021-06-07 17:38:55 UTC-0700	RenderQueueClusterRCSCapacityInstanceRole2FFFAE5A	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:38:55 UTC-0700	RenderQueueLogGroupWrapperC747BBF5	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:38:55 UTC-0700	RenderQueueCluster479052D0	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:38:54 UTC-0700	RenderQueueClusterRCSCapacityInstanceProfile1AB32556	
    DELETE_COMPLETE	-
    2021-06-07 17:38:54 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupECD6DE1E	
    DELETE_COMPLETE	-
    2021-06-07 17:38:54 UTC-0700	RenderQueueClusterRCSCapacityInstanceRoleDefaultPolicy7681C464	
    DELETE_COMPLETE	-
    2021-06-07 17:38:53 UTC-0700	RenderQueueClusterRCSCapacityInstanceRoleDefaultPolicy7681C464	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:38:53 UTC-0700	RenderQueueClusterRCSCapacityInstanceProfile1AB32556	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:38:53 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupECD6DE1E	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:38:52 UTC-0700	RenderQueueClusterRCSCapacityLaunchConfig07113D34	
    DELETE_COMPLETE	-
    2021-06-07 17:38:52 UTC-0700	RenderQueueClusterRCSCapacityLaunchConfig07113D34	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:38:51 UTC-0700	RenderQueueClusterRCSCapacityASG95F0EF27	
    DELETE_COMPLETE	-
    2021-06-07 17:34:52 UTC-0700	RQCertTable5F69A6AA	
    DELETE_COMPLETE	-
    2021-06-07 17:34:38 UTC-0700	RenderQueueAcmCertTableE23A135F	
    DELETE_COMPLETE	-
    2021-06-07 17:34:23 UTC-0700	RenderQueueTlsCaCertPemTable1654FF94	
    DELETE_COMPLETE	-
    2021-06-07 17:34:22 UTC-0700	RQCertGeneratorServiceRoleB03157FE	
    DELETE_COMPLETE	-
    2021-06-07 17:34:22 UTC-0700	RQCertPassphrase00C8D549	
    DELETE_COMPLETE	-
    2021-06-07 17:34:21 UTC-0700	RQCertTable5F69A6AA	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:21 UTC-0700	RQCertPassphrase00C8D549	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:21 UTC-0700	RQCertGeneratorServiceRoleB03157FE	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:20 UTC-0700	RQCertGeneratorServiceRoleDefaultPolicyC38F25A1	
    DELETE_COMPLETE	-
    2021-06-07 17:34:19 UTC-0700	RQCertGeneratorServiceRoleDefaultPolicyC38F25A1	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:19 UTC-0700	RQCertGeneratorCD9C63B1	
    DELETE_COMPLETE	-
    2021-06-07 17:34:16 UTC-0700	RenderQueueLBSecurityGroup3E7CD539	
    DELETE_COMPLETE	-
    2021-06-07 17:34:11 UTC-0700	RQCertGeneratorCD9C63B1	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:11 UTC-0700	RQCert	
    DELETE_COMPLETE	-
    2021-06-07 17:34:09 UTC-0700	RenderQueueTlsRcsCertBundleTable23BF6F1D	
    DELETE_COMPLETE	-
    2021-06-07 17:34:08 UTC-0700	SingletonLambda2d20d8f27b84444eb738c75b499a9eaaServiceRole0A680CE2	
    DELETE_COMPLETE	-
    2021-06-07 17:34:07 UTC-0700	RQCert	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:07 UTC-0700	SingletonLambda2d20d8f27b84444eb738c75b499a9eaaServiceRole0A680CE2	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:07 UTC-0700	RenderQueueAcmCertTableE23A135F	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:06 UTC-0700	SingletonLambda2d20d8f27b84444eb738c75b499a9eaaServiceRoleDefaultPolicyCE44710D	
    DELETE_COMPLETE	-
    2021-06-07 17:34:05 UTC-0700	SingletonLambda2d20d8f27b84444eb738c75b499a9eaaServiceRoleDefaultPolicyCE44710D	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:34:04 UTC-0700	SingletonLambda2d20d8f27b84444eb738c75b499a9eaaDA03150B	
    DELETE_COMPLETE	-
    2021-06-07 17:33:57 UTC-0700	SingletonLambda2d20d8f27b84444eb738c75b499a9eaaDA03150B	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:56 UTC-0700	RenderQueueAcmCert1A501BFD	
    DELETE_FAILED	Received response status [FAILED] from custom resource. Message returned: Timeout error Error: Timeout error at AcmCertificateImporter.handler (/var/task/lib/custom-resource/simple-resource.js:58:136) at Runtime.importCert [as handler] (/var/task/x509-certificate/acm-handlers.js:166:26) at Runtime.handleOnce (/var/runtime/Runtime.js:66:25) (RequestId: 65972ad4-6daf-4da4-9e03-0fafff48f2b9)
    2021-06-07 17:33:54 UTC-0700	RenderQueueTlsCaCertPemGeneratorServiceRoleBF35D10F	
    DELETE_COMPLETE	-
    2021-06-07 17:33:54 UTC-0700	RenderQueueTlsCaCertPemPassphrase2234E9C3	
    DELETE_COMPLETE	-
    2021-06-07 17:33:53 UTC-0700	RenderQueueTlsCaCertPemGeneratorServiceRoleBF35D10F	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:53 UTC-0700	RenderQueueTlsCaCertPemPassphrase2234E9C3	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:53 UTC-0700	RenderQueueTlsCaCertPemTable1654FF94	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:52 UTC-0700	RenderQueueTlsCaCertPemGeneratorServiceRoleDefaultPolicy977B3455	
    DELETE_COMPLETE	-
    2021-06-07 17:33:51 UTC-0700	RenderQueueTlsCaCertPemGeneratorServiceRoleDefaultPolicy977B3455	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:50 UTC-0700	RenderQueueTlsCaCertPemGenerator37207ABB	
    DELETE_COMPLETE	-
    2021-06-07 17:33:44 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionServiceRoleBCF2E2B7	
    DELETE_COMPLETE	-
    2021-06-07 17:33:43 UTC-0700	RenderQueueTlsCaCertPemGenerator37207ABB	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:43 UTC-0700	RenderQueueTlsCaCertPem45E587A5	
    DELETE_COMPLETE	-
    2021-06-07 17:33:42 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionServiceRoleBCF2E2B7	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:42 UTC-0700	RenderQueueClusterRCSCapacityASG95F0EF27	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:41 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionServiceRoleDefaultPolicyC9ACC32E	
    DELETE_COMPLETE	-
    2021-06-07 17:33:41 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionServiceRoleDefaultPolicyC9ACC32E	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:40 UTC-0700	RenderQueueTlsRcsCertBundleGeneratorServiceRole60AAB8F4	
    DELETE_COMPLETE	-
    2021-06-07 17:33:40 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionAD796CC3	
    DELETE_COMPLETE	-
    2021-06-07 17:33:39 UTC-0700	RenderQueueTlsRcsCertBundlePassphraseA34CE0E4	
    DELETE_COMPLETE	-
    2021-06-07 17:33:38 UTC-0700	RenderQueueTlsRcsCertBundleGeneratorServiceRole60AAB8F4	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:38 UTC-0700	RenderQueueTlsRcsCertBundlePassphraseA34CE0E4	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:38 UTC-0700	RenderQueueTlsRcsCertBundleTable23BF6F1D	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:38 UTC-0700	RenderQueueTlsCaCertPem45E587A5	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:37 UTC-0700	RenderQueueTlsRcsCertBundleGeneratorServiceRoleDefaultPolicy67A73EDB	
    DELETE_COMPLETE	-
    2021-06-07 17:33:36 UTC-0700	RenderQueueTlsRcsCertBundleGeneratorServiceRoleDefaultPolicy67A73EDB	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:36 UTC-0700	SingletonLambda085534161fc94be9a818609a31ae1b5bServiceRole6504823C	
    DELETE_COMPLETE	-
    2021-06-07 17:33:36 UTC-0700	RenderQueueTlsRcsCertBundleGenerator43BCCC77	
    DELETE_COMPLETE	-
    2021-06-07 17:33:34 UTC-0700	SingletonLambda085534161fc94be9a818609a31ae1b5bServiceRole6504823C	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:33 UTC-0700	SingletonLambda085534161fc94be9a818609a31ae1b5b3FBD84FB	
    DELETE_COMPLETE	-
    2021-06-07 17:33:33 UTC-0700	RenderQueueClusterRCSCapacityLifecycleHookDrainHookTopic64B430DD	
    DELETE_COMPLETE	-
    2021-06-07 17:33:33 UTC-0700	RenderQueueClusterRCSCapacityLifecycleHookDrainHookTopic64B430DD	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:32 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionAD796CC3	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:32 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionAllowInvokestudiobuilderreleaseintegrationprodServiceRenderQueueClusterRCSCapacityLifecycleHookDrainHookTopic3734FF73FB403FF9	
    DELETE_COMPLETE	-
    2021-06-07 17:33:29 UTC-0700	RenderQueueTlsRcsCertBundleGenerator43BCCC77	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:28 UTC-0700	RenderQueueTlsRcsCertBundle604062E3	
    DELETE_COMPLETE	-
    2021-06-07 17:33:27 UTC-0700	SingletonLambda085534161fc94be9a818609a31ae1b5b3FBD84FB	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:26 UTC-0700	RCSImagesThinkboxEcrProviderD767DD76	
    DELETE_COMPLETE	-
    2021-06-07 17:33:26 UTC-0700	RenderQueueClusterRCSCapacityLifecycleHookDrainHookRole38C12737	
    DELETE_COMPLETE	-
    2021-06-07 17:33:25 UTC-0700	RenderQueueLBSecurityGroup3E7CD539	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:25 UTC-0700	RenderQueueRCSTaskTaskRole651C2F8C	
    DELETE_COMPLETE	-
    2021-06-07 17:33:25 UTC-0700	RenderQueueRCSTaskExecutionRoleCE854605	
    DELETE_COMPLETE	-
    2021-06-07 17:33:24 UTC-0700	RenderQueueClusterRCSCapacityLifecycleHookDrainHookRole38C12737	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:24 UTC-0700	RenderQueueLB235D35F4	
    DELETE_COMPLETE	-
    2021-06-07 17:33:24 UTC-0700	RenderQueueLBPublicListenerECSGroup05494E95	
    DELETE_COMPLETE	-
    2021-06-07 17:33:24 UTC-0700	RenderQueueLBPublicListenerECSGroup05494E95	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:24 UTC-0700	RenderQueueLB235D35F4	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:24 UTC-0700	RenderQueueAcmCert1A501BFD	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:24 UTC-0700	RenderQueueTlsRcsCertBundle604062E3	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:23 UTC-0700	RenderQueueClusterRCSCapacityLifecycleHookDrainHookRoleDefaultPolicy3EDB00D3	
    DELETE_COMPLETE	-
    2021-06-07 17:33:23 UTC-0700	RCSImagesThinkboxEcrProviderD767DD76	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:23 UTC-0700	RenderQueueRCSTaskExecutionRoleCE854605	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:23 UTC-0700	RenderQueueRCSTaskTaskRole651C2F8C	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:23 UTC-0700	RenderQueueRCSTaskA9AE70D3	
    DELETE_COMPLETE	-
    2021-06-07 17:33:23 UTC-0700	RenderQueueClusterRCSCapacityLifecycleHookDrainHookRoleDefaultPolicy3EDB00D3	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:23 UTC-0700	RenderQueueLBPublicListenerBBF15D5F	
    DELETE_COMPLETE	-
    2021-06-07 17:33:22 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupstudiobuilderreleaseintegrationprodDataDocumentDbCluster43EC2FD4IndirectPortto752A2EA4	
    DELETE_COMPLETE	-
    2021-06-07 17:33:22 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupstudiobuilderreleaseintegrationprodNetworkStudioDefaultVpcInterfaceEndpointSG2F5F2A39443to2E61CB8A	
    DELETE_COMPLETE	-
    2021-06-07 17:33:22 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupstudiobuilderreleaseintegrationprodDataRepositoryFileSystem3EBF781F2049to57825939	
    DELETE_COMPLETE	-
    2021-06-07 17:33:22 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupfromstudiobuilderreleaseintegrationprodServiceRenderQueueLBSecurityGroupF73D05864433A6B891FB	
    DELETE_COMPLETE	-
    2021-06-07 17:33:22 UTC-0700	RenderQueueClusterRCSCapacityLifecycleHookDrainHook109F3D21	
    DELETE_COMPLETE	-
    2021-06-07 17:33:22 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionTopic1CF5F07D	
    DELETE_COMPLETE	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupfromstudiobuilderreleaseintegrationprodServiceRenderQueueLBSecurityGroupF73D05864433A6B891FB	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueRCSTaskA9AE70D3	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupstudiobuilderreleaseintegrationprodDataRepositoryFileSystem3EBF781F2049to57825939	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueLBPublicListenerBBF15D5F	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueClusterRCSCapacityLifecycleHookDrainHook109F3D21	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionTopic1CF5F07D	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupstudiobuilderreleaseintegrationprodNetworkStudioDefaultVpcInterfaceEndpointSG2F5F2A39443to2E61CB8A	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueClusterRCSCapacityDrainECSHookFunctionAllowInvokestudiobuilderreleaseintegrationprodServiceRenderQueueClusterRCSCapacityLifecycleHookDrainHookTopic3734FF73FB403FF9	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:21 UTC-0700	RenderQueueClusterRCSCapacityInstanceSecurityGroupstudiobuilderreleaseintegrationprodDataDocumentDbCluster43EC2FD4IndirectPortto752A2EA4	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:33:20 UTC-0700	RenderQueueAlbEc2ServicePatternService5B6692FB	
    DELETE_COMPLETE	-
    2021-06-07 17:27:42 UTC-0700	RenderQueueAlbEc2ServicePatternDNSFAD438DE	
    DELETE_COMPLETE	-
    2021-06-07 17:27:20 UTC-0700	GetStreamingImageIdsFunctionServiceRoleA3B38BB6	
    DELETE_COMPLETE	-
    2021-06-07 17:27:18 UTC-0700	GetStreamingImageIdsFunctionServiceRoleA3B38BB6	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:27:17 UTC-0700	GetStreamingImageIdsFunctionServiceRoleDefaultPolicy0D6BA32B	
    DELETE_COMPLETE	-
    2021-06-07 17:27:17 UTC-0700	GetStreamingImageIdsFunctionServiceRoleDefaultPolicy0D6BA32B	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:27:16 UTC-0700	GetStreamingImageIdsFunctionA6262AB6	
    DELETE_COMPLETE	-
    2021-06-07 17:27:11 UTC-0700	GetStreamingImageIdsProviderframeworkonEventServiceRole0CB8F0E7	
    DELETE_COMPLETE	-
    2021-06-07 17:27:09 UTC-0700	GetStreamingImageIdsProviderframeworkonEventServiceRole0CB8F0E7	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:27:09 UTC-0700	GetStreamingImageIdsFunctionA6262AB6	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:27:08 UTC-0700	GetStreamingImageIdsProviderframeworkonEventServiceRoleDefaultPolicy7A27FC73	
    DELETE_COMPLETE	-
    2021-06-07 17:27:07 UTC-0700	GetStreamingImageIdsProviderframeworkonEventServiceRoleDefaultPolicy7A27FC73	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:27:07 UTC-0700	GetStreamingImageIdsProviderframeworkonEvent0B6233A6	
    DELETE_COMPLETE	-
    2021-06-07 17:27:03 UTC-0700	StudioComponentComputeFarm	
    DELETE_COMPLETE	-
    2021-06-07 17:27:00 UTC-0700	GetStreamingImageIdsProviderframeworkonEvent0B6233A6	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:59 UTC-0700	GetStreamingImageIds	
    DELETE_COMPLETE	-
    2021-06-07 17:26:55 UTC-0700	StudioComponentComputeFarm	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:54 UTC-0700	GetStreamingImageIds	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:54 UTC-0700	LaunchProfile	
    DELETE_COMPLETE	-
    2021-06-07 17:26:52 UTC-0700	RenderQueueAlbEc2ServicePatternService5B6692FB	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:51 UTC-0700	RenderQueueWaitForStableServiceECSWaitLambdaRole427691CA	
    DELETE_COMPLETE	-
    2021-06-07 17:26:50 UTC-0700	SingletonLambda2e19e24316ee4d1aa3c918d35eddd446ServiceRole29B967AC	
    DELETE_COMPLETE	-
    2021-06-07 17:26:49 UTC-0700	RenderQueueWaitForStableServiceECSWaitLambdaRole427691CA	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:49 UTC-0700	SingletonLambda2e19e24316ee4d1aa3c918d35eddd446ServiceRole29B967AC	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:48 UTC-0700	RenderQueueWaitForStableServiceECSWaitD5508F84	
    DELETE_COMPLETE	-
    2021-06-07 17:26:47 UTC-0700	SingletonLambda2e19e24316ee4d1aa3c918d35eddd44620E11C08	
    DELETE_COMPLETE	-
    2021-06-07 17:26:46 UTC-0700	LaunchProfile	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:46 UTC-0700	LaunchProfileWorkstation	
    DELETE_COMPLETE	-
    2021-06-07 17:26:42 UTC-0700	RenderQueueWaitForStableServiceECSWaitD5508F84	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:41 UTC-0700	RenderQueueWaitForStableService4B92A8D2	
    DELETE_COMPLETE	-
    2021-06-07 17:26:41 UTC-0700	SingletonLambda2e19e24316ee4d1aa3c918d35eddd44620E11C08	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:40 UTC-0700	RenderWorkers2111BA37	
    DELETE_COMPLETE	-
    2021-06-07 17:26:40 UTC-0700	RenderQueueTlsRcsCertBundleGeneratorLogRetention589815D3	
    DELETE_COMPLETE	-
    2021-06-07 17:26:40 UTC-0700	VersionDeadlineResourceF80FEF97	
    DELETE_COMPLETE	-
    2021-06-07 17:26:40 UTC-0700	SingletonLambda2e19e24316ee4d1aa3c918d35eddd446LogRetention970D13ED	
    DELETE_COMPLETE	-
    2021-06-07 17:26:40 UTC-0700	RenderQueueWaitForStableServiceECSWaitLogRetention34AA979A	
    DELETE_COMPLETE	-
    2021-06-07 17:26:40 UTC-0700	RenderQueueTlsCaCertPemGeneratorLogRetentionBC3B4D20	
    DELETE_COMPLETE	-
    2021-06-07 17:26:40 UTC-0700	SingletonLambda085534161fc94be9a818609a31ae1b5bLogRetention633A28FB	
    DELETE_COMPLETE	-
    2021-06-07 17:26:40 UTC-0700	RQCertGeneratorLogRetention36A4B329	
    DELETE_COMPLETE	-
    2021-06-07 17:26:40 UTC-0700	RenderWorkers2111BA37	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:39 UTC-0700	SingletonLambda2d20d8f27b84444eb738c75b499a9eaaEventInvokeConfigA72831A7	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	CDKMetadata	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderQueueLBSecurityGrouptostudiobuilderreleaseintegrationprodServiceRenderQueueClusterRCSCapacityInstanceSecurityGroupACBF07F84433950E8238	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderWorkersfromstudiobuilderreleaseintegrationprodDataWorkstationToDeadlineF81033A056032560400212A1AE	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataWorkstationToDeadlineF81033A05603256040from2D9BD8E9	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderQueueLBSecurityGroupstudiobuilderreleaseintegrationprodDataWorkstationToDeadlineF81033A04433from59075507	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataFSxFileSystems74984C0A988to8924C7C3	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodNetworkLicenseServers52500F52ALLTRAFFICtoF051917E	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataFSxFileSystems74984C0A445to40AE10C6	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderQueueLBSecurityGroupfromstudiobuilderreleaseintegrationprodDataWorkstationToDeadlineF81033A04433189648E4	
    DELETE_COMPLETE	-
    2021-06-07 17:26:39 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodNetworkStudioDefaultVpcInterfaceEndpointSG2F5F2A39443to2CDB68BE	
    DELETE_COMPLETE	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataFSxFileSystems74984C0A10211023to6A4AA7A8	
    DELETE_COMPLETE	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueRCSTaskExecutionRoleDefaultPolicyE3743EFF	
    DELETE_COMPLETE	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueLBSecurityGroupfromstudiobuilderreleaseintegrationprodServiceRenderWorkersE104E3104433620F0F45	
    DELETE_COMPLETE	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataFSxFileSystems74984C0A5985to7CF6759F	
    DELETE_COMPLETE	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueRCSTaskTaskRoleDefaultPolicyC938E530	
    DELETE_COMPLETE	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersfromstudiobuilderreleaseintegrationprodDataWorkstationToDeadlineF81033A056032560400212A1AE	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	CDKMetadata	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataFSxFileSystems74984C0A988to8924C7C3	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodNetworkLicenseServers52500F52ALLTRAFFICtoF051917E	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueWaitForStableServiceECSWaitLogRetention34AA979A	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataFSxFileSystems74984C0A445to40AE10C6	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueTlsRcsCertBundleGeneratorLogRetention589815D3	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueTlsCaCertPemGeneratorLogRetentionBC3B4D20	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueLBSecurityGroupfromstudiobuilderreleaseintegrationprodServiceRenderWorkersE104E3104433620F0F45	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueLBSecurityGroupfromstudiobuilderreleaseintegrationprodDataWorkstationToDeadlineF81033A04433189648E4	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataWorkstationToDeadlineF81033A05603256040from2D9BD8E9	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RQCertGeneratorLogRetention36A4B329	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodNetworkStudioDefaultVpcInterfaceEndpointSG2F5F2A39443to2CDB68BE	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataFSxFileSystems74984C0A5985to7CF6759F	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	VersionDeadlineResourceF80FEF97	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueLBSecurityGrouptostudiobuilderreleaseintegrationprodServiceRenderQueueClusterRCSCapacityInstanceSecurityGroupACBF07F84433950E8238	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	SingletonLambda2d20d8f27b84444eb738c75b499a9eaaEventInvokeConfigA72831A7	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueWaitForStableService4B92A8D2	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueRCSTaskTaskRoleDefaultPolicyC938E530	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueAlbEc2ServicePatternDNSFAD438DE	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	SingletonLambda2e19e24316ee4d1aa3c918d35eddd446LogRetention970D13ED	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	LaunchProfileWorkstation	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	SingletonLambda085534161fc94be9a818609a31ae1b5bLogRetention633A28FB	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueLBSecurityGroupstudiobuilderreleaseintegrationprodDataWorkstationToDeadlineF81033A04433from59075507	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderWorkersstudiobuilderreleaseintegrationprodDataFSxFileSystems74984C0A10211023to6A4AA7A8	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:38 UTC-0700	RenderQueueRCSTaskExecutionRoleDefaultPolicyE3743EFF	
    DELETE_IN_PROGRESS	-
    2021-06-07 17:26:34 UTC-0700	studiobuilderreleaseintegrationprodService	
    DELETE_IN_PROGRESS	User Initiated

@ddneilson ddneilson self-assigned this Jun 8, 2021
@ddneilson ddneilson removed the needs-triage This issue or PR still needs to be triaged. label Jun 8, 2021
@ddneilson
Copy link
Contributor

Thanks Claire.

It looks like our timeout is set way too low for the lambda (30 seconds).
This is a theory, of course...

  1. The error we're seeing is this -- https://github.com/aws/aws-rfdk/blob/8a55f32124e2cfbadb33437d2d0494580a9eebac/pack[…]-rfdk/lib/lambdas/nodejs/lib/custom-resource/simple-resource.ts -- which only happens if the lambda's code doesn't complete in time. Timeout: 30s
  2. On Delete, the ACM Cert Importer is doing an exponential backoff waiting for all users of the certificate to be deleted -- https://github.com/aws/aws-rfdk/blob/8a55f32124e2cfbadb33437d2d0494580a9eebac/pack[…]es/aws-rfdk/lib/lambdas/nodejs/x509-certificate/acm-handlers.ts
  3. The backoffs are at around 19s after the 7th attempt to query ACM; that's most of the 30s timeout.
  4. The ALB seems to have been deleted at around 17:33:25
  5. The Cert delete started at 17:33:24

So, I think that it's theoretically possible that the ACM service hasn't yet registered that the certificate is no longer in use before we timeout of the Lambda behind the Custom Resource; resulting in the delete failing.

@ddneilson
Copy link
Contributor

I just re-read the original description, Claire. The logs posted & original description are actually two separate issues. In the original we see:

UnrecognizedClientException -- The security token included in the request is invalid.

This is an indication that the temporary credentials that the stack operation were running under have expired.

@ddneilson ddneilson mentioned this issue Jun 8, 2021
Closed
@jusiskin jusiskin linked a pull request Jun 22, 2021 that will close this issue
fix(core): increase lambda timeout for X509Certificate* constructs #476
Merged
@jusiskin
Copy link
Contributor

Resolved by #476

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ddneilson ddneilson

Labels
bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(core): increase lambda timeout for X509Certificate* constructs jusiskin/aws-rfdk
3 participants
@ddneilson @jusiskin @aws-painec