Backstage pattern timeout on load balancer creation

[shubhamk-cdtn]

Describe the bug
When I try to deploy backstage pattern, the eks-loadbalancer resource creation is timing out with errors like

Received response status [FAILED] from custom resource. Message returned: TimeoutError: {"state":"TIMEOUT","reason":"Waiter has timed out"} at checkExceptions (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/util-waiter/dist-cjs/index.js:59:26) at waitUntilFunctionActiveV2 (/var/runtime/node_modules/@aws-sdk/client-lambda/dist-cjs/index.js:5826:49) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async defaultInvokeFunction (/var/task/outbound.js:1:1024) at async invokeUserFunction (/var/task/framework.js:1:2287) at async onEvent (/var/task/framework.js:1:369) at async Runtime.handler (/var/task/cfn-response.js:1:1676) (RequestId: 16ad0de3-1e44-438b-814c-ac208323c802)

On searching for the issue in cloudwatch logs, I get this

2024-07-26T16:49:08.745Z	885eae61-9575-48f9-84d1-c2058a66a94f	INFO	[provider-framework] executing user function arn:aws:lambda:ap-south-1:323155100510:function:backstage-blueprint-awscdkawseksKu-Handler886CB40B-TCMBwnyiTMag with payload {
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:ap-south-1:323155100510:function:backstage-blueprint-awscd-ProviderframeworkonEvent-8xHpBQWOgq7v",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:ap-south-1:323155100510:stack/backstage-blueprint/faf29dd0-4b6c-11ef-83b5-027ab549ec09",
    "RequestId": "16ad0de3-1e44-438b-814c-ac208323c802",
    "LogicalResourceId": "backstageblueprintawsloadbalancercontrollermanifestawsloadbalancercontrollerServiceAccountResource42E6EEE9",
    "ResourceType": "Custom::AWSCDK-EKS-KubernetesResource",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:ap-south-1:323155100510:function:backstage-blueprint-awscd-ProviderframeworkonEvent-8xHpBQWOgq7v",
        "PruneLabel": "aws.cdk.eks/prune-c8e52c43061de408d9284c707a9eaa0c6f9cfaef0f",
        "ClusterName": "backstage-blueprint",
        "Manifest": "[{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\",\"labels\":{\"aws.cdk.eks/prune-c8e52c43061de408d9284c707a9eaa0c6f9cfaef0f\":\"\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"},\"annotations\":{\"eks.amazonaws.com/role-arn\":\"arn:aws:iam::323155100510:role/backstage-blueprint-backstageblueprintawsloadbalanc-bYL4sAgx0WEu\"}}}]",
        "RoleArn": "arn:aws:iam::323155100510:role/backstage-blueprint-backstageblueprintCreationRoleE-sYYVkymowvO8"
    }
}

2024-07-26T16:52:23.572Z	885eae61-9575-48f9-84d1-c2058a66a94f	INFO	[provider-framework] CREATE failed, responding with a marker physical resource id so that the subsequent DELETE will be ignored

2024-07-26T16:52:23.603Z	885eae61-9575-48f9-84d1-c2058a66a94f	INFO	[provider-framework] submit response to cloudformation https://cloudformation-custom-resource-response-apsouth1.s3.ap-south-1.amazonaws.com//arn%3Aaws%3Acloudformation%3Aap-south-1%3A323155100510%3Astack/backstage-blueprint/faf29dd0-4b6c-11ef-83b5-027ab549ec09%7CbackstageblueprintawsloadbalancercontrollermanifestawsloadbalancercontrollerServiceAccountResource42E6EEE9%7C16ad0de3-1e44-438b-814c-ac208323c802?*** {
    "Status": "FAILED",
    "Reason": "TimeoutError: {\"state\":\"TIMEOUT\",\"reason\":\"Waiter has timed out\"}\n    at checkExceptions (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/util-waiter/dist-cjs/index.js:59:26)\n    at waitUntilFunctionActiveV2 (/var/runtime/node_modules/@aws-sdk/client-lambda/dist-cjs/index.js:5826:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async defaultInvokeFunction (/var/task/outbound.js:1:1024)\n    at async invokeUserFunction (/var/task/framework.js:1:2287)\n    at async onEvent (/var/task/framework.js:1:369)\n    at async Runtime.handler (/var/task/cfn-response.js:1:1676)",
    "StackId": "arn:aws:cloudformation:ap-south-1:323155100510:stack/backstage-blueprint/faf29dd0-4b6c-11ef-83b5-027ab549ec09",
    "RequestId": "16ad0de3-1e44-438b-814c-ac208323c802",
    "PhysicalResourceId": "AWSCDK::CustomResourceProviderFramework::CREATE_FAILED",
    "LogicalResourceId": "backstageblueprintawsloadbalancercontrollermanifestawsloadbalancercontrollerServiceAccountResource42E6EEE9"
}

Lambda is timing out I think

To Reproduce

make pattern backstage deploy

cc: @freschri

[freschri]

@shubhamk-cdtn thanks for reaching out. how many times did you try? can you clean all and try again? thanks

[shubhamk-cdtn]

@freschri multiple times actually. Tried on fresh aws account without any success unfortunately