(ecs-patterns): Creating a NetworkLoadBalancedFargateService with an imported NetworkLoadBalancer creates the Listener in the stack the NLB was looked up in, creating a circular dependency

[rishi-kulkarni]

Describe the bug

Hi all,

I created a NetworkLoadBalancedFargateService using a NLB that I looked up by tag, as it is associated with a number of other services at my organization. The service is created as part of an EcsStack and the lookup happens in a LookupStack.

This all went fine and got deployed into all of our various environments. However, when trying to make a change to the resources provisioned, the EcsStack and the LookupStack have become circularly dependent due to the fact that, for some reason, the Listener and the TargetGroup created by the EcsStack is actually part of the LookupStack's CloudFormation template:

# in lookup-stack.template.json
  "...": {
   "Type": "AWS::ElasticLoadBalancingV2::Listener",
   "Properties": {
    "DefaultActions": [
     {
      "TargetGroupArn": {
       "Ref": "..."
      },
      "Type": "forward"
     }
    ],
    "LoadBalancerArn": "...",
    "Port": ..,
    "Protocol": "TCP"
   },
   "Metadata": {
    "aws:cdk:path": "..."
   }
  },
  "...": {
   "Type": "AWS::ElasticLoadBalancingV2::TargetGroup",
   "Properties": {
    "Port": ...,
    "Protocol": "TCP",
    "TargetType": "ip",
    "VpcId": "..."
   },
   "Metadata": {
    "aws:cdk:path": "..."
   }

This is surprising, and when I was running some tests on how to break this circular dependency, I noticed that creating a Listener as an L2 construct in the EcsStack properly associates it with the EcsStack CloudFormation template rather than the LookupStack template. I'm also now at a loss for how to break this dependency - especially given that this service is deployed into many accounts, so ideally it would be fixable in a single CDK deployment.

Expected Behavior

My expectation is that Resources created in a stack are associated with the CloudFormation template of THAT stack, not any other stack. Especially not one that immediately creates a circular dependency.

Current Behavior

The Listener and Target group created by the NetworkLoadBalancedFargateService L3 construct are part of the stack that looked up the NLB, rather than the stack creating the NetworkLoadBalancedFargateService construct.

I then tried to just move the lookup of the NLB into the LookupStack, but I get the circular dependency error:

Export {account-name}stack:ExportsOutputRefdev...PublicListenerECSGroup1D50C3306D93B20F cannot be deleted as it is in use by {account-name}-ecs-stack

Reproduction Steps

Create a NetworkLoadBalancer outside of CDK that has some identifiable tag.

Make two stacks:

Stack 1:
Look up NLB by tag.
Return the NLB.

Stack 2:
Take the NLB returned by Stack 1 as an argument.
Create a NetworkLoadBalancedFargateService with the above NLB as the NLB.

The listener and target group of Stack 2's service will be part of Stack 1.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.92.0

Framework Version

No response

Node.js Version

v20.2

OS

Ubuntu

Language

Go

Language Version

1.21.0

Other information

I can see what I should have done in the first place, but now I'm more interested in figuring out how I can unwind these mutually-dependent stacks without service disruption in production.

[peterwoodworth]

The resources are being created in the LookupStack because they are being added through the method on INLB, which will add them in the same scope as the NLB which is the correct behavior. We should probably document that your NLB needs to be in the same stack as the L3 pattern construct.

You can get around the error you're experiencing with a bit of work, see this blogpost on the topic

[rishi-kulkarni]

Interesting. Creating a NewNetworkListener in the ECS stack using the looked up NLB puts the Listener in the ECS stack. Is that incorrect behavior?

[peterwoodworth]

That is correct behavior, since you are handling the creation of the NetworkListener yourself and creating the NetworkListener in the scope of the ECS Stack