(aws-ecs): doesn't configure security group on target group used in alb with forward action that follows non-terminating authenticate action

[sakfa]

What is the problem?

If one creates an ApplicationLoadBalancer with a listener which forwards traffic to TargetType.IP target group and then attaches a FargateService to that target group (via service.attachToApplicationTargetGroup(targetGroup) call), CDK will automatically configure security groups rules:

• ALB security group to allow outgoing connection on correct port to that fargate service
• inbound connection on that fargate service from ALB

which is actually quite impressive. The issue is, when listener action is not merely a "Forward" action but rather a non-terminating "authenticateOidc" action (with "Forward" as next), this relation seems to break and security groups are not configured well.

Reproduction Steps

1. Define an ALB with forward action forwarding to FargateService via IP address target group and attachToApplicationTargetGroup call:

import * as cdk from "aws-cdk-lib";
import {aws_ec2, aws_ecr_assets, aws_ecs, aws_elasticloadbalancingv2} from "aws-cdk-lib";
import {ApplicationProtocol, ListenerAction} from "aws-cdk-lib/aws-elasticloadbalancingv2";
import path from "path";

export class DemoStack extends cdk.Stack {
    constructor(scope: cdk.App, id: string) {
        super(scope, id);
        const vpc = new aws_ec2.Vpc(this, "VPC", {});
        const alb = new aws_elasticloadbalancingv2.ApplicationLoadBalancer(this, "ALB", {
            internetFacing: true,
            vpc
        });
        const listener = alb.addListener("Listener", {protocol: ApplicationProtocol.HTTP, port: 80, });
        const targetGroup = new aws_elasticloadbalancingv2.ApplicationTargetGroup(this, 'Target', {
            vpc,
            targetType: aws_elasticloadbalancingv2.TargetType.IP,
            port: 8000,
            protocol: ApplicationProtocol.HTTP,
        });
        listener.addAction("DefaultAction", {
            action: ListenerAction.forward([targetGroup], {})
        });

        const taskDefinition = new aws_ecs.FargateTaskDefinition(this, "Task", {});
        taskDefinition.addContainer("Container", {
            image: aws_ecs.ContainerImage.fromDockerImageAsset(new aws_ecr_assets.DockerImageAsset(this, "Image", {
                directory: path.join(__dirname, "docker/")
            })),
            portMappings: [{containerPort: 8000, hostPort: 8000}]
        });
        const cluster = new aws_ecs.Cluster(this, "Cluster", { vpc });
        const service = new aws_ecs.FargateService(this, "ProxyService", {
            taskDefinition,
            cluster,
        });

        service.attachToApplicationTargetGroup(targetGroup);
    }
}

run cdk synth, proper security groups are created:

  ProxyServiceSecurityGroupfromDemoALBSecurityGroup39ED51AC80000016DABB:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      IpProtocol: tcp
      Description: Load balancer to target
      FromPort: 8000
      GroupId:
        Fn::GetAtt:
          - ProxyServiceSecurityGroup86509986
          - GroupId
      SourceSecurityGroupId:
        Fn::GetAtt:
          - ALBSecurityGroup8B8624F8
          - GroupId
      ToPort: 8000
    Metadata:
      aws:cdk:path: Demo/ProxyService/SecurityGroup/from DemoALBSecurityGroup39ED51AC:8000

and corresponding ALBSecurityGrouptoDemoProxyServiceSecurityGroup1E6FC1A280009B0210A1:

but now replace

        listener.addAction("DefaultAction", {
            action: ListenerAction.forward([targetGroup], {})
        });

with composite action:

        listener.addAction("DefaultAction", {
            action: ListenerAction.authenticateOidc({
                clientId: "xxx",
                clientSecret: SecretValue.plainText("yyy"),
                authorizationEndpoint: "https://auth",
                tokenEndpoint: "https://token",
                userInfoEndpoint: "https://userinfo",
                issuer: "https://demo",
                next: ListenerAction.forward([targetGroup])
            })
        });

and security groups are not created and ALB can't reach fargate service.

What did you expect to happen?

security groups created regardless of if forward action is top level action or a 'next' on non-terminating action (AFAIK there are 2 - authenticateOidc and authenticateCogito)

What actually happened?

No target security group changes were made, ALB's only group blocks all outbound traffic:

  ALBSecurityGroup8B8624F8:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Automatically created Security Group for ELB DemoALB4D859AA0
      SecurityGroupEgress:
        - CidrIp: 255.255.255.255/32
          Description: Disallow all traffic
          FromPort: 252
          IpProtocol: icmp
          ToPort: 86

Fargate service did not allow inbound.

CDK CLI Version

2.8.0 (build 8a5eb49)

Framework Version

No response

Node.js Version

v17.5.0

OS

MacOs

Language

Typescript

Language Version

No response

Other information

No response

[sakfa]

Note just in case someone stumbles on this before its fixed:

• to workaround this while hacking my prototypes I added a dummy rule with condition that will never match:

        listener.addAction("DummyAction", {
            priority: 1000,
            action: ListenerAction.forward([this.targetGroup]),
            conditions: [ ListenerCondition.hostHeaders(["matches.nothing.example.com"])]
        });

• for prod I will just configure security groups manually.

[sakfa]

Also another somewhat related issue:

when ALB uses authenticateOidc action it needs to communicate with identity provider (ALB fetches token from token endpoint and then user claim from user info endpoint). Identity Provider will typically live in the public internet.

right now CDK default - unlike clicking in console - blocks egress Internet access from ALB. This causes ALB to fail with tricky to debug 500 Internal Server Error during execution of oauth flow.

I think when using authenticateOidc action with ALB it would make sense for CDK to configure security group on ALB to allow egress access to 0.0.0.0/0 on port 443

[Kilowhisky]

This was a fun one to triage on my setup. Took the whole existing ALB down until i found that it had altered the existing SG outbound rules, removing all existing rules, and installing just the single outbound rule for the new service.

My setup is slightly different as i have an existing LB. Here's the GIST:
https://gist.github.com/Kilowhisky/9185d4885b94aad10136cc84aeb07300

[hrvg]

Also another somewhat related issue:

when ALB uses authenticateOidc action it needs to communicate with identity provider (ALB fetches token from token endpoint and then user claim from user info endpoint). Identity Provider will typically live in the public internet.

right now CDK default - unlike clicking in console - blocks egress Internet access from ALB. This causes ALB to fail with tricky to debug 500 Internal Server Error during execution of oauth flow.

I think when using authenticateOidc action with ALB it would make sense for CDK to configure security group on ALB to allow egress access to 0.0.0.0/0 on port 443

Also in case someone comes here from trying to solve HTTP 500: Internal server error issues on load balancer provisioned through aws_cdk.ecs_patterns, here is my CDK fix:

load_balanced_ecs_service = ecs_patterns.ApplicationLoadBalancedFargateService(...)
load_balanced_ecs_service.listener.add_action(
            'AuthAction',
            action=elbv2.ListenerAction.authenticate_oidc(...)
)
load_balancer_egress_sg = ec2.SecurityGroup(self, "LBEgressSecurityGroup", vpc=vpc)
load_balancer_egress_sg.add_egress_rule(
    peer=ec2.Peer.any_ipv4(),
    connection=ec2.Port.tcp(443)
)
load_balanced_ecs_service.load_balancer.add_security_group(load_balancer_egress_sg)

I resorted to provision a new security group and add it to the load balancer as I was unable to directly act on the automatically provisioned load balancer security from load_balanced_ecs_service.load_balancer.security_groups. This is equivalent though.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.