(@aws_cdk) Python constructs do not implement a compatible interface

[pradoz]

Describe the bug

Similar (old) issue: aws/aws-cdk#15651

aws_ec2.SubnetSelection cannot be initialized from a list of aws_ec2.Subnet's because aws_ec2.ISubnet is not a compatible interface

Fix

We yanked construct==10.3.1 and released construct==10.3.2 with correct dependency constraints.
Make sure your project is not using construct==10.3.1 and not using typeguard==4.3.0

Workaround

In requirements.txt, pin the version of typeguard:

typeguard==2.13.3

Expected Behavior

Passing a list of aws_ec2.Subnet's to an aws_ec2.SubnetSelection can initialize properly

Current Behavior

Error Message:

Traceback (most recent call last):
  File "/home/myuser/testdir/app2/app.py", line 34, in <module>
    App2(app, 'App2', env=env)
  File "/home/myuser/testdir/.venv/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in __call__
    inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/testdir/app2/app.py", line 27, in __init__
    ec2.SubnetSelection(subnets=subnet_construct_list)  # this goes boom
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/testdir/.venv/lib/python3.11/site-packages/aws_cdk/aws_ec2/__init__.py", line 85174, in __init__
    check_type(argname="argument subnets", value=subnets, expected_type=type_hints["subnets"])
  File "/home/myuser/testdir/.venv/lib/python3.11/site-packages/aws_cdk/aws_ec2/__init__.py", line 2602, in check_type
    typeguard.check_type(value=value, expected_type=expected_type, collection_check_strategy=typeguard.CollectionCheckStrategy.ALL_ITEMS) # type:ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/testdir/.venv/lib/python3.11/site-packages/typeguard/_functions.py", line 106, in check_type
    check_type_internal(value, expected_type, memo)
  File "/home/myuser/testdir/.venv/lib/python3.11/site-packages/typeguard/_checkers.py", line 861, in check_type_internal
    checker(value, origin_type, args, memo)
  File "/home/myuser/testdir/.venv/lib/python3.11/site-packages/typeguard/_checkers.py", line 433, in check_union
    raise TypeCheckError(f"did not match any element in the union:\n{formatted_errors}")
typeguard.TypeCheckError: list did not match any element in the union:
  Sequence[aws_cdk.aws_ec2.ISubnet]: item 0 is not compatible with the ISubnet protocol because it has no method named '__jsii_proxy_class__'
  NoneType: is not an instance of NoneTypeTraceback (most recent call last):

Reproduction Steps

requirements.txt

aws-cdk-lib==2.162.0
constructs>=10.0.0,<11.0.0

requirements-dev.txt

pytest==6.2.5
boto3

app1/app.py

import os

from aws_cdk import (
    App,
    Environment,
    Stack,
    aws_ec2 as ec2,
    aws_ssm as ssm,
)
from constructs import Construct


class App1(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)
        vpc = ec2.Vpc(
            self, 'Vpc',
            max_azs=1,
            create_internet_gateway=False,
            subnet_configuration=[
                ec2.SubnetConfiguration(
                    name='PublicSubnets',
                    subnet_type=ec2.SubnetType.PUBLIC,
                    cidr_mask=26,
                ),
                ec2.SubnetConfiguration(
                    name='PrivateSubnets',
                    subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS,
                    cidr_mask=20,
                ),
            ],
            nat_gateways=1,
            restrict_default_security_group=True,
        )
        private_subnets = ','.join([s.subnet_id for s in vpc.private_subnets])
        ssm.StringParameter(
            self, 'SubnetParam',
            parameter_name='/my/subnets',
            string_value=private_subnets,
        )


env = Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'),
                  region=os.getenv('CDK_DEFAULT_REGION'))

app = App()
App1(app, 'App1', env=env)
app.synth()

app2/app.py

import os
import boto3

from aws_cdk import (
    App,
    Environment,
    Stack,
    aws_ec2 as ec2,
)
from constructs import Construct


def get_parameter() -> list:
    ssm_client = boto3.client('ssm', region_name='us-east-2')
    response = ssm_client.get_parameter(Name='/my/subnets')
    val = response['Parameter']['Value']
    return val.split(',')


class App2(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)
        subnet_list = get_parameter()
        subnet_construct_list = [
            ec2.Subnet.from_subnet_id(self, s, subnet_id=s) for s in subnet_list
        ]
        ec2.SubnetSelection(subnets=subnet_construct_list)  # this goes boom


env = Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'),
                  region=os.getenv('CDK_DEFAULT_REGION'))

app = App()
App2(app, 'App2', env=env)
app.synth()

Deploy:

$ python3 -m venv .venv && source .venv/bin/activate
$  pip install -r requirements.txt -r requirements-dev.txt
$  cdk deploy -vv --require-approval never --app "python3 app1/app.py"
$  cdk deploy -vv --require-approval never --app "python3 app2/app.py"

Observe the error. I also confirmed that a print statement for subnet_list prints ['subnet-0b39XXXXXX']

Possible Solution

Not sure

Additional Information/Context

Package Version

---

attrs 24.2.0
aws-cdk.asset-awscli-v1 2.2.206
aws-cdk.asset-kubectl-v20 2.1.3
aws-cdk.asset-node-proxy-agent-v6 2.1.0
aws-cdk.cloud-assembly-schema 38.0.1
aws-cdk-lib 2.162.0
boto3 1.35.38
botocore 1.35.38
cattrs 23.2.3
constructs 10.3.1
importlib_resources 6.4.5
iniconfig 2.0.0
jmespath 1.0.1
jsii 1.103.1
packaging 24.1
pip 24.0
pluggy 1.5.0
publication 0.0.3
py 1.11.0
pytest 6.2.5
python-dateutil 2.9.0.post0
s3transfer 0.10.3
setuptools 65.5.0
six 1.16.0
toml 0.10.2
typeguard 4.3.0
typing_extensions 4.12.2
urllib3 2.2.3

SDK version used

CDK 2.162.0 (build c8d7dd3), python 3.11, node 18 & 20

Environment details (OS name and version, etc.)

Amazon Linux 2023 & Ubuntu 20.04 on WSL

[pradoz]

Our team has also found a smaller example that doesn't work and produces a much more obscure error. Stack contents:

        vpc: aws_ec2.Vpc = aws_ec2.Vpc(
            self,
            "Vpc",
            max_azs=1,
            subnet_configuration=[
                aws_ec2.SubnetConfiguration(
                    name="PublicSubnets",
                    subnet_type=aws_ec2.SubnetType.PUBLIC,
                    cidr_mask=26,
                ),
                aws_ec2.SubnetConfiguration(
                    name="PrivateSubnets",
                    subnet_type=aws_ec2.SubnetType.PRIVATE_WITH_EGRESS,
                    cidr_mask=20,
                ),
            ],
            nat_gateways=1,
            restrict_default_security_group=True,
        )

        aws_ec2.SecurityGroup(
            self,
            "SecurityGroup",
            vpc=vpc,
            allow_all_outbound=True,
            allow_all_ipv6_outbound=False,
            description="This is a test",
        )

Error:

typeguard.TypeCheckError: aws_cdk.aws_ec2.Vpc is not compatible with the IVpc protocol because its 'add_client_vpn_endpoint' method has mandatory keyword-only arguments in its declaration: cidr, server_certificate_arn

[silv-io]

We're running into similar issues with aws_cdk.BootstraplessSynthesizer and aws_cdk.IReusableStackSynthesizer:

typeguard.TypeCheckError: aws_cdk.BootstraplessSynthesizer did not match any element in the union:
  aws_cdk.IReusableStackSynthesizer: is not compatible with the IReusableStackSynthesizer protocol because its 'add_docker_image_asset' method has mandatory keyword-only arguments in its declaration: source_hash
  NoneType: is not an instance of NoneType

[MrDWilson]

I'm also experiencing this on deploys that were working yesterday before the aws-cdk upgrade. I believe pinning the typeguard version to <3 will fix for now, but would be good to get this resolved.

[mrgrain]

Yes, the workaround is to pin typeguard==2.13.3

[mrgrain]

This seems to be intentional behavior in typeguard >=3

https://github.com/agronholm/typeguard/blob/cf25d56dc0dbf6bb2f51ea29da8436b368ed4857/src/typeguard/_checkers.py#L171-L181

But I don't understand yet why. PEP3102 explicitly allows keyword-only arguments without defaults.

[mrgrain]

This check seems to have been added ages ago, so the pre-conditions to trigger this check must have changed
https://github.com/agronholm/typeguard/blob/cf25d56dc0dbf6bb2f51ea29da8436b368ed4857/docs/versionhistory.rst?plain=1#L518

[mrgrain]

Check seems to pass with typeguard==3.0.2 if I add @typing.runtime_checkable to the protocol.
Same for typeguard-4.0.0 and typeguard-4.1.0 and typeguard-4.2.1

=> typeguard 4.3.0 is introducing this issue.

[mrgrain]

Seems to be introduced by this commit: agronholm/typeguard@241d120#diff-0ac00416d20efb96e963a903c7c8f8e7a5070064af5b4673ad441c7e3114265eR727

[mrgrain]

We yanked the 10.3.1 release of constructs from PyPI as a quick mitigation for most users.
https://pypi.org/project/constructs/10.3.1/

Opened an issue with typeguard to better understand this requirement agronholm/typeguard#495

[mrgrain]

Another example

from aws_cdk import App, Stack
from aws_cdk import aws_lambda, aws_s3
from aws_cdk import aws_s3_notifications as s3_notify

app = App()
stack = Stack(app, "bucket-stack")

bucket = aws_s3.Bucket(stack, "Bucket")

bucket.add_event_notification(
    event=aws_s3.EventType.OBJECT_CREATED,
    dest=s3_notify.LambdaDestination(
        aws_lambda.Function.from_function_arn(
            stack,
            "ObjectCreatedLambda",
            "arn:aws:lambda:us-west-2:460106496004:function:ObjectCreatedLambda-qQkkxlUaClaJ",
        )
    ),
)

app.synth()

[Hektor-V]

It appears the issue has now been resolved, and I've confirmed this with AWS, who directed me to this solution. We implemented the same workaround yesterday by specifying the Typeguard version. I appreciate everyone who worked on resolving this issue in a timely manner.

notes:
Successful deployment: Typeguard was downloaded twice:
Downloading typeguard-4.3.0-py3-none-any.whl.metadata (3.7 kB)
Downloading typeguard-2.13.3-py3-none-any.whl.metadata (3.6 kB)

Failed deployment: Typeguard was downloaded only once:
Downloading typeguard-4.3.0-py3-none-any.whl (35 kB)

I am not sure why typeguard is installed twice every time aws-cdk-lib is ran but it's how all our release look.

[mrgrain]

I am not sure why typeguard is installed twice every time aws-cdk-lib is ran but it's how all our release look.

This might be a quirk of your package manager. Using poetry I can see typeguard being downloaded only once.

[Hektor-V]

I am not sure why typeguard is installed twice every time aws-cdk-lib is ran but it's how all our release look.

This might be a quirk of your package manager. Using poetry I can see typeguard being downloaded only once.

Good suggestion but looks like to me how pip handles the install?
Reason I created a fresh virtual environment using python -m venv .venv, then ran a pip install aws-cdk-lib --no-cache-dir. Even then I still get typeguard to appear twice.

[mrgrain]

Reason I created a fresh virtual environment using python -m venv .venv, then ran a pip install aws-cdk-lib --no-cache-dir. Even then I still get typeguard to appear twice.

Yeah, I see the same. I think pip isn't just very clever in resolving dependencies. aws-cdk-lib currently still incorrectly declares compatibility with typeguard-4.3.0. We are working on a fix for that. Because constructs has been fixed, you will end using a compatible version of typeguard. I guess pip just downloads both before making this choice.

[mrgrain]

I actually got the dependency via transitivity. I mentioned explicitly stating the version to make it easier to reproduce.

In fact, upon checking the changes to my Poetry lock file, I noticed that a particular commit caused "typeguard" to bump from 2.13 to 4.3. That was due to a cascade of upgrades allowed by the fact that aws-cdk-asset-kubectl-v20 declared support for typeguard = ">=2.13.3,<5.0.0"

[mrgrain]

This might be resolved in typeguard 4.4.0
agronholm/typeguard#465

[BwL1289]

@mrgrain I just tried with 4.4.0 and got the following:

File "/usr/local/lib/python3.12/site-packages/constructs/__init__.py", line 800, in __init__
--
958 | check_type(argname="argument scope", value=scope, expected_type=type_hints["scope"])
959 | TypeError: check_type() got an unexpected keyword argument 'argname'
960 | [22:47:11] failed command: python3 app.py