Packages installed in the layer are different that the ones from aws_lambda_powertools[all]

[ericbn]

Hi! 👋

Below is the list of installed packaged I get with a lambda using the arn:aws:lambda:us-east-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:17 layer, obtained with this code:

import pkg_resources


def lambda_handler(event, context):
    print(*sorted(f"{i.key}=={i.version}" for i in pkg_resources.working_set), sep="\n")

Output:

aws-lambda-powertools==2.5.0
aws-xray-sdk==2.11.0
awslambdaric==2.0.4
boto3==1.20.32
botocore==1.23.32
fastjsonschema==2.16.2
jmespath==0.10.0
pip==22.0.4
pydantic==1.10.2
python-dateutil==2.8.2
s3transfer==0.5.2
setuptools==58.1.0
simplejson==3.17.2
six==1.16.0
typing-extensions==4.4.0
urllib3==1.26.9
wrapt==1.14.1

And these are the packages I get by installing aws_lambda_powertools[all,aws-sdk]==2.5.0:

❯ python3.9 -m venv venv
❯ . venv/bin/activate
❯ pip install 'aws_lambda_powertools[all,aws-sdk]==2.5.0'
❯ pip freeze --local
aws-lambda-powertools==2.5.0
aws-xray-sdk==2.11.0
boto3==1.26.42
botocore==1.29.42
fastjsonschema==2.16.2
jmespath==1.0.1
pydantic==1.10.4
python-dateutil==2.8.2
s3transfer==0.6.0
six==1.16.0
typing_extensions==4.4.0
urllib3==1.26.13
wrapt==1.14.1

Here's the difference (removed pip and setuptools from the left, which is the list from the layer):

Questions:

1. Shouldn't the package versions all be the same between the layer and the aws_lambda_powertools extra(s)?
2. Shouldn't awslambdaric and boto3 already be included in aws_lambda_powertools[all] (the "all" extra), or some other extra that is intended to match the packages in the layer?

I'm assuming it's a good practice to add aws_lambda_powertools[all]==2.5.0 to the dev dependencies and then deploy your lambdas only with extra dependencies beyond those already included as dev, because I was assuming those dev dependencies would be the exact same ones already provided in the arn:aws:lambda:us-east-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:17 layer. In other words, I was expecting the diff above to perfectly match. I see it's not the case.

(Side question: Any plans on updating the boto3 version in next releases of the layer and the package?)

[heitorlessa]

Hey @ericbn, thanks for another great question ;)

TL;DR: This is because we remove what's already available in Lambda when building the Layer, whereas locally you'll be bringing that as a peer dependency because of AWS X-Ray SDK.

Answering the questions now:

Shouldn't the package versions all be the same between the layer and the aws_lambda_powertools extra(s)?

Not in the context you ran with pkg_resources, because (A) it brings additional libraries like awslambdaric only available at runtime, (B) we're setting a minimum boto3 version to match with what's available at Lambda runtime at the time of writing (quite old), (C) and X-Ray SDK depends on boto3 1.11.3 or higher so pip/poetry/hatch/etc will upgrade boto3 either way as all includes [tracer].

Shouldn't awslambdaric and boto3 already be included in aws_lambda_powertools[all] (the "all" extra), or some other extra that is intended to match the packages in the layer?

awslambdaric won't be part because it's specific to working with the Lambda Runtime API (we don't use it), however boto3 is a dependency of [tracer] (X-Ray SDK) so it's being inherited in whatever most up-to-date.

You can try this by downloading the actual Layer we build for Lambda, which will already have no awslambdaric via: aws lambda get-layer-version-by-arn --arn arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:17 --region eu-west-1

In the Lambda Layer however, we do the following as part of the build process:

• Install Cython and compile the latest version of pydantic (TODO: we should be more careful here @rubenfonseca, to avoid Pydantic V2 breakage)
• Install all packages ([all])
• Then finally, we remove everything that is already available in the Lambda Runtime to optimize the final package size (~97%) since AWS SDK is the bulk of it.

Any plans on updating the boto3 version in next releases of the layer and the package?

Could you send us this question from your corporate email? Lambda team is aware is outdated and not ideal, so we'd like to do a +1 on your behalf to help prioritize it.

What can we do them? (our own question for next steps)

The ideal scenario would be Lambda Runtime to have more regular updates. As it's outside of our control, we have two options here:

A. Improve our messaging (I somehow thought we did) in the Local development section that the AWS SDK available at Lambda Runtime may not be up to date. Therefore, customers should bundle AWS SDK into their dependencies (Layer won't) if they need it.

B. Potentially lock aws-sdk to the version we know it's available in the Lambda Runtime (and keep an eye on it), so pip install aws-lambda-powertools[all,aws-sdk] will always match Lambda Runtime. Except, this leaves customers to potential deadlocks where dependency resolution may fail IF X-Ray SDK changes their minimum version at some point, or whatever dependency a customer brings may also depend on a different AWS SDK minimum version

I think we should do A regardless. As for B, I think it can be fragile as we won't know when it might break - instead I'd like us to keep asking Lambda to prioritize more regular updates so this becomes less of an issue for everyone.

Please let us know!!

Thank you,
Heitor Lessa

[ericbn]

Oi @heitorlessa. And thanks for another detailed answer!

All makes sense, and I didn't know about the X-Ray SDK dependencies playing a part on it.

Good point on option B potentially causing dependency resolution issues due to locked versions.

I agree option A is easier to maintain. Under Local development, I might have missed "Powertools relies on the AWS SDK bundled in the Lambda runtime" or missed what else does that imply. Maybe the Lambda Layer section could mention that what's already available in the Lambda Runtime is removed from the Layer. Perhaps mention that it means specifically boto, botocore and their dependencies. Perhaps even list which packages are contained in the Layer; of course in case it's feasible to keep that up to date (oh I wish the Lambda Runtime would have that documented).

Thanks again!

[rubenfonseca]

@heitorlessa regarding this

Install Cython and compile the latest version of pydantic (TODO: we should be more careful here @rubenfonseca, to avoid Pydantic V2 breakage)

We pin the pydantic dependency to 1.x.x. So when they release v2 we will still use v1 everywhere, including the layer.

[heitorlessa]

Converted this into an issue and will be updating the documentation this week -- thanks a lot @ericbn once again. On Lambda Runtime having it documented, we continue to ask the Lambda team for that quality improvement.

Instead of documenting what packages and their versions available in the Layer, I'll extract this "hidden" piece on how customers can peek into the contents, and outline what goes in the Layer and how they can download its details for scanning or anything.

[heitorlessa]

Took a few hours to find the right placement and experiment with the right wording in an inclusive and global English manner - #1978

Added screenshots of what the final results look like too :)

Appreciate the patience while we were dealing with other issues in the code base.

[ericbn]

Hey, you're the one being super patient here. Appreciate the great work you're doing with this tool. Just added my comments to #1978. Looks great!