Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(lambda-python-alpha): bundling hash logic includes auth tokens in PIP urls, causing an unnecessary rebuild #27903

Merged
merged 6 commits into from
Dec 23, 2023

Conversation

kcp-chewie
Copy link
Contributor

@kcp-chewie kcp-chewie commented Nov 8, 2023

Update the bundler hash logic to ignore the secret token that is included in the URL when fetching packages from Code Artifact. This token changes constantly, and prevents the results of a previous build from being reused, along with causing lambdas to be unnecessarily redeployed anytime the CDK is built, even if no code is changed.

This implementation strips the token from the hash, but does not change anything else about the hash.

Open question
Currently this logic will error if an invalid URL string is passed for PIP_INDEX_URL or PIP_EXTRA_INDEX_URL. The build would fail later anyway when those URLs are being used, but I'm happy to try/catch this logic block to be more robust if that is preferred.

Closes #27331.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. labels Nov 8, 2023
@aws-cdk-automation aws-cdk-automation requested a review from a team November 8, 2023 23:04
@github-actions github-actions bot added the p2 label Nov 8, 2023
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@kcp-chewie kcp-chewie changed the title Sanitize secret tokens in cache key when bundling lambdas fix(core): Sanitize secret tokens in cache key when bundling lambdas Nov 8, 2023
@kcp-chewie kcp-chewie changed the title fix(core): Sanitize secret tokens in cache key when bundling lambdas fix(core): sanitize secret tokens in cache key when bundling lambdas Nov 8, 2023
@kcp-chewie kcp-chewie force-pushed the sanitize-cache-key branch 2 times, most recently from 77c0adb to 54e89c9 Compare November 9, 2023 00:13
@aws-cdk-automation aws-cdk-automation dismissed their stale review November 9, 2023 06:15

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@aws-cdk-automation aws-cdk-automation dismissed their stale review November 9, 2023 22:22

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Nov 9, 2023
Copy link
Contributor

@lpizzinidev lpizzinidev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍
Just some minor adjustments in my opinion.
Also, can you please change the title to describe the bug (not the solution) as per guidelines?

packages/aws-cdk-lib/core/lib/asset-staging.ts Outdated Show resolved Hide resolved
packages/aws-cdk-lib/core/lib/asset-staging.ts Outdated Show resolved Hide resolved
@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Nov 10, 2023
@kcp-chewie kcp-chewie changed the title fix(core): sanitize secret tokens in cache key when bundling lambdas fix(core): bundling hash logic includes auth tokens in PIP urls, causing an unnecessary rebuild Nov 10, 2023
Copy link
Contributor Author

@kcp-chewie kcp-chewie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review!

packages/aws-cdk-lib/core/lib/asset-staging.ts Outdated Show resolved Hide resolved
Copy link
Contributor

@lpizzinidev lpizzinidev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks 👍

@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Nov 11, 2023
@kcp-chewie
Copy link
Contributor Author

Anything I can do to help this one get reviewed/merged?

CC @lpizzinidev

Copy link
Contributor

@kaizencc kaizencc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kcp-chewie. I surmise that this will cause unnecessary rebuilds for the first time people using PIP_INDEX_URL update their CDK with this change, but I suppose we are okay with that.

@kaizencc kaizencc changed the title fix(core): bundling hash logic includes auth tokens in PIP urls, causing an unnecessary rebuild fix(lambda-python-alpha): bundling hash logic includes auth tokens in PIP urls, causing an unnecessary rebuild Dec 22, 2023
@github-actions github-actions bot added the effort/medium Medium work item – several days of effort label Dec 22, 2023
@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Dec 22, 2023
Copy link
Contributor

mergify bot commented Dec 22, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Copy link
Contributor

mergify bot commented Dec 23, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: b679b04
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 00331a7 into aws:main Dec 23, 2023
10 checks passed
Copy link
Contributor

mergify bot commented Dec 23, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@kcp-chewie
Copy link
Contributor Author

Thanks @kaizencc !

For reference, this should only cause an unnecessary rebuild for anyone PIP_INDEX_URL with basic auth username/password and a static password. Folks using PIP_INDEX_URL without any auth should see no rebuild, and folks using PIP_INDEX_URL with a rotating token (like AWS CodeArtifact users) will see 1 final rebuild.

paulhcsun pushed a commit to paulhcsun/aws-cdk that referenced this pull request Jan 5, 2024
… PIP urls, causing an unnecessary rebuild (aws#27903)

Update the bundler hash logic to ignore the secret token that is included in the URL when fetching packages from Code Artifact. This token changes constantly, and prevents the results of a previous build from being reused, along with causing lambdas to be unnecessarily redeployed anytime the CDK is built, even if no code is changed.

This implementation strips the token from the hash, but does not change anything else about the hash. 

Open question
Currently this logic will error if an invalid URL string is passed for PIP_INDEX_URL or PIP_EXTRA_INDEX_URL. The build would fail later anyway when those URLs are being used, but I'm happy to try/catch this logic block to be more robust if that is preferred.

Closes aws#27331.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/medium Medium work item – several days of effort p2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

aws_lambda_python_alpha: Using PIP_INDEX_URL and Codeartifact cause Docker to rebuild lambda every synth
5 participants