Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(lambda-python): commands run non-sequentially on Graviton when building container image #22398

Merged
merged 2 commits into from
Oct 12, 2022

Conversation

mmaeng
Copy link
Contributor

@mmaeng mmaeng commented Oct 6, 2022

fixes #22012

There were reports that Dockerfile RUN commands are done out of order when building the container image on Graviton. I combined all the separate RUN commands into a single command and using the shell to explicitly enforce the sequence.


All Submissions:

Adding new Unconventional Dependencies:

  • This PR adds new unconventional dependencies following the process described here

New Features

  • Have you added the new feature to an integration test?
  • [] Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Oct 6, 2022

@github-actions github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/small Small work item – less than a day of effort p1 labels Oct 6, 2022
@aws-cdk-automation aws-cdk-automation requested a review from a team October 6, 2022 16:25
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

@mmaeng
Copy link
Contributor Author

mmaeng commented Oct 6, 2022

Docker image was built using updated Dockerfile. Packages installed within the image

bash-4.2# pip freeze
attrs==22.1.0
backports.cached-property==1.0.2
CacheControl==0.12.11
cachy==0.3.0
certifi==2022.9.24
cffi==1.15.1
charset-normalizer==2.1.1
cleo==1.0.0a5
crashtest==0.3.1
cryptography==38.0.1
distlib==0.3.6
dulwich==0.20.46
filelock==3.8.0
html5lib==1.1
idna==3.4
importlib-metadata==4.13.0
importlib-resources==5.9.0
jaraco.classes==3.2.3
jeepney==0.8.0
jsonschema==4.16.0
keyring==23.9.3
lockfile==0.12.2
more-itertools==8.14.0
msgpack==1.0.4
packaging==21.3
pexpect==4.8.0
pipenv==2022.4.8
pkginfo==1.8.3
pkgutil_resolve_name==1.3.10
platformdirs==2.5.2
poetry==1.2.1
poetry-core==1.2.0
poetry-plugin-export==1.1.1
ptyprocess==0.7.0
pycparser==2.21
pylev==1.4.0
pyparsing==3.0.9
pyrsistent==0.18.1
requests==2.28.1
requests-toolbelt==0.9.1
SecretStorage==3.3.3
shellingham==1.5.0
six==1.16.0
tomlkit==0.11.5
typing_extensions==4.3.0
urllib3==1.26.12
virtualenv==20.16.5
virtualenv-clone==0.5.7
webencodings==0.5.1
zipp==3.8.1

bash-4.2# pip --version
pip 22.2.2 from /usr/app/venv/lib/python3.7/site-packages/pip (python 3.7)

@TheRealAmazonKendra TheRealAmazonKendra changed the title fix(aws-lambda-python): commands run non-sequentially on Graviton when building container image fix(lambda-python): commands run non-sequentially on Graviton when building container image Oct 12, 2022
@TheRealAmazonKendra TheRealAmazonKendra added pr-linter/exempt-test The PR linter will not require test changes pr-linter/exempt-integ-test The PR linter will not require integ test changes labels Oct 12, 2022
@aws-cdk-automation aws-cdk-automation dismissed their stale review October 12, 2022 20:44

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

@mergify
Copy link
Contributor

mergify bot commented Oct 12, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 9bce521
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit e427fd6 into aws:main Oct 12, 2022
@mergify
Copy link
Contributor

mergify bot commented Oct 12, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify bot pushed a commit that referenced this pull request Oct 19, 2022
…ll (#22512)

This is another attempt at fixing #22012. #22398 intended to fix it, but the issue still occurs when building on arm64 machines with both `compatible_runtimes=[lambdas.Runtime.PYTHON_3_9]` and `compatible_architectures=[lambdas.Architecture.ARM_64]`

The core problem is that the build leaves root-owned files under `/tmp/pip-cache`, which then cause permission problems when subsequent layers calling `pip install` are unable to create temporary directories. Example error:

```
WARNING: Building wheel for alembic-utils failed: [Errno 13] Permission denied: '/tmp/pip-cache/wheels/5c'
```

Expanding on the idea within #22398, this PR adds one more step to the chain of operations, removing any temporary files from the cache that are no longer needed (because the packages are now installed), and ensuring that subsequent layers are able to create temporary files inside the cache directories.

Comparing the contents of the Docker image before and after this change...

Before:
```
$ ls -aFl /tmp/*cache
/tmp/pip-cache:
total 16
drwxrwxrwx  4 root root 4096 Oct 15 00:57 ./
drwxrwxrwt  1 root root 4096 Oct 15 00:58 ../
drwxr-xr-x 18 root root 4096 Oct 15 00:57 http/
drwxr-xr-x  2 root root 4096 Oct 15 00:57 selfcheck/

/tmp/poetry-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 00:57 ./
drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../
```

After:
```
$ ls -aFl /tmp/*cache
/tmp/pip-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 01:00 ./
drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../

/tmp/poetry-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 00:59 ./
drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../
```

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
mrgrain pushed a commit to mrgrain/aws-cdk that referenced this pull request Oct 24, 2022
…ll (aws#22512)

This is another attempt at fixing aws#22012. aws#22398 intended to fix it, but the issue still occurs when building on arm64 machines with both `compatible_runtimes=[lambdas.Runtime.PYTHON_3_9]` and `compatible_architectures=[lambdas.Architecture.ARM_64]`

The core problem is that the build leaves root-owned files under `/tmp/pip-cache`, which then cause permission problems when subsequent layers calling `pip install` are unable to create temporary directories. Example error:

```
WARNING: Building wheel for alembic-utils failed: [Errno 13] Permission denied: '/tmp/pip-cache/wheels/5c'
```

Expanding on the idea within aws#22398, this PR adds one more step to the chain of operations, removing any temporary files from the cache that are no longer needed (because the packages are now installed), and ensuring that subsequent layers are able to create temporary files inside the cache directories.

Comparing the contents of the Docker image before and after this change...

Before:
```
$ ls -aFl /tmp/*cache
/tmp/pip-cache:
total 16
drwxrwxrwx  4 root root 4096 Oct 15 00:57 ./
drwxrwxrwt  1 root root 4096 Oct 15 00:58 ../
drwxr-xr-x 18 root root 4096 Oct 15 00:57 http/
drwxr-xr-x  2 root root 4096 Oct 15 00:57 selfcheck/

/tmp/poetry-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 00:57 ./
drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../
```

After:
```
$ ls -aFl /tmp/*cache
/tmp/pip-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 01:00 ./
drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../

/tmp/poetry-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 00:59 ./
drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../
```

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/small Small work item – less than a day of effort p1 pr-linter/exempt-integ-test The PR linter will not require integ test changes pr-linter/exempt-test The PR linter will not require test changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

aws-lambda-python: Layer build fails
3 participants