Poor error message when tmp dir is mounted as noexec with CLIv2

[ctaintor]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
Currently if you try to run the CLIv2 in an environment where the tmp directory is mounted noexec you get a very confusing error message. This makes it hard to figure out how to fix the issue and also leads to question on why the CLI needs to have tmp mounted with exec.

This bug is reported in #5047 but it was unfortunately closed since there is a workaround. I opened this to track the poor error message

SDK version number

aws-cli/2.1.22 Python/3.7.3 Linux/4.14.214-160.339.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off

Platform/OS/Hardware/Device

• Amazon Linux 2 - EC2 (but really any Linux)

To Reproduce (observed behavior)

• create a docker image or EC2 instance that has tmpdir mounted noexec. Example can be found in MemoryError thrown in aws cli 2 in buster-slim docker image #5047
• run aws - you will get output like this

$ aws
Traceback (most recent call last):
  File "PyInstaller/loader/pyiboot01_bootstrap.py", line 127, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/codebuild/output/src893450325/src/repos/awscli/.tox/exe/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module
  File "ctypes/__init__.py", line 543, in <module>
  File "ctypes/__init__.py", line 273, in _reset_cache
MemoryError
[19632] Failed to execute script pyiboot01_bootstrap

the workaround for now is to set $TMPDIR (or $HOME since Python will fallback to this for tmpdir)

$ TMPDIR=/root aws

usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help

aws: error: the following arguments are required: command

Expected behavior
I'd expect either:

• the AWS CLI works even when tmpdir is mounted noexec
  OR
• the error message clearly states the

Logs/output

Additional context

[kdaily]

Hi @ctaintor,

Thanks for reporting. I'm not able to reproduce with only a /tmp filesystem mounted with noexec. In order to get this error, I need to also make the root directory read-only. On a container image where I installed the AWS CLI V2 bundle, it's successful with noexec-mounted /tmp but with a rw root filesystem:

> docker run -it --rm --tmpfs /tmp:rw,noexec,nosuid,size=65536k aws-cli-5047 aws --version
aws-cli/2.1.22 Python/3.7.3 Linux/4.19.121-linuxkit exe/x86_64.debian.10 prompt/off

As well as with exec-mounted /tmp but read only root filesystem:

> docker run -it --rm --tmpfs /tmp:rw,exec --read-only aws-cli-5047 aws --version
aws-cli/2.1.22 Python/3.7.3 Linux/4.19.121-linuxkit exe/x86_64.debian.10 prompt/off

But not without both:

> docker run -it --rm --tmpfs /tmp:rw,noexec,nosuid,size=65536k --read-only aws-cli-5047 aws --version
Traceback (most recent call last):
  File "PyInstaller/loader/pyiboot01_bootstrap.py", line 127, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/codebuild/output/src464632960/src/repos/awscli/.tox/exe/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module
  File "ctypes/__init__.py", line 543, in <module>
  File "ctypes/__init__.py", line 273, in _reset_cache
MemoryError
[8] Failed to execute script pyiboot01_bootstrap

I can confirm the strace that was posted in #5047 (comment). Going to have to look into it further to see what PyInstaller is doing there - if the behavior can be changed and if we can even catch this error to give a better message. As I understand it, this happen before any AWS CLI code is run, so making the change there would not help.

[ctaintor]

thanks for digging further - what you say about root directory makes sense since for us it is with a security hardened versino of AL2. I'm looking forward to understanding what PyInstaller is doing :)

[kdaily]

@ctaintor,

I don't have any updates on what PyInstaller is doing in this case. However, we have a new proposal for a source distribution that would probably be a better fit for security hardened cases where you would like to maintain complete control over the build and install:

#6352

I would say that once that is made fully available it would solve your use case.

[kdaily]

Closing this out. Please track on the progress of the source distribution as that will be the path forward for cases like this.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.