failure to mock aws auth - NoCredentialsError

[dazza-codes]

Forgive me for assuming the @mock_s3 decorator magically takes care of everything or for missing some RTD about this; happy to be corrected and pointed to the right RTD page.

On gitlab CI pipeline runner, there are no AWS credentials available and the @mock_s3 decorator is not mocking the credentials correctly, resulting in a test failure like:

/opt/conda/lib/python3.7/site-packages/botocore/auth.py:425: in add_auth
    super(S3SigV4Auth, self).add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.auth.S3SigV4Auth object at 0x7f3634cdedd8>
request = <botocore.awsrequest.AWSRequest object at 0x7f3634cde978>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/opt/conda/lib/python3.7/site-packages/botocore/auth.py:357: NoCredentialsError
________________________ test_download_missing_s3_file _________________________

s3_bucket_name = 'bucket'
test_cache = S3Cache(cache_dir=PosixPath('/builds/jupiterintel/gis-dataprocessing/s3_test_cache'), cache_persist=False)

    @mock_s3
    def test_download_missing_s3_file(s3_bucket_name, test_cache):
        # mock an s3 bucket but no file
        conn = boto3.resource('s3', region_name='us-east-1')
>       conn.create_bucket(Bucket=s3_bucket_name)

[MarkKewley]

I'm having similar issues as well, even adding something like AWS_ACCESS_KEY_ID=fake AWS_SECRET_ACCESS_KEY=fake python setup.py test doesn't appear to work. It looks as though the mocking isn't working because locally I see Sending http request: <PreparedRequest [GET]> and on gitlab we see Sending http request: <AWSPreparedRequest

Versions

boto3 - 1.9.97
botocore - 1.12.97
moto - 1.3.7

[dazza-codes]

Update, creating a bucket fails with permission denied errors.

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

$ conda --version
conda 4.6.7
$ python --version
Python 3.7.1

$ pip freeze | grep -i -E 'moto|boto'
boto==2.49.0
boto3==1.9.102
botocore==1.12.102
moto==1.3.7

Actual AWS credentials are working file, i.e.

$ aws s3 ls
<snipped long list of results>
$ echo $?
0

error and backtrace

TL;DR: botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied

    @mock_s3
    def test_download_missing_s3_file(s3_bucket_name, test_cache):
        # mock an s3 bucket but no file
        conn = boto3.resource('s3', region_name='us-east-1')
>       conn.create_bucket(Bucket=s3_bucket_name)

tests/test_raster_s3_io.py:472: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../miniconda3/envs/gis-dataprocessing/lib/python3.7/site-packages/boto3/resources/factory.py:520: in do_action
    response = action(self, *args, **kwargs)
../../../miniconda3/envs/gis-dataprocessing/lib/python3.7/site-packages/boto3/resources/action.py:83: in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
../../../miniconda3/envs/gis-dataprocessing/lib/python3.7/site-packages/botocore/client.py:357: in _api_call
    return self._make_api_call(operation_name, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.client.S3 object at 0x7f58880066a0>, operation_name = 'CreateBucket', api_params = {'Bucket': 'bucket'}

    def _make_api_call(self, operation_name, api_params):
        operation_model = self._service_model.operation_model(operation_name)
        service_name = self._service_model.service_name
        history_recorder.record('API_CALL', {
            'service': service_name,
            'operation': operation_name,
            'params': api_params,
        })
        if operation_model.deprecated:
            logger.debug('Warning: %s.%s() is deprecated',
                         service_name, operation_name)
        request_context = {
            'client_region': self.meta.region_name,
            'client_config': self.meta.config,
            'has_streaming_input': operation_model.has_streaming_input,
            'auth_type': operation_model.auth_type,
        }
        request_dict = self._convert_to_request_dict(
            api_params, operation_model, context=request_context)
    
        service_id = self._service_model.service_id.hyphenize()
        handler, event_response = self.meta.events.emit_until_response(
            'before-call.{service_id}.{operation_name}'.format(
                service_id=service_id,
                operation_name=operation_name),
            model=operation_model, params=request_dict,
            request_signer=self._request_signer, context=request_context)
    
        if event_response is not None:
            http, parsed_response = event_response
        else:
            http, parsed_response = self._make_request(
                operation_model, request_dict, request_context)
    
        self.meta.events.emit(
            'after-call.{service_id}.{operation_name}'.format(
                service_id=service_id,
                operation_name=operation_name),
            http_response=http, parsed=parsed_response,
            model=operation_model, context=request_context
        )
    
        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied

../../../miniconda3/envs/gis-dataprocessing/lib/python3.7/site-packages/botocore/client.py:661: ClientError

[dazza-codes]

It's possible this could be related to using boto outside a mock context anywhere in the project. Seems like if any modules anywhere in the project library or in tests initialize a boto.resource('s3') and hang on to it (or not?), the entire mock framework gets broken by one outstanding instance somewhere. This is hard to confirm, but tracking down all the calls to boto.resource and isolating them in lazy-loaders is part of the solution.

[kgutwin]

See #1926 for a possible solution - you can call boto3.setup_default_session() after your mock starts but before you create/recreate boto3 clients or resources in order to get the mock working properly.

[dazza-codes]

For gitlab, creating fake values for the AWS credentials as environment variables worked OK. All the code is careful not to create or retain any module vars/consts, class or instance vars that retain instantiation of any boto3 resources on imports. All instantiation of boto3 resources is lazy.

[spulec]

Can someone please test this PR? #2285

[spulec]

Going to close given #2285, but feel free to reopen if this still is happening.