Import and/or client order now appears to matter between boto3 and moto

[kgutwin]

I was eagerly expecting #1907 and moto 1.3.7 to resolve my failing tests... however, even with all relevant modules up to date, I'm still getting botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the XXX operation: The security token included in the request is invalid.

(for the record, here's my current module versions:)

boto==2.49.0
boto3==1.9.37
botocore==1.12.37
moto==1.3.7

Trying to reproduce the problem and narrow it down led me to an odd test case. For me, the following code completes successfully:

import boto3
from moto import mock_iam
iam_resource = boto3.resource('iam')

with mock_iam():
    user = iam_resource.create_user(UserName='john')
    print(user)

While this results in the ClientError mentioned previously:

import boto3
iam_resource = boto3.resource('iam')
from moto import mock_iam

with mock_iam():
    user = iam_resource.create_user(UserName='john') # throws ClientError
    print(user)

Note that the only difference is that the mock_iam import now appears after the definition of iam_resource.

Of course, you may think as I did, we may just need to re-initialize the iam_resource instance - but sadly that doesn't work:

import boto3
iam_resource = boto3.resource('iam')
from moto import mock_iam

with mock_iam():
    iam_resource = boto3.resource('iam')
    user = iam_resource.create_user(UserName='john') # still throws ClientError
    print(user) 

For me, this is the problem - in my test suite, I have one test file which loads boto3 resources but no mocks, then subsequent test files load the mocks before the boto3 resources, but ClientError is still thrown. If I run pytest with just one test file at a time, all tests pass.

Is this a case of user error? Is there some way I can "reset" boto3's clients and resources after they've first initialized, so that moto can intercept the requests properly? Or is this a moto bug?

[kgutwin]

I think I've figured out how to work around this issue. Calling boto3's setup_default_session() method causes the default session to be regenerated, with moto's hooks in place if moto has been activated.

The following works:

import boto3
iam_resource = boto3.resource('iam')
from moto import mock_iam

with mock_iam():
    boto3.setup_default_session()
    iam_resource = boto3.resource('iam')
    user = iam_resource.create_user(UserName='john') # succeeds
    print(user)

If this is a FAQ, it might need to be added to the documentation.

[ajhodges]

Also ran into this issue, thanks so much for your setup_default_session workaround. I would definitely say this is a moto issue.

[hingen]

after using mock_s3, it was still making calls to the AWS service. setup_default_session worked for me thx :)

[luispollo]

I have the same problem, except the workaround above does not work for me... Changing the order of imports doesn't fix it either. ☹️

Versions:

boto                         2.49.0
boto3                        1.9.118
botocore                     1.12.118
moto                         1.3.7

I'm on Python 2.7, in case that matters at all.

[dazza-codes]

I had to refactor all my package code to use lazy boto3 resource instantiations. Then everything works OK, even without the setup_default_session option. On gitlab CI, providing bogus environment variables for the AWS credentials worked OK too.

So instead of:

@dataclass
class S3Util:
    client = boto3.resource('s3')

Use properties to return on-demand and garbage collect them. Let boto3 manage the resource pools etc. and don't hang on to them. Something like:

class S3Util:
    @property
    def client(self):
        return boto3.resource('s3')

[jyun5000]

adding in boto3.setup_default_session() inside of the test with the @decorator/@mock_sns worked fine for me.

@mock_sns
def test_sns():
    boto3.setup_default_session()
    sns_client = boto3.client(service_name='sns', region_name='us-west-2',
                              aws_access_key_id='foo', aws_secret_access_key='boo')

    sns_client.create_topic(Name='test-topic')```

[krtkvrm]

I solved it using

botocore==1.12.85
git+https://github.com/spulec/moto.git@df493ea18de22b4533073eee3e296686019911c2

Installing via commit till moto>1.3.8

[spulec]

As mentioned, this should be fixed in master, but feel free to reopen if there are still issues.

[awiddersheim]

I ran into this today and it was solved using the boto3.setup_default_session() call. I'm not really sure where things were going wrong exactly but it lead me here. I'm using fairly new stuff:

moto==1.3.14.dev326
boto3==1.9.81
botocore==1.12.81

@spulec Do you know which commit fixes this so I can see if I'm using the right versions?

[spulec]

Feel free to try the very latest version: 1.3.15.dev393

[brady-gsa]

@spulec I'm also seeing this using the latest master, but can't call boto3.setup_default_session() since I'm using Terraform, not Python. Any suggestions?