Moto 1.3 causes tests to fail when assert_all_requests_are_fired = True

[jakul]

Moto v1.3 causes my tests to fail if I set responses.mock.assert_all_requests_are_fired = True.

I set responses.mock.assert_all_requests_are_fired = True to make sure that my application is behaving as expected, but since it doesn't make all the requests to all the moto urls then my test fails.

How to recreate

craig@amogato2:/tmp
$ virtualenv /tmp/moto-error
New python executable in /tmp/moto-error/bin/python
Installing setuptools, pip, wheel...done.

craig@amogato2:/tmp
$ source /tmp/moto-error/bin/activate

[moto-error] craig@amogato2:/tmp
$ pip install pytest-responses moto

... output removed ...

[moto-error] craig@amogato2:/tmp
$ echo -e "import responses\nfrom moto import *\nresponses.mock.assert_all_requests_are_fired = True\n@mock_dynamodb2\ndef test_pass():\n    pass" > test.py

[moto-error] craig@amogato2:/tmp
$ pytest test.py
======================================== test session starts ========================================
platform darwin -- Python 2.7.10, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /private/tmp, inifile:
plugins: responses-0.3.0
collected 1 item

test.py F

============================================= FAILURES ==============================================
_____________________________________________ test_pass _____________________________________________

args = (), kwargs = {}, result = None

    def wrapper(*args, **kwargs):
        self.start(reset=reset)
        try:
            result = func(*args, **kwargs)
        finally:
>           self.stop()

/tmp/moto-error/lib/python2.7/site-packages/moto/core/models.py:73:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/moto-error/lib/python2.7/site-packages/moto/core/models.py:65: in stop
    self.disable_patching()
/tmp/moto-error/lib/python2.7/site-packages/moto/core/models.py:220: in disable_patching
    responses_mock.stop()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <responses.RequestsMock object at 0x10c59ab90>, allow_assert = True

    def stop(self, allow_assert=True):
        self._patcher.stop()
        if not self.assert_all_requests_are_fired:
            return
        if not allow_assert:
            return
        not_called = [m for m in self._matches if m.call_count == 0]
        if not_called:
            raise AssertionError(
                'Not all requests have been executed {0!r}'.format([(
>                   match.method, match.url) for match in not_called]))
E           AssertionError: Not all requests have been executed [(u'GET', <_sre.SRE_Pattern object at 0x10d53d030>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x7fcd835e9530>), (u'GET', <_sre.SRE_Pattern object at 0x10de469b8>), (u'GET', <_sre.SRE_Pattern object at 0x7fcd835e9f10>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'GET', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10d53d030>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x7fcd835e9530>), (u'DELETE', <_sre.SRE_Pattern object at 0x10de469b8>), (u'DELETE', <_sre.SRE_Pattern object at 0x7fcd835e9f10>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'DELETE', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10d53d030>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x7fcd835e9530>), (u'HEAD', <_sre.SRE_Pattern object at 0x10de469b8>), (u'HEAD', <_sre.SRE_Pattern object at 0x7fcd835e9f10>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'HEAD', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10d53d030>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x7fcd835e9530>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10de469b8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x7fcd835e9f10>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'OPTIONS', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10d53d030>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x7fcd835e9530>), (u'PATCH', <_sre.SRE_Pattern object at 0x10de469b8>), (u'PATCH', <_sre.SRE_Pattern object at 0x7fcd835e9f10>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PATCH', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10d53d030>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x7fcd835e9530>), (u'POST', <_sre.SRE_Pattern object at 0x10de469b8>), (u'POST', <_sre.SRE_Pattern object at 0x7fcd835e9f10>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'POST', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10d53d030>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x7fcd835e9530>), (u'PUT', <_sre.SRE_Pattern object at 0x10de469b8>), (u'PUT', <_sre.SRE_Pattern object at 0x7fcd835e9f10>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>), (u'PUT', <_sre.SRE_Pattern object at 0x10ed9d1f8>)]

/tmp/moto-error/lib/python2.7/site-packages/responses.py:587: AssertionError
===================================== 1 failed in 1.21 seconds ======================================

[moto-error] craig@amogato2:/tmp
$ pip install moto==1.2

... output removed ...

[moto-error] craig@amogato2:/tmp
$ pytest test.py
======================================== test session starts ========================================
platform darwin -- Python 2.7.10, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /private/tmp, inifile:
plugins: responses-0.3.0
collected 1 item

test.py .

===================================== 1 passed in 0.90 seconds ======================================

[moto-error] craig@amogato2:/tmp
$ pip freeze
asn1crypto==0.24.0
attrs==17.4.0
aws-xray-sdk==0.95
backports.ssl-match-hostname==3.5.0.1
backports.tempfile==1.0
backports.weakref==1.0.post1
boto==2.48.0
boto3==1.7.8
botocore==1.10.8
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
cookies==2.2.1
cryptography==2.2.2
docker==3.2.1
docker-pycreds==0.2.2
docutils==0.14
enum34==1.1.6
funcsigs==1.0.2
futures==3.2.0
idna==2.6
ipaddress==1.0.22
Jinja2==2.10
jmespath==0.9.3
jsondiff==1.1.1
jsonpickle==0.9.6
MarkupSafe==1.0
mock==2.0.0
more-itertools==4.1.0
moto==1.2.0
pbr==4.0.2
pluggy==0.6.0
py==1.5.3
pyaml==17.12.1
pycparser==2.18
pytest==3.5.1
pytest-responses==0.3.0
python-dateutil==2.6.1
pytz==2018.4
PyYAML==3.12
requests==2.18.4
responses==0.9.0
s3transfer==0.1.13
six==1.11.0
urllib3==1.22
websocket-client==0.47.0
Werkzeug==0.14.1
wrapt==1.10.11
xmltodict==0.11.0

[jakul]

I know that responses supports multiple mocks, so maybe the solution is for moto to use it's own mock instance, not the global one

[spulec]

I'm not sure there is a way for us to solve this. Although responses allows multiple mocks, I don't believe it will work properly with multiple mocks for the same global requests namespace.

I would be happy to be proven wrong though.

[spulec]

I believe this should be solved with #2294, but let me know if there are still issues.