Should Moto mock external requests?

[spulec]

While moto is active, should requests to other sites (ex http://www.google.com) still work?

I'm leaning toward yes. At the same time, I think it is important that any amazon service that is not implemented yet throw an error (see #1254).

I believe this means we should add a passthrough for * , but add a fake error responses for *amazonaws.com*

Thoughts?

[thehesiod]

personally I'd like to see the redirection work through proxying, without any patching to the underlying http client, perhaps requiring users to explicitly set an AWS client's proxy, ex: boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))

[killthrush]

+1 for passthroughs to external sites and throwing an error for unsupported services. I'm hearing from others that requests to AWS interleaved with calls to other services is common.

Additionally, clean stress-free unpatching of moto is still important IMO even if we had default passthroughs. To get the best test coverage, I use both moto and localstack, the latter of which is hitting a docker container on localhost. I don't generally need to combine the use of these two technologies in a single test case, but within a single run of the suite both will be used. If moto is not unpatched properly, it breaks tests based on localstack and also anything else that happens to use requests. So far, I've not yet been able to get unpatching to work, even with the current release.

[spulec]

@killthrush thanks for the feedback.

Just to confirm, the unpatching is still failing on 1.3.2? If so, any chance you could add some example code to #1026. I'd really like to get that fixed.

[killthrush]

Thanks @spulec - I'm working on a repo for this. I've reproduced the problem with 1.3.2, but only if I run tests a certain way. I think there may be something more subtle going on here. Will keep you posted.

[killthrush]

Alright, I think I've narrowed this down to something more specific. v1.3.2 fixes the mock problem when running pytest from the command line or when running pytest directly using python -m pytest test_dir. However PyCharm's pytest runner still fails for some reason on this particular case. Would it be more helpful if I just wrote up a bug for this? I don't want to detract from your original question ^^^

[spulec]

@killthrush yes please!

[killthrush]

@spulec - never mind, the PyCharm issue was my mistake. I was writing up a bug report and I noticed that my config was pointing to the wrong virtualenv =/

So #1026 is indeed fixed and there's no bug here. Thanks! This was a really good fix 🎉

[spulec]

@killthrush thanks for following up!

[kingbuzzman]

@spulec yeah, we should stop mocking the full requests lib, its just more trouble than its worth. The biggest complain i get from my coworkers about moto is this.
@thehesiod not against this, does that mean that moto will be the mock server? Will this just become another https://github.com/localstack/localstack ??

[spulec]

@kingbuzzman can you clarify a bit: are the complaints about mocking urls beyond AWS or mucking with the requests module in general?

[kingbuzzman]

@spulec the issue is mocking urls beyond what is needed to make aws work.

[spulec]

@kingbuzzman got it; very helpful, thanks.

[nbcsteveb]

@spulec hi! i'm loving moto, it has saved me countless hours.

A specific integration case i'm dealing with right now:

Only during initial integration test development, it is helpful to allow a mock SQS send/receive for one unit but still allow another unit to have outbound requests to other URLs.

Having to disable the decorator just to get both units to work means having to hit my staging SQS queue until I can mock the second unit.

This is a small headache of using moto, I haven't looked deeper to see how to fix however some workarounds are floating around the issue list. (like this one: #1026 (comment))

[wikier]

Any progress on this?

[thehesiod]

@kingbuzzman I haven't been following things much, how do moto and localstack compare? They seem very similar? This is how I currently use moto with process isolation between services and w/o mocking external requests: http://hesiod.blogspot.com/2017/10/mocking-aws-lambdas.html My idea for proxy support is that you'd have one endpoint instead that then routes the AWS calls to each individual aws service endpoint, instead of having an endpoint_url per service.

[kingbuzzman]

@thehesiod Well like everything atlassian.. localstack promises a lot and delivers on very little. As of right now, localstack and moto are close, but there are clear differences between the two. The second you start doing more than mocking the boto calls and start having a feature full/partial of what those boto calls do -- you're going to be in localstack territory.

In the spirit of honesty. I've used localstack very little/at all -- to the point that anytime i have an issue and i say to myself, "neat! i can use localstack for this" and always follow it with a quick look at their website to say: "#$%^ they dont support my feature!". Needless to say, take everything i say with a grain of salt.

@spulec whats the deal with the mocking of external requests other than moto?

[csolamx]

Hello all,

From other threads it seems that this issue would be solved by version 1.0.0. I just installed 1.3.5 today but it seems that it's still mocking all requests. I open connections to other sites while using DynamoDB... is there a workaround for this, or should I upgrade any other package that moto depends on?

Thank you!

[thehesiod]

@csolamx I don't think this is "solved" yet, see my post above for how I accomplish this

[spulec]

This will potentially be solved with #1847

[spulec]

It would be great to have some people test this PR: #2294

[dmulter]

Already did, guess my thumbs up was a little too subtle.

[e271828-]

@spulec This should definitely be fixed. Non-boto* requests within a module should be able to reach * unless separately mocked. We're currently working around it via:

import responses
responses.add_passthru('https://')
responses.add_passthru('http://')

ahead of the moto-decorated classes in our tests, which is ugly.