Optional exponential backoff

[joshowen]

Amazon recommends using exponential backoff on 5xx errors and on ProvisionedThroughputExceededException responses. I haven't tested this code yet, but wanted to see if there is interest in including it in the project. If so I'll clean it up and get some documentation written.

[joshowen]

@jlafon @danielhochman curious on your thoughts

[jlafon]

@joshowen What about using botocore's implementation?

[danielhochman]

My opinion:

I'm a fan of handling retries at the application layer. As an example, botocore's retry logic is not well understood. When this kind of handling is on by default, it's sure to behave unexpectedly during an outage or throttling, and can mask real issues with the application in the case that it does recover operations. Some may also want more granular retry logic (e.g. per operation), in which case you have to rewrite it yourself anyways.

If we do ship something like this I would vote for it to be off by default.

[joshowen]

I actually didn't know botocore had an implementation included. This PR has retrying disabled by default, and I'm currently using it in production without any issues. Is there any interest in including it in the project?

[joshowen]

@jlafon any thoughts on including this functionality in the project? Happy to rewrite to use BotoCore's implementation if you want it included.

[jlafon]

@joshowen Using botocore's sounds like a good idea.

[danielhochman]

superseded by #135