Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backoff code inside function #18

Open
luckydonald opened this issue Sep 17, 2016 · 11 comments
Open

Backoff code inside function #18

luckydonald opened this issue Sep 17, 2016 · 11 comments

Comments

@luckydonald
Copy link

luckydonald commented Sep 17, 2016

I am calling some functions of a lib i wanna backoff.
Problem is, I cannot modify the lib.

import backoff
from requests.exceptions import TimeoutError
from somewhere import library

So I am looking for something like:

with backoff.on_exception(backoff.expo, TimeoutError, max_tries=8):
    library.some_function(yada, blah, foo, bar)
# end if

or

while backoff.on_exception(backoff.expo, TimeoutError, max_tries=8):
    library.some_function(yada, blah, foo, bar)
# end if

Not sure what is possible to do.

Edit (2017-04-27): Calling just one function can be solved like seen below

@bgreen-litl
Copy link
Member

I like this idea. In particular, the context manager variant of it. That said, I've tried to keep the API surface of this library as small as possible, and I'd definitely want to spend some time and make sure we're getting it right if we were going to make a non-trivial expansion of it. I do think it's possible to have a function act as both a decorator and a context manager and so I think the API you suggested is possible. But the implementation may vary or be limited depending on the python version in use. Right now we support all the way back to 2.6!

(If think you probably know, but the way to do with with the current API is to create a thin wrapping function around the 3rd party function which you then decorate. If it's really just a 1 for 1, param for param wrapper it does end up seeming a bit awkward.)

@luckydonald
Copy link
Author

Yeah, workaround is

@backoff.whatever(...)
def retry(*args, **kwargs):
    return lib.func(*args, **kwargs)

@bgreen-litl bgreen-litl added this to the v2.0 milestone Nov 18, 2016
@bgreen-litl
Copy link
Member

bgreen-litl commented Nov 22, 2016

I spent a little time exploring the context manager variant of this. Unfortunately, it's not possible to retry blocks of code this way.

See https://www.python.org/dev/peps/pep-0343/ - specifically the comments about Anonymous Block Statements under Motivation and Summary.

Having thought about this more, I'd recommend one of two best practices for backoff:

  1. If the function call you want to backoff is used in more than one function in your code, declare a shared utility wrapper function decorated as desired. The same as what we discussed above.

     @backoff.on_exception(backoff.expo, foolib.FooException, max_tries=3)
     def foolib_call(*args, **kwargs):
         foolib.call(*args, **kwargs)
    
  2. But perhaps more commonly, if the call only happens in one place, or (requires different backoff params per call), declare and decorate a local scope function inline when needed. Note you don't need to bother with *args and **kwargs in this case as you can just reference any variables available in the calling scope directly. This isn't much wordier than the context manager proposal above, and it has the distinct advantage of actually being possible.

     def main():
         arg = 'foo'
    
         @backoff.on_exception(backoff.expo, foolib.FooException, max_tries=3)
         def call():
             return foolib.call(arg)
    
         result = call()
    

@bgreen-litl bgreen-litl removed this from the v2.0 milestone Nov 22, 2016
@luckydonald
Copy link
Author

It the problem of a with is that it can't be rerun, and a while that it can't catch exceptions,
how about combining them?

for retry in backoff.on_exception(backoff.expo, TimeoutError, max_tries=8):
   with retry:
      library.some_function(yada, blah, foo, bar)
   # end with
# end for

The for would handle the looping and return a context manager for the with statement.
The context manager (retry) in the with statement would then handle the exception, and notify the for loop to go another round or be done.

@pquentin
Copy link
Contributor

@bgreen-litl Could you consider adding this in the documentation? I believe context managers would be more natural, too bad they don't work for retrying (except if @luckydonald suggestion works).

Anyway, I think many of us would appreciate seeing this explanation in the documentation.

Thank you for this great library! 👍

@bgreen-litl
Copy link
Member

@pquentin Thank you for the suggestion. I think adding these best practices to the documentation is a fine idea. That documentation is getting a bit crowded as a simple README, but I'll see if we can fit one more thing in before we need to blow it up into something bigger.

Also, I haven't completely given up on the original context manager idea. I still think it's a good one. It would need to be structured a little differently than @luckydonald originally suggested, but I think there is a path to a context manager version of backoff that might work. I've had a few false starts on an implementation for it, but the idea I've been working with is something like this:

retry = backoff.on_exception(backoff.expo, requests.Exception)
with retry(foolib.call) as retry_call:
    retry_call('arg1','arg2')

The important difference from the original suggestion is that the value of the context manager needs to be a function rather than an anonymous block of code, which won't work unfortunately.

I have this issue under the 2.0 milestone, and I am still working on it. Any insights or ideas are appreciated.

I do worry that my version of the context manager idea above might not be much better or more natural or succinct than the local scope decorated function I documented. I want to make sure that if there's a trade off in terms of complexity that it's actually worth it...

@pquentin
Copy link
Contributor

(It looks like this specific issue is no longer under the 2.0 milestone, at least according to GitHub.)

You're right that a local function sounds less magical and simpler even if it's slightly more verbose. And it's more well, TOOWTDI. :)

@luckydonald
Copy link
Author

luckydonald commented Apr 27, 2017

@pquentin, to which of my suggestion did you refer?

@bgreen-litl, What about the for+with?
It also looks handy for normal code you don't really wan't to put in an extra function.

So 2 lines would be still less than writing it yourself every time:

for retry in backoff.retry_on_exception(backoff.expo, TimeoutError, max_tries=5):
   with retry:
     # >>> do stuff here <<<
   # end with
# end for

Now manually:

timeout = backoff.expo()
for try_count in range(5):
  try:
     # >>> do stuff here <<<
     break  # because executed successfully
  expect TimeoutError as e:
    if try_count == 5:
      raise e
    # end if
   sleep(timeout.next())
  # end try
# end for  

@luckydonald
Copy link
Author

luckydonald commented Apr 27, 2017

I just realized:
If it is a just a single function - even in a library - you could supply the decorator yourself:
So as solution to just the problem written in the original posting (calling a single function) this should be sufficient:

from somewhere.library import some_function as _some_function
import backoff
from requests.exceptions import TimeoutError

some_function = backoff.on_exception(backoff.expo, TimeoutError, max_tries=8)(_some_function)

some_function(yada, blah, foo, bar)

@shawnzhu
Copy link

I just adopted the code example from #18 (comment) into IBM-Cloud/sql-query-clients#65

@luckydonald Thank you!

@mikepii
Copy link

mikepii commented Nov 30, 2023

Somewhat related: #210 proposes supporting wrapping a Callable created by functools.partial(). It's already close to working, but backoff would need to log repr(callable) rather than callable.__name__ in the "backing off" message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants