Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Library is not ready for async usage #3

Open
lun-4 opened this issue Jun 22, 2022 · 2 comments
Open

Library is not ready for async usage #3

lun-4 opened this issue Jun 22, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@lun-4
Copy link

lun-4 commented Jun 22, 2022

At the moment, the documentation of booru says

This lib should be run in async context, and it's recommended to use asyncio

However, the library depends on requests, a very good HTTP library that is blocking, which means that calls to boorus will also block, which is not aligned with the asyncio model.

Because of that, the following piece of code, inspired by the README:

import asyncio
import booru

async def main():
    gel = booru.Gelbooru()
    # search for 1000 random cat_girls, in parallel batches of 100 at once to the Gelbooru API
    results = await asyncio.gather(*[
        gel.search("cat_girl sort:random")
        for _ in range(10)
    ])
    results = [booru.resolve(res) for res in results]
    print(results)

asyncio.run(main())

Will run each booru search sequentially, because requests.get blocks every other coroutine from doing any work. So, if gelbooru takes 1 second per request, that script would take a total of 10 seconds, where it would be expected for it to take around 1-3 seconds depending on network conditions and whatnot.

On my use-case, I create hundreds of tasks that are forced to operate sequentially, even though I want to have say, 10 of them running in parallel.

The library should either

  • Consider an async HTTP library for its internals, like aiohttp, or
  • Don't recommend itself as being used in an async context, as that is misleading.
@sinkaroid
Copy link
Owner

Sorry, didn't expect requests will stopping the flow, actually you right, definitely will consider this issue for future relase.
Thanks for remind me

@sinkaroid sinkaroid added the enhancement New feature or request label Jun 24, 2022
@sinkaroid
Copy link
Owner

I just finished rebranding this lib with aiohttp, it is should working as expected. Feel free to make test cases again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants