Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asynchronous task iterator as argument to Executor.submit() #360

Merged
merged 3 commits into from
May 13, 2021

Conversation

azawlocki
Copy link
Contributor

@azawlocki azawlocki commented Apr 29, 2021

Current status: seems to work, needs polishing, more testing, a convincing example and documentation.

Update: CI fails due to

  • For Python 3.6: no asyncio.run() function present in asyncio (can be fixed easily by replacing it with asyncio.get_event_loop().run_until_complete()
  • Wrong license for the aioconsole package this branch depends on

Update: Ready for review.

Added an integration test. An excerpt:

        # We use an async task generator that yields tasks removed from
        # an async queue. Each computed task will potentially spawn
        # new tasks -- this is made possible thanks to using async task
        # generator as an input to `executor.submit()`.

        task_queue = asyncio.Queue()

        # Seed the queue with the first task:
        await task_queue.put(Task(data=3))

        async def input_generator() -> AsyncGenerator[Task]:
            """Task generator yields tasks removed from `queue`."""
            while True:
                task = await task_queue.get()
                if task.data == 0:
                    break
                yield task

        async for task in executor.submit(worker, input_generator()):
            print("task result:", task.result, file=sys.stderr)
            for n in range(task.result):
                await task.queue.put(Task(data=task.result - 1))

        print("all done!", file=sys.stderr)

@azawlocki azawlocki requested a review from a team April 29, 2021 16:45
@azawlocki azawlocki marked this pull request as draft April 29, 2021 16:47
@azawlocki azawlocki self-assigned this Apr 29, 2021
@azawlocki azawlocki changed the title Asynchronous task iterator as argument to Executor.submit() [WIP] Asynchronous task iterator as argument to Executor.submit() Apr 29, 2021
@shadeofblue shadeofblue self-requested a review May 10, 2021 08:26
@azawlocki azawlocki changed the title [WIP] Asynchronous task iterator as argument to Executor.submit() Asynchronous task iterator as argument to Executor.submit() May 11, 2021
@azawlocki azawlocki marked this pull request as ready for review May 12, 2021 08:46
@azawlocki azawlocki force-pushed the az/async-task-generator branch 4 times, most recently from d02f808 to 3045f11 Compare May 12, 2021 10:54
Copy link
Collaborator

@shadeofblue shadeofblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good as far as I could tell

@azawlocki azawlocki merged commit 5561d39 into master May 13, 2021
@azawlocki azawlocki deleted the az/async-task-generator branch May 13, 2021 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants