Feature: Graceful shutdown

[sallory]

Hi

I would like to use FastStream in my project. However, zero downtime is crucial for me. Here's how I see it:

Handling SIGTERM -> the app should stop receiving new messages.

for handler in broker.handlers.values():
    await handler.close()

But not close the connection (because we need to proceed with all current messages, at least try)

Handling SIGKILL (INT) -> app can close handlers and the connection (now it works the same with SIGTERM and SIGINT)

We can wait for all tasks to be completed (I can kill it with a second signal) or wait for some timeout limit, like stop_grace_period in docker swarm.

In my case, tasks run a maximum of 15 secs, so I can do it like this:

async def on_shutdown():
    for handler in broker.handlers.values():
        await handler.close()
    await asyncio.sleep(15)

app.on_shutdown(on_shutdown)

But it's not a solution I guess

I'm thinking about waiting for all tasks:

tasks = asyncio.all_tasks() # should remove current task
done, pending = asyncio.wait(tasks, stop_grace_period)
# and we can inform that pending tasks are not completed

This code should run before App._shutdown and only if SIGTERM was received

I can try to implement it and create mr, what do you think guys?

[davorrunje]

@sallory that's a great proposal! We would truly appreciate if you could create a PR for this feature.

[Lancetnik]

Probably, we should just add asyncio.Lock to all handlers, take it in consume and wait for release in stop (with some timeout)

Broker already waits for all handlers shutdown, so we need to just stop them gracefully