fix(repo): Added panic hooks and reworked graceful shutdown

[0xterminator]

No description provided.

[lostman]

I'm not sure if this is what we want. Flushing async streams is tricky and we'd be better off assuming the publisher may die in the middle of publishing a block.

I talked to @0xterminator about it yesterday, and I think that the best and simplest thing we can do is this:

1. Ensure we publish the entire block N before we start publishing N+1.
2. Re-publish the last block on startup.

Currently, we look up the last published block L and we start at L+1. I have a PR that changes that to restart at L, precisely because the publisher may die before publishing inputs, receipts, and so on.

[lostman]

👍

[0xterminator]

thx

[lostman]

This overrides the default panic hook, so if the publisher panics, we won't get the panic message.

[0xterminator]

We will, the first thing I am doing in printing the stack above!

[lostman]

Flushing streams after panic is unreliable. async may be in a compromised state.

[0xterminator]

Its a usual practice to at least attempt some cleanup. Usually if we die, we might just end up in a really bad state. Also , note that both methods flush_await_all_streams and stop_fuel do not throw exceptions, i.e. they cannot fail whatsoever.

[lostman]

Yes, you can do sync cleanup but not grab the Tokio runtime. Here's an example:

use tokio::runtime::Handle;
use std::panic;

fn set_panic_hook() {
    panic::set_hook(Box::new(|panic_info| {
        println!("Panic occurred: {}", panic_info);
        // can panic
        let handle = Handle::current();
        // can panic
        handle.spawn(async {
            println!("Performing async cleanup in panic hook...");
            // can panic
            tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
            println!("Async cleanup completed.");
        });
    }));
}

#[tokio::main]
async fn main() {
    set_panic_hook();
    panic!("This is a test panic!");
}

Panic occurred: panicked at src/main.rs:27:5:
This is a test panic!
Performing async cleanup in panic hook...
<EOF>

Run it several times and you'll see that you may not even make it to

Performing async cleanup in panic hook...

[Jurshsmith]

@lostman that happens because the owner process, main dies, right? At that point, we're leveraging the auto-destructor from Tokio -- so graceful shutdown of processes still happens and we simply need our operations to be recoverable from panics.

[Jurshsmith]

To make it more fine-grained, we could use CancellationToken & TaskTracker and similar APIs (like tokio-graceful-shutdown) but since our operations are supposed to be idempotent, I think this solution is quite ergonomic. wdyt?

[Jurshsmith]

LGTM 👍🏾