Design and implement graceful shutdown for Zebra

[teor2345]

Motivation

In #1637, we check is_shutting_down in the CheckpointVerifier and network service.

But we'd like to implement graceful shutdown, where Zebra's endless loops wait on a shutdown condition or future.

We might also be able to implement this automatically using shutdown_timeout:
https://docs.rs/tokio/1.6.1/tokio/runtime/struct.Runtime.html#method.shutdown_timeout

Here's tokio's advice on graceful shutdown:
https://tokio.rs/tokio/topics/shutdown

Ideas

• Design graceful shutdown for Zebra
  • support graceful shutdown
  • support immediate shutdown on double Ctrl-C: https://dev.to/talzvon/handling-unix-kill-signals-in-rust-55g6
  • deal with shutdown from the state service debug_stop_at_height
• Implement the design
• Test the design (Add restart acceptance tests #1666)

Alternative designs:

• endless loops wait on a boolean condition, like is_shutting_down
• endless loops select on a shutdown future, like zebra-network's cancel handles
• each spawned task waits on select between a shutdown future and the actual task closure
  • replace tokio::spawn with zebra::spawn
  • could use a global watch channel with 3 states: Running, ShuttingDown, ExitNow
  • code that needs to exit can use the watch sender to set ShuttingDown
• do we need a new zebra-shutdown crate? (current users: zebra-state, zebrad)
• should library crates return a task error / service readiness error instead, so the app can shut down?
  • check if any other services can error
• do we also need to use an at_exit handler? Should it just set ExitNow?
• how do we handle panics?
  • currently we abort, but we could propagate them to the calling task, or set ExitNow

Related Issues

• Alternative faster fix Ignore some panics during shutdown using the panic handler #1677

• Related fix Interrupt handler does not work when a blocking task is running #1351

• Shutdown and restart tests Add restart acceptance tests #1666

• Original fix Fix shutdown panics #1637

Specific issues that this fix solves:

• Spurious MustUseOneshotSender panic on zebrad shutdown #1574
• Spurious checkpoint verifier JoinError panic on zebrad shutdown #1576
• Fatal panic: state service commit block failed: verified checkpoints must be committed transactionally: CommitFinalized(Closed) #2055
• Panic during shutdown in StateService::call(Request::CommitBlock) #2209
• Panic during shutdown: transaction verifier is always ready: Closed #3053

[conradoplg]

Here's one panic that can happen when Ctrl+C'ing Zebra

Error

ClientRequest oneshot sender must not be dropped before send: Canceled

Metadata

key	value
version	1.0.0-alpha.13+4.g86ff25c
Zcash network	Mainnet
state version	5
branch	finalized-state-in-chain-verification
git commit	86ff25c
commit timestamp	2021-07-16T06:26:12+00:00
target triple	x86_64-unknown-linux-gnu
build profile	debug
location	zebra-network/src/peer/client.rs:260:26

SpanTrace

SpanTrace:
   0: zebrad::components::sync::obtain_tips
             at zebrad/src/components/sync.rs:367
   1: zebrad::components::sync::sync
             at zebrad/src/components/sync.rs:262
   2: zebrad::application::
           with zebrad="86ff25c" net="Main"
             at zebrad/src/application.rs:365

[teor2345]

We're gradually doing this as errors come up.

[teor2345]

This seems to work and has worked for months.