Secret Connection network timeouts

[tarcieri]

Copying/pasting this from the original issue at tendermint/tmkms#310

The Secret Connection implementation presently has no notion of timeouts for things like connecting/"dialing" validators or reading from/writing to sockets.

Without timeouts, there are many cases which could potentially deadlock indefinitely inside of blocking I/O operations, and at least one user has experienced this.

Here are a few recommendations:

• futures/tokio: this is the up-and-coming ecosystem solution to this general problem, and the one I'd recommend. It's a full asynchronous event model which solves, among other things, timeouts. When async/await support ships in Rust 1.38 (scheduled to be released in August), migrating from blocking I/O should be fairly straightforward.
• libc crate + poll(2) system call: if we wanted to stick with blocking I/O, the poll(2) system call, as invoked through the libc crate, can be used to determine I/O readiness prior to performing a blocking I/O call, and also takes a timeout as an argument. This would probably be the lowest impact way to implement timeouts as it wouldn't involve switching away from blocking I/O.

I would suggest migrating to futures-rs/Tokio to solve this problem, and leveraging the async/await syntax when it lands later this month.

[tarcieri]

If anyone wants to take a crack at switching Secret Connection to use Tokio, they have just migrated to std::future upstream and released an alpha that supports it:

https://tokio.rs/blog/2019-08-alphas/

Using it will require nightly (soon to be 1.38 beta) Rust

[tarcieri]

Another option (announced yesterday) is async-std, which is a sort of drop-in async replacement for std::io:

• https://async.rs/blog/announcing-async-std/
• https://github.com/async-rs/async-std

It allows easily adding timeouts which leverage futures while using an API that looks nearly identical to std::io (with the addition of async/await):

async fn get() -> io::Result<Vec<u8>> {
    let mut stream = TcpStream::connect("example.com:80").await?;
    stream.write_all(b"GET /index.html HTTP/1.0\r\n\r\n").await?;

    let mut buf = vec![];

    io::timeout(Duration::from_secs(5), async {
        stream.read_to_end(&mut buf).await?
        Ok(buf)
    })
}

[liamsi]

@xla want to pair up on this soon?

[tarcieri]

@liamsi FWIW I copied Secret Connection back to the KMS repo with the goal of working on async support over there (see #27)

[ebuchman]

Can we close this here then?

[liamsi]

Yes, doesn't make sense to have it open here. There used to be an open issue there tendermint/tmkms#310 but this is also tracked by tendermint/tmkms#352.