Timeout in HTTP::Server spec with preview_mt

[straight-shoota]

Some HTTP::Server specs occasionally time out in CI after reaching the 15 second timout.
Example:

• HTTP::Server #bind fails after listen (Duration (p95): 11ms, 98% success rate)
• HTTP::Server #listen fails after listen (Duration (p95): 7ms, 99% success rate)
• HTTP::Server doesn't raise on accept after close #2692 (Duration (p95): 8s, 99% success rate)

Stats from https://app.circleci.com/insights/github/crystal-lang/crystal/workflows/nightly_release/tests?branch=master

We could consider increasing the timeout similar to #13043. But that would just be ignoring the symptoms. It really shouldn't take that long to execute these specs. The 95-percentile of the duration being several orders of magnitude below the timeout confirms that.

Maybe there is some kind of deadlock or perhaps all available threads are saturated for such a long time. It's flaky. We need to investigate what's actually going on and figure out what can be done to improve this.

[ysbaddaden]

The problems are easily reproduced on local, though I never get the 15s timeout. During some run I either get an infinite hang (strace shows repeated epoll_wait) or a libevent warn about a bad file descriptor... which is reminiscent of #2692 that is quoted in one of the reported failing spec.

Indeed, this is a race condition induced by MT: we're listening and closing a server from different fibers that happen to be run on different threads, and are no longer synchronized, so a spec suddenly closes a server while it's being started, ...

[ysbaddaden]

To reproduce:

$ bin/crystal build spec/std/http/server/server_spec.cr -Dpreview_mt

Then repeatedly run ./server_spec. It usually fails every half dozen runs for me.

[ysbaddaden]

This is hard to fix properly. So far only adding a few sleep 0.001 avoids the issue 😞