OSX unit tests segfault

[jszwedko]

Discussed in discord: https://discordapp.com/channels/742820443487993987/746070604192415834/760546620559065148

Example: https://github.com/timberio/vector/pull/4193/checks?check_run_id=1183596467

Appears to have been introduced by #4187

[leebenson]

FWIW, I'm pretty sure I saw this even before #4187. It's especially hard to diagnose because it doesn't always happen, it happens even after all tests have already passed, and I can't reproduce it locally in macOS 10.15.6.

There are similar issues on rust-lang:

• FFI to C on MacOS causes segmentation fault (works on Ubuntu)
• No issues on Linux, segfault on OS X (but only when building with –release)
• https://users.rust-lang.org/t/segfaults-under-mac-os-with-multithreaded-code/18724
• https://users.rust-lang.org/t/without-a-single-line-of-unsafe-code-i-am-getting-segfault-and-i-cant-figure-out-why-since-i-cant-stack-trace/40017
• [stable regression] #[should_panic] tests segfault in certain OS X configurations with 1.44.0 rust-lang/rust#73030 (perhaps not directly related, but we were bitten by this in Investigate #[should_panic] flakiness in macOS CI tests #4056 and removed related tests in chore: Remove retry until after timeout test #4050)
• RUST_BACKTRACE on macOS can trigger crashes
• Spurious LLDB segfault (signal 11)

Some of these are still open. Others were fixed in earlier Rust versions. Some may not be relevant. Some may be compiler issues, or due to external libs.

When using Travis, we could SSH into a test and perform some more in-depth testing againt the environment. I'm not sure what the options are for GH Actions. It's really hard to test because make test never segfaults locally, with an identical toolchain.

One thing to note is that I saw segfaults (although not signal 11 specifically) when setting codegen-units = 1, when trying to work around a known LLVM issue with older macOS installations and parallel compilations, in environments other than macOS. In all cases, they were due to OOM errors. I'm wondering if this could also be memory pressure, although I don't know why it would specifically affect only macOS and not other OSs.

[leebenson]

Do we have a recent example of where this is still surfacing? I haven't seen it personally for a while.

[jszwedko]

I also haven't seen this in a quite a bit. I think we can close until it crops up again.