FFI to C on MacOS causes segmentation fault (works on Ubuntu)

[ExSidius]

The following crate illustrates the issue I'm running into.

https://github.com/exsidius/rust_libpg_query

Running cargo test on Ubuntu causes no errors but on Mac will (inconsistently cause Seg. Faults).

The errors codes are as follows -

error: process didn't exit successfully: `/Users/ExSidius/Dropbox/actual-code/rust_libpg_query/target/debug/deps/pg_query-ebb8d6e50e159947` (signal: 11, SIGSEGV: invalid memory reference)

Occasionally, I will also run into a Utf8Error (may or may not be part of the same bug) -

---- normalize::tests::normalize_bad_query stdout ----
"unterminated dollar-quoted string at or near \"$$$\""
thread 'normalize::tests::normalize_bad_query' panicked at 'called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 2, error_len: Some(1) }', src/libcore/result.rs:1009:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:211
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:221
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at src/libstd/panicking.rs:491
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:398
   6: std::panicking::try::do_call
             at src/libstd/panicking.rs:325
   7: core::char::methods::<impl char>::escape_debug
             at src/libcore/panicking.rs:95
   8: core::result::unwrap_failed
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/macros.rs:26
   9: <core::result::Result<T, E>>::unwrap
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/result.rs:808
  10: pg_query::error::handle_error
             at src/error.rs:55
  11: pg_query::normalize::normalize
             at src/normalize.rs:17
  12: pg_query::normalize::tests::normalize_bad_query
             at src/normalize.rs:81
  13: pg_query::normalize::tests::normalize_bad_query::{{closure}}
             at src/normalize.rs:78
  14: core::ops::function::FnOnce::call_once
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
  15: <F as alloc::boxed::FnBox<A>>::call_box
             at src/libtest/lib.rs:1471
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:673
  16: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:102

I believe I'm using CStrings appropriately (and the fact that it works on one operating system and not another seems suspicious.

Meta

Ubuntu -

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic

ose
rustc 1.33.0 (2aa4c46cf 2019-02-28)
binary: rustc
commit-hash: 2aa4c46cfdd726e97360c2734835aa3515e8c858
commit-date: 2019-02-28
host: x86_64-unknown-linux-gnu
release: 1.33.0
LLVM version: 8.0

macOS -

ProductName:	Mac OS X
ProductVersion:	10.14.4
BuildVersion:	18E205e

rustc 1.33.0 (2aa4c46cf 2019-02-28)
binary: rustc
commit-hash: 2aa4c46cfdd726e97360c2734835aa3515e8c858
commit-date: 2019-02-28
host: x86_64-apple-darwin
release: 1.33.0
LLVM version: 8.0

Let me know if there's anything I can do to help debug.

[sfackler]

This is almost certainly a problem with the library, and you happen to get lucky on Ubuntu.

[nagisa]

Let me know if there's anything I can do to help debug.

Minimising to something that is a small, self-contained reproducer would help a lot.

[ExSidius]

@nagisa sounds good - let me work on that.

@sfackler I don't have a tonne of experience with C libraries and ffi. I can say for certain that when I run the C function independently (gcc/cc compiled) on both Linux and Mac, and run the same function repeatedly, I cannot reproduce the error. What would you need from me in order to convince you that this is an FFI issue? (in addition to a smaller reproducible program).

[sfackler]

I'd want to see some specific evidence that the compiler did a specific thing wrong.

[ExSidius]

That's fair. I'm not entirely sure how to go about that but will continue debugging and do my best to get you something credible and useful.

[Stargateur]

you should try to use automatic tool like https://github.com/rust-lang/rust-bindgen instead of doing the binding yourself.

Also, try to run your program with tools such as valgrind

[ExSidius]

Will close this until I have more information.

[parisholley]

@sfackler i just hit this as well... you can check out a fork of a c library I am trying to build a wrapper for. and this error is strange, because i commented out all of my C calls that pass the CString as an arg, so i'm not even sending it anywhere, and can get it to segfault repeatedly.

git clone -n git@github.com:parisholley/Device-Detection.git
cd Device-Detection
git checkout rust
git lfs fetch
cd rust
cargo test --features pattern pattern_init -- --nocapture

a few things are interesting though, notice these two lines:

https://github.com/parisholley/Device-Detection/blob/7d04b4a99526bff53e35843c9a2347a4b955d110/rust/pattern.rs#L127-L132

If i comment out line 131, and only call the detect function once, there is no problem. it is only when called a second time, that SIGSEGV is thrown. I have tried using both into_raw() and as_ptr() with the same results.

what makes this EVEN more strange, is if in that same function, i comment out 78 and 111, so that only 74/75 (CString) is live, no segfault. if i do the opposite, and comment out 74/75, but leave 78/111 live, no segfault. it is only when the two, seemingly unrelated calls are in the same function, that things go crazy.

my environment:
`
rustc 1.40.0-nightly (1721c96 2019-10-12)
mac os 10.14.6
llvm 9.0.0

Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
`