Crashes on run on macOS Sonoma 14.0 Beta (23A5312d)

[nabijaczleweli]

Forwarding because I see Curl_macos_init <- ZN3std4sync4once4Once9call_once28$u7b$$u7b$closure$u7d$$u7d$17hac714a257a9e32a7E <- _ZN4curl9INIT_CTOR9init_ctor17h6cadb2258443a9ccE in backtrace.

Downstream: nabijaczleweli/cargo-update#240

Using cargo 1.71.0 (cfd3bbd8f 2023-06-08)

cargo install-update immediately crashes.

*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFString stringByStandardizingPath]: unrecognized selector sent to instance 0x600001908440'
*** First throw call stack:
(
	0   CoreFoundation                      0x0000000192774960 __exceptionPreprocess + 176
	1   libobjc.A.dylib                     0x000000019226deb4 objc_exception_throw + 60
	2   CoreFoundation                      0x000000019282646c -[NSObject(NSObject) __retain_OA] + 0
	3   CoreFoundation                      0x00000001926deb24 ___forwarding___ + 1572
	4   CoreFoundation                      0x00000001926de440 _CF_forwarding_prep_0 + 96
	5   Foundation                          0x00000001937afd80 -[NSProcessInfo arguments] + 188
	6   CoreFoundation                      0x00000001927f0094 __getDefaultArguments_block_invoke + 96
	7   libdispatch.dylib                   0x0000000192475910 _dispatch_client_callout + 20
	8   libdispatch.dylib                   0x000000019247714c _dispatch_once_callout + 32
	9   CoreFoundation                      0x00000001927efa30 _addBackstopValuesForIdentifierAndSource + 640
	10  CoreFoundation                      0x00000001926aa3b4 __81-[_CFXPreferences(SourceAdditions) withNamedVolatileSourceForIdentifier:perform:]_block_invoke + 144
	11  CoreFoundation                      0x00000001927ef6d8 -[_CFXPreferences withNamedVolatileSourceForIdentifier:perform:] + 272
	12  CoreFoundation                      0x00000001926b0724 -[CFPrefsSearchListSource addNamedVolatileSourceForIdentifier:] + 136
	13  CoreFoundation                      0x000000019282e94c __108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke.155 + 296
	14  CoreFoundation                      0x000000019282e5f4 -[_CFXPreferences withSearchLists:] + 84
	15  CoreFoundation                      0x00000001926abc78 __108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 300
	16  CoreFoundation                      0x000000019282e7a0 -[_CFXPreferences withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 384
	17  CoreFoundation                      0x00000001926ab5a0 -[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 156
	18  CoreFoundation                      0x00000001926ab4c8 _CFPreferencesCopyAppValueWithContainerAndConfiguration + 112
	19  SystemConfiguration                 0x0000000193438488 SCDynamicStoreCopyProxiesWithOptions + 180
	20  cargo-install-update                0x0000000102861a68 Curl_macos_init + 16
	21  cargo-install-update                0x0000000102850574 curl_global_init + 176
	22  cargo-install-update                0x000000010283e504 _ZN3std4sync4once4Once9call_once28_$u7b$$u7b$closure$u7d$$u7d$17hac714a257a9e32a7E + 36
	23  cargo-install-update                0x0000000102a94d28 _ZN3std10sys_common4once5queue4Once4call17h5a1c3f8d4cad741fE + 1056
	24  cargo-install-update                0x000000010283e62c _ZN4curl9INIT_CTOR9init_ctor17h6cadb2258443a9ccE + 96
	25  dyld                                0x00000001922c55c8 ___ZZNK5dyld46Loader25findAndRunAllInitializersERNS_12RuntimeStateEENK3$_0clEv_block_invoke + 168
	26  dyld                                0x000000019230a920 ___ZNK5dyld313MachOAnalyzer18forEachInitializerER11DiagnosticsRKNS0_15VMAddrConverterEU13block_pointerFvjEPKv_block_invoke.209 + 340
	27  dyld                                0x00000001922fdc60 ___ZNK5dyld39MachOFile14forEachSectionEU13block_pointerFvRKNS0_11SectionInfoEbRbE_block_invoke + 496
	28  dyld                                0x00000001922a52fc _ZNK5dyld39MachOFile18forEachLoadCommandER11DiagnosticsU13block_pointerFvPK12load_commandRbE + 300
	29  dyld                                0x00000001922fcc98 _ZNK5dyld39MachOFile14forEachSectionEU13block_pointerFvRKNS0_11SectionInfoEbRbE + 192
	30  dyld                                0x000000019230a434 _ZNK5dyld313MachOAnalyzer18forEachInitializerER11DiagnosticsRKNS0_15VMAddrConverterEU13block_pointerFvjEPKv + 516
	31  dyld                                0x00000001922c1798 _ZNK5dyld46Loader25findAndRunAllInitializersERNS_12RuntimeStateE + 448
	32  dyld                                0x00000001922c7b14 _ZNK5dyld416JustInTimeLoader15runInitializersERNS_12RuntimeStateE + 36
	33  dyld                                0x00000001922c1b4c _ZNK5dyld46Loader23runInitializersBottomUpERNS_12RuntimeStateERN5dyld35ArrayIPKS0_EE + 220
	34  dyld                                0x00000001922c5654 _ZZNK5dyld46Loader38runInitializersBottomUpPlusUpwardLinksERNS_12RuntimeStateEENK3$_1clEv + 112
	35  dyld                                0x00000001922c1ccc _ZNK5dyld46Loader38runInitializersBottomUpPlusUpwardLinksERNS_12RuntimeStateE + 304
	36  dyld                                0x00000001922e6ad4 _ZN5dyld44APIs25runAllInitializersForMainEv + 464
	37  dyld                                0x00000001922a9f34 _ZN5dyld4L7prepareERNS_4APIsEPKN5dyld313MachOAnalyzerE + 3192
	38  dyld                                0x00000001922a8f44 start + 1948
)
libc++abi: terminating due to uncaught exception of type NSException

[mitsuhiko]

I ran into this in rye. Since I do not have a mac beta I cannot reproduce this myself but if I were to guess, this might be an initialization order issue. Today curl_global_init is invoked in a __DATA,__mod_init_func section:

curl-rust/src/lib.rs

Lines 138 to 151 in 3dc0872

	#[cfg_attr(target_os = "macos", link_section = "__DATA,__mod_init_func")]
	#[cfg_attr(target_os = "windows", link_section = ".CRT$XCU")]
	pub static INIT_CTOR: extern "C" fn() = {
	/// This is the body of our constructor function.
	#[cfg_attr(
	any(target_os = "linux", target_os = "android"),
	link_section = ".text.startup"
	)]
	extern "C" fn init_ctor() {
	init();
	}
	init_ctor
	};

There is a decent chance that this now runs too early and something that curl's init requires on macOS is not initialized yet.

[djc]

Interestingly, we recently got a similar crash reported on Windows:

rust-lang/rustup#3457

[Ryu-ga]

TL;DR. Link CoreServices to binaries

In Sonoma, some CoreFoundation functions require CoreServices, but dyld of Sonoma cannot load CoreServices when it should do, like cxx_global_init in gettext, MacOS_init in libcurl.
So, some programs like aria2(gettext), cargo(libcurl) didn't work.

[kornelski]

Linking with CoreServices hasn't helped in my case, but switching to system curl did:

curl-sys = { version = "0.4", features = ["force-system-lib-on-osx"] }

[krishnaTORQUE]

I am facing the same issue in stable sonoma 14.0.
Any update?
Thanks

[leepa]

Workaround works for me thanks @kornelski

[speelbarrow]

Fix for this issue has been implemented in curl/curl@6ab7e19 and is pending the next release.

[KenjiEmura]

Please let us know when the fix is released, thanks!

[jcamiel]

Hi,

In my comprehension, if someone uses the curl crate to produce a binary, this binary will potentially crash if the linked libcurl is the current system one.

In other words, even if curl is patched, a Rust binary linking to the system libcurl will crash unless:

• the Rust binary itself links explicitly to CoreServices
• or the Rust binary links to a patched non-system libcurl (which seems to be what the brew version of cargo is doing in ZhongRuoyu/homebrew-core@1023367)
• or Sonoma is updated with a patched system libcurl

Given this, shouldn't we always link a Rust binary using curl crate to CoreServices on Sonoma?

[ZhongRuoyu]

In my comprehension, if someone uses the curl crate to produce a binary, this binary will potentially crash if the linked libcurl is the current system one.

For what it's worth, I've seen several similar bug reports and none of them seem to be using the system libcurl. See, for example, the stack trace in this issue description, where Curl_macos_init is from cargo-install-update rather than the system libcurl.4.dylib. It's very likely that curl-sys built its own libcurl, which was statically linked into the executable.

To further verify this, see the minimal reproducer below.

Minimal reproducer

Cargo.toml:

[package]
name = "curl-repro"
version = "0.1.0"

[dependencies]
# This doesn't work:
curl = { version = "0.4.44", features = ["static-curl"] }
# This works:
# curl = { version = "0.4.44", features = ["force-system-lib-on-osx"] }

src/main.rs:

fn main() {
    let _ = curl::easy::Easy::new();
    println!("OK");
}

Using the static-curl feature would lead to NSInvalidArgumentException, while force-system-lib-on-osx would work normally.

which seems to be what the brew version of cargo is doing in ZhongRuoyu/homebrew-core@1023367

Note that Homebrew's cargo from the rust formula is using the system libcurl ¹; the patch was on cargo-c to work around the issue seen here. Due to another unrelated issue (see Homebrew/homebrew-core#146547 (comment)), the Homebrew-provided curl is used instead of the system libcurl.

Linkage of Homebrew's cargo

$ otool -L `brew --prefix rust`/bin/cargo
/usr/local/opt/rust/bin/cargo:
        /usr/local/opt/libgit2@1.6/lib/libgit2.1.6.dylib (compatibility version 1.6.0, current version 1.6.4)
        /usr/local/opt/libssh2/lib/libssh2.1.dylib (compatibility version 2.0.0, current version 2.1.0)
        /usr/local/opt/openssl@3/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
        /usr/local/opt/openssl@3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
        /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
        /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
        /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

Given this, shouldn't we always link a Rust binary using curl crate to CoreServices on Sonoma?

I do think this is a reasonable solution. Upstream (curl) is already doing is (see curl/curl#11894) and it should be in an upcoming release.

Footnotes

1. https://github.com/Homebrew/homebrew-core/blob/3fbedd0d9444b59b7e60810853dead25306ed558/Formula/r/rust.rb#L106-L110 ↩

[jcamiel]

Thanks for the clarifications @ZhongRuoyu!

[SomeoneToIgnore]

To note, there's no need to wait for the next curl release, but rather link it as @ZhongRuoyu proposed.
#530 should be enough to fix it.

[SomeoneToIgnore]

Does anybody know who's able to review #530 ?
Kind of a bummer that the fix is one line (and a minor release) away, and still it is broken for weeks (if not months at this point).

[SomeoneToIgnore]

I can confirm that the issue is resolved and my related projects work on Sonoma now.

To fix the issue, one needs to upgrade to https://github.com/alexcrichton/curl-rust/releases/tag/curl-sys-0.4.67
For projects that use this package, run cargo update -p curl-sys to update to the package that has the issue fixed.
For already installed binaries like cargo-update, you'll have to reinstall them with cargo install --force cargo-update