julia 1.9.0-beta2 fails to build from source on macOS

[fxcoudert]

We are seeing the build failure on Homebrew CI at Homebrew/homebrew-core#119456
The error seems to be lld: error: LC_DYLD_INFO_ONLY not found.

Sysimage built. Summary:
Base ────────  15.284967 seconds 41.4134%
Stdlibs ─────  21.622479 seconds 58.5843%
Total ───────  36.908299 seconds
Warning: git information unavailable; versioning information limited
 cd /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/base && if ! JULIA_BINDIR=/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/bin WINEPATH="/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/bin;$WINEPATH" JULIA_NUM_THREADS=1  /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/bin/julia -O3 -C "generic;cortex-a57;thunderx2t99;armv8.2-a,crypto,fullfp16,lse,rdm" --output-o /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib/julia/sys-o.a.tmp  --startup-file=no --warn-overwrite=yes --sysimage /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib/julia/sys.ji /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/contrib/generate_precompile.jl 1; then echo '*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***'; false; fi 
lld: error: LC_DYLD_INFO_ONLY not found in /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib/libjulia.dylib
lld: error: LC_DYLD_INFO_ONLY not found in /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib/libjulia-internal.dylib
ERROR: failed process: Process(setenv(`/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/tools/lld -flavor darwin -arch arm64 -undefined dynamic_lookup -platform_version macos 12.6 12.3 '' -dylib -o /private/tmp/jl_kzKcXi/compiled/v1.9/jl_eltPtw -all_load /private/tmp/jl_kzKcXi/compiled/v1.9/jl_YcrnFt '' -L/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib -L/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib/julia -L/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib -ljulia -ljulia-internal`,["MFLAGS=- --jobserver-fds=3,4 -j", "PATH=/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/tools:/opt/homebrew/Library/Homebrew/shims/mac/super:/opt/homebrew/opt/cmake/bin:/opt/homebrew/opt/brotli/bin:/opt/homebrew/opt/gettext/bin:/opt/homebrew/opt/libidn2/bin:/opt/homebrew/opt/openssl@1.1/bin:/opt/homebrew/opt/openldap/bin:/opt/homebrew/opt/rtmpdump/bin:/opt/homebrew/opt/lz4/bin:/opt/homebrew/opt/xz/bin:/opt/homebrew/opt/zstd/bin:/opt/homebrew/opt/curl/bin:/opt/homebrew/opt/gcc/bin:/opt/homebrew/opt/libgit2/bin:/opt/homebrew/opt/mbedtls@2/bin:/opt/homebrew/opt/p7zip/bin:/opt/homebrew/opt/pcre2/bin:/opt/homebrew/opt/metis/bin:/opt/homebrew/opt/suite-sparse/bin:/usr/bin:/bin:/usr/sbin:/sbin", "CXX=clang++ -mmacosx-version-min=12", "USE_SYSTEM_ZLIB=1", "HOMEBREW_USER_AGENT_CURL=Homebrew/3.6.16-97-ge76c55e (Macintosh; arm64 Mac OS X 12.6) curl/7.79.1", "HOMEBREW_BREW_DEFAULT_GIT_REMOTE=https://github.com/Homebrew/brew", "HOMEBREW_PRODUCT=Homebrew", "BUILDROOT=/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2", "MAKEFLAGS= --jobserver-fds=3,4 -j -- TAGGED_RELEASE_BANNER=Built\\ by\\ Homebrew\\ (v1.9.0-beta2) JULIA_CPU_TARGET=generic;cortex-a57;thunderx2t99;armv8.2-a,crypto,fullfp16,lse,rdm MACOSX_VERSION_MIN=12 LOCALBASE=/opt/homebrew PYTHON=python3 USE_BLAS64=0 LIBLAPACKNAME=libopenblas LIBLAPACK=-lopenblas LIBBLASNAME=libopenblas LIBBLAS=-lopenblas USE_SYSTEM_P7ZIP=1 USE_SYSTEM_ZLIB=1 USE_SYSTEM_PATCHELF=1 USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_CURL=1 USE_SYSTEM_NGHTTP2=1 USE_SYSTEM_LIBSSH2=1 USE_SYSTEM_MBEDTLS=1 USE_SYSTEM_UTF8PROC=1 USE_SYSTEM_LIBSUITESPARSE=1 USE_SYSTEM_MPFR=1 USE_SYSTEM_GMP=1 USE_SYSTEM_LAPACK=1 USE_SYSTEM_BLAS=1 USE_SYSTEM_OPENLIBM=1 USE_SYSTEM_PCRE=1 USE_SYSTEM_CSL=1 sysconfdir=/opt/homebrew/etc prefix=/opt/homebrew/Cellar/julia/1.9.0-beta2 USE_BINARYBUILDER=0 VERBOSE=1", "MAKEOVERRIDES=\${-*-command-variables-*-}"  …  "HOMEBREW_GITHUB_PACKAGES_AUTH=***", "ZERO_AR_DATE=1", "HOMEBREW_ANALYTICS_IDS=UA-76679469-1", "USE_SYSTEM_PCRE=1", "HOMEBREW_LIBRARY_PATHS=/opt/homebrew/opt/openssl@1.1/lib:/opt/homebrew/opt/openldap/lib:/opt/homebrew/opt/curl/lib:/opt/homebrew/opt/mbedtls@2/lib:/opt/homebrew/opt/openblas/lib:/opt/homebrew/lib:/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries", "JULIA_CPU_TARGET=generic;cortex-a57;thunderx2t99;armv8.2-a,crypto,fullfp16,lse,rdm", "HOMEBREW_ARCHFLAGS=", "USE_BINARYBUILDER=0", "HOMEBREW_GIT_CONFIG_FILE=/opt/homebrew/.git/config", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd, ::Base.DevNull, ::Vararg{Any}; wait::Bool)
   @ Base ./process.jl:480
 [3] run
   @ ./process.jl:477 [inlined]
 [4] link_image (repeats 2 times)
   @ ./linking.jl:144 [inlined]
 [5] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2055
 [6] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:2016
 [7] top-level scope
   @ none:3
ERROR: LoadError: failed process: Process(`/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/bin/julia -O0 --sysimage /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib/julia/sys.ji --trace-compile=/private/tmp/jl_kzKcXi/jl_ilK1exWvIN --startup-file=no -Cnative -e 'pushfirst!(DEPOT_PATH, "/private/tmp/jl_kzKcXi");
Base.PRECOMPILE_TRACE_COMPILE[] = "/private/tmp/jl_kzKcXi/jl_WX45ybgvab";
Base.compilecache(Base.PkgId("__PackagePrecompilationStatementModule"), "/private/tmp/jl_kzKcXi/__PackagePrecompilationStatementModule/src/__PackagePrecompilationStatementModule.jl")
# NOTE: these were moved to the end of Base.jl. TODO: move back here.
# # Used by Revise & its dependencies
# while true  # force inference
# delete!(push!(Set{Module}(), Base), Main)
# m = first(methods(+))
# delete!(push!(Set{Method}(), m), m)
# empty!(Set())
# push!(push!(Set{Union{GlobalRef,Symbol}}(), :two), GlobalRef(Base, :two))
# (setindex!(Dict{String,Base.PkgId}(), Base.PkgId(Base), "file.jl"))["file.jl"]
# (setindex!(Dict{Symbol,Vector{Int}}(), [1], :two))[:two]
# (setindex!(Dict{Base.PkgId,String}(), "file.jl", Base.PkgId(Base)))[Base.PkgId(Base)]
# (setindex!(Dict{Union{GlobalRef,Symbol}, Vector{Int}}(), [1], :two))[:two]
# (setindex!(IdDict{Type, Union{Missing, Vector{Tuple{LineNumberNode, Expr}}}}(), missing, Int))[Int]
# Dict{Symbol, Union{Nothing, Bool, Symbol}}(:one => false)[:one]
# Dict(Base => [:(1+1)])[Base]
# Dict(:one => [1])[:one]
# Dict("abc" => Set())["abc"]
# pushfirst!([], sum)
# get(Base.pkgorigins, Base.PkgId(Base), nothing)
# sort!([1,2,3])
# unique!([1,2,3])
# cumsum([1,2,3])
# append!(Int[], BitSet())
# isempty(BitSet())
# delete!(BitSet([1,2]), 3)
# deleteat!(Int32[1,2,3], [1,3])
# deleteat!(Any[1,2,3], [1,3])
# Core.svec(1, 2) == Core.svec(3, 4)
# # copy(Core.Compiler.retrieve_code_info(Core.Compiler.specialize_method(which(+, (Int, Int)), [Int, Int], Core.svec())))
# any(t->t[1].line > 1, [(LineNumberNode(2,:none),:(1+1))])
# break   # end force inference
# end
using Artifacts, Base.BinaryPlatforms, Libdl
artifacts_toml = abspath(joinpath(Sys.STDLIB, "Artifacts", "test", "Artifacts.toml"))
artifact_hash("HelloWorldC", artifacts_toml)
oldpwd = pwd(); cd(dirname(artifacts_toml))
macroexpand(Main, :(@artifact_str("HelloWorldC")))
cd(oldpwd)
artifacts = Artifacts.load_artifacts_toml(artifacts_toml)
platforms = [Artifacts.unpack_platform(e, "HelloWorldC", artifacts_toml) for e in artifacts["HelloWorldC"]]
best_platform = select_platform(Dict(p => triplet(p) for p in platforms))
dlopen("libjulia", RTLD_LAZY | RTLD_DEEPBIND)

'`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run
    @ ./process.jl:477 [inlined]
  [4] (::Main.anonymous.var"#1#5"{Set{String}, String})(prec_path::String)
    @ Main.anonymous /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/contrib/generate_precompile.jl:274
  [5] mktempdir(fn::Main.anonymous.var"#1#5"{Set{String}, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:760
  [6] mktempdir (repeats 2 times)
    @ ./file.jl:756 [inlined]
  [7] generate_precompile_statements()
    @ Main.anonymous /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/contrib/generate_precompile.jl:256
  [8] top-level scope
    @ /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/contrib/generate_precompile.jl:431
  [9] eval(m::Module, e::Any)
    @ Core ./boot.jl:370
 [10] top-level scope
    @ /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/contrib/generate_precompile.jl:10
in expression starting at /private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/contrib/generate_precompile.jl:7
*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
make[2]: *** [/private/tmp/julia-20221231-64074-1neh5ig/julia-1.9.0-beta2/usr/lib/julia/sys-o.a] Error 1
make[1]: *** [julia-sysimg-release] Error 2
make: *** [install] Error 2

Possible related links:

• https://leanprover-community.github.io/archive/stream/270676-lean4/topic/ld64.2Elld.20dynamic.20link.20errors.20on.20MacOS.html
• lld/mac fails to link against dylib with LC_DYLD_EXPORTS_TRIE llvm/llvm-project#54550
• llvm/llvm-project@94e0f8e

[fxcoudert]

If I read the LLVM commit right, that was fixed in LLVM 15.0.0, but Julia 1.9.0-beta2 uses LLVM 14.0.5.

[vchuravy]

I think we can backport that patch, and we might want allow to user to use the system ld instead of the one we ship with Julia

[fxcoudert]

For the record, this affects building Julia on all macOS versions when Xcode 14 / Apple clang 14 is used (both macOS 12 and 13, and both Intel and ARM processors).

[fingolfin]

Huh, I am building Julia all the time on macOS 12 with Xcode 14 / Apple clang 14 installed. So perhaps there is more to the story?

[fxcoudert]

@fingolfin do you use USE_BINARYBUILDER=0? Homebrew build options:

make VERBOSE=1 USE_BINARYBUILDER=0 prefix=/opt/homebrew/Cellar/julia/1.9.0-beta2 sysconfdir=/opt/homebrew/etc USE_SYSTEM_CSL=1 USE_SYSTEM_PCRE=1 USE_SYSTEM_OPENLIBM=1 USE_SYSTEM_BLAS=1 USE_SYSTEM_LAPACK=1 USE_SYSTEM_GMP=1 USE_SYSTEM_MPFR=1 USE_SYSTEM_LIBSUITESPARSE=1 USE_SYSTEM_UTF8PROC=1 USE_SYSTEM_MBEDTLS=1 USE_SYSTEM_LIBSSH2=1 USE_SYSTEM_NGHTTP2=1 USE_SYSTEM_CURL=1 USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_PATCHELF=1 USE_SYSTEM_ZLIB=1 USE_SYSTEM_P7ZIP=1 LIBBLAS=-lopenblas LIBBLASNAME=libopenblas LIBLAPACK=-lopenblas LIBLAPACKNAME=libopenblas USE_BLAS64=0 PYTHON=python3 LOCALBASE=/opt/homebrew MACOSX_VERSION_MIN=13

[fingolfin]

No, I don't -- the fact that you do (and the full build instructions) seems like critical information to include in the issue description, though :-)

[fingolfin]

[ Off-topic, but: Of course the fact that Homebrew does all these things is also why I tell all our users (i.e.: of a specific Julia package, I am not a core Julia developer) to avoid installing Julia via Homebrew or Debian or a bunch of other package managers; for us it is critical that the same libgmp, libmpfr, etc. etc. are used by Julia and by our BinaryBuild recipes, and Homebrew/Debian/etc break that assumption, which caused all kinds of subtle, hard to debug issues over the years ]

[fxcoudert]

In this case, it's none of the system versions that is causing the trouble, but the fact that Julia is actually using an outdated dependency :) if we built against LLVM 15, we would not have that issue. So there are two sides to that coin.

[vchuravy]

At the time we started the release process for Julia 1.9 LLVM 14 was the currently released version. LLVM is for many projects a special dependency since it is extremely hard to support more than one version. Additionally moving to a new LLVM version is a work intensive process and often introduces new bugs, not something we want to do right before we cut a release. (LLVM 15 is particularly problematic since it will no longer support older Mac versions)

We have two problems here. The first is what are the libraries that have this new LC_DYLD_EXPORTS_TRIE the pkgimage link against libjulia and libjulia-internal. Who is introducing this new (not backwards compatible flag?)

Secondly the issue is that we can't assume that the user has Xcode installed, that's why the Julia is bundling LLD. Now there are two solutions here that might be feasible. Firstly we can backport that patch to our tree of LLVM. Secondly I can provide a build system flag to not use the bundled LLD and instead use the system linker. If Homebrew can guarantee that one is available at all times.

[fxcoudert]

Firstly we can backport that patch to our tree of LLVM.

That seems like the best outcome. This will fix things not only for Homebrew, but also other users.

[kpamnany]

Secondly I can provide a build system flag to not use the bundled LLD and instead use the system linker.

For our problem, #48235, this solution would work.

[gbaraldi]

I was looking for a solution to this but there is a question here. We could have a use system ld envar, but then we would just ld from the path and use the appropriate flags, which in macos are slighly different from lld for example. Or we could have a more general use other linker, but I'm not sure how we would make that look.

[fxcoudert]

This is still occurring with 1.9.0-beta4

[KristofferC]

(I realize now we haven't done a new release since the PR that fixed this was merged)