Improve libjulia, rebuild libjulia 1.3.1 with the improvements

[fingolfin]

It is not clear to me right now why libjulia is not built for FreeBSD, so this PR is trying to figure it out, and then either solve it, or document it for future generations.

[fingolfin]

OK so I made two Julia PRs to fix warnings when compiling for FreeBSD, see JuliaLang/julia#38139 (also fixes a genuine threading bug, I think) and JuliaLang/julia#38140.

But unfortunately linking fails:

/workspace/srcdir/julia-1.4.2/usr/lib/libjulia.so: undefined reference to `llvm::cfg::Update<llvm::BasicBlock*>::dump() const'
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)

Googling that lead me to AcademySoftwareFoundation/OpenShadingLanguage#1069 which lead me to JuliaLang/julia#35364. And indeed, Julia 1.4 is missing -DNDEBUG. So now I got Julia 1.4 building on FreeBSD :-).

Next up is 1.5; there we run into a download issue:

[11:59:15] [ ! -e /workspace/srcdir/julia-1.5.1/usr/manifest/libwhich ] || make uninstall-libwhich
[11:59:15] mkdir -p /workspace/srcdir/julia-1.5.1/usr
[11:59:15] /usr/local/bin/tar -xmzf /workspace/srcdir/julia-1.5.1/usr-staging/libwhich-81e9723c0273d78493dc8c8ed570f68d9ce7e89e.tgz -C /workspace/srcdir/julia-1.5.1/usr
[11:59:15] echo 'libwhich-81e9723c0273d78493dc8c8ed570f68d9ce7e89e staged-uninstaller' > /workspace/srcdir/julia-1.5.1/usr/manifest/libwhich
[11:59:15]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[11:59:15]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[11:59:15] curl: (22) The requested URL returned error: 404 Not Found
[11:59:15] make[1]: *** [/workspace/srcdir/julia-1.5.1/deps/unwind.mk:110: /workspace/srcdir/julia-1.5.1/deps/srccache/LibUnwind.v1.3.1-4.x86_64-unknown-freebsd11.1-julia_version+1.5.tar.gz] Error 22
[11:59:15] make[1]: Leaving directory '/workspace/srcdir/julia-1.5.1/deps'
[11:59:15] make: *** [Makefile:60: julia-deps] Error 2

[fingolfin]

Ok now it works on 1.5, too 🎆

[fingolfin]

gotta still check that I broke nothing else... so I'll now enable a full build of all three versions in all platforms... for debugging, shouldn't be merged

[fingolfin]

I've enabled more platforms now, and am seeing build issues that I already saw for the Julia 1.4.2 build, see #1704 (comment) -- so this used to work, and likely some of the changes that were made since then to libjulia_jll (mostly by me...) broke it. On the upside, perhaps that means we can get ARM 32bit builds to work across all Julia versions again. We'll see.

[fingolfin]

While FreeBSD now works, I noticed that Julia 1.5 now also fails in ARM 32 builds, too, with the same error as discussed here. On the upside, that suggests that we might be able to fix this also for 1.3 and 1.4, too.

The first suggestion here is to to add -latomic to LDFLAGS, but that just results in different errors:

[23:03:55] Error relocating /usr/lib/csl-glibc-x86_64/libatomic.so.1: __atomic_exchange_16: symbol not found
[23:03:55] Error relocating /usr/lib/csl-glibc-x86_64/libatomic.so.1: __atomic_compare_exchange_16: symbol not found
[23:03:55] Error relocating /usr/lib/csl-glibc-x86_64/libatomic.so.1: __atomic_load_16: symbol not found
[23:03:55] Error relocating /usr/lib/csl-glibc-x86_64/libatomic.so.1: __atomic_store_16: symbol not found
[23:03:55] make[1]: *** [Makefile:205: julia_flisp.boot] Error 127

This issue is also discussed in JuliaLang/julia#36371 so perhaps @yuyichao has some insights?

I have a suspicion that one needs to set a different -march=ARCH flag to get atomics working (but the compiler wrappers reject that, I think?). Or perhaps set MARCH/MCPU/... build vars suitably... I experimented a bit but couldn't figure out something that worked, yet.

[giordano]

but the compiler wrappers reject that, I think?

You can allow with lock_microarchitecture=false. It should be safe to do for this package.

[fingolfin]

D'oh, actually we already use lock_microarchitecture=false I just didn't try to use -march because I thought it wouldn't work 😂

[fingolfin]

Wooohooo that did it :-)

[fingolfin]

Status:

• PowerPC in Julia 1.3 still fails, unsurprisingly, due to glibc version issues; fixing this likely requires rebuilding LLVM_jll 6.0.1, see PR Rebuild LLVM 6.0.1 #1915 (and also Add libLLVM 6.0.1 for use in libjulia 1.3.1 #1914)
• i686 musl builds fails linking with error libunwind.so.8: undefined reference to setcontext'([see e.g. here](https://dev.azure.com/JuliaPackaging/Yggdrasil/_build/results?buildId=6844&view=logs&j=11cc9737-c3b2-5b2e-49c1-3ecd4dd3473d&t=6864626c-b206-5daa-5d51-831805962223&l=2954)). I wonder if this is a bug inLibUnwind_jll`?
• somehow at some point the macOS builds for Julia 1.5 broke; I now see this:

[15:47:56]  x86_64-apple-darwin14-clang++ -stdlib=libc++ -mmacosx-version-min=10.8 -march=x86-64 -integrated-as -m64 -I/workspace/destdir/include -std=c++14 -fno-exceptions -fno-rtti -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -fasynchronous-unwind-tables -D_LARGEFILE_SOURCE -D_DARWIN_USE_64_BIT_INODE=1 -DSYSTEM_LLVM -DJULIA_NUM_THREADS=1 -DHAVE_SSP=1 -pipe -fPIC -fno-rtti -pedantic -D_FILE_OFFSET_BITS=64 -I/workspace/destdir/include -DNDEBUG -O3 -g -D_GNU_SOURCE -I. -I/workspace/srcdir/julia-1.5.1/src -I/workspace/srcdir/julia-1.5.1/src/flisp -I/workspace/srcdir/julia-1.5.1/src/support -I/workspace/destdir/include -I/workspace/srcdir/julia-1.5.1/usr/include -DLIBRARY_EXPORTS -I/workspace/srcdir/julia-1.5.1/deps/valgrind -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -fvisibility=hidden -fno-common -Wpointer-arith -Wundef -DJL_BUILD_ARCH='"x86_64"' -DJL_BUILD_UNAME='"Darwin"' -I -DLLVM_SHLIB "-DJL_SYSTEM_IMAGE_PATH=\"../lib/julia/sys.dylib\"" -DJL_NDEBUG -c /workspace/srcdir/julia-1.5.1/src/llvm-remove-ni.cpp -o llvm-remove-ni.o
[15:47:57] In file included from /workspace/srcdir/julia-1.5.1/src/signal-handling.c:117:
[15:47:57] In file included from ./signals-unix.c:218:
[15:47:57] ./signals-mach.c:462:135: error: too few arguments to function call, expected 5, have 4
[15:47:57]                 bt_size_cur += rec_backtrace_ctx((jl_bt_element_t*)bt_data_prof + bt_size_cur, bt_size_max - bt_size_cur - 1, uc, NULL);
[15:47:57]                                ~~~~~~~~~~~~~~~~~                                                                                      ^
[15:47:57] ./julia_internal.h:797:1: note: 'rec_backtrace_ctx' declared here

Anyway, I gotta run now, so I'll only debug this later. If somebody knows the above issue and has a solution, please chime in.

[benlorenz]

A fix (from JuliaLang/julia#36821) for the macOS 1.5 rec_backtrace_ctx error is scheduled for julia 1.5.3, see JuliaLang/julia#38122.
I am slightly surprised that it is not using the #ifdef LIBOSXUNWIND block, but it looks like USE_SYSTEM_LIBUNWIND == 1 doesn't set anything useful for $(OS) == Darwin, see https://github.com/JuliaLang/julia/blob/v1.5.1/Make.inc#L929.

[benlorenz]

And regarding libunwind, according to the readme it requires setcontext and getcontext which musl doesn't provide. For x86_64 libunwind has it's own implementation but not for i686, see libunwind/libunwind#69. So maybe the musl i686 build for LibUnwind should be disabled?

[fingolfin]

@benlorenz thanks for the help, I got the macOS port fixed. And I won't fret about 32bit musl builds, I'll just keep them disabled.

[fingolfin]

To proceed with this, I'll let all builds run one more time to see if I got it right. Then I will submit this in three staged PRs: the first updates libjulia 1.3; the next then 1.4; and finally 1.5.

[fingolfin]

This PR now only rebuilds Julia 1.3, and is to be ready for review and merging.

[staticfloat]

This is really impressive! Quite obviously a work of much patience.

[staticfloat]

Practically, this is fine, but theoretically there's no guarantee on the ordering of tags in a platform triplet.

I suggest we extend the bb command to allow the build script to ask for the platforms it wants. E.g. bb get-triplet --libgfortran --cxxabi.

[staticfloat]

(This is not a request on this PR, more a note that we should do this eventually)

[fingolfin]

But isn't the precise value forbb_full_target what is used to generate the file names of the binary archives we upload to GitHub? And the precise value of BB_TRIPLET_LIBGFORTRAN_CXXABI (resp. the other BB_TRIPLET* values derived from it) what is then used to generate the download URLs? So they need to match?

What am I missing?

[giordano]

No, the environment variable ${bb_full_target} within the build environment has the full concrete specification of the target, including C++ string ABI and libgfortran version, even if those details aren't relevant for the generated tarballs

[staticfloat]

I'm saying that your bash variable substitution here says "take $bb_full_target and remove everything after -julia_version", but if the triplet gets serialized as x86_64-linux-gnu-julia_version+1.6.0-cxx11-libgfortran3, then this obviously won't work.

Practically speaking, that's never going to happen because due to historical reasons we'll always put libgfortran and cxx11 first, but some day in the future we may have other tags that are relevant here and they may or may not be placed before/after the julia_version tag. So I'm saying it would be good for you as the builder to ask BB for a triplet containing a certain subset of tags.

[staticfloat]

No, the environment variable ${bb_full_target} within the build environment has the full concrete specification of the target, including C++ string ABI and libgfortran version, even if those details aren't relevant for the generated tarballs

While that is true, the Julia build system is smart enough to strip off the cxxabi and libgfortran tags for dependencies that don't care about them. The way Max is passing the target triplet in is correct, I am merely mentioning this for future-proofing.

[fingolfin]

@giordano ah of course 🤦 that's what I was missing, thanks

UPDATE OK I guess I was missing even more 😂

[fingolfin]

Yeah, I can of course change it to match something like -julia_version[^-]+ or so (except that it'll have to fit into a bash regex).
But since this works now, I hope we can merge this as-is, and I promise I'll soon submit a cleaned version, to future-proof this. But then other things that are waiting for this PR could already proceed now.

[fingolfin]

Actually I just fixed this anyway, ${bb_full_target/-julia_version+([^-])} should do it

[fingolfin]

Merge pretty please? 🥺