[Runner] Move csl_paths(host_platform) at the beginning of LD_LIBRARY_PATH

[giordano]

This should make it easier to run applications we build with the host toolchain. CC: @vchuravy

[vchuravy]

This keeps blocking new LLVM versions being built for Julia so it would be great to have a solution that allows running binaries built for the host.

[giordano]

Tested in JuliaPackaging/BinaryBuilder.jl#1032, unless I'm missing something, all failures are in Rust tests... Rust toolchain is hopelessly broken... 😢

[giordano]

This seems to cause some issues: JuliaPackaging/Yggdrasil#3249 (comment) I wonder whether we should override LD_LIBRARY_PATH inside the compiler wrappers

---

Reproducer (it should work at least with {i686,x86_64}-linux-musl-lbgfortran5 with GCC 9+):

int main() {
    return 0;
}

sandbox:${WORKSPACE}/srcdir/petsc-3.15.2 # cc -o conftest conftest.c -Wl,-rpath,/workspace/destdir/lib -L/workspace/destdir/lib -lmpifort -lquadmath -Wl,--trace
/opt/x86_64-linux-musl/bin/../x86_64-linux-musl/sys-root/usr/lib/crt1.o
/opt/x86_64-linux-musl/bin/../x86_64-linux-musl/sys-root/usr/lib/crti.o
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/crtbegin.o
/tmp/ccoBfNHm.o
/workspace/destdir/lib/libmpifort.so
/opt/x86_64-linux-musl/x86_64-linux-musl/lib64/libquadmath.so
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/libgcc.a
/opt/x86_64-linux-musl/x86_64-linux-musl/lib64/libgcc_s.so
/opt/x86_64-linux-musl/x86_64-linux-musl/lib64/libgcc_s.so.1
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/libgcc.a
/opt/x86_64-linux-musl/bin/../x86_64-linux-musl/sys-root/usr/lib/libc.so
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/libgcc.a
/opt/x86_64-linux-musl/x86_64-linux-musl/lib64/libgcc_s.so
/opt/x86_64-linux-musl/x86_64-linux-musl/lib64/libgcc_s.so.1
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/libgcc.a
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/crtend.o
/opt/x86_64-linux-musl/bin/../x86_64-linux-musl/sys-root/usr/lib/crtn.o
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `sinhq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `erfcq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `acosq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `tanq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `jnq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `strtoflt128@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `atan2q@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `nanq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `expq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `cexpq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `atanq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `atanhq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `lroundq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `truncq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `csinq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `csqrtq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `coshq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `ynq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `ccosq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `asinq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `quadmath_snprintf@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `tanhq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `fmaq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `clogq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `roundq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `fmodq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `copysignq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `fabsq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `sqrtq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `log10q@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `cosq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `asinhq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `logq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `acoshq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `sinq@QUADMATH_1.0'
/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/10.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/lib/csl-glibc-x86_64/libgfortran.so.5: undefined reference to `cabsq@QUADMATH_1.0'
collect2: error: ld returned 1 exit status

Needs MPICH_jll

[giordano]

Ok, this pull request is turning into a massive yak shaving.

Now I'm overriding LD_LIBRARY_PATH inside the compiler wrappers and I added a test based on the reproducer above. However, now I can't run the compiled program for i686-linux-gnu-libgfortran5. Note: this works on master, more specifically it works without 86114d6 (#157), this means that running the program only worked because the CSL paths have always been wrong. Running the test program results in the following error:

./test: error while loading shared libraries: libc.musl-i386.so.1: cannot open shared object file: No such file or directory

@staticfloat does this mean we aren't rejecting musl correctly? Any hints?

---

Edit: as an additional datapoint, this is what ldd shows:

sandbox:${WORKSPACE}/srcdir # ldd ./test
        linux-gate.so.1 (0xf7fbf000)
        libmpifort.so.12 => /workspace/destdir/lib/libmpifort.so.12 (0xf7f4f000)
        libquadmath.so.0 => /usr/lib/csl-glibc-i686/libquadmath.so.0 (0xf7ed9000)
        libc.so.6 => /lib/libc.so.6 (0xf7d01000)
        libmpi.so.12 => /workspace/destdir/lib/libmpi.so.12 (0xf71a2000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xf7181000)
        librt.so.1 => /lib/librt.so.1 (0xf7176000)
        libgfortran.so.5 => /usr/lib/csl-musl-i686/libgfortran.so.5 (0xf6ecd000)
        libm.so.6 => /lib/libm.so.6 (0xf6e03000)
        libgcc_s.so.1 => /usr/lib/csl-glibc-i686/libgcc_s.so.1 (0xf6de2000)
        /lib/ld-linux.so.2 => ldd (0xf7fc1000)
        libc.musl-i386.so.1 => not found

It looks like we load i686-linux-musl libgfortran

---

Edit 2: ha! LD_LIBRARY_PATH is messed up, it has CSL for i686-linux-musl before that for i686-linux-gnu:

sandbox:${WORKSPACE}/srcdir # echo $LD_LIBRARY_PATH 
/usr/lib/csl-musl-x86_64:/usr/lib/csl-glibc-x86_64:/usr/lib/csl-musl-i686:/usr/lib/csl-glibc-i686:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/i686-linux-gnu/i686-linux-gnu/lib64:/opt/i686-linux-gnu/i686-linux-gnu/lib:/workspace/destdir/lib64:/workspace/destdir/lib

This works instead:

sandbox:${WORKSPACE}/srcdir # LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/lib/csl-glibc-x86_64:/usr/lib/csl-glibc-i686:/usr/lib/csl-musl-i686:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/i686-linux-gnu/i686-linux-gnu/lib64:/opt/i686-linux-gnu/i686-linux-gnu/lib:/workspace/destdir/lib64:/workspace/destdir/lib ./test 
Hello World!

Yay!

[vchuravy]

However, now I can't run the compiled program for i686-linux-gnu-libgfortran5.

That sounds like a feature xD

[giordano]

Ok, I think I found a solution: generate CSL paths only for target and host platforms, at the moment we were generating them for all platforms, resulting in the error shown above: #157 (comment). The annoying thing is that the order is very important: target first only if it has an architecture different from the host, otherwise host first.

As a bonus point, now we can run C++ applications for i686-linux-musl which never worked, woooo! 🚀

Nevermind, this doesn't really work: we can run C++ applications for i686-linux-musl, but then on that platform we can't run C++ applications for the host. Reverting that change.