julia_setup() crashes R because of a strange dynamic library path

[lahvak]

On Debian Linux (buster/sid combination), julia_setup() crashes the R session:

> library(JuliaCall)
> julia_setup()
Julia version 1.0.3 at location /usr/bin will be used.
ERROR: could not load library "/usr/lib/x86_64-linux-gnu/../bin/../lib/x86_64-linux-gnu/julia/sys.so"
/usr/lib/x86_64-linux-gnu/../bin/../lib/x86_64-linux-gnu/julia/sys.so: cannot open shared object file: No such file or directory

For some reason, it is looking for a library in a really bizarre location. I can use Julia from the command line and from a Jupyter notebook, but not from R. I get the same error from terminal R session, nvim-R, and r-studio.

Session Info

R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux buster/sid

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.5.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.5.2

[Non-Contradiction]

Thank you for the feedback!
Yes, it is looking for Julia library in a really bizarre location. And it is also not common to see Julia at "usr/bin."
Is there a Julia binary in the folder of "usr/bin"?
If you pass the folder which contains the Julia binary to argument JULIA_HOME, what will happen?

By the way, what is the version of JuliaCall?

[lahvak]

Yes, there is Julia in /usr/bin. It is where the Debian packaging system installs it.

The version of JuliaCall is 0.16.5.

When I pass "/usr/bin/" to JULIA_HOME, the same thing happens:

> julia_setup(JULIA_HOME="/usr/bin/")
Julia version 1.0.3 at location /usr/bin will be used.
ERROR: could not load library "/usr/lib/x86_64-linux-gnu/../bin/../lib/x86_64-linux-gnu/julia/sys.so"
/usr/lib/x86_64-linux-gnu/../bin/../lib/x86_64-linux-gnu/julia/sys.so: cannot open shared object file: No such file or directory

[lahvak]

I downloaded Julia 1.1.0 and unpacked it in my home directory, and now when I pass the location of the julia 1.1.0 binary to JULIA_HOME, everything works fine.

It seems that there may be a problem with the way julia is packaged in Debian.

[Non-Contradiction]

Thanks!
At the time of JuliaCall development, Debian had an old version of Julia packaged, so it was not supported at that time. From then on, JuliaCall assumes that Julia generic binary package is used, so it may have a hard time to locate Julia libraries in a system specific way.

[Emmanuel-R8]

The error is still present. The problem is not the location of the binary, but that of the sys.so library, The library is present at /usr/lib/x86_64-linux-gnu/julia/sys.so (Ubuntu install). The error comes from searching in /usr/lib/x86_64-linux-gnu/../bin/../lib/x86_64-linux-gnu/julia/ is not possible (the path makes no sense unless a chunk of it is removed or replace ../bin by ../../bin).

I track the error to the C++ function juliacall_initialize() (JuliaCall.cpp) which is called by julia_setup(). But I couldn't track further.

[Non-Contradiction]

@Emmanuel-R8 Thanks for the feedback and the tracking down. After investigating, I believe there are actually some issues in julia itself or the way that julia is compiling on the system.
Could you try to run

using Libdl; Libdl.dlpath("libjulia.so")
Libdl.dlpath("sys.so")
Base.Sys.BINDIR
Base.LIBDIR

in julia and see if the results make sense?

And you could also try dyn.load("the location of sys.so") in R before julia_setup() and see if things work.

[Emmanuel-R8]

Here it is (on Julia 1.3.1 - all packages up to date).

julia> using Libdl

julia> Libdl.dlpath("libjulia.so")
ERROR: could not load library "libjulia.so"
libjulia.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] #dlopen#3(::Bool, ::typeof(dlopen), ::String, ::UInt32) at /build/julia-LY8gXc/julia-1.3.1+dfsg/usr/share/julia/stdlib/v1.3/Libdl/src/Libdl.jl:109
 [2] dlopen at /build/julia-LY8gXc/julia-1.3.1+dfsg/usr/share/julia/stdlib/v1.3/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
 [3] dlpath(::String) at /build/julia-LY8gXc/julia-1.3.1+dfsg/usr/share/julia/stdlib/v1.3/Libdl/src/Libdl.jl:232
 [4] top-level scope at REPL[2]:1

But this is OK.

julia> Libdl.dlpath("libjulia.so.1")
"/usr/bin/../lib/x86_64-linux-gnu/libjulia.so.1"

And this is good too:

julia> Libdl.dlpath("sys.so")
"/usr/lib/x86_64-linux-gnu/julia/sys.so"

julia> Base.Sys.BINDIR
"/usr/bin"

julia> Base.LIBDIR
"../lib/x86_64-linux-gnu"

On R, dyn.load("/usr/lib/x86_64-linux-gnu/julia/sys.so") runs without error or messages. Running it a second time immediately afterwords breaks (but I have no idea if that the expected behaviour):

> dyn.load("/usr/lib/x86_64-linux-gnu/julia/sys.so")
Two passes with the same argument (-juliaO0) attempted to be registered!

 *** caught segfault ***
address 0x2, cause 'memory not mapped'

It breaks even I run dyn.unload("/usr/lib/x86_64-linux-gnu/julia/sys.so") in between.

[toeserve]

I am having this issue too on Ubuntu with simply embedding Julia into a C program following Julia's example. It has nothing to do with R therefore.

[toeserve]

Update: If you use jl_init_with_image__threading("/usr/lib/x86_64-linux-gnu/julia/", "sys.so"); in place of jl_init
in the file where you are embedding Julia (this is th code out of a C file), it will work. This function may be exposed to R, I have not looked into that

[nosuz]

Hello,

I met the same error on Ubuntu 20.04 today. The version number of Julia is 1.4.1 installed from Ubuntu repository.

I have remove Julia package and installed 1.4.2 downloaded from the official Julia web page.
This new version of Julia works fine.

Thank you,

[liskin]

This is a bug in Julia itself, see JuliaLang/julia#32614 (comment) and JuliaLang/julia#32614 (comment). Until the issue is fixed in Julia, it might be wise for JuliaCall to invoke jl_init_with_image whenever JULIA_BINDIR is set, so that users can export JULIA_BINDIR=/usr/bin in their bashrc to fix this. What do you folks think? Shall I make a pull request?

[liskin]

My wife needed this sooner rather than later, so the PR is here: #143