Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add libjulia 1.4.2 #1704

Merged
merged 1 commit into from
Oct 15, 2020
Merged

Conversation

fingolfin
Copy link
Member

No description provided.

@giordano giordano added the julia 💜 ❤️ 💚 Builders and issues related to Julia and its dependencies label Sep 27, 2020
@fingolfin fingolfin closed this Sep 28, 2020
@fingolfin fingolfin reopened this Sep 28, 2020
@fingolfin fingolfin force-pushed the mh/libjulia-1.4 branch 5 times, most recently from 6f34c38 to 7caa40f Compare October 4, 2020 20:28
@fingolfin fingolfin closed this Oct 4, 2020
@fingolfin fingolfin reopened this Oct 4, 2020
L/libjulia/common.jl Outdated Show resolved Hide resolved
L/libjulia/common.jl Outdated Show resolved Hide resolved
@fingolfin
Copy link
Member Author

Still a build issue on Darwin (and for Linux on 32bit arm, but I am willing to skip that). Will now compared with the results for libjulia@1.5 at https://dev.azure.com/JuliaPackaging/Yggdrasil/_build/results?buildId=5860&view=results to see if I can spot a hint

@fingolfin
Copy link
Member Author

Regarding lapack, here is the core of the issue: on macOS, USE_SYSTEM_BLAS=1 means something slightly different; to quote Make.inc:

ifeq ($(USE_SYSTEM_BLAS), 1)
ifeq ($(OS), Darwin)
USE_BLAS64 := 0
USE_SYSTEM_LAPACK := 0
LIBBLAS := -L$(build_libdir) -lgfortblas
LIBBLASNAME := libgfortblas
else
LIBBLAS ?= -lblas
LIBBLASNAME ?= libblas
endif
else
LIBBLAS := -L$(build_shlibdir) -lopenblas
LIBBLASNAME := libopenblas
endif

So in particular, it expects a BLAS versions which does not contain LAPACK, which is why it tries to build it. This then leads to an issue because lapack 3.5.0 (the version used by Julia 1.4.2) insists on running a few testbinaries at the end of its build&install process; which of course doesn't work in a cross compiler. In lapack 3.9.0 (used in Julia 1.5) this has been abolished.

So there are several ways forward for dealing with the macOS issue

  1. play a bit around with USE_SYSTEM_BLAS and other settings to see if we can convinced Julia to use the OpenBLAS we provided it anyhow
  2. patch Julia's Make.inc so that USE_SYSTEM_BLAS uses OpenBLAS on macOS, too (since we really only are interested in the Julia headers and linking against libjulia.so/.dylib/.dll during build time the BLAS version really shouldn't matter
  3. patch lapack 3.5.0 to just not run those test binaries (but that requires patching a file the Julia build process downloads, which is a bit awkward

I played with option 1 and it actually worked (I think Julia just download the OpenBLAS BinaryBuilder binaries itself -- a slight inefficiency, but I don't think it matters).

@fingolfin fingolfin marked this pull request as ready for review October 8, 2020 09:28
Comment on lines 127 to 128
override BB_TRIPLET_LIBGFORTRAN_CXXABI=${bb_full_target}
override USE_BINARYBUILDER=1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this in an earlier attempt to get things working, but that made no difference... Now I wonder if this might break the Julia 1.5 build? Perhaps I should adjust this PR to also update libjulia 1.5, just to verify it still works right? (could also update it to 1.5.2 -- although there really isn't much of a point, as we only care about the headers and ABI, and want to use libjulia_jll only as a build dependency).

@fingolfin
Copy link
Member Author

I am giving up on the ARM 32bit version for now. If somebody is interested in this, we can still add it later.

Comment on lines +188 to +198
for p in platforms
p["julia_version"] = "$(version.major).$(version.minor)"
end
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last caveat: I've been (ab?)using the julia_version feature @staticfloat added some time ago, to help select the correct package versions. But I just realized that this seems (?) to affect the "platform string". So perhaps this is a bad idea and I should try again without it (I can always go back to this version if I can't do it without). Thoughts?

@fingolfin
Copy link
Member Author

OK to merge into this?

USE_SYSTEM_LAPACK=1
LIBLAPACKNAME=libopenblas
# USE_SYSTEM_GMP=1
#USE_SYSTEM_BLAS=1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Doesn't make a difference on Linux and fixes macOS

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then we should only do this for 1.4.2, since it did work for 1.5

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think this will even work with 1.5.x ?

But I've now changed it to restrict this change to macOS, where using USE_SYSTEM_BLAS=1 has the same issue as on 1.4.x: it does not use OpenBLAS, but rather the BLAS shipped with the OS by Apple, period. And that is missing lapack, so it then ends up downloading and building that (even in Julia 1.5); the only reason it works in 1.5 is that the lapack there is newer and more cross compilation friendly. But there is a download in any case. By not setting USE_SYSTEM_BLAS=1, we get Julia on macOS to use its own BLAS binaries, which works well even on macOS and with Julia 1.4 and 1.5 equally.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vchuravy does that sound plausible / acceptable? I've applied a corresponding change to this PR now.

In any case, I think it would be useful to verify this PR doesn't break Julia 1.5.x builds. I could push a commit which edits the Julia 1.5 builder, to trigger a build of that, and then later drop that commit again once 1.5 builds (again). Alternatively, I could do that in a separate PR. Thoughts?

@fingolfin fingolfin force-pushed the mh/libjulia-1.4 branch 2 times, most recently from 6e4a3f3 to 5267bc1 Compare October 13, 2020 08:19
@fingolfin fingolfin closed this Oct 13, 2020
@fingolfin fingolfin reopened this Oct 13, 2020
@fingolfin fingolfin closed this Oct 13, 2020
@fingolfin fingolfin reopened this Oct 13, 2020
@giordano giordano closed this Oct 13, 2020
@giordano giordano reopened this Oct 13, 2020
@fingolfin fingolfin closed this Oct 14, 2020
@fingolfin fingolfin reopened this Oct 14, 2020
@fingolfin fingolfin closed this Oct 14, 2020
@fingolfin fingolfin reopened this Oct 14, 2020
@fingolfin fingolfin mentioned this pull request Oct 14, 2020
@fingolfin fingolfin closed this Oct 15, 2020
@fingolfin fingolfin reopened this Oct 15, 2020
@vchuravy vchuravy merged commit db92f1d into JuliaPackaging:master Oct 15, 2020
@fingolfin fingolfin deleted the mh/libjulia-1.4 branch October 15, 2020 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
julia 💜 ❤️ 💚 Builders and issues related to Julia and its dependencies
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants