Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

julia: init julia_10-bin and julia_16-bin #123188

Merged
merged 2 commits into from May 17, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions pkgs/development/compilers/julia/1.0-bin.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
{ autoPatchelfHook, fetchurl, lib, makeWrapper, openssl, stdenv }:

stdenv.mkDerivation rec {
pname = "julia-bin";
version = "1.0.5";

src = {
x86_64-linux = fetchurl {
url = "https://julialang-s3.julialang.org/bin/linux/x64/${lib.versions.majorMinor version}/julia-${version}-linux-x86_64.tar.gz";
sha256 = "00vbszpjmz47nqy19v83xa463ajhzwanjyg5mvcfp9kvfw9xdvcx";
};
}.${stdenv.hostPlatform.system} or (throw "Unsupported system: ${stdenv.hostPlatform.system}");

# Julia’s source files are in different locations for source and binary
# releases. Thus we temporarily create symlinks to allow us to share patches
# with source releases.
Comment on lines +14 to +16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand this comment. AFAICT there's no overlap between the patches used for the source and binary derivations. All of the patches seem to be split up: patches/1.0, patches/1.0-bin, etc.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not referring to the location in the Nixpkgs source tree, but rather the paths in the patches. If we do not do this trick we would have to alter the patches themselves as the paths are different between the Julia release tarballs.

prePatch = ''
ln -s share/julia/stdlib/v${lib.versions.majorMinor version} stdlib
ln -s share/julia/test
'';
patches = [
# Source release Nix patch(es) relevant for binary releases as well.
./patches/1.0-bin/0002-nix-Skip-tests-that-require-network-access.patch
];
postPatch = ''
# Revert symlink hack.
rm stdlib test
'';

buildInputs = [ makeWrapper ];
nativeBuildInputs = [ autoPatchelfHook ];

installPhase = ''
runHook preInstall
cp -r . $out
# Setting `LD_LIBRARY_PATH` resolves `Libdl` failures. Not sure why this is
# only necessary on v1.0.x and a cleaner solution is welcome, but after
# staring at `strace` for a few hours this is as clean as I could make it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe 1.0 adds $out/lib to RPATH but links no libraries (using dlopen instead), so Nixpkgs automatic patchelf removes it from RPATH?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, that could be it. Will investigate.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I suspect you are bang on:

> objdump -x lib/libjulia.so | grep RPATH
  RPATH                $ORIGIN/julia:$ORIGIN
> objdump -x /nix/store/pxs6qkddkhpvk1hpkahgz4w18ahfd3m1-julia-bin-1.0.5/lib/libjulia.so | grep RPATH
>

Digging through the manual I thought that what I should do was to set dontPatchELF to true, but I still end up with the RPATH being removed.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is not easily addressable, I am happy to merge this as is and try to clean it up later.

wrapProgram $out/bin/julia \
--suffix LD_LIBRARY_PATH : $out/lib
runHook postInstall
'';

# Breaks backtraces, etc.
dontStrip = true;

doInstallCheck = true;
installCheckInputs = [ openssl ];
preInstallCheck = ''
# Some tests require read/write access to $HOME.
export HOME="$TMPDIR"
'';
installCheckPhase = ''
runHook preInstallCheck
# Command lifted from `test/Makefile`.
$out/bin/julia \
--check-bounds=yes \
--startup-file=no \
--depwarn=error \
$out/share/julia/test/runtests.jl
runHook postInstallCheck
'';

meta = {
description = "High-level, high-performance dynamic language for technical computing";
homepage = "https://julialang.org";
# Bundled and linked with various GPL code, although Julia itself is MIT.
license = lib.licenses.gpl2Plus;
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to this PR, but does this mean that we should chance the license from MIT to GPL 2+ for our source-based Julia packages as well?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nobody is sure what exactly our license metadata means in any non-trivial case, so …

Here it is clear: here a GPL2+ artifacy is downloaded.

maintainers = with lib.maintainers; [ ninjin raskin ];
platforms = [ "x86_64-linux" ];
};
}
73 changes: 73 additions & 0 deletions pkgs/development/compilers/julia/1.6-bin.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
{ autoPatchelfHook, fetchurl, lib, stdenv }:

stdenv.mkDerivation rec {
pname = "julia-bin";
version = "1.6.1";

src = {
x86_64-linux = fetchurl {
url = "https://julialang-s3.julialang.org/bin/linux/x64/${lib.versions.majorMinor version}/julia-${version}-linux-x86_64.tar.gz";
sha256 = "01i5sm4vqb0y5qznql571zap19b42775drrcxnzsyhpaqgg8m23w";
};
}.${stdenv.hostPlatform.system} or (throw "Unsupported system: ${stdenv.hostPlatform.system}");

# Julia’s source files are in different locations for source and binary
# releases. Thus we temporarily create a symlink to allow us to share patches
# with source releases.
Comment on lines +14 to +16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as my other comment.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise.

prePatch = ''
ln -s share/julia/test
'';
patches = [
# Source release Nix patch(es) relevant for binary releases as well.
./patches/1.6-bin/0002-nix-Skip-tempname-test-broken-in-sandbox.patch
./patches/1.6-bin/0003-nix-Skip-chown-tests-broken-in-sandbox.patch
./patches/1.6-bin/0005-nix-Enable-parallel-unit-tests-for-sandbox.patch
];
postPatch = ''
# Revert symlink hack.
rm test

# Julia fails to pick up our Certification Authority root certificates, but
# it provides its own so we can simply disable the test. Patching in the
# dynamic path to ours require us to rebuild the Julia system image.
substituteInPlace share/julia/stdlib/v${lib.versions.majorMinor version}/NetworkOptions/test/runtests.jl \
--replace '@test ca_roots_path() != bundled_ca_roots()' \
'@test_skip ca_roots_path() != bundled_ca_roots()'
'';

nativeBuildInputs = [ autoPatchelfHook ];

installPhase = ''
runHook preInstall
cp -r . $out
runHook postInstall
'';

# Breaks backtraces, etc.
dontStrip = true;

doInstallCheck = true;
preInstallCheck = ''
# Some tests require read/write access to $HOME.
export HOME="$TMPDIR"
'';
installCheckPhase = ''
runHook preInstallCheck
# Command lifted from `test/Makefile`.
$out/bin/julia \
--check-bounds=yes \
--startup-file=no \
--depwarn=error \
$out/share/julia/test/runtests.jl
runHook postInstallCheck
'';

meta = {
description = "High-level, high-performance dynamic language for technical computing.";
homepage = "https://julialang.org";
# Bundled and linked with various GPL code, although Julia itself is MIT.
license = lib.licenses.gpl2Plus;
maintainers = with lib.maintainers; [ ninjin raskin ];
platforms = [ "x86_64-linux" ];
};
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
From 4954b99efae367da49412edd31a7bd832ec62c69 Mon Sep 17 00:00:00 2001
From: Pontus Stenetorp <pontus@stenetorp.se>
Date: Mon, 15 Mar 2021 05:55:18 +0000
Subject: [PATCH 2/3] nix: Skip tests that require network access

Necessary as the Nix build sandbox does not permit network access.
---
stdlib/Sockets/test/runtests.jl | 40 ++++++++++++++++-----------------
test/file.jl | 4 ++--
2 files changed, 22 insertions(+), 22 deletions(-)

diff --git a/stdlib/Sockets/test/runtests.jl b/stdlib/Sockets/test/runtests.jl
index 6145f87616..9cc7a001e5 100644
--- a/stdlib/Sockets/test/runtests.jl
+++ b/stdlib/Sockets/test/runtests.jl
@@ -151,33 +151,33 @@ defaultport = rand(2000:4000)
end

@testset "getnameinfo on some unroutable IP addresses (RFC 5737)" begin
- @test getnameinfo(ip"192.0.2.1") == "192.0.2.1"
- @test getnameinfo(ip"198.51.100.1") == "198.51.100.1"
- @test getnameinfo(ip"203.0.113.1") == "203.0.113.1"
- @test getnameinfo(ip"0.1.1.1") == "0.1.1.1"
- @test getnameinfo(ip"::ffff:0.1.1.1") == "::ffff:0.1.1.1"
- @test getnameinfo(ip"::ffff:192.0.2.1") == "::ffff:192.0.2.1"
- @test getnameinfo(ip"2001:db8::1") == "2001:db8::1"
+ @test_skip getnameinfo(ip"192.0.2.1") == "192.0.2.1"
+ @test_skip getnameinfo(ip"198.51.100.1") == "198.51.100.1"
+ @test_skip getnameinfo(ip"203.0.113.1") == "203.0.113.1"
+ @test_skip getnameinfo(ip"0.1.1.1") == "0.1.1.1"
+ @test_skip getnameinfo(ip"::ffff:0.1.1.1") == "::ffff:0.1.1.1"
+ @test_skip getnameinfo(ip"::ffff:192.0.2.1") == "::ffff:192.0.2.1"
+ @test_skip getnameinfo(ip"2001:db8::1") == "2001:db8::1"
end

@testset "getnameinfo on some valid IP addresses" begin
@test !isempty(getnameinfo(ip"::")::String)
- @test !isempty(getnameinfo(ip"0.0.0.0")::String)
- @test !isempty(getnameinfo(ip"10.1.0.0")::String)
- @test !isempty(getnameinfo(ip"10.1.0.255")::String)
- @test !isempty(getnameinfo(ip"10.1.255.1")::String)
- @test !isempty(getnameinfo(ip"255.255.255.255")::String)
- @test !isempty(getnameinfo(ip"255.255.255.0")::String)
- @test !isempty(getnameinfo(ip"192.168.0.1")::String)
- @test !isempty(getnameinfo(ip"::1")::String)
+ @test_skip !isempty(getnameinfo(ip"0.0.0.0")::String)
+ @test_skip !isempty(getnameinfo(ip"10.1.0.0")::String)
+ @test_skip !isempty(getnameinfo(ip"10.1.0.255")::String)
+ @test_skip !isempty(getnameinfo(ip"10.1.255.1")::String)
+ @test_skip !isempty(getnameinfo(ip"255.255.255.255")::String)
+ @test_skip !isempty(getnameinfo(ip"255.255.255.0")::String)
+ @test_skip !isempty(getnameinfo(ip"192.168.0.1")::String)
+ @test_skip !isempty(getnameinfo(ip"::1")::String)
end

@testset "getaddrinfo" begin
- let localhost = getnameinfo(ip"127.0.0.1")::String
- @test !isempty(localhost) && localhost != "127.0.0.1"
- @test !isempty(getalladdrinfo(localhost)::Vector{IPAddr})
- @test getaddrinfo(localhost, IPv4)::IPv4 != ip"0.0.0.0"
- @test try
+ let localhost = getnameinfo(ip"::")::String
+ @test_skip !isempty(localhost) && localhost != "127.0.0.1"
+ @test_skip !isempty(getalladdrinfo(localhost)::Vector{IPAddr})
+ @test_skip getaddrinfo(localhost, IPv4)::IPv4 != ip"0.0.0.0"
+ @test_skip try
getaddrinfo(localhost, IPv6)::IPv6 != ip"::"
catch ex
isa(ex, Sockets.DNSError) && ex.code == Base.UV_EAI_NONAME && ex.host == localhost
diff --git a/test/file.jl b/test/file.jl
index e86476f975..579276f82c 100644
--- a/test/file.jl
+++ b/test/file.jl
@@ -874,8 +874,8 @@ if !Sys.iswindows() || (Sys.windows_version() >= Sys.WINDOWS_VISTA_VER)
else
@test_throws ErrorException symlink(file, "ba\0d")
end
-@test_throws ArgumentError download("good", "ba\0d")
-@test_throws ArgumentError download("ba\0d", "good")
+@test_skip @test_throws ArgumentError download("good", "ba\0d")
+@test_skip @test_throws ArgumentError download("ba\0d", "good")

###################
# walkdir #
--
2.29.3

Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
From ffe227676352a910754d96d92e9b06e475f28ff1 Mon Sep 17 00:00:00 2001
From: Pontus Stenetorp <pontus@stenetorp.se>
Date: Thu, 8 Apr 2021 04:25:19 +0000
Subject: [PATCH 2/6] nix: Skip `tempname` test broken in sandbox

Reported upstream:

https://github.com/JuliaLang/julia/issues/38873
---
test/file.jl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/file.jl b/test/file.jl
index 0f39bc7c14..bd4dd78f62 100644
--- a/test/file.jl
+++ b/test/file.jl
@@ -95,7 +95,7 @@ end
@test dirname(t) == tempdir()
mktempdir() do d
t = tempname(d)
- @test dirname(t) == d
+ @test_skip dirname(t) == d
end
@test_throws ArgumentError tempname(randstring())
end
--
2.29.3

Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
From b20357fb1044d2c100172b1d5cbdf6c6d9bd3590 Mon Sep 17 00:00:00 2001
From: Pontus Stenetorp <pontus@stenetorp.se>
Date: Thu, 8 Apr 2021 05:10:39 +0000
Subject: [PATCH 3/6] nix: Skip `chown` tests broken in sandbox

---
test/file.jl | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/file.jl b/test/file.jl
index bd4dd78f62..06fd4e49da 100644
--- a/test/file.jl
+++ b/test/file.jl
@@ -503,8 +503,8 @@ if !Sys.iswindows()
@test stat(file).gid == 0
@test stat(file).uid == 0
else
- @test_throws Base._UVError("chown($(repr(file)), -2, -1)", Base.UV_EPERM) chown(file, -2, -1) # Non-root user cannot change ownership to another user
- @test_throws Base._UVError("chown($(repr(file)), -1, -2)", Base.UV_EPERM) chown(file, -1, -2) # Non-root user cannot change group to a group they are not a member of (eg: nogroup)
+ @test_skip @test_throws Base._UVError("chown($(repr(file)), -2, -1)", Base.UV_EPERM) chown(file, -2, -1) # Non-root user cannot change ownership to another user
+ @test_skip @test_throws Base._UVError("chown($(repr(file)), -1, -2)", Base.UV_EPERM) chown(file, -1, -2) # Non-root user cannot change group to a group they are not a member of (eg: nogroup)
end
else
# test that chown doesn't cause any errors for Windows
--
2.29.3

Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
From 44c2c979c4f2222567ce65f506cf47fb87482348 Mon Sep 17 00:00:00 2001
From: Pontus Stenetorp <pontus@stenetorp.se>
Date: Thu, 8 Apr 2021 04:37:44 +0000
Subject: [PATCH 5/6] nix: Enable parallel unit tests for sandbox

Disabled by default due to lack of networking in the Nix sandbox. This
greatly speeds up the build process on a multi-core system.
---
test/runtests.jl | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/test/runtests.jl b/test/runtests.jl
index 2f9cd058bb..2f8c19fa32 100644
--- a/test/runtests.jl
+++ b/test/runtests.jl
@@ -83,8 +83,9 @@ prepend!(tests, linalg_tests)
import LinearAlgebra
cd(@__DIR__) do
n = 1
- if net_on
- n = min(Sys.CPU_THREADS, length(tests))
+ if net_on || haskey(ENV, "NIX_BUILD_CORES")
+ x = haskey(ENV, "NIX_BUILD_CORES") ? parse(Int, ENV["NIX_BUILD_CORES"]) : Sys.CPU_THREADS
+ n = min(x, Sys.CPU_THREADS, length(tests))
n > 1 && addprocs_with_testenv(n)
LinearAlgebra.BLAS.set_num_threads(1)
end
--
2.29.3

7 changes: 7 additions & 0 deletions pkgs/top-level/all-packages.nix
Original file line number Diff line number Diff line change
Expand Up @@ -11113,6 +11113,13 @@ in
julia-stable = julia_15;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should reshuffle some of these as well. In particular, I don't think it makes sense for any of the "flagship" attributes to point to julia_15 since that one's broken.

Also I think an alias julia_16 = julia_16-bin would make sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 julia_16 = julia_16-bin

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that this makes sense, but I would rather address that in a separate PR and/or issue.

julia = julia-lts;

julia_10-bin = callPackage ../development/compilers/julia/1.0-bin.nix { };
julia_16-bin = callPackage ../development/compilers/julia/1.6-bin.nix { };

julia-lts-bin = julia_10-bin;
julia-stable-bin = julia_16-bin;
julia-bin = julia-lts-bin;

jwasm = callPackage ../development/compilers/jwasm { };

knightos-genkfs = callPackage ../development/tools/knightos/genkfs { };
Expand Down