julia: 1.1 -> 1.3

[cstich]

This updates julia to version 1.3.1. I couldn't get some of the dependencies as defined in the original build script to work, so this is building the source binary that includes all dependencies as provided by upstream. Over time I guess it would make sense to work on replacing the bundled dependencies with native ones.

Motivation for this change

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • [x ] NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• [x ] Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[7c6f434c]

Hm, what happenned to all-packages.nix change adding julia_1_3 = … ?

[cstich]

Hm, what happenned to all-packages.nix change adding julia_1_3 = … ?

Short answer, I looked for the place where top-level packages are defined but could not find it. So thanks for pointing that out. I have added it now to all-packages.nix. Also this might possibly be a very beginner question but how do I actually test the package definition in all-packages.nix?

[7c6f434c]

Hm, your file misses the Darwin-only arguments, as revealed by ofBorg failure.

To test that your checkout where you prepare the PR has a usable Julia 1.3 package, do nix-build -o julia-1.3 /path/to/nixpkgs-checkout -A julia_1_3 to get a julia-1.3 symlink to the built package.

In general, I guess the Nix manual could be useful (note that Nix, Nixpkgs and NixOS have their own different manuals)

[cstich]

I'll look into this. Thanks for the pointers.

[GTrunSec]

julia test failed

│ %5 = Test.Expr(:comparison, %2, ==, %4) [20/1955] │ %6 = Test.Expr(:comparison, $(QuoteNode(:(test12806()))), :(==), $(QuoteNode(:((true, true, false, [1]))))) │ @_2 = Test.eval_test(%5, %6, $(QuoteNode(:(#= /build/source/test/core.jl:951 =#))), $(QuoteNode(false))) └── $(Expr(:leave, 1)) 3 ─ goto #9 4 ┄ $(Expr(:leave, 1)) 5 ─ #4111#_e = $(Expr(:the_exception)) │ %12 = #4111#_e isa Test.InterruptException └── goto #7 if not %12 6 ─ Test.rethrow() └── goto #8 7 ─ false 8 ┄ %17 = #4111#_e │ %18 = Base.getproperty(Test.Base, :catch_stack) │ %19 = (%18)() │ @_2 = Test.Threw(%17, %19, $(QuoteNode(:(#= /build/source/test/core.jl:951 =#)))) └── $(Expr(:pop_exception, :(%1))) 9 ┄ %22 = @_2 │ Test.do_test(%22, $(QuoteNode(:(test12806() == (true, true, false, [1]))))) └── return ), 0x0000000000000001) … Ptr{Nothing} @0x00007fffc9b84b25, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b849d8, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b8487e, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b846f8, Ptr{Nothing} @0x00007fffc9b81c35, Ptr{Nothing} @0x00007ffff7af78ee, Ptr{Nothing} @0x0000000000000000] == Base.StackTraces.StackFrame[] Error in testset REPL: Test Failed at /build/source/usr/share/julia/stdlib/v1.3/REPL/test/replcompletions.jl:107

), 0x0000000000000001) … Ptr{Nothing} @0x00007fffc9b84b25, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b849d8, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b8487e, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b846f8, Ptr{Nothing} @0x00007fffc9b81c35, Ptr{Nothing} @0x00007ffff7af78ee, Ptr{Nothing} @0x0000000000000000] == Base.StackTraces.StackFrame[]
Error in testset REPL:
Test Failed at /build/source/usr/share/julia/stdlib/v1.3/REPL/test/replcompletions.jl:107
Expression: count(isequal("REPL"), c) == 1
Evaluated: 2 == 1
Error in testset precompile:
Error During Test at none:1
Test threw exception
Expression: precompile

Error in testset stress:
Error During Test at none:1
Test threw exception
Expression: stress

ERROR: LoadError: Test run finished with errors
in expression starting at /build/source/test/runtests.jl:62
make[1]: *** [Makefile:25: all] Error 1
make[1]: Leaving directory '/build/source/test'
make: *** [Makefile:584: testall] Error 2
builder for '/nix/store/az57bwbvh5fd346i29q4cwkqcnxsmdm3-julia-1.3.1.drv' failed with exit code 2
error: build of '/nix/store/az57bwbvh5fd346i29q4cwkqcnxsmdm3-julia-1.3.1.drv' failed

[cstich]

julia test failed

This looks like it this issue: JuliaLang/julia#32377
The workaround seems to be to use multiple threads to build julia. How many threads are you using to build it?
Alternatively we can think about skipping that particular test, although I don't know how from the top of my hat.

EDIT: Grammar

[GTrunSec]

julia test failed

This looks like it this issue: JuliaLang/julia#32377
The workaround seems to be to use multiple threads to build the julia. How many threads are you using to build it?
Alternatively we can think about skipping that particular test, although I don't know who from the top of my hat.

I didn't change the default setting of julia_1.3. So I guess the default thread number was one. Do I need
modify julia_1.3.nix to set multiple threads ? or which arguments I can set to skip the particular test?

[cstich]

Parallel builds are enabled for julia_1.3. Can you check the environment variable NIX_BUILD_CORES and see how many cores you have enabled for building nix expressions?

And I unfortunately do not know how to skip tests.

[GTrunSec]

Parallel builds are enabled for julia_1.3. Can you check the environment variable NIX_BUILD_CORES and see how many cores you have enabled for building nix expressions?

And I unfortunately do not know how to skip tests.

before building LLVM(it did my cup core number), when compiling Julia that did a single thread.

/build/source/src/codegen.cpp:6415:34: warning: 'retval' may be used uninitialized in this function [-Wmaybe-uninitialized]              
 6415 |             ctx.builder.CreateRet(retval);                                                                                       
      |             ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~                                                                                        
    LINK usr/lib/libjulia.so.1.3                                                                                                         
Warning: git information unavailable; versioning information limited                                                                     
    CC ui/repl.o                                                                                                                         
    LINK usr/bin/julia                                                                                                                   
Warning: git information unavailable; versioning information limited                                                                     
    JULIA usr/lib/julia/corecompiler.ji    


///////////////       it's going to  single thread                                                                                  
essentials.jl                                                                                                                            
ctypes.jl                                                                                                                                
generator.jl                                                                                                                             
reflection.jl                                                                                                                            
options.jl                                                                                                                               
promotion.jl                                                                                                                             
tuple.jl                                                                                                                                 
pair.jl                                                                                                                                  
traits.jl                      

[GTrunSec]

I just overrode the argument of checkTarget ="testall" to none

julia_13 = pkgs.julia_13.overrideAttrs(oldAttrs: {checkTarget = "";});  

it passed the build.

[twhitehead]

I believe the issue is that the test system doesn't truly isolate tests. Specifically, the Main module space is shared by the test running system and the tests themselves. This has bad effects in the REPL module which has several implicit and explicit references to Main.

What is screwing up this particular test is that test/runtests.jl imports REPL to get REPL.Terminal. This adds the module REPL to the Main namespace and causes the using REP completion test in stdlib/REPL/test/replcompletions.jl to return the module REPL brought into scope by the test runner in addition to the expected package REPL.

I tried to see if this issue could be worked around by wrapping replcompletions.jl in a module Main statement to hide the global one. While it did resolve this issue, it causes a few of the other later tests to stop working for reasons I haven't fully figured out. My wild guess would be that it might have to do with my Main not having the magic global Main export everything automatically property.

A horrible hack to resolve the issue is to just change import REPL in test/runtests.jl to import REPL.Terminal and all REPL.Terminal. call prefixes to just Terminal. I only mention this to prove the point though, not to suggest it be done.

[twhitehead]

Sorry for the last post. Had intended to post that on the julia issue tracker. Didn't realize I had jumped repositories when clicking links.

As it happens though, I have been actually working on this in order to get Julia 1.3 compiling under Nix. The other issue I've ran into is the exceptions test set can explode rather spectacularly due to an earlier core module test exposing JuliaLang/julia#34579.

My personal recommendation would be to just disable both this one test and the core module test (test12806).

[twhitehead]

Forgot to mention, you will want to set USER (why is this set to root in the build environment?) and HOME as well or you will have a bunch of other test failures.

HOME=$NIX_BUILD_TOP
USER=$(whoami)

[cstich]

@twhitehead How do I selectively disable tests for julia?
Also HOME is set to "$NIX_BUILD_TOP". And I do not know why USER is set to root, I just copied things from version 1.1 over to this. At some point this whole expression could probably use a re-write but I unfortunately do not have time for this right now.

[twhitehead]

You either have to do some text substitution on the test file (e.g., use sed to put a # in front of the line that runs the test) or remove or zero the test file itself. I'll submit my suggested cleanups when I get the chance (currently just having problems with one more test that I'm tracking down).