Likely doesn't work in 1.11.x

[PallHaraldsson]

[See here: https://github.com/JuliaLang/julia/issues/56805 This part is false alarm, I was looking into use of jl_array_size [here] that was removed in Julia, while only functions jl_array_sizehint* are defined here and used... kind of regret helping out since the testing took a long time and broke/restarted my computer because of OOM, rare except for OOMs...]

I do still see in code:

Enzyme.jl/src/compiler/validation.jl

Lines 76 to 77 in 7c0823f

	"jl_array_sizehint",
	"ijl_array_sizehint",

but even though it and more gone from Julia 1.11, I think it actually doesn't matter here. If so, then drop it and more that is gone and tag a new release for Julia 1.11. Not sure, might be actively bad over there:

Enzyme.jl/src/rules/llvmrules.jl

Lines 2207 to 2208 in 7c0823f

	register_handler!(
	("jl_array_sizehint", "ijl_array_sizehint"),

I'm not sure if the test is showing that, I expected a segfault, it's likely failing for one more reason (only):

1.11 is supposed to be supported (#1358), but likely is broken (on 1.11.1):

Test Summary:        | Pass  Error  Total  Time
Simple Complex tests |   18      1     19  7.0s
ERROR: LoadError: Some tests did not pass: 18 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/pharaldsson/.julia/packages/Enzyme/haqjK/test/runtests.jl:512
ERROR: Package Enzyme errored during testing

Maybe nothing much to see here:

(@v1.11) pkg> test Enzyme
     Testing Enzyme
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ Pkg.Operations ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:1902
      Status `/tmp/jl_OqKzGM/Project.toml`
...
⌅ [86de99a1] LLVM_jll v16.0.6+4
...
Precompiling project for configuration --code-coverage=none --color=yes --check-bounds=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none...
  14 dependencies successfully precompiled in 287 seconds. 110 already precompiled.
     Testing Running tests...
┌ Info: Testing against
└   Enzyme_jll.libEnzyme = "/home/pharaldsson/.julia/artifacts/d4a52b1aa639e6d5a437ac40520d49ebb2890d4f/lib/libEnzyme-16.so"
Test Summary: | Pass  Total  Time
Const Return  |    2      2  0.7s
...
Test Summary:   | Pass  Broken  Total  Time
within_autodiff |    2       1      3  0.1s
...
Test Summary:             | Pass  Broken  Total  Time
Batch ActiveReverse Rules |    1       2      3  0.1s
...
Test Summary:      | Pass  Total  Time
Cholesky PotRF 3x3 |   36     36  2.9s
┌ Warning: TODO forward zero-set of memorycopy used memset rather than runtime type 
│ Caused by:
│ Stacktrace:
│  [1] copy
│    @ ./array.jl:350
│  [2] tcholsolv_lower
│    @ ~/.julia/packages/Enzyme/haqjK/test/internal_rules.jl:243
│  [3] #29
│    @ ~/.julia/packages/Enzyme/haqjK/test/internal_rules.jl:271
│  [4] #29
│    @ ~/.julia/packages/Enzyme/haqjK/test/internal_rules.jl:0
└ @ Enzyme.Compiler ~/.julia/packages/GPUCompiler/GnbhK/src/utils.jl:59
┌ Warning: TODO forward zero-set of memorycopy used memset rather than runtime type 
...
Test Summary:               | Pass  Total  Time
Absint sum vector of vector |    1      1  2.7s
┌ Warning: Using fallback BLAS replacements for (["cblas_cdotu_sub64_"]), performance may be degraded
└ @ Enzyme.Compiler ~/.julia/packages/GPUCompiler/GnbhK/src/utils.jl:59
┌ Warning: Using fallback BLAS replacements for (["cblas_cdotu_sub64_"]), performance may be degraded
└ @ Enzyme.Compiler ~/.julia/packages/GPUCompiler/GnbhK/src/utils.jl:59
┌ Warning: Using fallback BLAS replacements for (["cblas_zdotu_sub64_"]), performance may be degraded
...
Test Summary: | Pass  Total     Time
BLAS rules    | 5556   5556  4m10.5s
Test Summary:  | Pass  Total  Time
Internal tests |   29     29  1.6s
Test Summary: | Total  Time
Reflection    |     0  0.5s
; ModuleID = 'start'
source_filename = "start"
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-linux-gnu"

;  @ /home/pharaldsson/.julia/packages/Enzyme/haqjK/test/runtests.jl:249 within `sqrtsumsq2`
; Function Attrs: alwaysinline
define "enzyme_type"="{[-1]:Float@double}" double @julia_sqrtsumsq2_754439({}* noundef nonnull align 8 dereferenceable(24) "enzyme_type"="{[-1]:Pointer, [-1,0]:Pointer, [-1,0,-1]:Float@double, [-1,8]:Pointer, [-1,8,0]:Integer, [-1,8,1]:Integer, [-1,8,2]:Integer, [-1,8,3]:Integer, [-1,8,4]:Integer, [-1,8,5]:Integer, [-1,8,6]:Integer, [-1,8,7]:Integer, [-1,8,8]:Pointer, [-1,8,8,-1]:Float@double, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer}" "enzymejl_parmtype"="140379536534160" "enzymejl_parmtype_ref"="2" %0) local_unnamed_addr #0 {
top:
  %thread_ptr = call i8* asm "movq %fs:0, $0", "=r"() #18
  %tls_ppgcstack = getelementptr i8, i8* %thread_ptr, i64 -8
  %1 = bitcast i8* %tls_ppgcstack to {}****
...
; │┌ @ float.jl:493 within `*`
    %17 = fmul fast double %16, %2
    %18 = fmul fast double %10, %2
    %gc_slot_addr_0 = getelementptr inbounds [8 x {}*], [8 x {}*]* %gcframe15, i64 0, i64 2
    %19 = extractvalue { { {}*, i64, double, double, double, double* }, double } %_augmented.i, 0, 0
    store {}* %19, {}** %gc_slot_addr_0, align 16
; │└
; │┌ @ reducedim.jl:983 within `sum`
; ││┌ @ reducedim.jl:983 within `#sum#934`
; │││┌ @ reducedim.jl:987 within `_sum`
; ││││┌ @ reducedim.jl:987 within `#_sum#936`
; │││││┌ @ reducedim.jl:329 within `mapreduce`
; ││││││┌ @ reducedim.jl:329 within `#mapreduce#926`
; │││││││┌ @ reducedim.jl:337 within `_mapreduce_dim`
          call fastcc void @diffejulia__mapreduce_754384({}* nocapture nonnull readonly align 8 dereferenceable(24) %0, {}* nocapture readonly align 8 %1, double %18, { {}*, i64, double, double, double, double* } %subcache.i)
          call fastcc void @diffejulia__mapreduce_754384.7({}* nocapture nonnull readonly align 8 dereferenceable(24) %0, {}* nocapture readonly align 8 %1, double %17, { {}*, {}*, {}*, i64, {}*, {}*, double, double, double, double* } %subcache4.i)
          fence syncscope("singlethread") seq_cst
          %frame.prev14 = load {}*, {}** %frame.prev, align 8
          %20 = bitcast {}*** %tls_pgcstack to {}**
          store {}* %frame.prev14, {}** %20, align 8
          ret void
; └└└└└└└└
}

declare void @jl_throw({}*)

declare noalias nonnull {}** @julia.new_gc_frame(i32)

declare void @julia.push_gc_frame({}**, i32)

declare {}** @julia.get_gc_frame_slot({}**, i32)

; Function Attrs: nounwind willreturn allockind("alloc") allocsize(1) memory(argmem: read, inaccessiblemem: readwrite)
declare noalias nonnull {}* @julia.gc_alloc_bytes(i8*, i64, i64) #13

declare void @julia.pop_gc_frame({}**)

; Function Attrs: memory(argmem: readwrite, inaccessiblemem: readwrite)
declare void @julia.queue_gc_root({}*) #14

; Function Attrs: memory(argmem: readwrite, inaccessiblemem: readwrite)
declare void @ijl_gc_queue_root({}*) #14

; Function Attrs: nounwind willreturn allockind("alloc") allocsize(2) memory(argmem: read, inaccessiblemem: readwrite)
declare noalias nonnull {}* @ijl_gc_pool_alloc_instrumented(i8*, i32, i32, i64) #15

; Function Attrs: nounwind willreturn allockind("alloc") allocsize(1) memory(argmem: read, inaccessiblemem: readwrite)
declare noalias nonnull {}* @ijl_gc_big_alloc_instrumented(i8*, i64, i64) #13

; Function Attrs: nounwind willreturn allockind("alloc") allocsize(1) memory(argmem: read, inaccessiblemem: readwrite)
declare noalias nonnull {}* @ijl_gc_alloc_typed(i8*, i64, i64) #13

; Function Attrs: nocallback nofree nounwind willreturn memory(argmem: write)
declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg) #16

; Function Attrs: nocallback nofree nounwind willreturn memory(argmem: readwrite)
declare void @llvm.memcpy.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64, i1 immarg) #17

attributes #0 = { mustprogress nofree nounwind willreturn allockind("alloc,uninitialized") allocsize(1) memory(inaccessiblemem: readwrite) "enzyme_no_escaping_allocation" "enzymejl_world"="27210" }
attributes #1 = { nofree memory(argmem: readwrite, inaccessiblemem: readwrite) "enzyme_inactive" "enzyme_no_escaping_allocation" "enzymejl_world"="27210" }
attributes #2 = { noinline noreturn "enzyme_ta_norecur" "enzymejl_mi"="140379558484832" "enzymejl_rt"="140379652936144" "enzymejl_world"="27210" }
attributes #3 = { nofree "enzymejl_world"="27210" }
attributes #4 = { mustprogress nofree nounwind willreturn memory(inaccessiblemem: readwrite) "enzyme_inactive" "enzyme_no_escaping_allocation" "enzymejl_world"="27210" }
attributes #5 = { noreturn "enzyme_no_escaping_allocation" "enzymejl_world"="27210" }
attributes #6 = { noreturn "enzymejl_world"="27210" }
attributes #7 = { norecurse nosync nounwind speculatable willreturn memory(none) "enzyme_nocache" "enzyme_shouldrecompute" "enzymejl_world"="27210" }
attributes #8 = { noinline noreturn "enzyme_ta_norecur" "enzymejl_mi"="140379569787376" "enzymejl_rt"="140379652936144" "enzymejl_world"="27210" }
attributes #9 = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
attributes #10 = { mustprogress noinline "enzyme_parmremove"="3" "enzyme_ta_norecur" "enzymejl_mi"="140379713999952" "enzymejl_rt"="140379591438352" "enzymejl_world"="27210" }
attributes #11 = { mustprogress "enzyme_ta_norecur" "enzymejl_mi"="140379728730512" "enzymejl_rt"="140379591438352" "enzymejl_world"="27210" }
attributes #12 = { alwaysinline "enzymejl_world"="27210" }
attributes #13 = { nounwind willreturn allockind("alloc") allocsize(1) memory(argmem: read, inaccessiblemem: readwrite) }
attributes #14 = { memory(argmem: readwrite, inaccessiblemem: readwrite) }
attributes #15 = { nounwind willreturn allockind("alloc") allocsize(2) memory(argmem: read, inaccessiblemem: readwrite) }
attributes #16 = { nocallback nofree nounwind willreturn memory(argmem: write) }
attributes #17 = { nocallback nofree nounwind willreturn memory(argmem: readwrite) }
attributes #18 = { nounwind }
attributes #19 = { nounwind willreturn memory(inaccessiblemem: readwrite) "enzyme_no_escaping_allocation" }
attributes #20 = { noreturn }
attributes #21 = { mustprogress noreturn willreturn }

!llvm.module.flags = !{!0, !1}

!0 = !{i32 2, !"Dwarf Version", i32 4}
!1 = !{i32 2, !"Debug Info Version", i32 3}

Test Summary:          | Pass  Broken  Total  Time
Recursion optimization |    4       1      5  2.6s
Test Summary:          | Total  Time
Batch Reverse sret fix |     0  0.5s
Test Summary: | Total  Time
Batch Closure |     0  0.2s
Test Summary: | Pass  Total   Time
Simple tests  |   79     79  14.9s
Test Summary:               | Pass  Total  Time
Deferred and deferred thunk |    8      8  1.2s
Test Summary:    | Pass  Total  Time
Deferred upgrade |    1      1  0.3s
Simple Complex tests: Error During Test at /home/pharaldsson/.julia/packages/Enzyme/haqjK/test/runtests.jl:512
  Got exception outside of a @test
  Enzyme compilation failed due to an internal error.
   Please open an issue with the code to reproduce and full error log on github.com/EnzymeAD/Enzyme.jl To toggle more information for debugging (needed for bug reports), set Enzyme.Compiler.VERBOSE_ERRORS[] = true (default false)
  
  Stacktrace:
   [1] getindex
     @ ./essentials.jl:917
   [2] _mapreduce
     @ ./reduce.jl:435
   [3] _mapreduce_dim
     @ ./reducedim.jl:337
   [4] mapreduce
     @ ./reducedim.jl:329
   [5] _sum
     @ ./reducedim.jl:987
   [6] _sum
     @ ./reducedim.jl:986
   [7] sum
     @ ./reducedim.jl:982
   [8] sumsq3
     @ ~/.julia/packages/Enzyme/haqjK/test/runtests.jl:563
   [9] sumsq3
     @ ~/.julia/packages/Enzyme/haqjK/test/runtests.jl:0
  
  Stacktrace:
    [1] (::Enzyme.Compiler.var"#getparent#69"{Context, LLVM.Function, LLVM.IntegerType, Int64, Dict{LLVM.PHIInst, LLVM.PHIInst}, Dict{LLVM.PHIInst, LLVM.PHIInst}, LLVM.PHIInst, LLVM.BitCastInst, IRBuilder})(v::Value, offset::Value, hasload::Bool)
      @ Enzyme.Compiler ~/.julia/packages/Enzyme/haqjK/src/llvm/transforms.jl:888
    [2] (::Enzyme.Compiler.var"#getparent#69"{Context, LLVM.Function, LLVM.IntegerType, Int64, Dict{LLVM.PHIInst, LLVM.PHIInst}, Dict{LLVM.PHIInst, LLVM.PHIInst}, LLVM.PHIInst, LLVM.BitCastInst, IRBuilder})(v::Value, offset::Value, hasload::Bool)
      @ Enzyme.Compiler ~/.julia/packages/Enzyme/haqjK/src/llvm/transforms.jl:777
    [3] nodecayed_phis!(mod::LLVM.Module)
      @ Enzyme.Compiler ~/.julia/packages/Enzyme/haqjK/src/llvm/transforms.jl:891
    [4] optimize!(mod::LLVM.Module, tm::TargetMachine)
      @ Enzyme.Compiler ~/.julia/packages/Enzyme/haqjK/src/compiler/optimize.jl:582
    [5] codegen(output::Symbol, job::CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, toplevel::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
      @ Enzyme.Compiler ~/.julia/packages/Enzyme/haqjK/src/compiler.jl:4096
    [6] codegen
      @ ~/.julia/packages/Enzyme/haqjK/src/compiler.jl:3223 [inlined]
    [7] _thunk(job::CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, postopt::Bool)
      @ Enzyme.Compiler ~/.julia/packages/Enzyme/haqjK/src/compiler.jl:5273
    [8] _thunk
      @ ~/.julia/packages/Enzyme/haqjK/src/compiler.jl:5273 [inlined]
    [9] cached_compilation
      @ ~/.julia/packages/Enzyme/haqjK/src/compiler.jl:5324 [inlined]
   [10] thunkbase(mi::Core.MethodInstance, World::UInt64, FA::Type{<:Annotation}, A::Type{<:Annotation}, TT::Type, Mode::Enzyme.API.CDerivativeMode, width::Int64, ModifiedBetween::NTuple{N, Bool} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool)
      @ Enzyme.Compiler ~/.julia/packages/Enzyme/haqjK/src/compiler.jl:5434
   [11] thunk_generator(world::UInt64, source::LineNumberNode, FA::Type, A::Type, TT::Type, Mode::Enzyme.API.CDerivativeMode, Width::Int64, ModifiedBetween::NTuple{N, Bool} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, self::Any, fakeworld::Any, fa::Type, a::Type, tt::Type, mode::Type, width::Type, modifiedbetween::Type, returnprimal::Type, shadowinit::Type, abi::Type, erriffuncwritten::Type, runtimeactivity::Type)
      @ Enzyme.Compiler ~/.julia/packages/Enzyme/haqjK/src/compiler.jl:5601
   [12] autodiff
      @ ~/.julia/packages/Enzyme/haqjK/src/Enzyme.jl:446 [inlined]
   [13] autodiff(mode::ReverseMode{false, false, FFIABI, true, false}, f::var"#sumsq3#106", ::Type{Active}, args::Duplicated{Vector{ComplexF64}})
      @ Enzyme ~/.julia/packages/Enzyme/haqjK/src/Enzyme.jl:524
   [14] macro expansion
      @ ~/.julia/packages/Enzyme/haqjK/test/runtests.jl:566 [inlined]
   [15] macro expansion
      @ ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Test/src/Test.jl:1700 [inlined]
   [16] top-level scope
      @ ~/.julia/packages/Enzyme/haqjK/test/runtests.jl:513
   [17] include(fname::String)
      @ Main ./sysimg.jl:38
   [18] top-level scope
      @ none:6
   [19] eval
      @ ./boot.jl:430 [inlined]
   [20] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:296
   [21] _start()
      @ Base ./client.jl:531
Test Summary:        | Pass  Error  Total  Time
Simple Complex tests |   18      1     19  7.0s
ERROR: LoadError: Some tests did not pass: 18 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/pharaldsson/.julia/packages/Enzyme/haqjK/test/runtests.jl:512
ERROR: Package Enzyme errored during testing

[vchuravy]

The handlers are independent of version, we could likely be better and not register handlers for runtime functions that don't exist on Julia, but we handle jl_array_sizehint not being present gracefully.

[PallHaraldsson]

Thanks, what I thought,but what about the error itself in the test?