Unreachable reached/illegal instruction by generating instances of random concrete types

[Seelengrab]

Hi!

The code for generating crashes can be found in this gist. Simply include that and run generate(Any) to (non-deterministically, sadly) crash julia with something similar to this (with varying degrees of depth, depending on how deep the generate rabbit hole goes):

Unreachable reached at 0x7f370cbe36d2

signal (4): Illegal instruction
in expression starting at none:1
generate at /home/<snip>/PropCheck/src/PropCheck.jl:105
iterate at ./generator.jl:47 [inlined]
collect_to! at ./array.jl:732
collect_to_with_first! at ./array.jl:710
unknown function (ip: 0x7f370cbe3682)
collect at ./array.jl:691
generate at /home/<snip>/PropCheck/src/PropCheck.jl:105
generate at /home/<snip>/PropCheck/src/PropCheck.jl:111
iterate at ./generator.jl:47 [inlined]
collect at ./array.jl:686
generate at /home/<snip>/PropCheck/src/PropCheck.jl:105
generate at /home/<snip>/PropCheck/src/PropCheck.jl:88
top-level scope at ./none:1
jl_toplevel_eval_flex at /home/<snip>/julia/src/toplevel.c:834
jl_toplevel_eval_flex at /home/<snip>/julia/src/toplevel.c:790
jl_toplevel_eval_flex at /home/<snip>/julia/src/toplevel.c:790
jl_toplevel_eval_in at /home/<snip>/julia/src/toplevel.c:883
eval at ./boot.jl:331
exec_options at ./client.jl:272
_start at ./client.jl:506
jfptr__start_54721 at /home/<snip>/julia/usr/lib/julia/sys.so (unknown line)
jl_apply at /home/<snip>/julia/ui/../src/julia.h:1690 [inlined]
true_main at /home/<snip>/julia/ui/repl.c:106
main at /home/<snip>/julia/ui/repl.c:227
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at ./julia/julia (unknown line)
Allocations: 4879465 (Pool: 4878483; Big: 982); GC: 6
Illegal instruction (core dumped)

The machine this specific crash was generated on looked like this:

Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Xeon(R) CPU E5-2430L v2 @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, ivybridge)

but I could also create the crash on this:

Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4

as well as this:

Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, westmere)

but not on this:

Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libllvm-9.0.1 (ORCJIT, ivybridge)

, where the julia process never seemed to crash but instead seemed to be stuck in an infinite loop.

The command that was run was basically julia -e 'include("unreachablereached.jl"); while true; generate(Any); end' (in order to ensure that it crashed, at least at some point), but even just running generate(Any) with that file included seemed to result in an endless loop/crash.

The machine with the IvyBridge Xeon CPU also was helpful in that I could create an rr trace with it, but the first one I made was wrong so I'll make a new one.

Anyhow, I hope I'm not doing something monumentally stupid in my code, I'd really like it if this could work 😅

I haven't tested master yet, but I plan to do that next.

[Seelengrab]

Trying to get an rr trace on master was a little... challenging.. but it did crash there as well.

[ Info: Uploading Trace directory

signal (11): Segmentation fault
in expression starting at none:0
_ZNK4llvm10BasicBlock13getTerminatorEv at /home/<snip>/julia/usr/bin/../lib/libLLVM-9jl.so (unknown line)
unknown function (ip: 0x7f6d3be8db9f)
Allocations: 39874104 (Pool: 39864252; Big: 9852); GC: 48
Segmentation fault (core dumped)

[Seelengrab]

I've got a trace from 1.5.1 (same machine as before, the IvyBridge Xeon) now.

https://s3.amazonaws.com/julialang-dumps/reports/2020-09-17T16-22-11-Seelengrab.tar.zst

This one should actually include the crash.

[vtjnash]

MWE from that particular rr trace:

julia> @code_typed fieldtypes(Module)
CodeInfo(
    @ reflection.jl:704 within `fieldtypes'
1 ─ %1 = Base.:(var"#12#13")::Core.Const(Base.var"#12#13")
│   %2 = Core.typeof(T)::Core.Const(DataType)
│   %3 = Core.apply_type(%1, %2)::Core.Const(Base.var"#12#13"{DataType})
│   %4 = %new(%3, T)::Core.Const(Base.var"#12#13"{DataType}(Module))
│   %5 = Base.fieldcount(T)::Core.Const(2)
│        Base.ntupleany(%4, %5)::Union{}
└──      unreachable
) => Union{}

julia> fieldtypes(Module)
(Symbol, Any)

edit: or more precisely:

julia> f() = fieldtype(Module, 1)
f (generic function with 1 method)

julia> @code_typed f()
CodeInfo(
    @ REPL[10]:1 within `f'
1 ─     Main.fieldtype(Main.Module, 1)::Union{}
└──     unreachable
) => Union{}

[Seelengrab]

Wow, that's amazing! Without rr, this would have been very hard to debug, I imagine. Let's hope that this is the only one :D

[vtjnash]

Not too bad without it, all I did was print the backtrace frame (p jl_(jl_gdblookuplinfo($pc))) and then confirm that caused the crash in a regular Julia session too. It's a regression in v1.5 caused by 41f07ac, of which Jeff said "this seems misleading..." 😜

[Seelengrab]

Well, without rr I certainly wouldn't have been able to give you the opportunity for a traced backtrace 😅 If this fixes the issue, I'm happy :)