Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sporadic fatal errors during precompilation #48360

Closed
maleadt opened this issue Jan 20, 2023 · 12 comments
Closed

Sporadic fatal errors during precompilation #48360

maleadt opened this issue Jan 20, 2023 · 12 comments
Labels
bug Indicates an unexpected problem or unintended behavior heisenbug This bug occurs unpredictably

Comments

@maleadt
Copy link
Member

maleadt commented Jan 20, 2023

Using 1.9-beta3, I've ran into a segfault during precompilation a couple of times. I haven't been able to reproduce consistently, but there seems to be a pattern:

julia> using CUDA
[ Info: Precompiling CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
error: <inline asm>:1:2: invalid character in input
        ��


[50814] signal (11.1): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x7f81329a2eec)
exit at /usr/lib/libc.so.6 (unknown line)
_ZN4llvm11LLVMContext8diagnoseERKNS_14DiagnosticInfoE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNSt17_Function_handlerIFvRKN4llvm12SMDiagnosticEbRKNS0_9SourceMgrERSt6vectorIPKNS0_6MDNodeESaISA_EEEZNS0_28MachineModuleInfoWrapperPass16doInitializationERNS0_6ModuleEEUlS3_bS6_SD_E_E9_M_invokeERKSt9_Any_dataS3_ObS6_SD_ at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm9MCContext8diagnoseERKNS_12SMDiagnosticE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser11DiagHandlerERKN4llvm12SMDiagnosticEPv at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageERNS_11raw_ostreamENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS8_INS_7SMFixItEEEb at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS6_INS_7SMFixItEEEb at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser3RunEbb at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmENS_9StringRefERKNS_15MCSubtargetInfoERKNS_15MCTargetOptionsEPKNS_6MDNodeENS_9InlineAsm10AsmDialectE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmEPKNS_12MachineInstrE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm10AsmPrinter16emitFunctionBodyEv at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13X86AsmPrinter20runOnMachineFunctionERNS_15MachineFunctionE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
operator() at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1187
_ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
emit at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:612
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEvPvRS7_ at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EERKNS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISK_vEEEENSL_IS5_vEENS_6detail12DenseMapPairIS5_SN_EEEEEE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
addModule at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1385
jl_add_to_ee at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1761
jl_add_to_ee at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1783 [inlined]
_jl_compile_codeinst at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:235
jl_generate_fptr_impl at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:440
jl_compile_method_internal at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2278 [inlined]
jl_compile_method_internal at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2196
_jl_invoke at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2673 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2863
print at ./strings/io.jl:46
show at ./stream.jl:263 [inlined]
print at ./strings/io.jl:35
jfptr_print_47970.clone_1 at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2681 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2863
print_to_string at ./strings/io.jl:144
string at ./strings/io.jl:185 [inlined]
isopen at ./stream.jl:381 [inlined]
check_open at ./stream.jl:387 [inlined]
displaysize at ./stream.jl:569
jfptr_displaysize_52890.clone_1 at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2681 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2863
#handle_message#3 at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Logging/src/ConsoleLogger.jl:124
kwcall at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Logging/src/ConsoleLogger.jl:106
unknown function (ip: 0x7f8079300524)
_jl_invoke at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2681 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2863
jl_apply at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/julia.h:1874 [inlined]
jl_f__call_latest at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:818 [inlined]
kwcall at ./essentials.jl:813
unknown function (ip: 0x7f80d8127dee)
_jl_invoke at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2681 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2863
macro expansion at ./logging.jl:365 [inlined]
profile_printing_listener at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Profile/src/Profile.jl:49
#3 at ./threadingconstructs.jl:340
jfptr_YY.3_58552.clone_1 at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2681 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/gf.c:2863
jl_apply at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/julia.h:1874 [inlined]
start_task at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/task.c:1092
Allocations: 37232133 (Pool: 37219699; Big: 12434); GC: 57
ERROR: Failed to precompile CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] to "/home/tim/Julia/depot/compiled/v1.9/CUDA/jl_RVLMBJ".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2195
 [3] compilecache
   @ ./loading.jl:2068 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1712
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1567
 [6] macro expansion
   @ ./loading.jl:1555 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1518

or

julia> using oneAPI
[ Info: Precompiling oneAPI [8f75cd03-7ff8-4ecb-9b8f-daf728133b1b]
error: <inline asm>:1:3: invalid character in input
        PÉ


[3542258] signal (11.128): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x7f43d17504a6)
exit at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
main at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/cli/loader_exe.c:62
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 40361685 (Pool: 40355332; Big: 6353); GC: 86
ERROR: Failed to precompile oneAPI [8f75cd03-7ff8-4ecb-9b8f-daf728133b1b] to "/home/tim/Julia/depot/compiled/v1.9/oneAPI/jl_2fs5QR".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2195
 [3] compilecache
   @ ./loading.jl:2068 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1712
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1567
 [6] macro expansion
   @ ./loading.jl:1555 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1518

and also

julia> using oneAPI
[ Info: Precompiling oneAPI [8f75cd03-7ff8-4ecb-9b8f-daf728133b1b]
pure virtual method called
terminate called without an active exception

These errors are temporary; most of the time simply retrying the import makes it work.

Using 1.9-beta3 from juliaup:

julia> versioninfo(; verbose=true)
Julia Version 1.9.0-beta3
Commit 24204a73447 (2023-01-18 07:20 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  uname: Linux 6.1.6-arch1-3 #1 SMP PREEMPT_DYNAMIC Mon, 16 Jan 2023 12:51:23 +0000 x86_64 unknown
  CPU: AMD Ryzen 9 5950X 16-Core Processor:
                 speed         user         nice          sys         idle          irq
       #1-32  2200 MHz      45242 s       1326 s       9963 s   19919188 s       1110 s
  Memory: 62.220149993896484 GB (58765.95703125 MB free)
  Uptime: 62440.49 sec
  Load Avg:  0.41  0.55  0.41
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, znver3)
  Threads: 32 on 32 virtual cores
Environment:
  JULIA_PKG_DEVDIR = /home/tim/Julia/pkg
  JULIA_PKG_SERVER_REGISTRY_PREFERENCE = eager
  JULIA_PKG_USE_CLI_GIT = true
  JULIA_NUM_THREADS = 32
  JULIA_DEPOT_PATH = /home/tim/Julia/depot
  HOME = /home/tim
  PATH = /usr/local/sbin:/usr/local/bin:/usr/bin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/tim/.local/bin
  TERM = xterm-256color
  CUDA_PATH = /opt/cuda
@maleadt maleadt added the bug Indicates an unexpected problem or unintended behavior label Jan 20, 2023
@vtjnash vtjnash added this to the 1.9 milestone Jan 25, 2023
@maleadt
Copy link
Member Author

maleadt commented Feb 2, 2023

Seems to happen to packages that do artifact selection:

❯ /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/julia -Cnative -J/home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --compiled-modules=yes --project=/home/tim/Julia/pkg/NNlibCUDA/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/home/tim/Julia/depot"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/home/tim/Julia/depot/packages/CUDNN_jll/y2KvU/.pkg")
include("/home/tim/Julia/depot/packages/CUDNN_jll/y2KvU/.pkg/select_artifacts.jl")
' --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.9.0
[CUDNN]
arch = "x86_64"
git-tree-sha1 = "39edbd07a46d182c2681130c16ff339251297514"
lazy = true
libc = "glibc"
cuda = "11.0"
os = "linux"

    [[CUDNN.download]]
    sha256 = "7aedcce1519a244c79f1c57f894b66f1acdf9faba96d3cb15c464f0d229de8af"
    url = "https://github.com/JuliaBinaryWrappers/CUDNN_jll.jl/releases/download/CUDNN-v8.6.0+3/CUDNN.v8.6.0.x86_64-linux-gnu-cuda+11.0.tar.gz"
error: <inline asm>:1:2: invalid character in input
        �'


[44597] signal (11.128): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x7f483bd69eec)
exit at /usr/lib/libc.so.6 (unknown line)
_ZN4llvm11LLVMContext8diagnoseERKNS_14DiagnosticInfoE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNSt17_Function_handlerIFvRKN4llvm12SMDiagnosticEbRKNS0_9SourceMgrERSt6vectorIPKNS0_6MDNodeESaISA_EEEZNS0_28MachineModuleInfoWrapperPass16doInitializationERNS0_6ModuleEEUlS3_bS6_SD_E_E9_M_invokeERKSt9_Any_dataS3_ObS6_SD_ at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm9MCContext8diagnoseERKNS_12SMDiagnosticE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser11DiagHandlerERKN4llvm12SMDiagnosticEPv at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageERNS_11raw_ostreamENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS8_INS_7SMFixItEEEb at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS6_INS_7SMFixItEEEb at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser3RunEbb at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmENS_9StringRefERKNS_15MCSubtargetInfoERKNS_15MCTargetOptionsEPKNS_6MDNodeENS_9InlineAsm10AsmDialectE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmEPKNS_12MachineInstrE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm10AsmPrinter16emitFunctionBodyEv at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13X86AsmPrinter20runOnMachineFunctionERNS_15MachineFunctionE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
operator() at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1187
_ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
emit at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:612
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEvPvRS7_ at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EERKNS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISK_vEEEENSL_IS5_vEENS_6detail12DenseMapPairIS5_SN_EEEEEE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
addModule at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1385
jl_add_to_ee at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1761
jl_add_to_ee at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1783 [inlined]
_jl_compile_codeinst at /cache/build/default-amdci4-7/julialang/julia-release-1-dot-9/src/jitlayers.cpp:235
free(): corrupted unsorted chunks

[44597] signal (6.-6): Aborted
in expression starting at none:0
zsh: IOT instruction  /home/tim/Julia/depot/juliaup/julia-1.9.0-beta3+0.x64.linux.gnu/bin/julia

The above only reproduced once, so I didn't get to make an rr recording...

@vchuravy
Copy link
Member

vchuravy commented Feb 2, 2023

Yeah getting which function we die on would be good...
Can we register a handler that prints the whole function? @pchintalapudi

@pchintalapudi
Copy link
Member

I doubt this is the result of a bad function; it seems more like memory corruption. That being said, dumping every module and redirecting output to a file is probably the closest thing to dumping the last function. If lucky, you could try branching on verifyModule to print.

@KristofferC KristofferC added the heisenbug This bug occurs unpredictably label Feb 14, 2023
@KristofferC
Copy link
Sponsor Member

Since this issue is so hard to reproduce I don't think we can keep it on the 1.9 milestone indefinitely. So I will remove it and it will just have to get fixed when there is enough information to do something actionable.

@KristofferC KristofferC removed this from the 1.9 milestone Feb 20, 2023
@rafaqz
Copy link
Contributor

rafaqz commented Mar 16, 2023 •
edited
Loading

I've had similar segfauts that reference CUDNN. Unfortunately I lost the stack trace besides the same invalid character in input section.

@rafaqz
Copy link
Contributor

rafaqz commented Mar 16, 2023 •
edited
Loading

For me the context is always either add or dev of a package that is already used in the current session. It may be a slightly different bug?

julia> Pkg.add(url="https://github.com/rafaqz/MakieDraw.jl")
     Cloning git-repo `https://github.com/rafaqz/MakieDraw.jl`
    Updating git-repo `https://github.com/rafaqz/MakieDraw.jl`
   Resolving package versions...
error: <inline asm>:1:2: invalid character in input
        ��


[12977] signal (11.128): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x7f60191b83ac)
exit at /usr/lib/libc.so.6 (unknown line)
main at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/cli/loader_exe.c:62
unknown function (ip: 0x7f60191a078f)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 2998 (Pool: 2986; Big: 12); GC: 0
ERROR: failed process: Process(`/opt/julia/bin/julia -Cnative -J/opt/julia/lib/julia/sys.so --check-bounds=yes 
-g1 -O0 --color=no --history-file=no --startup-file=no --compiled-modules=yes --project=/home/raf/.julia/enviro
nments/v1.9/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/home/raf/.julia", "/opt/julia/local/share/
julia", "/opt/julia/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/home/raf/.julia/packages/CUDNN_jll/xLybE/.pkg")
include("/home/raf/.julia/packages/CUDNN_jll/xLybE/.pkg/select_artifacts.jl")
' --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.9.0`, ProcessSignaled(11)) 
[0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:710
  [4] collect_artifacts
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:696 [inlined]
  [5] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::Versi
onNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:742
  [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.T
ypes.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1361
  [7] add
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1343 [inlined]
  [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platf
orm::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tupl
e{Base.TTY}}})
    @ Pkg.API /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:275
  [9] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, Named
Tuple{(), Tuple{}}})
    @ Pkg.API /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [10] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [11] #add#24
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
 [12] add
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
 [13] add(; name::Nothing, uuid::Nothing, version::Nothing, url::String, rev::Nothing, path::Nothing, mode::Pkg
.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:173
 [14] top-level scope
    @ REPL[30]:1

julia>

@natema
Copy link
Contributor

natema commented Apr 20, 2023

I get the same error trying to add MKL.jl in Julia 1.9.0-rc2. I add it without problems using Julia 1.8.5, while I get the following with Julia 1.9:

(@v1.9) pkg> add MKL
   Resolving package versions...
error: <inline asm>:1:2: invalid character in input
        ��


[17926] signal (11.128): Segmentation fault
in expression starting at none:0
__run_exit_handlers at /lib64/libc.so.6 (unknown line)
exit at /lib64/libc.so.6 (unknown line)
_ZN4llvm11LLVMContext8diagnoseERKNS_14DiagnosticInfoE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNSt17_Function_handlerIFvRKN4llvm12SMDiagnosticEbRKNS0_9SourceMgrERSt6vectorIPKNS0_6MDNodeESaISA_EEEZNS0_28MachineModuleInfoWrapperPass16doInitializationERNS0_6ModuleEEUlS3_bS6_SD_E_E9_M_invokeERKSt9_Any_dataS3_ObS6_SD_ at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm9MCContext8diagnoseERKNS_12SMDiagnosticE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser11DiagHandlerERKN4llvm12SMDiagnosticEPv at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageERNS_11raw_ostreamENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS8_INS_7SMFixItEEEb at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS6_INS_7SMFixItEEEb at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser3RunEbb at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmENS_9StringRefERKNS_15MCSubtargetInfoERKNS_15MCTargetOptionsEPKNS_6MDNodeENS_9InlineAsm10AsmDialectE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmEPKNS_12MachineInstrE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm10AsmPrinter16emitFunctionBodyEv at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13X86AsmPrinter20runOnMachineFunctionERNS_15MachineFunctionE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
operator() at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1206
_ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
emit at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:631
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEvPvRS7_ at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EERKNS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISK_vEEEENSL_IS5_vEENS_6detail12DenseMapPairIS5_SN_EEEEEE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/../lib/julia/libLLVM-14jl.so (unknown line)
addModule at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1420
jl_add_to_ee at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1815
_jl_compile_codeinst at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:241
jl_generate_fptr_impl at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:460
jl_compile_method_internal at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2348 [inlined]
jl_compile_method_internal at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2237
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2750 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
print at ./strings/io.jl:46
show at ./stream.jl:263 [inlined]
print at ./strings/io.jl:35
jfptr_print_33427.clone_1 at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
print_to_string at ./strings/io.jl:144
string at ./strings/io.jl:185 [inlined]
isopen at ./stream.jl:381 [inlined]
check_open at ./stream.jl:387 [inlined]
displaysize at ./stream.jl:569
jfptr_displaysize_53371.clone_1 at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
#handle_message#3 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Logging/src/ConsoleLogger.jl:124
handle_message at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Logging/src/ConsoleLogger.jl:106
unknown function (ip: 0x7f02e0208698)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
#invokelatest#2 at ./essentials.jl:818
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
invokelatest at ./essentials.jl:813
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
macro expansion at ./logging.jl:365 [inlined]
profile_printing_listener at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Profile/src/Profile.jl:49
#3 at ./threadingconstructs.jl:341
jfptr_YY.3_59756.clone_1 at /home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
Allocations: 2995 (Pool: 2983; Big: 12); GC: 0
ERROR: failed process: Process(`/home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/bin/julia -Cnative -J/home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --compiled-modules=yes --project=/home/enatale/.julia/environments/v1.9/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/home/enatale/.julia", "/home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/local/share/julia", "/home/enatale/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/home/enatale/.julia/packages/CUDNN_jll/xLybE/.pkg")
include("/home/enatale/.julia/packages/CUDNN_jll/xLybE/.pkg/select_artifacts.jl")
' --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.9.0`, ProcessSignaled(11)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:721
  [4] collect_artifacts
    @ ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:707 [inlined]
  [5] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:753
  [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1390
  [7] add
    @ ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1372 [inlined]
  [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:275
  [9] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [10] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [11] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:412
 [12] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:390
 [13] do_cmd
    @ ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:380 [inlined]
 [14] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:557
 [15] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [16] invokelatest
    @ ./essentials.jl:813 [inlined]
 [17] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2647
 [18] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL ~/.julia/juliaup/julia-1.9.0-rc2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1298
 [19] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:514

@rafaqz
Copy link
Contributor

rafaqz commented Apr 21, 2023

This really seems to be a problem specifically with CUDNN_jll artifacts.

JuliaPackaging/Yggdrasil#6626

@ViralBShah
Copy link
Member

Is this still an issue?

@rafaqz
Copy link
Contributor

rafaqz commented Aug 7, 2023

It went away with CUDNN a few months ago, but theres something simililar reported with XGBoost in the Yggdrasil merhod linked above

@maleadt maleadt mentioned this issue Nov 10, 2023
Open
@KSepetanc
Copy link

In the latest Julia 1.10.4 the error still persists.

Precompiling CUDA with PackagePrecompiler or adding it as dependency and them compiling any other package will result in segmentation fault. #2428

@maleadt
Copy link
Member Author

maleadt commented Jul 2, 2024

JuliaGPU/CUDA.jl#2428 is unlikely to be related to this issue, which always reports an LLVM diagnostic error before exiting/segfaulting/aborting.

I haven't seen this specific (artifact selection-related) issue in ages, so I think we can close this, even though we never found the underlying issue...

@maleadt maleadt closed this as completed Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behavior heisenbug This bug occurs unpredictably
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@vchuravy @vtjnash @maleadt @ViralBShah @KristofferC @rafaqz @natema @pchintalapudi @KSepetanc