Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local runtime discovery does not work for external libraries (CUDNN, CUTENSOR) #1850

Closed
peter-fields opened this issue Mar 31, 2023 · 7 comments · Fixed by #1858
Closed

Local runtime discovery does not work for external libraries (CUDNN, CUTENSOR) #1850

peter-fields opened this issue Mar 31, 2023 · 7 comments · Fixed by #1858
Labels
bug Something isn't working installation CUDA is easy to install, right?

Comments

@peter-fields
Copy link

I am using CUDA and Flux on a HPC. There is no internet access when I mount a gpu via slurm so I ran CUDA.set_runtime_version!("local") , which CUDA.jl 4 requires for finding local installs. I ensure local installation of cuda and cudnn are loaded via the slurm module load command. importing Flux gives an error indicating cuDNN is not available for my platform, and broadcasting functions from Flux onto CuArrays does not work. See MWE below.

julia> using CUDA

julia> using Flux
┌ Error: cuDNN is not available for your platform (x86_64-linux-gnu-libgfortran4-cxx11-libstdcxx29-cuda+local-julia_version+1.7.3)
└ @ cuDNN ~/.julia/packages/cuDNN/tgAWl/src/cuDNN.jl:167

julia> x =cu(Float32.((rand(2,2))))
2×2 CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}:
 0.763084  0.126537
 0.709367  0.564856

julia> Flux.sigmoid.(x)
ERROR: UndefVarError: libcudnn not defined
Stacktrace:
  [1] macro expansion
    @ ~/.julia/packages/cuDNN/tgAWl/src/libcudnn.jl:694 [inlined]
  [2] macro expansion
    @ ~/.julia/packages/CUDA/N71Iw/src/pool.jl:312 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/cuDNN/tgAWl/src/libcudnn.jl:22 [inlined]
  [4] cudnnCreateActivationDescriptor(activationDesc::Vector{Ptr{cuDNN.cudnnActivationStruct}})
    @ cuDNN ~/.julia/packages/CUDA/N71Iw/lib/utils/call.jl:26
  [5] cuDNN.cudnnActivationDescriptor(::cuDNN.cudnnActivationMode_t, ::Vararg{Any})
    @ cuDNN ~/.julia/packages/cuDNN/tgAWl/src/descriptors.jl:39
  [6] #cudnnActivationForward!#1187
    @ ~/.julia/packages/cuDNN/tgAWl/src/activation.jl:22 [inlined]
  [7] #55
    @ ~/.julia/packages/NNlibCUDA/C6t0p/src/cudnn/activations.jl:12 [inlined]
  [8] materialize(bc::Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{2}, Nothing, typeof(σ), Tuple{CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}}})
    @ NNlibCUDA ~/.julia/packages/NNlibCUDA/C6t0p/src/cudnn/activations.jl:31
  [9] top-level scope
    @ REPL[6]:1
 [10] top-level scope
    @ ~/.julia/packages/CUDA/N71Iw/src/initialization.jl:163
Manifest.toml 

julia_version = "1.7.3"
manifest_format = "2.0"

[[deps.AbstractFFTs]]
deps = ["ChainRulesCore", "LinearAlgebra"]
git-tree-sha1 = "16b6dbc4cf7caee4e1e75c49485ec67b667098a0"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.3.1"

[[deps.Accessors]]
deps = ["Compat", "CompositionsBase", "ConstructionBase", "Dates", "InverseFunctions", "LinearAlgebra", "MacroTools", "Requires", "StaticArrays", "Test"]
git-tree-sha1 = "beabc31fa319f9de4d16372bff31b4801e43d32c"
uuid = "7d9f7c33-5ae7-4f3b-8dc6-eff91059b697"
version = "0.1.28"

[[deps.Adapt]]
deps = ["LinearAlgebra", "Requires"]
git-tree-sha1 = "cc37d689f599e8df4f464b2fa3870ff7db7492ef"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.6.1"

[[deps.ArgCheck]]
git-tree-sha1 = "a3a402a35a2f7e0b87828ccabbd5ebfbebe356b4"
uuid = "dce04be8-c92d-5529-be00-80e4d2c0e197"
version = "2.3.0"

[[deps.ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[deps.Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[deps.Atomix]]
deps = ["UnsafeAtomics"]
git-tree-sha1 = "c06a868224ecba914baa6942988e2f2aade419be"
uuid = "a9b6321e-bd34-4604-b9c9-b65b8de01458"
version = "0.1.0"

[[deps.BFloat16s]]
deps = ["LinearAlgebra", "Printf", "Random", "Test"]
git-tree-sha1 = "dbf84058d0a8cbbadee18d25cf606934b22d7c66"
uuid = "ab4f0b2a-ad5b-11e8-123f-65d77653426b"
version = "0.4.2"

[[deps.BangBang]]
deps = ["Compat", "ConstructionBase", "Future", "InitialValues", "LinearAlgebra", "Requires", "Setfield", "Tables", "ZygoteRules"]
git-tree-sha1 = "7fe6d92c4f281cf4ca6f2fba0ce7b299742da7ca"
uuid = "198e06fe-97b7-11e9-32a5-e1d131e6ad66"
version = "0.3.37"

[[deps.Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[deps.Baselet]]
git-tree-sha1 = "aebf55e6d7795e02ca500a689d326ac979aaf89e"
uuid = "9718e550-a3fa-408a-8086-8db961cd8217"
version = "0.1.1"

[[deps.CEnum]]
git-tree-sha1 = "eb4cb44a499229b3b8426dcfb5dd85333951ff90"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.2"

[[deps.CUDA]]
deps = ["AbstractFFTs", "Adapt", "BFloat16s", "CEnum", "CUDA_Driver_jll", "CUDA_Runtime_Discovery", "CUDA_Runtime_jll", "CompilerSupportLibraries_jll", "ExprTools", "GPUArrays", "GPUCompiler", "KernelAbstractions", "LLVM", "LazyArtifacts", "Libdl", "LinearAlgebra", "Logging", "Preferences", "Printf", "Random", "Random123", "RandomNumbers", "Reexport", "Requires", "SparseArrays", "SpecialFunctions", "UnsafeAtomicsLLVM"]
git-tree-sha1 = "f1f97192581609e8042839dff335c630a19b53d7"
uuid = "052768ef-5323-5732-b1bb-66c8b64840ba"
version = "4.1.1"

[[deps.CUDA_Driver_jll]]
deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"]
git-tree-sha1 = "a1efe2bfb96c10906dede10f1580aca0ae6d092f"
uuid = "4ee394cb-3365-5eb0-8335-949819d2adfc"
version = "0.4.0+2"

[[deps.CUDA_Runtime_Discovery]]
deps = ["Libdl"]
git-tree-sha1 = "d6b227a1cfa63ae89cb969157c6789e36b7c9624"
uuid = "1af6417a-86b4-443c-805f-a4643ffb695f"
version = "0.1.2"

[[deps.CUDA_Runtime_jll]]
deps = ["Artifacts", "CUDA_Driver_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"]
git-tree-sha1 = "96e06a24c89a9945c57278fd09fb717f71476d87"
uuid = "76a88914-d11a-5bdc-97e0-2f5a05c973a2"
version = "0.4.0+2"

[[deps.CUDNN_jll]]
deps = ["Artifacts", "CUDA_Runtime_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"]
git-tree-sha1 = "2918fbffb50e3b7a0b9127617587afa76d4276e8"
uuid = "62b44479-cb7b-5706-934f-f13b2eb2e645"
version = "8.8.1+0"

[[deps.ChainRules]]
deps = ["Adapt", "ChainRulesCore", "Compat", "Distributed", "GPUArraysCore", "IrrationalConstants", "LinearAlgebra", "Random", "RealDot", "SparseArrays", "Statistics", "StructArrays"]
git-tree-sha1 = "7d20c2fb8ab838e41069398685e7b6b5f89ed85b"
uuid = "082447d4-558c-5d27-93f4-14fc19e9eca2"
version = "1.48.0"

[[deps.ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "c6d890a52d2c4d55d326439580c3b8d0875a77d9"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "1.15.7"

[[deps.ChangesOfVariables]]
deps = ["ChainRulesCore", "LinearAlgebra", "Test"]
git-tree-sha1 = "485193efd2176b88e6622a39a246f8c5b600e74e"
uuid = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0"
version = "0.1.6"

[[deps.CommonSubexpressions]]
deps = ["MacroTools", "Test"]
git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7"
uuid = "bbf7d656-a473-5ed7-a52c-81e309532950"
version = "0.3.0"

[[deps.Compat]]
deps = ["Dates", "LinearAlgebra", "UUIDs"]
git-tree-sha1 = "7a60c856b9fa189eb34f5f8a6f6b5529b7942957"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "4.6.1"

[[deps.CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[deps.CompositionsBase]]
git-tree-sha1 = "455419f7e328a1a2493cabc6428d79e951349769"
uuid = "a33af91c-f02d-484b-be07-31d278c5ca2b"
version = "0.1.1"

[[deps.ConstructionBase]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "89a9db8d28102b094992472d333674bd1a83ce2a"
uuid = "187b0558-2788-49d3-abe0-74a17ed4e7c9"
version = "1.5.1"

[[deps.ContextVariablesX]]
deps = ["Compat", "Logging", "UUIDs"]
git-tree-sha1 = "25cc3803f1030ab855e383129dcd3dc294e322cc"
uuid = "6add18c4-b38d-439d-96f6-d6bc489c04c5"
version = "0.1.3"

[[deps.DataAPI]]
git-tree-sha1 = "e8119c1a33d267e16108be441a287a6981ba1630"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.14.0"

[[deps.DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "d1fff3a548102f48987a52a2e0d114fa97d730f0"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.13"

[[deps.DataValueInterfaces]]
git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6"
uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464"
version = "1.0.0"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.DefineSingletons]]
git-tree-sha1 = "0fba8b706d0178b4dc7fd44a96a92382c9065c2c"
uuid = "244e2a9f-e319-4986-a169-4d1fe445cd52"
version = "0.1.2"

[[deps.DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[deps.DiffResults]]
deps = ["StaticArraysCore"]
git-tree-sha1 = "782dd5f4561f5d267313f23853baaaa4c52ea621"
uuid = "163ba53b-c6d8-5494-b064-1a9d43ac40c5"
version = "1.1.0"

[[deps.DiffRules]]
deps = ["IrrationalConstants", "LogExpFunctions", "NaNMath", "Random", "SpecialFunctions"]
git-tree-sha1 = "a4ad7ef19d2cdc2eff57abbbe68032b1cd0bd8f8"
uuid = "b552c78f-8df3-52c6-915a-8e097449b14b"
version = "1.13.0"

[[deps.Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[deps.DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "2fb1e02f2b635d0845df5d7c167fec4dd739b00d"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.9.3"

[[deps.Downloads]]
deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[deps.DrWatson]]
deps = ["Dates", "FileIO", "JLD2", "LibGit2", "MacroTools", "Pkg", "Random", "Requires", "Scratch", "UnPack"]
git-tree-sha1 = "856bd680393b74b05517f8b7b9283fe8c8fd3284"
uuid = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1"
version = "2.12.4"

[[deps.ExprTools]]
git-tree-sha1 = "c1d06d129da9f55715c6c212866f5b1bddc5fa00"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.9"

[[deps.FLoops]]
deps = ["BangBang", "Compat", "FLoopsBase", "InitialValues", "JuliaVariables", "MLStyle", "Serialization", "Setfield", "Transducers"]
git-tree-sha1 = "ffb97765602e3cbe59a0589d237bf07f245a8576"
uuid = "cc61a311-1640-44b5-9fba-1b764f453329"
version = "0.2.1"

[[deps.FLoopsBase]]
deps = ["ContextVariablesX"]
git-tree-sha1 = "656f7a6859be8673bf1f35da5670246b923964f7"
uuid = "b9860ae5-e623-471e-878b-f6a53c775ea6"
version = "0.1.1"

[[deps.FileIO]]
deps = ["Pkg", "Requires", "UUIDs"]
git-tree-sha1 = "7be5f99f7d15578798f338f5433b6c432ea8037b"
uuid = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549"
version = "1.16.0"

[[deps.FileWatching]]
uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"

[[deps.FillArrays]]
deps = ["LinearAlgebra", "Random", "SparseArrays", "Statistics"]
git-tree-sha1 = "7072f1e3e5a8be51d525d64f63d3ec1287ff2790"
uuid = "1a297f60-69ca-5386-bcde-b61e274b549b"
version = "0.13.11"

[[deps.Flux]]
deps = ["Adapt", "CUDA", "ChainRulesCore", "Functors", "LinearAlgebra", "MLUtils", "MacroTools", "NNlib", "NNlibCUDA", "OneHotArrays", "Optimisers", "Preferences", "ProgressLogging", "Random", "Reexport", "SparseArrays", "SpecialFunctions", "Statistics", "StatsBase", "Zygote", "cuDNN"]
git-tree-sha1 = "e657a9aad824de4211606f113edd0b50d5e1f6db"
uuid = "587475ba-b771-5e3f-ad9e-33799f191a9c"
version = "0.13.14"

[[deps.FoldsThreads]]
deps = ["Accessors", "FunctionWrappers", "InitialValues", "SplittablesBase", "Transducers"]
git-tree-sha1 = "eb8e1989b9028f7e0985b4268dabe94682249025"
uuid = "9c68100b-dfe1-47cf-94c8-95104e173443"
version = "0.1.1"

[[deps.ForwardDiff]]
deps = ["CommonSubexpressions", "DiffResults", "DiffRules", "LinearAlgebra", "LogExpFunctions", "NaNMath", "Preferences", "Printf", "Random", "SpecialFunctions", "StaticArrays"]
git-tree-sha1 = "00e252f4d706b3d55a8863432e742bf5717b498d"
uuid = "f6369f11-7733-5829-9624-2563aa707210"
version = "0.10.35"

[[deps.FunctionWrappers]]
git-tree-sha1 = "d62485945ce5ae9c0c48f124a84998d755bae00e"
uuid = "069b7b12-0de2-55c6-9aab-29f3d0a68a2e"
version = "1.1.3"

[[deps.Functors]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "478f8c3145bb91d82c2cf20433e8c1b30df454cc"
uuid = "d9f16b24-f501-4c13-a1f2-28368ffc5196"
version = "0.4.4"

[[deps.Future]]
deps = ["Random"]
uuid = "9fa8497b-333b-5362-9e8d-4d0656e87820"

[[deps.GPUArrays]]
deps = ["Adapt", "GPUArraysCore", "LLVM", "LinearAlgebra", "Printf", "Random", "Reexport", "Serialization", "Statistics"]
git-tree-sha1 = "7a2e790b1e2e6f648cfb25c4500c5de1f7b375ef"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "8.6.5"

[[deps.GPUArraysCore]]
deps = ["Adapt"]
git-tree-sha1 = "1cd7f0af1aa58abc02ea1d872953a97359cb87fa"
uuid = "46192b85-c4d5-4398-a991-12ede77f4527"
version = "0.1.4"

[[deps.GPUCompiler]]
deps = ["ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "TimerOutputs", "UUIDs"]
git-tree-sha1 = "fd6431121f31fed05a5386ac88b9bb3f97fdfa69"
uuid = "61eb1bfa-7361-4325-ad38-22787b887f55"
version = "0.18.0"

[[deps.IRTools]]
deps = ["InteractiveUtils", "MacroTools", "Test"]
git-tree-sha1 = "0ade27f0c49cebd8db2523c4eeccf779407cf12c"
uuid = "7869d1d1-7146-5819-86e3-90919afe41df"
version = "0.4.9"

[[deps.InitialValues]]
git-tree-sha1 = "4da0f88e9a39111c2fa3add390ab15f3a44f3ca3"
uuid = "22cec73e-a1b8-11e9-2c92-598750a2cf9c"
version = "0.3.1"

[[deps.InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[deps.InverseFunctions]]
deps = ["Test"]
git-tree-sha1 = "49510dfcb407e572524ba94aeae2fced1f3feb0f"
uuid = "3587e190-3f89-42d0-90ee-14403ec27112"
version = "0.1.8"

[[deps.IrrationalConstants]]
git-tree-sha1 = "630b497eafcc20001bba38a4651b327dcfc491d2"
uuid = "92d709cd-6900-40b7-9082-c6be49f344b6"
version = "0.2.2"

[[deps.IteratorInterfaceExtensions]]
git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856"
uuid = "82899510-4779-5014-852e-03e436cf321d"
version = "1.0.0"

[[deps.JLD2]]
deps = ["FileIO", "MacroTools", "Mmap", "OrderedCollections", "Pkg", "Printf", "Reexport", "Requires", "TranscodingStreams", "UUIDs"]
git-tree-sha1 = "42c17b18ced77ff0be65957a591d34f4ed57c631"
uuid = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
version = "0.4.31"

[[deps.JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "abc9885a7ca2052a736a600f7fa66209f96506e1"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.4.1"

[[deps.JuliaVariables]]
deps = ["MLStyle", "NameResolution"]
git-tree-sha1 = "49fb3cb53362ddadb4415e9b73926d6b40709e70"
uuid = "b14d175d-62b4-44ba-8fb7-3064adc8c3ec"
version = "0.2.4"

[[deps.KernelAbstractions]]
deps = ["Adapt", "Atomix", "InteractiveUtils", "LinearAlgebra", "MacroTools", "SnoopPrecompile", "SparseArrays", "StaticArrays", "UUIDs", "UnsafeAtomics", "UnsafeAtomicsLLVM"]
git-tree-sha1 = "350a880e80004f4d5d82a17f737d8fcdc56c3462"
uuid = "63c18a36-062a-441e-b654-da1e3ab1ce7c"
version = "0.9.1"

[[deps.LLVM]]
deps = ["CEnum", "LLVMExtra_jll", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "f044a2796a9e18e0531b9b3072b0019a61f264bc"
repo-rev = "v4.17.1"
repo-url = "https://github.com/maleadt/LLVM.jl.git"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "4.17.1"

[[deps.LLVMExtra_jll]]
deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"]
git-tree-sha1 = "070e4b5b65827f82c16ae0916376cb47377aa1b5"
uuid = "dad2f222-ce93-54a1-a47d-0025e8a3acab"
version = "0.0.18+0"

[[deps.LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"

[[deps.LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[deps.LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[deps.LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[deps.LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[deps.Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[deps.LinearAlgebra]]
deps = ["Libdl", "libblastrampoline_jll"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[deps.LogExpFunctions]]
deps = ["ChainRulesCore", "ChangesOfVariables", "DocStringExtensions", "InverseFunctions", "IrrationalConstants", "LinearAlgebra"]
git-tree-sha1 = "0a1b7c2863e44523180fdb3146534e265a91870b"
uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
version = "0.3.23"

[[deps.Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[deps.MLStyle]]
git-tree-sha1 = "bc38dff0548128765760c79eb7388a4b37fae2c8"
uuid = "d8e11817-5142-5d16-987a-aa16d5891078"
version = "0.4.17"

[[deps.MLUtils]]
deps = ["ChainRulesCore", "Compat", "DataAPI", "DelimitedFiles", "FLoops", "FoldsThreads", "NNlib", "Random", "ShowCases", "SimpleTraits", "Statistics", "StatsBase", "Tables", "Transducers"]
git-tree-sha1 = "f69cdbb5b7c630c02481d81d50eac43697084fe0"
uuid = "f1d291b0-491e-4a28-83b9-f70985020b54"
version = "0.4.1"

[[deps.MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "42324d08725e200c23d4dfb549e0d5d89dede2d2"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.10"

[[deps.Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[deps.MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[deps.MicroCollections]]
deps = ["BangBang", "InitialValues", "Setfield"]
git-tree-sha1 = "629afd7d10dbc6935ec59b32daeb33bc4460a42e"
uuid = "128add7d-3638-4c79-886c-908ea0c25c34"
version = "0.1.4"

[[deps.Missings]]
deps = ["DataAPI"]
git-tree-sha1 = "f66bdc5de519e8f8ae43bdc598782d35a25b1272"
uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28"
version = "1.1.0"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[deps.MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[deps.NNlib]]
deps = ["Adapt", "ChainRulesCore", "LinearAlgebra", "Pkg", "Random", "Requires", "Statistics"]
git-tree-sha1 = "33ad5a19dc6730d592d8ce91c14354d758e53b0e"
uuid = "872c559c-99b0-510c-b3b7-b6c96a88d5cd"
version = "0.8.19"

[[deps.NNlibCUDA]]
deps = ["Adapt", "CUDA", "LinearAlgebra", "NNlib", "Random", "Statistics", "cuDNN"]
git-tree-sha1 = "f94a9684394ff0d325cc12b06da7032d8be01aaf"
uuid = "a00861dc-f156-4864-bf3c-e6376f28a68d"
version = "0.2.7"

[[deps.NaNMath]]
deps = ["OpenLibm_jll"]
git-tree-sha1 = "0877504529a3e5c3343c6f8b4c0381e57e4387e4"
uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3"
version = "1.0.2"

[[deps.NameResolution]]
deps = ["PrettyPrint"]
git-tree-sha1 = "1a0fa0e9613f46c9b8c11eee38ebb4f590013c5e"
uuid = "71a1bf82-56d0-4bbc-8a3c-48b961074391"
version = "0.1.5"

[[deps.NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[deps.OneHotArrays]]
deps = ["Adapt", "ChainRulesCore", "Compat", "GPUArraysCore", "LinearAlgebra", "NNlib"]
git-tree-sha1 = "f511fca956ed9e70b80cd3417bb8c2dde4b68644"
uuid = "0b1bfda6-eb8a-41d2-88d8-f5af5cad476f"
version = "0.2.3"

[[deps.OpenBLAS_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"]
uuid = "4536629a-c528-5b80-bd46-f80d51c5b363"

[[deps.OpenLibm_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "05823500-19ac-5b8b-9628-191a04bc5112"

[[deps.OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.5+0"

[[deps.Optimisers]]
deps = ["ChainRulesCore", "Functors", "LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "4b214125921ec010160ddb39931885e0a6585639"
uuid = "3bd65402-5787-11e9-1adc-39752487f4e2"
version = "0.2.17"

[[deps.OrderedCollections]]
git-tree-sha1 = "d78db6df34313deaca15c5c0b9ff562c704fe1ab"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.5.0"

[[deps.Parameters]]
deps = ["OrderedCollections", "UnPack"]
git-tree-sha1 = "34c0e9ad262e5f7fc75b10a9952ca7692cfc5fbe"
uuid = "d96e819e-fc66-5662-9728-84c9c7592b0a"
version = "0.12.3"

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "47e5f437cc0e7ef2ce8406ce1e7e24d44915f88d"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.3.0"

[[deps.PrettyPrint]]
git-tree-sha1 = "632eb4abab3449ab30c5e1afaa874f0b98b586e4"
uuid = "8162dcfd-2161-5ef2-ae6c-7681170c5f98"
version = "0.2.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.ProgressLogging]]
deps = ["Logging", "SHA", "UUIDs"]
git-tree-sha1 = "80d919dee55b9c50e8d9e2da5eeafff3fe58b539"
uuid = "33c8b6b6-d38a-422a-b730-caa89a2f386c"
version = "0.1.4"

[[deps.REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.Random123]]
deps = ["Random", "RandomNumbers"]
git-tree-sha1 = "7a1a306b72cfa60634f03a911405f4e64d1b718b"
uuid = "74087812-796a-5b5d-8853-05524746bad3"
version = "1.6.0"

[[deps.RandomNumbers]]
deps = ["Random", "Requires"]
git-tree-sha1 = "043da614cc7e95c703498a491e2c21f58a2b8111"
uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143"
version = "1.5.3"

[[deps.RealDot]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "9f0a1b71baaf7650f4fa8a1d168c7fb6ee41f0c9"
uuid = "c1ae055f-0cd5-4b69-90a6-9a35b1a98df9"
version = "0.1.0"

[[deps.Reexport]]
git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.2.2"

[[deps.Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.3.0"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[deps.Scratch]]
deps = ["Dates"]
git-tree-sha1 = "30449ee12237627992a99d5e30ae63e4d78cd24a"
uuid = "6c6a2e73-6563-6170-7368-637461726353"
version = "1.2.0"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.Setfield]]
deps = ["ConstructionBase", "Future", "MacroTools", "StaticArraysCore"]
git-tree-sha1 = "e2cc6d8c88613c05e1defb55170bf5ff211fbeac"
uuid = "efcf1570-3423-57d1-acb7-fd33fddbac46"
version = "1.1.1"

[[deps.ShowCases]]
git-tree-sha1 = "7f534ad62ab2bd48591bdeac81994ea8c445e4a5"
uuid = "605ecd9f-84a6-4c9e-81e2-4798472b76a3"
version = "0.1.0"

[[deps.SimpleTraits]]
deps = ["InteractiveUtils", "MacroTools"]
git-tree-sha1 = "5d7e3f4e11935503d3ecaf7186eac40602e7d231"
uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d"
version = "0.9.4"

[[deps.SnoopPrecompile]]
deps = ["Preferences"]
git-tree-sha1 = "e760a70afdcd461cf01a575947738d359234665c"
uuid = "66db9d55-30c0-4569-8b51-7e840670fc0c"
version = "1.0.3"

[[deps.Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[deps.SortingAlgorithms]]
deps = ["DataStructures"]
git-tree-sha1 = "a4ada03f999bd01b3a25dcaa30b2d929fe537e00"
uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c"
version = "1.1.0"

[[deps.SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[deps.SpecialFunctions]]
deps = ["ChainRulesCore", "IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"]
git-tree-sha1 = "ef28127915f4229c971eb43f3fc075dd3fe91880"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "2.2.0"

[[deps.SpinModels]]
deps = ["LinearAlgebra", "NNlib", "Random", "Statistics"]
git-tree-sha1 = "f91cfa2e313470bba46d1b7c5f011ed8e3295dd2"
repo-rev = "main"
repo-url = "https://github.com/wavengampruetikorn/SpinModels.jl"
uuid = "7ff35ac3-3e32-42ea-af81-1555e7609869"
version = "0.1.0"

[[deps.SplittablesBase]]
deps = ["Setfield", "Test"]
git-tree-sha1 = "e08a62abc517eb79667d0a29dc08a3b589516bb5"
uuid = "171d559e-b47b-412a-8079-5efa626c420e"
version = "0.1.15"

[[deps.StaticArrays]]
deps = ["LinearAlgebra", "Random", "StaticArraysCore", "Statistics"]
git-tree-sha1 = "b8d897fe7fa688e93aef573711cb207c08c9e11e"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.5.19"

[[deps.StaticArraysCore]]
git-tree-sha1 = "6b7ba252635a5eff6a0b0664a41ee140a1c9e72a"
uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c"
version = "1.4.0"

[[deps.Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[deps.StatsAPI]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "45a7769a04a3cf80da1c1c7c60caf932e6f4c9f7"
uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0"
version = "1.6.0"

[[deps.StatsBase]]
deps = ["DataAPI", "DataStructures", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"]
git-tree-sha1 = "d1bf48bfcc554a3761a133fe3a9bb01488e06916"
uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
version = "0.33.21"

[[deps.StructArrays]]
deps = ["Adapt", "DataAPI", "GPUArraysCore", "StaticArraysCore", "Tables"]
git-tree-sha1 = "521a0e828e98bb69042fec1809c1b5a680eb7389"
uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a"
version = "0.6.15"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[deps.TableTraits]]
deps = ["IteratorInterfaceExtensions"]
git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39"
uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
version = "1.0.1"

[[deps.Tables]]
deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "OrderedCollections", "TableTraits", "Test"]
git-tree-sha1 = "1544b926975372da01227b382066ab70e574a3ec"
uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
version = "1.10.1"

[[deps.Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[deps.Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[deps.TimerOutputs]]
deps = ["ExprTools", "Printf"]
git-tree-sha1 = "f2fd3f288dfc6f507b0c3a2eb3bac009251e548b"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.22"

[[deps.TranscodingStreams]]
deps = ["Random", "Test"]
git-tree-sha1 = "94f38103c984f89cf77c402f2a68dbd870f8165f"
uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa"
version = "0.9.11"

[[deps.Transducers]]
deps = ["Adapt", "ArgCheck", "BangBang", "Baselet", "CompositionsBase", "DefineSingletons", "Distributed", "InitialValues", "Logging", "Markdown", "MicroCollections", "Requires", "Setfield", "SplittablesBase", "Tables"]
git-tree-sha1 = "c42fa452a60f022e9e087823b47e5a5f8adc53d5"
uuid = "28d57a85-8fef-5791-bfe6-a80928e7c999"
version = "0.4.75"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.UnPack]]
git-tree-sha1 = "387c1f73762231e86e0c9c5443ce3b4a0a9a0c2b"
uuid = "3a884ed6-31ef-47d7-9d2a-63182c4928ed"
version = "1.0.2"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[deps.UnsafeAtomics]]
git-tree-sha1 = "6331ac3440856ea1988316b46045303bef658278"
uuid = "013be700-e6cd-48c3-b4a1-df204f14c38f"
version = "0.2.1"

[[deps.UnsafeAtomicsLLVM]]
deps = ["LLVM", "UnsafeAtomics"]
git-tree-sha1 = "ead6292c02aab389cb29fe64cc9375765ab1e219"
uuid = "d80eeb9a-aca5-4d75-85e5-170c8b632249"
version = "0.1.1"

[[deps.Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[deps.Zygote]]
deps = ["AbstractFFTs", "ChainRules", "ChainRulesCore", "DiffRules", "Distributed", "FillArrays", "ForwardDiff", "GPUArrays", "GPUArraysCore", "IRTools", "InteractiveUtils", "LinearAlgebra", "LogExpFunctions", "MacroTools", "NaNMath", "Random", "Requires", "SnoopPrecompile", "SparseArrays", "SpecialFunctions", "Statistics", "ZygoteRules"]
git-tree-sha1 = "4df8f470806a45a8630ac8f597304821dc8e8838"
uuid = "e88e6eb3-aa80-5325-afca-941959d7151f"
version = "0.6.59"

[[deps.ZygoteRules]]
deps = ["ChainRulesCore", "MacroTools"]
git-tree-sha1 = "977aed5d006b840e2e40c0b48984f7463109046d"
uuid = "700de1a5-db45-46bc-99cf-38207098b444"
version = "0.2.3"

[[deps.cuDNN]]
deps = ["CEnum", "CUDA", "CUDNN_jll"]
git-tree-sha1 = "3aa15aba7aad5be8b9b3c1b77a9b81e3e1357280"
uuid = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"
version = "1.0.2"

[[deps.libblastrampoline_jll]]
deps = ["Artifacts", "Libdl", "OpenBLAS_jll"]
uuid = "8e850b90-86db-534c-a0d3-1478176c7d93"

[[deps.nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[deps.p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"

Expected behavior

With CUDA@3.12.0, it was able to discover all local installtionas of cuda and cudnn and the MWE above would work fine. The cuda runtime version is 11.5 and cudnn is 8.3.1.

Version info

Details on Julia:

Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, cascadelake)

Details on CUDA:

CUDA runtime 11.5, local installation
CUDA driver 12.1
NVIDIA driver 470.82.1, originally for CUDA 11.5

Libraries: 
- CUBLAS: 11.7.3
- CURAND: 10.2.6
- CUFFT: 10.6.0
- CUSOLVER: 11.2.1
- CUSPARSE: 11.7.0
- CUPTI: 16.0.0
- NVML: 11.0.0+470.82.1

Toolchain:
- Julia: 1.7.3
- LLVM: 12.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

1 device:
  0: Tesla V100-PCIE-16GB (sm_70, 15.448 GiB / 15.782 GiB available)

Additional context
here is the MWE with JULIA_DEBUG=all

julia> using CUDA
┌ Debug: Using LLVM 12.0.1 at /software/julia-1.7.3-el8-x86_64/lib/julia/libLLVM-12jl.so
└ @ LLVM ~/.julia/packages/LLVM/HykgZ/src/LLVM.jl:86
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.5.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/qOVLJ/src/wrappers/x86_64-linux-gnu.jl:87
┌ Debug: Forward-compatible CUDA driver found at /home/pfields/.julia/artifacts/7a7fd08bbad6b42e7ed42fd2fc058e42039b075f/lib/libcuda.so; known to be version 12.1.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/qOVLJ/src/wrappers/x86_64-linux-gnu.jl:138
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/qOVLJ/src/wrappers/x86_64-linux-gnu.jl:167
┌ Debug: Requires conditionally ran code in 0.003001692 seconds: `RandomNumbers` detected `Random123`
└ @ Requires ~/.julia/packages/RandomNumbers/3pD1N/src/RandomNumbers.jl:38
┌ Debug: Requires conditionally ran code in 0.007870579 seconds: `Adapt` detected `StaticArrays`
└ @ Requires ~/.julia/packages/Adapt/xviDc/src/Adapt.jl:76
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME, CUDA_ROOT
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:286
┌ Debug: Looking for binary ptxas in /software/cuda-11.5-el8-x86_64
│   all_locations =
│    2-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/bin"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:156
┌ Debug: Found ptxas at /software/cuda-11.5-el8-x86_64/bin/ptxas
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:162
┌ Debug: Looking for binary nvdisasm in /software/cuda-11.5-el8-x86_64
│   all_locations =
│    2-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/bin"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:156
┌ Debug: Found nvdisasm at /software/cuda-11.5-el8-x86_64/bin/nvdisasm
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:162
┌ Debug: Looking for binary nvlink in /software/cuda-11.5-el8-x86_64
│   all_locations =
│    2-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/bin"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:156
┌ Debug: Found nvlink at /software/cuda-11.5-el8-x86_64/bin/nvlink
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:162
┌ Debug: Looking for binary compute-sanitizer in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_locations =
│    6-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/bin"
│     "/software/cuda-11.5-el8-x86_64/extras/compute-sanitizer"
│     "/software/cuda-11.5-el8-x86_64/extras/compute-sanitizer/bin"
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer"
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/bin"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:156
┌ Debug: Found compute-sanitizer at /software/cuda-11.5-el8-x86_64/bin/compute-sanitizer
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:162
┌ Debug: Looking for library cudart, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libcudart.so"
│   all_locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libcudart.so at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Looking for library nvvm, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libnvvm.so"
│   all_locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Did not find nvvm
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:131
┌ Debug: Looking for library cufft, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libcufft.so"
│   all_locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libcufft.so at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Looking for library cublas, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libcublas.so"
│   all_locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libcublas.so at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Looking for library cusparse, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libcusparse.so"
│   all_locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libcusparse.so at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Looking for library cusolver, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libcusolver.so"
│   all_locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libcusolver.so at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Looking for library cusolverMg, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libcusolverMg.so"
│   all_locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libcusolverMg.so at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Looking for library curand, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libcurand.so"
│   all_locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libcurand.so at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Looking for library cupti, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer or /software/cuda-11.5-el8-x86_64/extras/CUPTI
│   all_names =
│    1-element Vector{String}:
│     "libcupti.so"
│   all_locations =
│    16-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/extras/CUPTI/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libcupti.so at /software/cuda-11.5-el8-x86_64/extras/CUPTI/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Looking for library nvToolsExt, no specific version, in /software/cuda-11.5-el8-x86_64 or /software/cuda-11.5-el8-x86_64/extras/compute-sanitizer or /software/cuda-11.5-el8-x86_64/compute-sanitizer or /software/cuda-11.5-el8-x86_64/extras/CUPTI
│   all_names =
│    1-element Vector{String}:
│     "libnvToolsExt.so"
│   all_locations =
│    16-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/extras/CUPTI/libx64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:128
┌ Debug: Found libnvToolsExt.so at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:137
┌ Debug: Request to look for libcudadevrt 
│   locations =
│    4-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/extras/compute-sanitizer"
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer"
│     "/software/cuda-11.5-el8-x86_64/extras/CUPTI"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:390
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│   locations =
│    12-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/lib"
│     "/software/cuda-11.5-el8-x86_64/lib64"
│     "/software/cuda-11.5-el8-x86_64/extras/compute-sanitizer"
│     ⋮
│     "/software/cuda-11.5-el8-x86_64/extras/CUPTI"
│     "/software/cuda-11.5-el8-x86_64/extras/CUPTI/lib"
│     "/software/cuda-11.5-el8-x86_64/extras/CUPTI/lib64"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:419
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /software/cuda-11.5-el8-x86_64/lib64
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:426
┌ Debug: Request to look for libdevice
│   locations =
│    4-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/extras/compute-sanitizer"
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer"
│     "/software/cuda-11.5-el8-x86_64/extras/CUPTI"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:357
┌ Debug: Look for libdevice
│   locations =
│    4-element Vector{String}:
│     "/software/cuda-11.5-el8-x86_64"
│     "/software/cuda-11.5-el8-x86_64/nvvm/libdevice"
│     "/software/cuda-11.5-el8-x86_64/compute-sanitizer"
│     "/software/cuda-11.5-el8-x86_64/extras/CUPTI"
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:369
┌ Debug: Found unified device library at /software/cuda-11.5-el8-x86_64/nvvm/libdevice/libdevice.10.bc
└ @ CUDA_Runtime_Discovery ~/.julia/packages/CUDA_Runtime_Discovery/tnsVx/src/CUDA_Runtime_Discovery.jl:374

julia> using Flux
┌ Debug: Requires conditionally ran code in 0.012140274 seconds: `BangBang.NoBang` detected `StaticArrays`
└ @ Requires ~/.julia/packages/BangBang/90Cfb/src/NoBang/NoBang.jl:24
┌ Debug: Requires conditionally ran code in 0.006795824 seconds: `BangBang` detected `StaticArrays`
└ @ Requires ~/.julia/packages/BangBang/90Cfb/src/BangBang.jl:79
┌ Debug: Requires conditionally ran code in 0.009443282 seconds: `Accessors` detected `StaticArrays`
└ @ Requires ~/.julia/packages/Accessors/VsgVs/src/Accessors.jl:20
┌ Debug: Requires conditionally ran code in 0.004939166 seconds: `BangBang` detected `StructArrays`
└ @ Requires ~/.julia/packages/BangBang/90Cfb/src/BangBang.jl:82
┌ Error: cuDNN is not available for your platform (x86_64-linux-gnu-libgfortran4-cxx11-libstdcxx29-cuda+local-julia_version+1.7.3)
└ @ cuDNN ~/.julia/packages/cuDNN/tgAWl/src/cuDNN.jl:167
┌ Debug: Requires conditionally ran code in 0.052461289 seconds: `CUDA` detected `SpecialFunctions`
└ @ Requires ~/.julia/packages/CUDA/N71Iw/src/initialization.jl:130
┌ Debug: Requires conditionally ran code in 0.003496896 seconds: `NNlib` detected `ForwardDiff`
└ @ Requires ~/.julia/packages/NNlib/ydqxJ/src/NNlib.jl:93
┌ Debug: `init_primary_task`: Initializing `PRIMARY_TASK` channel...
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:24
┌ Debug: `init_primary_task`: Initializing `PRIMARY_TASK` channel... DONE
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:36
┌ Debug: `init_each_thread`: Initializing `EACH_THREAD` channels...
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:57
┌ Debug: `init_each_thread`: Initializing `EACH_THREAD` channels... DONE
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:68
┌ Debug: `init_taskpool`: Initializing `TASKPOOL_REQUEST` channels...
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/taskpool.jl:85
┌ Debug: Processing request run_future
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:7
┌ Debug: `init_taskpool`: Initializing `TASKPOOL_REQUEST` channels... DONE
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/taskpool.jl:88
┌ Debug: `init_background_taskpool`: Initializing `BACKGROUND_TASKPOOL_REQUEST` channels...
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/taskpool.jl:92
┌ Debug: Processing request run_future
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:7
┌ Debug: `init_background_taskpool`: Initializing `BACKGROUND_TASKPOOL_REQUEST` channels... DONE
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/taskpool.jl:95

julia> x =cu(Float32.((rand(2,2))))
2×2 CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}:
 0.758133   0.482096
 0.0829441  0.657127

julia> Flux.sigmoid.(x)
ERROR: UndefVarError: libcudnn not defined
Stacktrace:
  [1] macro expansion
    @ ~/.julia/packages/cuDNN/tgAWl/src/libcudnn.jl:694 [inlined]
  [2] macro expansion
    @ ~/.julia/packages/CUDA/N71Iw/src/pool.jl:312 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/cuDNN/tgAWl/src/libcudnn.jl:22 [inlined]
  [4] cudnnCreateActivationDescriptor(activationDesc::Vector{Ptr{cuDNN.cudnnActivationStruct}})
    @ cuDNN ~/.julia/packages/CUDA/N71Iw/lib/utils/call.jl:26
  [5] cuDNN.cudnnActivationDescriptor(::cuDNN.cudnnActivationMode_t, ::Vararg{Any})
    @ cuDNN ~/.julia/packages/cuDNN/tgAWl/src/descriptors.jl:39
  [6] #cudnnActivationForward!#1187
    @ ~/.julia/packages/cuDNN/tgAWl/src/activation.jl:22 [inlined]
  [7] #55
    @ ~/.julia/packages/NNlibCUDA/C6t0p/src/cudnn/activations.jl:12 [inlined]
  [8] materialize(bc::Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{2}, Nothing, typeof(σ), Tuple{CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}}})
    @ NNlibCUDA ~/.julia/packages/NNlibCUDA/C6t0p/src/cudnn/activations.jl:31
  [9] top-level scope
    @ REPL[5]:1
 [10] top-level scope
    @ ~/.julia/packages/CUDA/N71Iw/src/initialization.jl:163
@peter-fields peter-fields added the bug Something isn't working label Mar 31, 2023
@mtfishman
Copy link

We see a similar issue with CUDA.jl v4.1 not being able to find the cuTENSOR library, as discussed in ITensor/ITensors.jl#1107. At least for cuTENSOR, the issue seems to be that CUDA.jl v4.1 bumps support to CUDA 12.1 but cuTENSOR doesn't support CUDA 12.1 yet.

@mtfishman mtfishman mentioned this issue Apr 3, 2023
Merged
@maleadt
Copy link
Member

maleadt commented Apr 4, 2023

We see a similar issue with CUDA.jl v4.1 not being able to find the cuTENSOR library, as discussed in ITensor/ITensors.jl#1107. At least for cuTENSOR, the issue seems to be that CUDA.jl v4.1 bumps support to CUDA 12.1 but cuTENSOR doesn't support CUDA 12.1 yet.

That's a different issue. You 'just' need to downgrade the CUDA runtime version to one that CUTENSOR supports. Importing CUTENSOR.jl should warn about that (or at least mention that the library isn't available for your platform).

@maleadt
Copy link
Member

maleadt commented Apr 4, 2023

There is no internet access when I mount a gpu via slurm

As a workaround, it should be possible to download the artifacts on the login node by specifying the CUDA runtime version to use on the compute nodes. The documentation has some details on that, as it's similar to how containers work: https://cuda.juliagpu.org/stable/installation/overview/#Containers

@maleadt maleadt added the installation CUDA is easy to install, right? label Apr 4, 2023
@maleadt maleadt changed the title CUDA.jl 4 no longer discovers local installation of cudnn Local runtime discovery does not work for external libraries (CUDNN, CUTENSOR) Apr 4, 2023
@maleadt
Copy link
Member

maleadt commented Apr 4, 2023

If anybody wants to help fixing this, it would involve adding (optional) discovery of sublibraries to https://github.com/JuliaGPU/CUDA_Runtime_Discovery.jl, and use that JLL conditionally just like it's done now in CUDA.jl proper:

CUDA.jl/src/CUDA.jl

Lines 23 to 31 in 5c51766

import CUDA_Runtime_jll
if haskey(CUDA_Runtime_jll.preferences, "version") &&
CUDA_Runtime_jll.preferences["version"] == "local"
using CUDA_Runtime_Discovery
const CUDA_Runtime = CUDA_Runtime_Discovery
else
using CUDA_Runtime_jll
const CUDA_Runtime = CUDA_Runtime_jll
end

@mtfishman
Copy link

We see a similar issue with CUDA.jl v4.1 not being able to find the cuTENSOR library, as discussed in ITensor/ITensors.jl#1107. At least for cuTENSOR, the issue seems to be that CUDA.jl v4.1 bumps support to CUDA 12.1 but cuTENSOR doesn't support CUDA 12.1 yet.

That's a different issue. You 'just' need to downgrade the CUDA runtime version to one that CUTENSOR supports. Importing CUTENSOR.jl should warn about that (or at least mention that the library isn't available for your platform).

Indeed, this is the error that was reported:

Error: cuTENSOR is not available for your platform (x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-cuda+12.1-julia_version+1.8.5)
└ @ cuTENSOR ~/.julia/packages/cuTENSOR/WTpqy/src/cuTENSOR.jl:99
[...]

It's a bit opaque since it doesn't specifically say the issue is the CUDA version so maybe that could be improved a bit, if possible (say by reporting the closest platform that is supported).

I guess I was hoping this could get handled in some way by the Julia package manager (for example by setting an upper bound on the compat of CUDA in cuTENSOR.jl), but I see that probably isn't possible right now since the version of CUDA is set by whatever is already installed on the system and isn't controlled by CUDA.jl.

@maleadt
Copy link
Member

maleadt commented Apr 4, 2023

It's a bit opaque since it doesn't specifically say the issue is the CUDA version so maybe that could be improved a bit, if possible (say by reporting the closest platform that is supported).

Yes, I agree. The whole transformation to JLLs is fairly new, so some issues are expected (hence this was a breaking release).

I guess I was hoping this could get handled in some way by the Julia package manager (for example by setting an upper bound on the compat of CUDA in cuTENSOR.jl)

Sadly, that's not possible. This was explicitly decided against by the Pkg developers because it would make package resolution a much harder problem.

@mloubout
Copy link
Contributor

mloubout commented Apr 5, 2023

I am running into the same issue. I have CUDA setup through the nvidia HPC SDK and cudnn as a separate one. All the path are setup properly but afterI set CUDA.set_runtime_version!("local") it fails to find cuDNN. Is there a "quick" workaround to force it to use my system libs because it prevents me from running any ML right now.

A separate note, and not sure if it is related, even with "local" runtime version, CUDA seems to still load libcuda.so from an artifact (debug says forward-compatible CUDA driver) from CUDA_driver_jll. From the documentation I though setting the config to "local" should be preventing CUDA to load any jll and only use the local install.

@mloubout mloubout mentioned this issue Apr 5, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working installation CUDA is easy to install, right?
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support runtime discovery for non-toolkit libraries (CUTENSOR, CUDNN, CUQUANTUM) mloubout/CUDA.jl
4 participants
@maleadt @mloubout @mtfishman @peter-fields