-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CUSPARSE is broken in CUDA.jl 1.2 #322
Comments
Can't reproduce on master/1.2 with Julia 1.5 or 1.4. Can you verify in a clean environment? |
I'm so sorry @maleadt , obviously |
can confirm that this is broken and it includes using CUDA
using CUDA.CUSPARSE
using SparseArrays
a = rand(10,10); a[a.<0.9] .= 0
a = CuSparseMatrixCSC(sparse(a));
b = CUDA.rand(10);
c=a*b results in This is with a CUDA.jl 1.2, CUDA 9.2, Julia 1.4.2. The Julia environment was fresh but I installed all packages that I needed for my current project (Flux + DiffEqFlux + some small utilities) alongside CUDA.jl. I use the pre-installed CUDA version on the HPC as the compute nodes have no internet connection for downloading artifacts. |
Looks a lot like #256 |
I don't think this has ever worked. The reason it returned nothing is because of badly-attached docstrings, but the underlying cause is that you're performing a mixed Fkoat64 x Float32 multiplication that's not supported by CUSPARSE. Furthermore CUSPARSE seems to only support matrix-vector multiplication on the BCSR format, so this works:
I agree that this isn't particularly user-friendly though. |
Oh, you are right. I reduced my problem to a wrong, hastily written minimal example it seems. In this case it really was the Float32, Float64 mix up. (a side note: is it not possible to output a warning if |
@maleadt : From where do you take:
I roughly observed the following behaviour when porting my current project from I also do not want to derail from the problem that @michel2323 has posted, but I think this might be very similar if not the same issue. edit: MWE: Julia 1.3, CuArrays.jl (v.1.7.3), CUDA 9.2 (pre-installed on the HPC I am using) using CuArrays
using CuArrays.CUSPARSE
using SparseArrays
a = rand(Float32, 10,10); a[a.<0.9] .= 0
a = CuSparseMatrixCSC(sparse(a));
b = rand(Float32,10);
b = CuArray(b)
a*b # == something Julia 1.4.2, CUDA.jl (1.2), CUDA 9.2 (pre-installed on the HPC I am using) using CUDA
using CUDA.CUSPARSE
using SparseArrays
a = rand(Float32, 10,10); a[a.<0.9] .= 0
a = CuSparseMatrixCSC(sparse(a));
b = rand(Float32,10);
b = CuArray(b)
a*b # == nothing |
@maleadt why is this closed? I don't think it is fixed. Personally, I have code that working just fine with the old versions that is not anymore and @michel2323 problem doesn't seem to be solved either. care to explain it? |
Sorry, I didn't see your comment with additional code before closing it. |
Found the culprit: |
Although I plan to look at this, I don't think I'll have it block the next release (since it's currently broken already, and there's other issues that are fixed already on the master branch). |
Julia 1.4.2 CUDA v1.2.1 CUDA Version: 10.2 using CUDA
using CUDA.CUSPARSE
using SparseArrays
a = rand(Float32, 10,10); a[a.<0.9] .= 0
a = CuSparseMatrixCSC(sparse(a));
b = rand(Float32,10,3);
b = CuArray(b)
a*b
# ERROR: UndefVarError: cusparseScsrmm2 not defined While using CUDA
using CUDA.CUSPARSE
using SparseArrays
a = rand(Float64, 10,10); a[a.<0.9] .= 0
a = CuSparseMatrixCSC(sparse(a));
b = rand(Float32,10,3);
b = CuArray(b)
a*b
# 10×3 CuArray{Float64,2}: |
Another issues is that sparse matrix * matrix works, but sparse matrix * vector does not: using CUDA
using CUDA.CUSPARSE
using SparseArrays
a = rand(Float64, 10,10); a[a.<0.9] .= 0
a = CuSparseMatrixCSC(sparse(a));
b = rand(Float32,10,3);
b = CuArray(b)
a*b # works
a*b[:,1] # returns nothing |
With #351, some cases work again already but CSC support is still missing. |
First, thank you for Although #351 looks like it fixed this issue, if I run the script above using CUDA
using CUDA.CUSPARSE
using SparseArrays
a = rand(Float64, 10,10); a[a.<0.9] .= 0
a = CuSparseMatrixCSC(sparse(a));
b = rand(Float32,10,3);
b = CuArray(b)
a*b # causes scalar operation warning
a*b[:,1] # errors I am getting: ┌ Warning: Performing scalar operations on GPU arrays: This is very slow, consider disallowing these operations with `allowscalar(false)`
└ @ GPUArrays ~/.julia/packages/GPUArrays/eVYIC/src/host/indexing.jl:43
ERROR: LoadError: MethodError: no method matching mv!(::Char, ::Float64, ::CuSparseMatrixCSC{Float64}, ::CuArray{Float32,1}, ::Float64, ::CuArray{Float64,1}, ::Char)
Closest candidates are:
mv!(::Char, ::Float64, ::CuSparseMatrixBSR{Float64}, ::CuArray{Float64,1}, ::Float64, ::CuArray{Float64,1}, ::Char) at /home/niklas/.julia/packages/CUDA/dZvbp/lib/cusparse/wrappers.jl:186
mv!(::Char, ::Float32, ::CuSparseMatrixBSR{Float32}, ::CuArray{Float32,1}, ::Float32, ::CuArray{Float32,1}, ::Char) at /home/niklas/.julia/packages/CUDA/dZvbp/lib/cusparse/wrappers.jl:186
mv!(::Char, ::Complex{Float64}, ::CuSparseMatrixBSR{Complex{Float64}}, ::CuArray{Complex{Float64},1}, ::Complex{Float64}, ::CuArray{Complex{Float64},1}, ::Char) at /home/niklas/.julia/packages/CUDA/dZvbp/lib/cusparse/wrappers.jl:186
...
Stacktrace:
[1] mul!(::CuArray{Float64,1}, ::CuSparseMatrixCSC{Float64}, ::CuArray{Float32,1}) at /home/niklas/.julia/packages/CUDA/dZvbp/lib/cusparse/interfaces.jl:12
[2] *(::CuSparseMatrixCSC{Float64}, ::CuArray{Float32,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/matmul.jl:51
[3] top-level scope at /home/niklas/repos/SpatialEchoStateNetwork/test.jl:11
[4] include(::String) at ./client.jl:457
[5] top-level scope at REPL[1]:1
in expression starting at /home/niklas/repos/SpatialEchoStateNetwork/test.jl:11 I am on CUDA.jl v1.3.3 and julia 1.5: (SpatialEchoStateNetwork) pkg> st
Project SpatialEchoStateNetwork v0.1.0
Status `~/repos/SpatialEchoStateNetwork/Project.toml`
[7d9fca2a] Arpack v0.4.0
[052768ef] CUDA v1.3.3
[d9f16b24] Functors v0.1.0
[37e2e46d] LinearAlgebra
[2f01184e] SparseArrays |
This has not been merged in 1.3.x, since it drops support for CUDA 10.1 and earlier. |
#409 should fix most of the issues reported here, please test. Do note that you need to exactly match up array types, many of the scalar iterations reported here were due to combining e.g. a Float32 and Float64 array. We could improve that, of course, but let's just get the missing functionality back first. |
I keep coming back to this, but I still have several problems with CUSPARSE when migrating from the old CuArrays.jl to CUDA.jl . Things that used to work don't work anymore. With Julia 1.4.2 and CUDA.jl 1.3.3, I get for using CUDA
using CUDA.CUSPARSE
using SparseArrays
a = rand(Float64, 10,10); a[a.<0.9] .= 0
a = CuSparseMatrixCSC(sparse(a));
b = rand(Float64,10,3);
b = CuArray(b);
a*b the following error
There are some other problems as well but I haven't had the time yet to sit down and find a proper MWE. |
I believe the above fix is only in CUDA.jl 2.0. Try upgrading to the latest version (you may need Julia 1.5 as well). |
Thanks for the tip. Unfortunately Flux.jl restricts me from updating to CUDA.jl 2.0. |
*
,mult!
and possibly other operations are broken with::CuSparseMatrixCSR
typeof(A*v)
returnsNothing
andmul!(w, A, v)
returns all zeros inw
with v1.2, whereas it returns the correct vector with v1.1. Maybe it's linked to #259 . I tested with CUDA 10.2 and CUDA 11.Details on CUDA:
The text was updated successfully, but these errors were encountered: