General purpose broadcast for sparse CSR matrices.

[maleadt]

Adapts the implementation from last week, adding general purpose broadcast support (i.e. involving multiple sparse inputs, dense arrays, scalars, etc). The trick here is to have a separate kernel count the nnz values per row so that we can allocate an output container that will fit all values, and use those values as row offsets (after scanning them). The only disadvantage of that approach is that it's synchronizing, to access the total nnz count, but performance seems pretty good nonetheless:

julia> using BenchmarkTools

julia> using SparseArrays, CUDA, CUDA.CUSPARSE

julia> cx = sprand(Float32, 1024, 1024, 0.1);

julia> cy = sprand(Float32, size(cx)..., 0.1);

julia> @benchmark cx .+ cy
BenchmarkTools.Trial: 4707 samples with 1 evaluation.
 Range (min … max):  959.167 μs …   3.586 ms  ┊ GC (min … max): 0.00% … 53.32%
 Time  (median):       1.007 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):     1.060 ms ± 288.209 μs  ┊ GC (mean ± σ):  5.39% ± 11.41%

  █▆█▄▂                                                         ▁
  █████▃▁▁▃▃▁▄▅▃▁▁▁▁▁▁▁▁▁▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▄▆▆▆▆▆▆█▇▇▇▆▆▆ █
  959 μs        Histogram: log(frequency) by time       2.54 ms <

 Memory estimate: 2.40 MiB, allocs estimate: 8.

julia> x = CuSparseMatrixCSR(cx);

julia> y = CuSparseMatrixCSR(cy);

julia> @benchmark CUDA.@sync x .+ y
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  152.027 μs …  18.858 ms  ┊ GC (min … max): 0.00% … 46.43%
 Time  (median):     160.127 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   184.608 μs ± 320.993 μs  ┊ GC (mean ± σ):  0.94% ±  0.66%

  █▅▁                                                           ▁
  █████▅▃▁▄▄▁▁▁▁▁▁▄▁▃▁▁▄▃▃▄▁▄▄▄▁▁▃▃▃▅▅▆▆▅▆▅▅▅▅▁▄▄▄▄▄▄▄▄▁▁▄▄▆▅▆▅ █
  152 μs        Histogram: log(frequency) by time        905 μs <

 Memory estimate: 16.53 KiB, allocs estimate: 293.

Still a WIP, I want to at least add CSC support.
cc @kshyatt @Roger-luo

[codecov]

Codecov Report

Merging #1380 (1c75de7) into master (ad5aa8b) will decrease coverage by 0.38%.
The diff coverage is 41.09%.

@@            Coverage Diff             @@
##           master    #1380      +/-   ##
==========================================
- Coverage   78.03%   77.64%   -0.39%     
==========================================
  Files         121      121              
  Lines        8942     8966      +24     
==========================================
- Hits         6978     6962      -16     
- Misses       1964     2004      +40     

Impacted Files	Coverage Δ
lib/cusparse/CUSPARSE.jl	77.27% <ø> (ø)
lib/cusparse/broadcast.jl	42.50% <41.09%> (-14.32%)	⬇️
lib/cusparse/array.jl	66.66% <0.00%> (-0.42%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad5aa8b...1c75de7. Read the comment docs.