CUDA: fix --split-mode row race condition

[JohannesGaessler]

Fixes #8801 (comment) .

The problem is that --split-mode row uses multiple CUDA streams to overlap data transfer with computation but on CUDA GPUs that are Volta or newer each of these streams allocates a temporary buffer for the stream-k fixup. And because it's not safe to allocate temporary buffers with multiple CUDA streams in parallel this leads to a race condition. However, because both stream-k decomposition and --split-mode row mitigates tail effects it's fine to just not use stream-k if multiple parallel CUDA streams are in use; the performance difference with 3x RTX 4090 is ~1%.

Long-term it may make sense to think about how we can safely handle multiple parallel CUDA streams. I think a good approach would be to move the parallelism from something CUDA-specific to something at the level of GGML compute graphs. That would also enable the reuse of the code for other backends.

[slaren]

You could use a different pool per stream. Since the pools are objects now, it should be a simple change. I thought about moving the implementation of tensor parallelism to ggml_backend_sched, but it is a lot of work for no clear benefit outside of datacenter GPUs with very fast interconnects, and I think it will make backend-specific optimizations harder to implement.