[QST] CPU vs GPU Time Comparison

[mtsol]

Hi, I plugged in spark 3.1 with rapids, and everything setup.
GPU: Tesla T4 16gb

the overall spark flow is giving less completion time on cpu as compared to gpu, I want to know what I am doing wrong because gpu is supposed to be faster.

[revans2]

The GPU is not a silver bullet when it comes to performance. There are so many reasons why the CPU could be faster than the GPU when running a query. Often this can be fixed with proper tuning. https://nvidia.github.io/spark-rapids/docs/FAQ.html#why-is-my-query-in-gpu-mode-slower-than-cpu-mode can help you start to look/debug why it might be slower. But there are some situations where the GPU inherently will not be faster than the CPU.

1. If the size of the data is so small that the overhead of launching kernels is more than actually doing the computation. This also includes individual tasks being very small. If tasks are too small you can often tune things and fix it. If the entire input data is too small then it is hard to fix that.
2. The processing is very cache coherent. The GPU is fast, but if the bottleneck is memory access, not actual computation and the processing fits in the CPU's cache, then it is going to be very difficult for the GPU to beat the CPU in those cases.
3. There are operations that process a lot of data and cannot be done on the GPU yet. This includes many UDFs. If the plugin encounters operations that it does not know about yet it will fall back to doing them on the CPU. But to do this we often need to move data from the GPU to the CPU. This can be expensive to do and often thrashes the CPU cache slowing down the processing on the CPU.
4. You are I/O bound and not computation bound. If you have slow networking on slow disks it is possible that the GPU is not able to speed anything up because all of the time is spent doing I/O and not actually doing computation.

It is also possible that you found a performance bug where there is an operator that we have implemented badly for some cases. If you could share with us details about the query and your data we can probably help you debug why it is slower. You can also look at the Spark UI and compare a CPU run to a GPU run. You can try to match up stages and see which ones are taking longer on the GPU vs the CPU to debug what is happening. We do have a few tools that can help with some of this, https://nvidia.github.io/spark-rapids/docs/spark-profiling-tool.html but it is not perfect.

[mtsol]

This happened in a way that, the code was actually faster on GTX 1660 super as compared to cpu; but when the code was taken to Tesla T4, it was slower as compared to cpu. We have a lot of udf operations going on in a loop.