[QST]some question about nsight system

[chenrui17]

What is your question?

my test data is gzip compress ，picture shows 2 stream executed on gpu , as you see, ptds is on , and my first 2 question is :

1. as red block, there are 2 times memory copy from host to device ,between is inflate_kernel i guess is decompression file , what's differences of 2 times memory copy ? and at the same time ,the first copy is pageable ,and the second is pinned , and after the second copy , there are gpuBuildStringDictionaryIndex and gpuDecodePageData and I guess this is decode parquet file ，and why the second copy size every time is 8bytes ,and looks like is Inefficiency .
2. as yellow block, there is no gpu action , and now Where is the bottleneck ,Why does this happen and how to optimize it

my last question is my nsight system looks like abnormal ，java tag is empty ,as shown in the figure below and what i expect is operators nvtx like " buffer file split " ,"Parquet readBatch Acquire GPU "

my nsys trace way is
first , i stop a worker process of node A, and modify spark-class file ,replaced "exec "${CMD[@]}" with "exec nsys profile --trace-fork-before-exec=true --trace=cuda,openacc,nvtx "${CMD[@]}"
second , i restart a worker process by "./start-slave.sh spark://master:7077"
finally, i run spark-shell and run a query,
after that , i go to node A ,and kill the woker process which is doesn't launched by "nsys profile -"

[jlowe]

Thanks for the traces, @chenrui17!

For the first question, this appears to be within libcudf's Parquet decoder, as all the behavior in question occurs within that read_parquet sample within libcudf. If you have a small repro case, it would be great to file a question in the https://github.com/rapidsai/cudf project, or if you attach it here I can do the same.

If you have configured a sufficient pinned memory pool for the SQL plugin, the buffer being handed to the libcudf parquet decoder will be pinned memory. Assuming you have configured enough pinned memory to hold the parquet data across the concurrent tasks, pageable transfers may be from CPU buffers computed within the libcudf call.

For the second question about missing Java traces, cudf Java does not generate NVTX traces by default and will only do so if the system property ai.rapids.cudf.nvtx.enabled=true. The NVTX profiling guide details how to setup this property and how to setup profiling in general. It sounds like your general approach to profiling works, and I suspect the system property isn't being set which would explain how you're getting libcudf traces but not Java traces.