Decreasing cpu bottleneck when running on gpu

[NoaMills-USDA]

I am running code that is very similar to the quickstart code, but with 1000 training images from spacenet. I have created a singularity image out of the docker image so I can run raster vision on a gpu node of an hpc. My code runs very slowly, and when I use nvtop to look at the utilization, I see that the cpu utilization is at 100%, and the gpu utilization is at 0%. I saw in the documentation here that it is recommended to set a root temporary directory to avoid re-downloading files. Is this a common fix to cpu bottlenecks? Should the root temporary directory point to a path in the container?

[AdeelH]

Is the GPU even being detected? What does the Using device: line in the logs say?

The doc page you linked is talking specifically about multi-GPU environments, so shouldn't be relevant here. But to answer the question: yes, the path should be a path accessible from inside the container.

[NoaMills-USDA]

It says Using devide: cuda. Any other reasons why my code might be cpu-bound?

[NoaMills-USDA]

I have some additional information. I'm running my code in an apptainer (formerly singularity) container on a compute cluster. If I run my code on a gpu mig instance of an nvidia-a100 gpu with 10 gigs of vram, it takes about an hour and a half to run, and the gpu utilization and memory utilization are basically zero. However, if I run my code on a full nvidia-a100 gpu with 80 gigs of ram, it runs in 20 minutes and surprisingly has higher gpu and memory utilization, ranging from 20-80%. I'm wondering why it's not using the gpu memory in the first case, and why it's taking up a large portion of the memory in the second case. My batch size is 2, and I'm using a resnet50 backbone. Is there anything else that could be taking up lots of space and causing it to not use gpu memory?

[AdeelH]

Low GPU utilization probably means most of the time is being spent reading the data.

• How many CPU cores does each machine have? Perhaps the default num_workers=4 helps performance with the larger machine with more cores, and hurts performance with the smaller machine?
• Is the data being streamed from remote files or downloaded?
• A batch size of 2 is pretty small and shouldn't be causing problems. How many bands are being read? Are they spread across multiple files?

[NoaMills-USDA]

I will try playing around with the number of cpus per gpu, and the number of workers. The data is already downloaded and stored on disk. I'm reading three bands, not spread across files.