Training and inference time on older GPUs

[primecai]

Hi,

Many thanks for the great work. I tried to run the code on an TITANXp, but it generally takes ~2 hours to complete the default 100k iterations (in headless mode), and several minutes to even render an image. Is this the expected results on older GPUs? The command I am running is:

python scripts/run.py --scene=fox \
                      --save_mesh=instant-ngp-results/fox.obj \
                      --screenshot_transforms=data/nerf/fox/transforms.json \
                      --screenshot_dir=instant-ngp-results/ \
                      --width=1080 --height=1920 --n_steps=100000

Best.

[Tom94]

Hi there, yes, older GPUs are expected to be quite a bit (on the order of 10x) slower. This is mostly caused by their lack of hardware acceleration for fp16 calculations (in the case of consumer Pascal GPUs) and lack of hardware matrix multiply acceleration.

That said, the default of 100k iteration is extremely large and usually not needed for good reconstruction. I recommend using something on the order of 20k iterations to save time. There's generally no measurable improvement after ~35k iterations due to the learning rate decay schedule (see nerf/base.json).

Cheers!

[primecai]

Thanks for the reply. I'll try lowering the resolution as well as less iterations. Is it also normal that rendering an 1080×1920 fox image takes several minutes (~10 min) on these older architecture GPUs?

Best.

[Tom94]

In the case of run.py, I would expect ~15 seconds per image, not minutes.

My 3090 needs ~170ms for a regular 1080p frame, so I estimate 15 seconds by multiplying that by 8 samples per pixel (as configured in run.py for anti aliasing) and by 10 for being an older GPU.

The above command line will render 50 screenshots -- are you perhaps referring to the entire rendering duration? That would be consistent with ~15 seconds per image x 50 images == 12.5 min

[primecai]

I timed the rendering time as follows in run.py:

			for idx in args.screenshot_frames:
				start_time = time.time()
				f = ref_transforms["frames"][int(idx)]
				cam_matrix = f["transform_matrix"]
				testbed.set_nerf_camera_matrix(np.matrix(cam_matrix)[:-1,:])
				outname = os.path.join(args.screenshot_dir, os.path.basename(f["file_path"]))

				# Some NeRF datasets lack the .png suffix in the dataset metadata
				if not os.path.splitext(outname)[1]:
					outname = outname + ".png"

				print(f"rendering {outname}")
				image = testbed.render(args.width or int(ref_transforms["w"]), args.height or int(ref_transforms["h"]), args.screenshot_spp, True)
				os.makedirs(os.path.dirname(outname), exist_ok=True)
				write_image(outname, image)
				print("--- %s seconds ---" % (time.time() - start_time), flush=True)

It seems on average, it takes ~2 minutes to render a single 1080x1920 frame on an TITANXp. The beginning of the building output is as follows:

-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- The CUDA compiler identification is NVIDIA 11.2.152
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- No release type specified. Setting to 'Release'.
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Using X11 for window creation
-- Found X11: /usr/include   
-- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so
-- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Found GLEW: /usr/include (found version "2.1.0") 
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- !!! Warning OptiX_INSTALL_DIR not set in environment. using default
-- OptiX_INSTALL_DIR value: /usr/local/NVIDIA-OptiX-SDK-7.3.0-linux64-x86_64
-- OptiX headers (optix.h and friends) not found.
CMake Warning at CMakeLists.txt:169 (message):
  OptiX was not found.  Neural graphics primitives will still compile and run
  correctly.  However, SDF training in 'raystab' and 'pathescape' modes will
  be significantly slower.


-- Found Python: /itet-stor/shecai/net_scratch/conda_envs/instant-ngp/bin/python3.8 (found suitable version "3.8.12", minimum required is "3.7") found components: Interpreter Development Development.Module Development.Embed 
-- pybind11 v2.7.1 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Targeting GPU architectures: 61
CMake Warning at dependencies/tiny-cuda-nn/CMakeLists.txt:105 (message):
  Fully fused MLPs do not support GPU architectures of 70 or less.  Falling
  back to CUTLASS MLPs.  Remove GPU architectures 70 and lower to allow
  maximum performance


-- Configuring done
-- Generating done
-- Build files have been written to: /home/shecai/Desktop/instant-ngp/build
[ 10%] Building C object dependencies/glfw/src/CMakeFiles/glfw_objects.dir/init.c.o
[ 12%] Building C object dependencies/glfw/src/CMakeFiles/glfw_objects.dir/context.c.o
[  9%] Building CUDA object dependencies/tiny-cuda-nn/src/CMakeFiles/tiny-cuda-nn.dir/common.cu.o

Is there anything missing?