RuntimeError: selected index k out of range

[PercyXiao]

I encountered the following error when running your program, Scene Segmentation: Instructions to train KP-FCNN on a scene segmentation task (S3DIS): RuntimeError: selected index k out of range.

[PercyXiao]

Sorry for borther you and thanks for your contribution!
when I am training my own datasets I have met the problem as follow:

HKRoad mean IoU = 93.0%
e001-i0000 => L=3.070 acc=100% / t(ms): 5527.1 55.9 152.0)
e001-i0005 => L=3.289 acc= 95% / t(ms): 7.8 84.1 119.1)
e001-i0010 => L=3.146 acc= 95% / t(ms): 4.7 85.3 123.9)
e001-i0015 => L=3.173 acc= 98% / t(ms): 3.0 92.5 124.3)
e001-i0020 => L=3.142 acc=100% / t(ms): 1.9 87.3 125.1)
e001-i0025 => L=3.593 acc=100% / t(ms): 1.2 87.8 127.3)
e001-i0030 => L=2.861 acc=100% / t(ms): 0.9 83.7 124.7)
e001-i0036 => L=3.155 acc= 94% / t(ms): 0.7 78.9 120.5)
e001-i0042 => L=2.951 acc= 98% / t(ms): 0.6 73.4 116.6)
e001-i0048 => L=2.741 acc= 99% / t(ms): 0.5 72.5 116.4)
e001-i0053 => L=3.129 acc=100% / t(ms): 0.5 74.7 121.2)
e001-i0058 => L=3.366 acc=100% / t(ms): 0.4 80.7 127.8)
e001-i0064 => L=3.103 acc= 91% / t(ms): 0.4 76.0 122.4)
e001-i0069 => L=3.355 acc= 99% / t(ms): 0.4 77.9 122.3)
e001-i0074 => L=3.099 acc=100% / t(ms): 0.4 82.7 126.6)
e001-i0080 => L=3.215 acc=100% / t(ms): 0.4 77.4 121.9)
e001-i0086 => L=2.938 acc=100% / t(ms): 0.4 75.9 120.7)
e001-i0091 => L=3.276 acc=100% / t(ms): 0.4 80.3 124.5)
e001-i0096 => L=2.591 acc= 97% / t(ms): 0.4 85.9 130.7)
e001-i0101 => L=3.035 acc= 99% / t(ms): 0.4 82.6 127.8)
e001-i0106 => L=2.963 acc= 93% / t(ms): 0.3 85.0 129.6)
e001-i0112 => L=2.811 acc= 99% / t(ms): 0.4 79.4 125.0)
e001-i0117 => L=3.018 acc= 99% / t(ms): 0.4 80.7 125.2)
e001-i0123 => L=2.919 acc= 99% / t(ms): 0.4 76.1 122.1)
e001-i0128 => L=3.337 acc= 94% / t(ms): 0.4 78.1 123.4)
e001-i0133 => L=3.099 acc= 94% / t(ms): 0.4 77.8 123.5)
e001-i0138 => L=3.445 acc= 85% / t(ms): 0.4 79.2 123.7)
e001-i0143 => L=3.117 acc= 80% / t(ms): 0.4 80.0 123.1)
e001-i0148 => L=3.177 acc=100% / t(ms): 0.4 82.7 126.6)
e001-i0153 => L=2.949 acc= 99% / t(ms): 0.3 81.7 126.1)
e001-i0158 => L=2.901 acc=100% / t(ms): 0.4 83.0 133.4)
e001-i0164 => L=2.746 acc=100% / t(ms): 0.4 79.7 129.1)
e001-i0169 => L=2.949 acc=100% / t(ms): 0.3 79.8 127.0)
e001-i0175 => L=2.845 acc=100% / t(ms): 0.4 79.7 124.5)
e001-i0181 => L=3.046 acc= 95% / t(ms): 0.4 78.8 119.6)
e001-i0186 => L=2.868 acc= 98% / t(ms): 0.4 80.0 122.0)
e001-i0192 => L=3.193 acc= 88% / t(ms): 0.5 75.5 119.9)
e001-i0197 => L=2.704 acc=100% / t(ms): 0.4 79.3 122.9)
e001-i0203 => L=2.959 acc= 98% / t(ms): 0.5 76.7 123.9)
e001-i0208 => L=2.949 acc=100% / t(ms): 0.4 80.6 124.9)
e001-i0214 => L=2.733 acc= 96% / t(ms): 0.4 79.7 122.4)
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [993,0,0], thread: [49,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [993,0,0], thread: [50,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [993,0,0], thread: [51,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "train_HKRoad.py", line 328, in
trainer.train(net, training_loader, test_loader, config)
File "/home/user02/Documents/Test/Experiment/KPConv/utils/trainer.py", line 188, in train
outputs = net(batch, config)
File "/home/user02/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/user02/Documents/Test/Experiment/KPConv/models/architectures.py", line 332, in forward
x = block_op(x, batch)
File "/home/user02/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/user02/Documents/Test/Experiment/KPConv/models/blocks.py", line 635, in forward
x = self.KPConv(q_pts, s_pts, neighb_inds, x)
File "/home/user02/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/user02/Documents/Test/Experiment/KPConv/models/blocks.py", line 312, in forward
neighb_row_bool, neighb_row_inds = torch.topk(in_range, new_max_neighb.item(), dim=1)
RuntimeError: selected index k out of range

I sincerely hope you can help me with the answer.

[HuguesTHOMAS]

Go at the place where the bug happens: "blocks.py", line 312, try printing the shapes and of the tensors there, Try printing the max values if the indices, etc. It should help you debug what happens.

[colinzhenli]

Hi Percy, have you resolved the issue? I encountered the same CUDA index out of bounds problem in other lines of block.py when training KP-FCNN on a scene segmentation task (S3DIS). I thoroughly checked the indices range, and everything seems to work fine. However, when the error occurs, I cannot retrieve any values from the input q_pts, s_pts, neighb_inds, and x tensors. I'm uncertain if it's a CUDA environment issue or if there are problems when preparing data batches.