Time taken to train. custome training estiamted to take 3000 days :(

[prashkmr]

Hi,
Thanks a lot for opensourcing the code.
I was triying to use your code for a differnet dataset.

I wanted to know that with your configuration of dataset of size: 2,64,1024 and ~50k samples for KITTI and ~ 297,737 for nuscenes.
how much time does it take to train the model for the 5,00,000 epochs.

I wanted to ask this because I have only 2048 samples 1 epoch is taking 10 minutes, and by that calculation it would take around ~3000 days to train.

Am I missing something , or is the training procedure something differnet, or you downsample the dataset.
Please let me know.

It is of huge importance to me as I am using your model as a major part of my research.

Eagerly awaiting your response.

Thank you.

[vzyrianov]

You don't need to train the model for 500,000 epochs. You should be able to get reasonable samples in about a day of training.

[prashkmr]

Thanks for the quick repsonse. Really appreciate it.

I also was wondering what would be the process to train the model for inpainting when we have a numpy(A.npy) to train and another numpy (B.npy) to inpaint.
A.npy: KITTI original lidar scan in range image format.
B.npy: KITTI lidar scans for inpainting.

The documentation was not very clear to me on how to proceed with inpaiting prcodeure. Could you elaborate on how to move forward with the inpainitng process from scratch the above.

Thanking you in anticipation.

[prashkmr]

I also did not understand this code in the ncsn_runner.py

at this line:

lidargen/LiDARGen/runners/ncsn_runner.py

Line 366 in 4c3226c

width = int(np.sqrt(self.config.sampling.batch_size))

refer_images = refer_images.to(self.config.device)
width = int(np.sqrt(self.config.sampling.batch_size))
init_samples = torch.rand(width, width, self.config.data.channels,
self.config.data.image_size,
self.config.data.image_width,
device=self.config.device)
init_samples = data_transform(self.config, init_samples)

Given the refer_images is of shape (batch_size, 2, 64,1024) what is it exactly doing.

Ideallly for every image in the refer_images we should have one inpainted image, but this is not happening in this case.

Can you clarify what this is doing and what to do to achieve this "for every image in the refer_images we should have one inpainted image,".