Support for PandaSet dataset

[JPenshornTAP]

Hey,
I'm interested in creating a DataParser for the PandaSet dataset. Looking through the existing forks, I found this fork from @pierremerriaux-leddartech with a WIP (?) DataParser for PandaSet. Do you have any further information regarding this fork or the support of PandaSet?

Kind regards
Jonas

[pierremerriaux-leddartech]

Hi @JPenshornTAP,
Yes this branch is supporting pandaset datareader. Now, we are working to integrate it in the a refact to reduce RAM usage.
A example below of reconstruction

sequence_011_cam_car_move.mp4

Pierre

[szhang963]

@pierremerriaux-leddartech Hi, did you only use one camera for reconstruction? Have you tried to use multiple cameras in mars?

[pierremerriaux-leddartech]

Only one camera in mars on my side due to memory actual memory consumption, but I didt before with nerfstudio nerfacto 5 and 6 camera with mask on dynamic object

sequence_11_free_trajectory.mp4

2023-09-21_234748.mp4

[szhang963]

@pierremerriaux-leddartech Waw, it's cool, but did you use the depth of images for reconstruction? And, Did you please tell me how to set a new camera's trajectory in mars? Besides, have you tried to use multiple cameras for a small number of frames? Do you know if it worked? Thanks in advance.

[j-pens]

@pierremerriaux-leddartech Great work with the dataparser! However, the RAM load is quite intense. Do you have any update regarding the current work on reducing the RAM usage? Is there anything I can support you with?

[pierremerriaux-leddartech]

Hi @j-pens, yes we are working with @xBeho1der and the memory consumption is really better, but for now we have still a bug and the reconstruction is not good as the current version.

[szhang963]

@pierremerriaux-leddartech did you try to fix it by 85?

[j-pens]

Hi @pierremerriaux-leddartech and @xBeho1der! I managed to reduce the memory consumption which enabled me to train on all 80 frames of a sequence. This should also work for more frames, e.g. from other cameras but more testing would be necessary. Based on my understanding of the code and the pipeline my proposed fix should have no impact on the reconstruction/ rendering quality of the pipeline, as the data passed to the pipeline should be the same. I tested it for the PandaSet data parser but the same adjustment should be possible for the other data parsers as well. It's similar to the fix in #85 that @szhang963 referenced.

Would it be possible to integrate with the PandaSet fork and the other data parsers? Which pull requests would make sense and in which repository/ fork? Also should we reopen this issue or create a new one?

[Nplace-su]

@j-pens I think submitting a PR to the main branch of this repo is the best choice since it is a general fix for all data parsers.

[OscIing]

Hi @pierremerriaux-leddartech and @j-pens, I have noticed the great work you have done for supporting Pandaset and appreciate your further research. However, I have a question that may have occurred in your earlier reproducing: how did you generate latent code without car-nerf? Also noticing issue #5 , I wonder if u guys select Nerfacto and save car-latents. If you could provide some insights, I would greatly appreciate it! (sorry for that in case not opening a new issue to bother other guys, i choose set my question here)

[j-pens]

Hi @OscIing, for initialisation you can just generate random latent codes like this
Then you can load the latents, e.g. from KITTI-mot and optimise them during training by setting optimise_latents to True in car_nerf.py. Hope this helps

[szhang963]

@j-pens Hi, is it necessary to use car latent for car reconstruction? I can get a relatively good result using the code for car reconstruction.

model=SceneGraphModelConfig(
    background_model=NerfactoModelConfig(),
    object_model_template=NerfactoModelConfig(),
    object_representation="object-wise",
    object_ray_sample_strategy="remove-bg",
),

Could you show the difference between the two methods?
Thanks a lot.

[Nplace-su]

@j-pens Hi, is it necessary to use car latent for car reconstruction? I can get a relatively good result using the code for car reconstruction.

model=SceneGraphModelConfig(
    background_model=NerfactoModelConfig(),
    object_model_template=NerfactoModelConfig(),
    object_representation="object-wise",
    object_ray_sample_strategy="remove-bg",
),

Could you show the difference between the two methods? Thanks a lot.

@szhang963
It's not necessary. Depending on the dataset you use, sometimes using the Nerfacto model for each object will give you better results. This has also been discussed in the appendix of the StreetGaussian paper.

[Yurri-hub]

Hi @OscIing, for initialisation you can just generate random latent codes like this Then you can load the latents, e.g. from KITTI-mot and optimise them during training by setting optimise_latents to True in car_nerf.py. Hope this helps

@j-pens Hi !
I tried it, but I got some error. Is there any suggestion to solve it?

Traceback (most recent call last):
  File "/root/autodl-tmp/light/UDNeRF/nerfstudio/nerfstudio/scripts/train.py", line 270, in <module>
    entrypoint()
  File "/root/autodl-tmp/light/UDNeRF/nerfstudio/nerfstudio/scripts/train.py", line 261, in entrypoint
    main(
  File "/root/autodl-tmp/light/UDNeRF/nerfstudio/nerfstudio/scripts/train.py", line 246, in main
    launch(
  File "/root/autodl-tmp/light/UDNeRF/nerfstudio/nerfstudio/scripts/train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/root/autodl-tmp/light/UDNeRF/nerfstudio/nerfstudio/scripts/train.py", line 100, in train_loop
    trainer.train()
  File "/root/anaconda3/envs/mars/lib/python3.9/site-packages/nerfstudio/engine/trainer.py", line 259, in train
    loss, loss_dict, metrics_dict = self.train_iteration(step)
  File "/root/anaconda3/envs/mars/lib/python3.9/site-packages/nerfstudio/utils/profiler.py", line 127, in inner
    out = func(*args, **kwargs)
  File "/root/anaconda3/envs/mars/lib/python3.9/site-packages/nerfstudio/engine/trainer.py", line 483, in train_iteration
    self.optimizers.optimizer_scaler_step_all(self.grad_scaler)
  File "/root/anaconda3/envs/mars/lib/python3.9/site-packages/nerfstudio/engine/optimizers.py", line 131, in optimizer_scaler_step_all
    grad_scaler.step(optimizer)
  File "/root/anaconda3/envs/mars/lib/python3.9/site-packages/torch/cuda/amp/grad_scaler.py", line 370, in step
    self.unscale_(optimizer)
  File "/root/anaconda3/envs/mars/lib/python3.9/site-packages/torch/cuda/amp/grad_scaler.py", line 284, in unscale_
    optimizer_state["found_inf_per_device"] = self._unscale_grads_(optimizer, inv_scale, found_inf, False)
  File "/root/anaconda3/envs/mars/lib/python3.9/site-packages/torch/cuda/amp/grad_scaler.py", line 229, in _unscale_grads_
    torch._amp_foreach_non_finite_check_and_unscale_(grads,
NotImplementedError: Could not run 'aten::_amp_foreach_non_finite_check_and_unscale_' with arguments from the 'CPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::_amp_foreach_non_finite_check_and_unscale_' is only available for these backends: [CUDA, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMeta, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].

CUDA: registered at aten/src/ATen/RegisterCUDA.cpp:43986 [kernel]
BackendSelect: fallthrough registered at ../aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
Python: registered at ../aten/src/ATen/core/PythonFallbackKernel.cpp:144 [backend fallback]
FuncTorchDynamicLayerBackMode: registered at ../aten/src/ATen/functorch/DynamicLayer.cpp:491 [backend fallback]
Functionalize: registered at aten/src/ATen/RegisterFunctionalization_3.cpp:22788 [kernel]
Named: registered at ../aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
Conjugate: registered at ../aten/src/ATen/ConjugateFallback.cpp:17 [backend fallback]
Negative: registered at ../aten/src/ATen/native/NegateFallback.cpp:19 [backend fallback]
ZeroTensor: registered at ../aten/src/ATen/ZeroTensorFallback.cpp:86 [backend fallback]
ADInplaceOrView: fallthrough registered at ../aten/src/ATen/core/VariableFallbackKernel.cpp:63 [backend fallback]
AutogradOther: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradCPU: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradCUDA: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradHIP: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradXLA: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradMPS: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradIPU: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradXPU: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradHPU: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradVE: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradLazy: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradMeta: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradMTIA: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradPrivateUse1: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradPrivateUse2: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradPrivateUse3: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
AutogradNestedTensor: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:15232 [autograd kernel]
Tracer: registered at ../torch/csrc/autograd/generated/TraceType_0.cpp:16728 [kernel]
AutocastCPU: fallthrough registered at ../aten/src/ATen/autocast_mode.cpp:487 [backend fallback]
AutocastCUDA: fallthrough registered at ../aten/src/ATen/autocast_mode.cpp:354 [backend fallback]
FuncTorchBatched: registered at ../aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:815 [backend fallback]
FuncTorchVmapMode: fallthrough registered at ../aten/src/ATen/functorch/VmapModeRegistrations.cpp:28 [backend fallback]
Batched: registered at ../aten/src/ATen/LegacyBatchingRegistrations.cpp:1073 [backend fallback]
VmapMode: fallthrough registered at ../aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
FuncTorchGradWrapper: registered at ../aten/src/ATen/functorch/TensorWrapper.cpp:210 [backend fallback]
PythonTLSSnapshot: registered at ../aten/src/ATen/core/PythonFallbackKernel.cpp:152 [backend fallback]
FuncTorchDynamicLayerFrontMode: registered at ../aten/src/ATen/functorch/DynamicLayer.cpp:487 [backend fallback]
PythonDispatcher: registered at ../aten/src/ATen/core/PythonFallbackKernel.cpp:148 [backend fallback]