Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore no-SLAM, fix replica import, CUDA 11.7, etc #12

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

jrpowers
Copy link
Contributor

@jrpowers jrpowers commented Jan 1, 2023

This PR addresses a few things I ran into trying to test NeRF-SLAM:

  • I could not run out of the box on Ubuntu 22.04 LTS with CUDA 11.3, so I updated to CUDA 11.7 (the first version with Ubuntu 22.04 support)
  • Then I also had to bump Pytorch on the way, since older Pytorch wasn't available with CUDA 11.7
  • I had trouble building the customized gtsam, so I found a smaller set of edits that I could compile
  • Even after that, I found some gtsam APIs had changed, esp. regarding Quaternions and identity(). I am not sure how it was working for @ToniRV since I didn't spot changes in the gtsam fork that would have addressed it. So I updated a few lines to compile.
  • I tried to run without --slam and found it wasn't working, so I fixed a few things so that could work.
  • I noticed some case-sensitivity issues when unzipping datasets, and noticed the Replica sample ZIP had apparently been renamed too.
  • I temporarily pointed gtsam to my fork for the purposes of reviewing the PR, and I also made a PR on the upstream gtsam with just 2 lines which would make it compatible with NeRF-SLAM (at least the parts I ran).

@jrpowers
Copy link
Contributor Author

jrpowers commented Jan 1, 2023

Looks like the gtsam guys may accept the minimal PR, then we could update to use the plain develop branch. borglab/gtsam#1361

@ToniRV
Copy link
Owner

ToniRV commented Jan 2, 2023

Awesome! Thanks!

@ZZy129326999
Copy link

ZZy129326999 commented Jan 26, 2023

I'm worried.
AttributeError: module 'droid_backends' has no attribute 'reduced_camera_matrix'

H, v, Q, E, w = droid_backends.reduced_camera_matrix(
    self.cam0_T_world,
    self.world_T_body, # TODO(remove, unnecessary, given the math)
    self.cam0_idepths,
    self.cam0_intrinsics[0],
    self.cam0_T_body,
    self.cam0_idepths_sensed,
    gru_estimated_flow, 
    gru_estimated_flow_weight, # TODO: we should pass as well previous pose covariances!, so to not lose information when optimizing
    damping,
    ii, jj, kf0, kf1)

@kadirkun
Copy link

Hi guys, I encountered with the same issue. I was able to come up with the same solution without looking at this PR.
Now I see AttributeError: module 'droid_backends' has no attribute 'reduced_camera_matrix' error as @ZZy129326999.

Is there any reason why this PR is not merged? @ToniRV
@ZZy129326999 were you able to solve this camera matrix issue?

@kadirkun
Copy link

Hi guys, I encountered with the same issue. I was able to come up with the same solution without looking at this PR. Now I see AttributeError: module 'droid_backends' has no attribute 'reduced_camera_matrix' error as @ZZy129326999.

Is there any reason why this PR is not merged? @ToniRV @ZZy129326999 were you able to solve this camera matrix issue?

I solved the reduced_camera_matrix error by replacing the droid-slam's src folder with the src folder provided by the nerf-slam repo. After recompiling droid-slam, the droid_backends module had the reduced_camera_matrix function.

@zhangshuoneu
Copy link

wonderful pr, thanks very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants