Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[probably] Infinite cuda extention build #116

Closed
kst179 opened this issue Nov 14, 2022 · 4 comments · Fixed by #137
Closed

[probably] Infinite cuda extention build #116

kst179 opened this issue Nov 14, 2022 · 4 comments · Fixed by #137

Comments

@kst179
Copy link

kst179 commented Nov 14, 2022

Hi, first of all: that's a great work, thank you for that. But I've had some problems on the first launch.

I've tried to run nerfstudio with instant-ngp model, and faced the [probably] infinite extention build when nerfacc's C++ functions are called. At least compilation did not finished successfully for about 10 minutes or even more (yeah, I am kinda impatient).

minimal example:

>>> import nerfacc
>>> nerfacc.ContractionType.AABB.to_cpp_version()
# Out: (<infinitly bouncing ball>) NerfAcc: Setting up CUDA (This may take a few minutes the first time)

I've tried to build it manualy, by just calling (I suppose build config was already generated by torch.utils.cpp_extention.load after first call)

$ cd ~/.cache/torch_extensions/py38_cu116/nerfacc_cuda && ninja -j 4

and voila, it took about 1 minute to compile (and it is showing more detailed progress 🙃). If not specify num workers it is killed because consumes all memory on my laptop and wants even more.

Hope it will help others, who facing the same problem and maybe you can fix it somehow.

nerfacc version: 0.2.1 (last compatible with nerfstudio)
nvcc version: 11.6
torch version: 1.12.1 (+cu116)
os: ubuntu 20.04.4 LTS
memory: 15.3 Gb
processor: core i7 @ 3.30GHz x8
gpu: RTX 3050 Laptop
@Zeju1997
Copy link

Thank you so much for sharing. :D

@liruilong940607
Copy link
Collaborator

Interesting! Thanks for sharing. I'll find some time to look into this

@theNded
Copy link

theNded commented Dec 29, 2022

It also happens from my side. Here are my observations:

  1. I had nerfacc configured in a testing environment with python3.9.
  2. Later I configured another environment for nerfstudio with python3.8, depending on nerfacc.
  3. The infinite loop happened when I was playing with nerfstudio, i.e., the python3.8 environment.
  4. I checked .cache and nerfacc only appeared in py39_cu113. Nothing was found in py38_cu133.
  5. After deleting all the cache, the compilation was completed for python3.8 correctly.
    Therefore I suspect the issue comes from the upstream torch's jit module.

@liruilong940607
Copy link
Collaborator

It should be fixed. Please reopen it if the problem still exists

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants