Torch library list not found

[AndreiMiculita]

Hi! I'm trying to compile DiffDVR but running into some issues. Was hoping someone could help. I want to use the same dependencies as the ones in the repo. I'm on Linux Mint 20.3, which is based on Ubuntu 20.04.
In order, I have installed:

• CUDA 11.0 (using the official runfile)
• cuDNN v8.1.1 (the highest version compatible with 11.0, as per the cuDNN archive)
• created a Python 3.8 environment with Torch 1.8.0 (based on the environment.yml file in the repo)
• libglm-dev, libglfw3-dev and libglew-dev as they were needed for compilation.

The error I get when going to the build dir and running cmake .. is:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
TORCH_LIB_c10
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui
TORCH_LIB_c10_cuda
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui
TORCH_LIB_torch_cpu
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui
TORCH_LIB_torch_cuda
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui
TORCH_LIB_torch_python
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui

Scrolling up in the output reveals the following problem:
Torch: full library list: /usr/lib/libtorch.so;TORCH_LIB_c10-NOTFOUND;TORCH_LIB_c10_cuda-NOTFOUND;/usr/lib/libtorch.so;TORCH_LIB_torch_cpu-NOTFOUND;TORCH_LIB_torch_cuda-NOTFOUND;TORCH_LIB_torch_python-NOTFOUND

So it seems like these libraries are not found. I tried to sudo apt-get install libtorch3-dev but apt says it's already installed (I assume it happened when conda installed the pytorch package?), and there are no libraries available with the c10, c10_cuda etc suffixes. How would I make sure these libraries are found, so I can compile the project?

[AndreiMiculita]

Update: managed to run cmake (and make) successfully by running the following cmake command (which I found in #1). If anyone finds this, remember to change the paths to fit your particular installation of conda. Might be a good idea to put this in the readme, took me quite a long time to finally find this solution (which you can see in the edit history :P).

cmake .. -DTORCH_PATH=/home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch -DTorch_DIR=/home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch/share/cmake/Torch -DPYTHON_LIBRARY=/home/andrei/miniconda3/envs/py38torch18/lib/libpython3.8.so -DPYTHON_EXECUTABLE=/home/andrei/miniconda3/envs/py38torch18/bin/python

[shamanDevel]

Hi, glad that you figured it out. I still haven't figured out, how to reliably detect the PyTorch installation, the paths, and the libraries used. Pull requests are welcome :)
Until then, I added a note to the Readme as suggested, see cbee71c

[AndreiMiculita]

@shamanDevel Unfortunately, I ran into more issues. Hope it's fine if I use this thread to document them. I'm using the Docker container in #1.

I'm trying to run pytests/tests/camera/run_entropy2d.py (so, I can just use the /opt/DiffDVR/ directory in the container). Here is what I do:

• docker run -it xetaiz/diffdvr bash
  • Then, in the container: cd /opt/DiffDVR/pytests/ && nano tests/camera/run_entropy2d.py
  • add import sys; sys.path.append('/opt/DiffDVR/bin'); sys.path.append('/opt/DiffDVR/bin/pytests') to top of file
  • pip install imageio because this particular script uses it
  • python tests/camera/run_entropy2d.py

The error I get is:

Traceback (most recent call last):
  File "tests/camera/run_entropy2d.py", line 15, in <module>
    from diffdvr import Renderer, CameraOnASphere, Entropy, ColorMatches, Settings, setup_default_settings, \
ModuleNotFoundError: No module named 'diffdvr'

Should I be running it another way? The readme just says to navigate to pytests and run the script with python.

[shamanDevel]

Hi, this is quite strange as the modules in 'diffdvr' are just python files in 'pytests'. Adding the pytests to sys.path should not be necessary as this is already your current working directory.
Just to make sure, the working directory is correct and if you run assert(os.path.exists("./diffdvr/__init__.py")) at the very top, it succeeds?

[AndreiMiculita]

Managed to get it to work by running export PYTHONPATH=. before python tests/camera/run_entropy2d.py.

Also, oddly enough, the assert you suggested passed successfully, even before that.

Might be a Docker thing.

I also had to add import sys; sys.path.append('/opt/DiffDVR/bin') to the top of run_entropy2d.py, and change the '../../config-files/...' paths at the end of it to instead point to './config-files/...'