libcuda.so is not in found but libcuda.so.1 is.

[PiotrCzapla]

Version

Microsoft Windows [Version 10.0.22000.778]

WSL Version

• WSL 2
• WSL 1

Kernel Version

5.10.102.1

Distro Version

Ubuntu 22.04

Other Software

cuda 10.7 , miniconda , pytorch

Repro Steps

Install CUDA as described on nvidia website using option 1:
https://docs.nvidia.com/cuda/wsl-user-guide/index.html

then install python3.9 and try the following code that shows the issue :

$ python
Python 3.9.12 (main, Apr  5 2022, 06:56:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import cdll
>>> cdll.LoadLibrary('libcuda.so.1')
<CDLL 'libcuda.so.1', handle 562f22ae4160 at 0x7f74fb662c10>
>>> cdll.LoadLibrary('libcuda.so')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/piotr/miniconda3/lib/python3.9/ctypes/__init__.py", line 460, in LoadLibrary
    return self._dlltype(name)
  File "/home/piotr/miniconda3/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcuda.so: cannot open shared object file: No such file or directory

I'm loading the library manually so that you don't have to install pytorch and timm, and then try to run training of 'resnet26d', as this triggers the loading of libcudnn_cnn_infer.so.8 which is linked against libcuda.so instead of libcuda.so.1 .

Expected Behavior

It should be possible to load libcuda.so without specifing /usr/lib/wsl/lib in the LD_LIBRARY_PATH.

>>> cdll.LoadLibrary('libcuda.so')
should return a library handle

Actual Behavior

Unfortunately somehow the ldconfig fails to cache the libcuda.so and subsequently it does not find it unless /usr/lib/wsl/lib is in the LD_LIBRARY_PATH.

Diagnostic Logs

libcuda.so is in the /usr/lib/wsl/lib along with libcuda.so.1 and libcuda.so.1.1 . All files are the same.

It looks like ldconfig somehow does not save the path to libcuda.so, you can observe this by running:

$  ldconfig  /usr/lib/wsl/lib -v  -n
libnvwgf2umx.so -> libnvwgf2umx.so
        libnvidia-opticalflow.so.1 -> libnvidia-opticalflow.so.1
        libnvidia-ml.so.1 -> libnvidia-ml.so.1
        libnvidia-encode.so.1 -> libnvidia-encode.so.1
        libnvdxdlkernels.so -> libnvdxdlkernels.so
        libnvcuvid.so.1 -> libnvcuvid.so.1
        libdxcore.so -> libdxcore.so
        libd3d12core.so -> libd3d12core.so
        libd3d12.so -> libd3d12.so
/sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

        libcuda.so.1 -> libcuda.so.1.1

The libcuda.so is not being picked up, even though strace shows that ldconfig is reading that file:
strace ldconfig /usr/lib/wsl/lib -n

newfstatat(AT_FDCWD, "/usr/lib/wsl/lib", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/wsl/lib/glibc-hwcaps", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/cache/ldconfig/aux-cache", O_RDONLY) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/usr/lib/wsl/lib", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
newfstatat(3, "", {st_mode=S_IFDIR|0555, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(3, 0x555556da4ee0 /* 18 entries */, 32768) = 672
newfstatat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 141464, PROT_READ, MAP_SHARED, 4, 0) = 0x7fe867b6e000
munmap(0x7fe867b6e000, 141464)          = 0
close(4)                                = 0
newfstatat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so.1", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so.1", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 141464, PROT_READ, MAP_SHARED, 4, 0) = 0x7fe867b6e000
munmap(0x7fe867b6e000, 141464)          = 0
close(4)                                = 0
newfstatat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so.1.1", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so.1.1", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 141464, PROT_READ, MAP_SHARED, 4, 0) = 0x7fe867b6e000
munmap(0x7fe867b6e000, 141464)          = 0
close(4)                                = 0
newfstatat(AT_FDCWD, "/usr/lib/wsl/lib/libd3d12.so"

[PiotrCzapla]

I've found a way to get the content of lib cache and the libcuda.so and libcuda.so.1.1 is not there, even though it is in the wsl folder.

$ ldconfig -p| grep libcuda
        libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.0
        libcudart.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so
        libcuda.so.1 (libc6,x86-64) => /usr/lib/wsl/lib/libcuda.so.1
$ ls -la /usr/lib/wsl/lib/libcuda*
-r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so
-r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so.1
-r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so.1.1

And here is how this works if you specify LD_LIBRARY_PATH

$ LD_LIBRARY_PATH=/usr/lib/wsl/lib/ python
Python 3.9.12 (main, Apr  5 2022, 06:56:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import cdll
>>> cdll.LoadLibrary('libcuda.so.1')
<CDLL 'libcuda.so.1', handle 55f038a352a0 at 0x7ff7bcdc68e0>
>>> cdll.LoadLibrary('libcuda.so')
<CDLL 'libcuda.so', handle 55f038a46510 at 0x7ff7bcd2eac0>
>>> cdll.LoadLibrary('libcuda.so.1.1')
<CDLL 'libcuda.so.1.1', handle 55f038a470a0 at 0x7ff7bcd2dc70>

[PiotrCzapla]

For anyone facing the following issue:

Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory130 00:00<00:00]
Please make sure libcudnn_cnn_infer.so.8 is in your library path!

when traing put LD_LIBARAR_PATH=/usr/lib/wsl/lib/:$LD_LIBARAR_PATH to your .bashrc . It should fix the problem.

[kryptan]

I think the correct line to add to the .bashrc file is:

export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

[matsuu]

try this:

cd /usr/lib/wsl/lib
# backup
sudo mv libcuda.so.1 libcuda.so.1.backup
sudo mv libcuda.so libcuda.so.backup
# create symlink manually
sudo ln -s libcuda.so.1.1 libcuda.so.1
sudo ln -s libcuda.so.1 libcuda.so
# update cache
sudo ldconfig

[Lonas13]

I think the correct line to add to the .bashrc file is:

export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

THANK YOU, THIS FINALLY FIXED IT FOR ME!!!

[Abdulla060]

I think the correct line to add to the .bashrc file is:

export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

I think this reply should be in the official docs

[fengyuexingzi]

THANK YOU, THIS FINALLY FIXED IT FOR ME!!!

[ptn77]

Thank you very much! Cuda worked in VSCode running on windows, but in my WSL, I had to create the symbolic link in order for it to work on my Ubuntu WSL.

[microsoft-github-policy-service]

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!