Shared Memory while multi-gpu?

[Bigfield77]

Hello,

I am able to load llama3 70b instruct in 5.0bpw using exllamav2_hf in ooba if I am expose only on GPU (Extremely slow)

If i expose my 2 GPUs (3090s), there is not enough vram to load it and it does not fall back to using shared memory like it does when using a single GPU.

Is this something that is possible to enable, something like falling back to shared memory on the last available gpu?

I am running under windows

[Bigfield77]

do you know if that a limitation with exllamav2, pytorch or the nvidia driver suite?

[turboderp]

I'm not really sure. This would be down to the NVIDIA driver, and there isn't to my knowledge any way you can control the sysmem fallback behavior from software. (?)

[Bigfield77]

Thanks for your answer!

I was asking because i saw something like that in the 'transformers' loader of oobabooga:

I will try to open a feature request with nvidia if i can find out where

Thanks for your time