advice on getting an RTX4090 to work as an eGPU on Ubuntu

[hamishcunningham]

I've been working with a couple of GTX cards in eGPU cases which work well, but I needed more VRAM so I've now got a shiney new RTX4090 :)

The trouble is the driver fails to load, so e.g. nvidia-smi gives couldn't communicate with the NVIDIA driver. I've googled a lot of linux/GPU and Nvidia forum posts but nothing seems directly relevant :(

I've tried two eGPUs (Sonnet and Razor); I'm running from a Lenovo X1 Carbon with 22.04 and nvidia-driver-530. The GPU powers up, and when I authorize the Thunderbolt connection it changes its lighting and so on, but the driver connection never works. Syslog says:

Jun 29 09:49:15 hc-x1b kernel: [  464.299277] NVRM: No NVIDIA GPU found.
Jun 29 09:49:15 hc-x1b kernel: [  464.300871] nvidia-nvlink: Unregistered Nvlink Core, major device number 508

Any advice on how to debug this?

Thanks!

[kristiankielhofner]

I've never used and don't really know anything about eGPU, etc but as a starting step does the device appear in lspci output?

Driver version 530 absolutely supports the RTX 4090 so there's likely an issue somewhere between eGPU/Thunderbolt and the PCI subsystem.

[deepcoder]

I am real PCI noob, so I might be out of my league here. What age is your motherboard? While I have not worked with eGPU for awhile, did try to get P40 to run with a motherboard that supported GTX10x0's was an adventure in frustration and failure. Grab a beer and a little time and look through the dmesg output from first boot. Look for any BAR memory allocation failure messages and/or ACPI allocation failure messages. It's rocket science beyond me for just plugging into a local PCI slot, so I can only guess what a thunderbolt abstraction adds. Good hunting!

[hamishcunningham]

Thanks both. The eGPU enclosure shows up on lspci as usual, but not the actual card. I'm suspicious of the PSU so I've ordered another, will report back when it arrives!

[tensiondriven]

I'd be interested to follow your progress.

I've been doing a lot with LLM's recently and my biggest constraint has been VRAM, and thus video cards, and thus PCI ports. Even though the bandwidth over eGPU is limited, if it can reach PCIe Gen 3 x 4 rates, I'll be happy with it.

Here's a guide on how to set up an eGPU under Linux (apologies if you've already done the googling):

https://mattkubilus.medium.com/setting-up-an-egpu-on-ubuntu-c87d4c04cea4

My guess would be that there's some kind of driver needed to make the PCI device available to linux. But this is a very general, hand-wavy suggestion as I don't have concrete experience here.

Another option might be to skip the eGPU entirely, and pick up a $200ish Dell box that would allow you to add the GPU directly. The necessary horsepower are in the GPU, and can probably be driven with any CPU from the past 5-10 years.

[hamishcunningham]

Thanks Jonathon. As I noted at the start of the discussion I have it working well with several other cards, so this discussion is specifically about the RTX4090. This is quite a power-hungry card; I'm trying a new PSU and will update.

[kristiankielhofner]

@tensiondriven - "pick up a $200ish Dell box" - this is exactly the approach I've been suggesting. Math breaks down like so:

• Pascal-something (for most Willow users): $100
• Dell/HP/whatever little tower, etc from eBay: $200

$300. Then you add (lets say) four Willow devices. You're now at $550 total (with Willow power supplies). This may seem like a lot but compare it to Raspberry Pi approaches from other projects:

• Qty 1 Raspberry Pi as the "server" (kind of a loose term because it's usually just HA). With enclosure, SD, power supply, etc: $100.
• Qty 4 Raspberry Pi as Willow drop-ins. With everything above + speaker, microphone array, etc and you have to assemble them, come up with an enclosure that works, etc: $150 each.

So now you're at $700 and the experience is frankly terrible (I call it unusable). In terms of power efficiency, with the approaches you see out there each Raspberry Pi is more-or-less pegged out and using ~5w each. That's 25w 24/7. My el-cheapo Dell minitower idles at 25-30w and I haven't even bothered to optimize that. Considering each Willow device takes something like 100mw the total power consumption ends up being roughly equivalent too.

Plus now you're in a position to do FAR more with that tower than just host HA (and even do just HA significantly better).

[deepcoder]

Good write up, thx! Interesting, there seem to be a boat load of Tesla P40's available on eBay, while about 50 to 70 USD more than 1070's. Seems like the 20+ GB of VRAM would be a good thing for running a combo of speeach in, Willow, Chatbot, TTS out. I get that it is the 'next one out of support' however seems like it might run for several years till the next evolution. Pluses seem to be int8, 23 GB VRAM, minuses seem to be more power hungry, more complex PCI, and you have to add cooling. Thx for your input.

[kristiankielhofner]

With the Tesla P40 the memory helps but the LLM performance is essentially unusable. Currently WIS uses less than half of the VRAM (4GB or so) of my 8GB GTX 1070 - and that's with all Whisper models, TTS, and speaker verification loaded. We also haven't optimized TTS or speaker verification for memory use (quantization, etc). The Tesla P4/GTX 1070 can still provide sub one second response times all in (with STT, command execution in HA, and TTS response). At best my RTX 3090 takes a few seconds to generate LLM responses with text in/out. See this thread showing a Tesla P41 generating 0.61 tokens/s and taking almost 50 seconds for a response.

For smaller form factors, no additional power/cooling, etc we typically recommend the Tesla P4. Given that the Pascal series is still supported by CUDA 12 (which isn't even supported by pytorch stable yet) and we don't use even half of the VRAM there should be at least several years of support for the most recent toolkits, frameworks, etc.

[hamishcunningham]

The solution was to a) replace the PSU in my eGPU (I'm now running a Corsair RM850e) and b) enable the "proposed" updates in APT, then install nvidia-driver-535. For some reason driver 530 doesn't manage to detect the 4090 in this configuration.

Thanks for the suggestions everyone.