Nvidia not getting detected on systems using nvidia-container-toolkit

[FrantaNautilus]

Upon installation of harbor I have noticed that the Nvidia GPU is not being used, based on the output of harbor cmd -h which does not include any compose files for Nvidia and the readings from nvtop. The reason for this seems to be the command docker info | grep -q "Runtimes:.*nvidia" which gives false return value because the nvidia-container-runtime is not installed. However nvidia-container-runtime is apparently deprecated by Nvidia and 'nvidia-container-toolkit' is the only needed component for running containers with Nvidia GPU acceleration. For this reason the detection of the Nvidia GPU should use different mechanism, e.g. command nvidia-container-toolkit -version.

Link to Nvidia detection code from harbor.sh in question:
https://github.com/av/harbor/blob/f25978a445099884f248f1a2c70fd70d734f1636/harbor.sh#L206C5-L209C7

[av]

Hi, thank you for trying out Harbor and for such a detailed bug report!

Just fixed it in v0.1.17, switching to the nvidia-container-toolkit detection.

I think the drawback will be that it will fail to detect the case when the toolkit is installed, but have not been configured with:

sudo nvidia-ctk runtime configure --runtime=docker

but I didn't test that

[FrantaNautilus]

Thank you for the virtually immediate reply and fix! Your speed of developing this software is incredible.
As for the configuration, I never configured the docker this way you and Nvidia container toolkit documentation suggests. Instead I only used the docker with the flag --gpus all and everything worked as expected.
I will test both the fix and the outcome of using sudo nvidia-ctk runtime configure --runtime=docker on my configuration and report back to this issue.

[FrantaNautilus]

I am sorry it took me so long, but I have the results. I have tested the v0.1.17 on a systems with and without configured 'nvidia-container-toolkit' (that is 'nvidia-container-toolkit' is installed on both but on the first it is configured a runtime and on the second it is only installed) and 'harbor' works on both without issues, tested with default setup ('ollama' + 'webui').

[av]

That's awesome, thanks for a thorough test!

[av]

I'm closing this for now, but please feel free to re-open if there'll be anything to revisit