Add a pre-check whether host/node is GPU-aware before running any other GPU commands

[bast]

Follow-up from #40. This pre-check should not be really needed since the subprocess is implemented to hopefully never crash or hang but I would still feel better if we do a quick check first before running other commands.

[lars-t-hansen]

For nvidia we could maybe:

• check for /dev/nvidia*
• check to see if nvidia-smi exists
• try to run nvidia-smi -L to see if there's any output at all

I'd love for there to be something absolutely solid but those are my best ideas so far, for run-time detection. There's module avail but then we take a dependency on module plus I know for a fact we have non-NVIDIA machines that have the NVIDIA stack.

Other options are:

• command line switch to sonar ps to tell it to look for nvidia cards, but this adds friction to setup
• compile-time switch, but this adds friction to deployment

[lars-t-hansen]

There's some variant of lspci | egrep 'VGA.*(NVIDIA Corporation|Advanced Micro Devices)' I guess. On my systems this finds all the cards. Indeed, finding the NVIDIA string means we should be able to run nvidia-smi, finding the AMD string that we should be able to run rocm-smi.

[lars-t-hansen]

A complementary approach is that used by the sysinfo prototype: it uses Go's LookPath primitive to resolve the paths for nvidia-smi and rocm-smi to ensure that the programs are found in the path but not in the current directory. This doesn't quite address the core of the problem - we want to be sure that the probe programs don't hang - but may in any case be desirable.

[lars-t-hansen]

As #188 shows, the presence of nvidia-smi is by itself insufficient.