Use NVIDIA's gpu-operator for GPU node support

[mboersma]

/kind feature

Describe the solution you'd like

#1002 implemented the "nvidia-gpu" flavor via postKubeadmCommands recommended by NVIDIA, as explained in this comment.

But NVIDIA's gpu-operator seems like a cleaner, more future-proof solution. We should investigate whether it supports containerd now and whether the current implementation could be replaced with gpu-operator.

Anything else you would like to add:

See the discussion in #426 and the current implementation in #1002.

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[jessehu]

NVIDIA's gpu-operator doesn't support containerd yet per discussion in NVIDIA/gpu-operator#7

[nader-ziada]

/assign @mboersma

[rosskukulinski]

Containerd 1.4 support is now live in gpu-operator 1.4. +100 for leveraging gpu-operator

[shysank]

@mboersma are you working on this? If not, can I pick this one up?

[mboersma]

/assign @shysank

@shysank I am not currently working on this, so please have at it (and thank you).

When I had looked at it in December, the issue was that restarting a node made Kubernetes lose track of the GPU device, which didn't seem to be a problem with the existing Pre|PostKubeAdmCommand approach. Hopefully that is fixed now or you can find a workaround.

[shysank]

There is an issue with gpu operator compatibility with containerd v1.3.0. tldr; containerd expects a default_runtime_name field to be set to nvidia. I have opened a pr which I believe is the fix. Will have to wait for the nvidia folks' confirmation, and the timeline for it to be available.