Add support for CDI

[almaslennikov]

https://github.com/container-orchestrated-devices/container-device-interface

[almaslennikov]

@abdallahyas We don't have any copyrights in the repo, should we copyright all files or drop the check?

[abdallahyas]

@abdallahyas We don't have any copyrights in the repo, should we copyright all files or drop the check?

@almaslennikov we can not drop because it is the company policy, so we can copy the copy right message from any other project and add it to only the newly added files. The CI will only check newly added files, and would ignore the modified files

[almaslennikov]

@e0ne addressed your comments and rebased the PR, PTAL

[e0ne]

LGTM. thanks for addressing my comments

[adrianchiris]

Hey @almaslennikov appologies for the delayed review !

[adrianchiris]

i have deployed the changes, something is off with the cdi spec, expect further feedback :)

[adrianchiris]

I have deployed device plugin in cdi mode.

the cdi specification generated is as follows:

cdiVersion: 0.5.0
containerEdits: {}
devices:
- containerEdits:
    deviceNodes:
    - hostPath: /dev/infiniband/issm0
      path: /dev/infiniband/issm0
      permissions: rw
    - hostPath: /dev/infiniband/umad0
      path: /dev/infiniband/umad0
      permissions: rw
    - hostPath: /dev/infiniband/uverbs0
      path: /dev/infiniband/uverbs0
      permissions: rw
    - hostPath: /dev/infiniband/rdma_cm
      path: /dev/infiniband/rdma_cm
      permissions: rw
    - hostPath: /dev/infiniband/issm1
      path: /dev/infiniband/issm1
      permissions: rw
    - hostPath: /dev/infiniband/umad1
      path: /dev/infiniband/umad1
      permissions: rw
    - hostPath: /dev/infiniband/uverbs1
      path: /dev/infiniband/uverbs1
      permissions: rw
    - hostPath: /dev/infiniband/rdma_cm
      path: /dev/infiniband/rdma_cm
      permissions: rw
  name: "0"
- containerEdits:
    deviceNodes:
    - hostPath: /dev/infiniband/issm0
      path: /dev/infiniband/issm0
      permissions: rw
    - hostPath: /dev/infiniband/umad0
      path: /dev/infiniband/umad0
      permissions: rw
    - hostPath: /dev/infiniband/uverbs0
      path: /dev/infiniband/uverbs0
      permissions: rw
    - hostPath: /dev/infiniband/rdma_cm
      path: /dev/infiniband/rdma_cm
      permissions: rw
    - hostPath: /dev/infiniband/issm1
      path: /dev/infiniband/issm1
      permissions: rw
    - hostPath: /dev/infiniband/umad1
      path: /dev/infiniband/umad1
      permissions: rw
    - hostPath: /dev/infiniband/uverbs1
      path: /dev/infiniband/uverbs1
      permissions: rw
    - hostPath: /dev/infiniband/rdma_cm
      path: /dev/infiniband/rdma_cm
      permissions: rw
  name: "1"
kind: rdma/net-rdma

the list of devices goes on up until rdmaHcaMax num is reached.

A couple of comments on this one:

1. the device kind is composed of vendor prefix and device class. in our case we should use nvidia.com as vendor prefix.
   later on it may be configurable if desired. currently, resource prefix is used as vendor and the default is rdma. this does not really fit in CDI kind.

2. device class used in this PR net-rdma fits well.

3. devices here are eventually PCI devices. so i think CDI device spec should reflect the actual device, its name should be the PCI device address. having duplicate devices (notice also there is duplicate mounts per device) here does not make much sense since eventually we can provide the same CDI device name in multiple device plugin Allocate calls. that means, we would a CDI device per PCI device selected. on Allocate call we may return either one or multiple annotations pointing to one or several CDI devices depending on the actual devices that are part of the pool.

4. current code does not support more than one resource pool since you are writing to the same file /var/run/cdi/rdma.yaml
   filename can be a combination of resource prefix + resource pool name. this will ensure uniqueness on the node as prefix+pool should be unique and a device may only be part of a single pool.

example of a cdi spec file as i envision it:

a resource pool with 2 devices:

cdiVersion: 0.5.0
containerEdits: {}
devices:
- containerEdits:
    deviceNodes:
    - hostPath: /dev/infiniband/issm0
      path: /dev/infiniband/issm0
      permissions: rw
    - hostPath: /dev/infiniband/umad0
      path: /dev/infiniband/umad0
      permissions: rw
    - hostPath: /dev/infiniband/uverbs0
      path: /dev/infiniband/uverbs0
      permissions: rw
    - hostPath: /dev/infiniband/rdma_cm
      path: /dev/infiniband/rdma_cm
      permissions: rw
  name: "0000:03:00.0"
- containerEdits:
    deviceNodes:
    - hostPath: /dev/infiniband/issm1
      path: /dev/infiniband/issm1
      permissions: rw
    - hostPath: /dev/infiniband/umad1
      path: /dev/infiniband/umad1
      permissions: rw
    - hostPath: /dev/infiniband/uverbs1
      path: /dev/infiniband/uverbs1
      permissions: rw
    - hostPath: /dev/infiniband/rdma_cm
      path: /dev/infiniband/rdma_cm
      permissions: rw
  name: "0000:03:00.1"
kind: nvidia.com/net-rdma