diff --git a/DEVICES.md b/DEVICES.md new file mode 100644 index 0000000..35b1985 --- /dev/null +++ b/DEVICES.md @@ -0,0 +1,62 @@ +# Device Management - Note: **Experimental** for Discussion Purposes + +## TLDR + +Devices attached to the host to/from the infrastructure need to be added, +deleted, and connected up on the infrastructure side +(to the dataplane, hypervisor, etc.). For supporting VMs & containers these +devices also need to be connected up on the host side. + +What used to be a simple virtual 'wire' when everything was on the host, now +requires additional coordination in the infrastructure and on the host. + +## Idenfication using PCI + +Given that the infrastructure connects to the host through PCI, +the PCI bus is used to uniquely identify a device. + +Devices can be created in 4 different ways: + +1. **PCI Enumeration**: At enumeration time devices can exist and be recognized +by the host. +2. **PCI Hotplug**: If hotplug is configured in the host and infrastructure, +devices can be added / removed dynamically using the PCI hotplug mechanism. +3. **SR-IOV**: Physical functions on the PCI bus can expose SR-IOV virtual +functions which can be treated as additional PCI devices by the host or by a +guest virtual machine. +4. **Sub-functions**: Sub-functions logically split a PCI device into +multiple independent functions that can be treated as separate devices. +these sub-functions are not exposed onto the PCI bus. + +Given these methods of device creation, a 3-tuple is used to +identify them on the infrastructure side: + +* {PCI Physical Function Handle, Virtual Function Index, Sub-function Index} + +From the perspective of the infrastructure device, it may not be aware of the +B:D.F (Bus:Device.Function) assigned to it on the host side, which is why a +PF handle is used. +A virtual function index of zero represents a PCI Physical Function. +A sub-function index of zero represents a device without sub-functions. + +This 3-tuple is sufficient to support the first 2 cases where the infrastructure +is serving devices to the host directly through PCI, and there is no explicit +coordination between the infrastructure and the host other than devices being +present on the PCI bus. + +## Host/Guest Agent Functionality + +In scenarios where there is coordiination between the host and the +infrastructure (for example when VMs and/or containers need to be associated +with their specific devices) an agent needs to be run in each of the hosts and +guests looking to coordinate devices with the infrastructure. + +Ultimately these agents are taking their direction from orchestration (either +directly or in-directly via the infrastructure). A secure communication +channel is established between the agent(s) and the infrastructure where the +following associations are made: + +* Mapping from Bus:Device.Function, Sub-function -> Device 3-tuple + +Having this mapping on the agent side enables the connectivity of VMs and +containers to the correct device, identified by their 3-tuple.