Passthrough enhancement and development discussion

[HikariKnight]

Introduction

Hello, I am the developer of Quickpassthrough (name still up for discussion) and the developer of ls-iommu, i am currently testing a major rework of quickpassthrough which is yet to be merged into the main branch.
However due to the complexity and quirkiness of VFIO and passthrough setups, consider this alpha stage software, use at your own risk.

NOTE: Laptops will never be officially supported as they are even harder to get properly working compared to desktops (plus the majority of them use 3D Controllers instead of VGA Compatible Controllers, which only the latter tend to support passthrough), See ls-iommu issue 3 and lantian's excellent write up about how bad 3D controller passthrough is.

The project was started when @flexiondotorg mentioned on stream last year that he would be interested in adding passthrough support to quickemu when we started talking about gpu passthrough, however i explained it would be hard without first simplifying the host configuration setup (which is where quickpassthrough comes in).
The proof of concept bash scripts have been tested on multiple systems for about 1 year now to good success, so i have moved to start making something more maintainable.

Currently quickpassthrough in the dev branch can setup and bind a spare GPU to the vfio_pci driver on systems using either:

• initramfs-tools
• dracut
• mkinitcpio
• modprobe

NOTE: this requires you to have 2 gpus (either 2 dedicated gpus or 1 igpu and 1 dedicated gpu, however Intel Arc gpus have not been tested as i lack one and i have not found anyone willing to test with an arc gpu)

it supports editing grub2 (/etc/default/grub), grubby and kernelstub when it comes to adding kernel arguments to the bootloader (if neither is detected it will spit out the arguments you need to add).

Anyways, i have gotten to the point in the development that i will be needing some info about how and what quickemu would like quickpassthrough to prepare for it (and maybe start exploring the thought of getting support for that implemented in quickemu)

Things that needs to be done in quickemu to have passthrough working

Not that much honestly if all we want is to passthrough a gpu.

• looking-glass might have to be included in the virtio drivers on windows if passthrough is enabled (they are working on a potential IDD which will replace the host application), otherwise vnc will have to be configured to make the VM accessible while it is being set up
• VNC should be setup on the windows VM so that it can be reached and controlled as the spice display will not work properly while a GPU is passed through.
• Make sure that no more than 1 passthrough VM can be active at a time.

This means at minimum quickemu will just have to make sure it cannot run 2 VMs using vfio at the same time and then you can bind the GPU of your choice to vfio_pci (manually or using quickpassthrough) and just add something like -device vfio-pci,host=0000:09:00.0,romfile=/usr/share/kvm/vfio_rx570.rom -device vfio-pci,host=0000:09:00.1 to the extra_args in the VM config and then run the VM as sudo or have the user in a correctly configured group (im a libvirt user so i dont really know the exact permissions needed but i would assume being in the kvm group or similar would be enough?).

If we want to be able to pass through other pci devices too like an USB controller (from the motherboard or pcie slots) in order to be able to use game controllers or other devices that require the whole USB controller to be sent to the VM, then we need quickemu to be able to unbind the ones the user want to use in the VM before the VM starts, then bind them back to the system once the VM is shut off.

What about single gpu passthrough (since i know this will be asked about)

NOTE: I have removed this from my scope due to complexity, i have left the original text below, in case it proves useful for people.

This requires a completely different configuration setup than dual GPU passthrough (1 gpu for host, 1 gpu for VM), so much so that quickpassthrough does not handle this yet, and i am planning to only support "standardized" setups (ex: full DEs only and systemd), if the user wants to run it on anything beyond that, they will have to edit the generated scripts themselves.

If we want to go down this rabbit hole of single GPU passthrough then quickemu will need the ability to stop the display manager (gdm, lightdm, sddm, etc) along with xorg and/or wayland... THEN run a helper script to prepare the single GPU for passthrough and then run the VM once all of that is shut off and prepared. Then do it all in reverse when the VM is shut down.
The current problem is that if quickemu attempts this now, it will close the session which quickemu is running in, thereby killing the process before any GPU gets unbound and any VM starts 😅.

As an example this is my Pre-Start VM hook script for libvirt when testing single GPU passthrough (on a system where the gpu is actually being used on the host), the release (VM shutdown) version is literally everything in reverse. libvirt uses a daemon of sorts to run these scripts before a VM starts and after a VM shuts down.

#!/bin/bash
# Helpful to read output when debugging
set -x

# Source the config file with the pci addresses
source /etc/libvirt/hooks/vfio.conf

# Stop the display manager, WM and other services so they no longer using the GPU (preventing it from being unbound)
systemctl stop display-manager.service
# Uncomment the following line if you use GDM and/or a window manager (replace awesome with your WM)
killall gdm-x-session
killall -9 awesome

# Replace hk with your username (this script does not run as $USER hence the hardcoded username)
pulse_pid=$(pgrep -u hk pulseaudio)
pipewire_pid=$(pgrep -u hk pipewire-media)
kill $pulse_pid
kill $pipewire_pid


# Unbind VTconsoles (change these to match your vtconsoles)
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind

# Unbind EFI-Framebuffer (uncomment if you use an nvidia card)
#echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

# Avoid a Race condition by waiting 5 seconds. This can be calibrated to be shorter or longer if required for your system
sleep 5

# Unload amdgpu
modprobe -r amdgpu

# Unload nvidia driver (uncomment if you are passing through an nvidia card)
#modprobe -r nvidia
#modprobe -r nvidia_drm
#modprobe -r gpu_sched
#modprobe -r ttm
#modprobe -r drm_kms_helper
#modprobe -r i2c_algo_bit
#modprobe -r drm
#modprobe -r snd_hda_intel

# Unbind the GPU from display driver (all this does is unbind the pci device to my knowledge)
virsh nodedev-detach pci_$DEVICE_VIDEO
virsh nodedev-detach pci_$DEVICE_AUDIO

# Load VFIO Kernel Module  
modprobe vfio
modprobe vfio_pci
modprobe vfio_iommu_type1 

This alone might be out of the scope of what quickemu is trying to accomplish if i were to be guess.

The config file (DRAFT)

Currently my draft of a vfio.conf file that can be added into a VM folder, looks like this (none of this is final and i want some input on which approach quickemu devs think are the better one)

GPU_PCI_ID=(pciaddr1 pciaddr2 pciaddr3...)
USB_CTL_ID=(pciaddr1 pciaddr2...)
GPU_ROMFILE="romfile.rom"

I do feel this is would be the more correct config to use.
However in this scenario, quickemu will have to generate the correct -device arguments for qemu and figure out if a romfile will be used in the args and which device to attach the romfile to.

Out of these only GPU_PCI_ID is mandatory. However a different (and better?) approach might be to write it like this

VFIO_ARG="-device vfio-pci,host=0000:09:00.0,romfile=/usr/share/kvm/vfio_rx570.rom -device vfio-pci,host=0000:09:00.1"
OTHER_PCI_IDS=(pciaddr1 pciaddr2 pciaddr3...)

In this file, the vfio arguments for qemu have already been generated by quickpassthrough and all quickemu will have to do is apply them to the launch line for the VM and then unbind the OTHER_PCI_IDS before starting the VM (and then rebind them when the VM shuts down), add -device vfio-pci,host=0000:XX:YY.Z to the qemu arguments and start the VM.

Other considerations will be if this conf file should reside in every VM folder you want passthrough to be enabled of if this shall be some generic config file that quickemu reads and only use on VMs that have something like VFIO=yes in their conf file?

PS: sorry for the long post but there is a lot of details required for all of this and i want to see what scope quickemu will be interested in going for, if there is anything that does not make sense for you or you need me to elaborate on then do not hesitate to ask me as i do have dyslexia and might have written it weirdly 😅