Virtual Machine with vGPU Unlock for single GPU desktop

This document catalogs my working setup for configuring an Arch Linux host OS to run a Windows 10 VM while passing through a virtualized GPU.

Sources

The resources I used during my own install. This document is a compilation of the instructions and files provided by these people.

Guides

• vGPU Wiki
• PCI passthrough via OVMF
• Proxmox 7 vGPU by wvthoog
• vGPU YouTube guide by Craft Computing
• Looking Glass
• Looking Glass YouTube guide by Pavol Elsig

Sources

• vGPU unlock scripts by DualCoder
• vGPU_LicenseBypass by KrutavShah
• vGPU unlock patch for kernel 5.12 by rupansh

Configuration

The what and how. Mostly described in suggested installation order unless otherwise stated.

Host

Operating system

Configuration of the host OS for virtualization using KVM and device passthrough using OVMF, IOMMU, and VFIO.

Arch Linux

Kernel 5.12.15-arch1-1

Kernel modules

• vfio
• vfio_iommu_type1
• vfio_pci
• vfio_virqfd

How to add

1. Edit /etc/mkinitcpio.conf
2. Add MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...) to the MODULES line.
3. Add HOOKS=(... modconf ...) to the HOOKS line.
4. Update update-initramfs -P (use mkinitpico -P for manjaro)

Boot parameters

• amd_iommu=on
• iommu=pt
• vfio_iommu_type1.allow_unsafe_interrupts=1
• kvm.ignore_msrs=1
• blacklist=nouveau

How to add

1. edit /etc/default/grub
2. Add to GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 blacklist=nouveau ..."

1. Update GRUB update-grub

Hardware

The physical devices present on my system.

• CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz
• GPU: NVIDIA GeForce GTX 1060 6GB
• Memory: 16MiB
• 1440p Monitor

Software

What needs to be installed.

Arch Packages

Packages installed using pacman.

linux

The OS kernel.

• Version: 5.12.15

How to install: First option

1. Install downgrade from AUR (using yay or other AUR helper)
2. downgrade linux
3. Select 5.12.15

How to install: From Arch archive

1. Download package curl -O https://archive.archlinux.org/packages/l/linux/linux-5.12.15.arch1-1-x86_64.pkg.tar.zst
2. Install with pacman -U linux-5.12.15.arch1-1-x86_64.pkg.tar.zst

linux-headers

Needed to build kernel modules, particularly the Nvidia driver.

• Version: 5.12.15

How to install: First option

1. Install downgrade from AUR (using yay or other AUR helper)
2. downgrade linux-headers
3. Select 5.12.15

From Arch archive

1. Download package curl -O https://archive.archlinux.org/packages/l/linux-headers/linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst
2. Install with pacman -U linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst

virt-manager

Used to create and run virtual machines.

libvirt

kvm

(Kernel-based Virtual Machine). A hypervisor. Used for virtualization.

qemu

looking-glass

Used to view VM in graphical display.

iptables-nft

For networking with the VM.

edk2-ovmf

dnsmasq

Used for VM networking.

mdevctl

Used to create vGPU devices

scream

Used to read audio stream from network.

How to setup

1. Install using package manager.
2. Create systemd-file

   [Unit]
   Description=Scream network audio client
   
   [Unit]
   Description=Scream Receiver
   After=pulseaudio.service network-online.target
   Wants=pulseaudio.service
   
   [Service]
   Type=simple
   ExecStartPre=/bin/sleep 3
   ExecStart=scream -i virbr0
   Restart=always
   RestartSec=1
   
   [Install]
   WantedBy=default.target
         

Python packages

Installed with pip.

frida

How to installIf using conda

1. Install python-pip using pacman.
2. Run /bin/pip install frida

Otherwise

1. pip install frida

Graphics setup

Nvidia Driver

Version: 460.73.01 merged and pre-unlocked This driver is a merged version of Nvidia’s vGPU driver and their standard driver. It allows for the host graphics to be displayed while the vGPU’s are active.Components

1. merged driver
2. vgpu_unlock
3. vgpu_unlock_5.12 patch

How to installMake sure all above arch packages and python packages are installed.Download thecomponents using curl -O the_linkMake the driver executable. chmod +x NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5.runExtract it. ./NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5.run -xcd NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5Apply the 5.12 patch patch -p0 < ../twelve.patchDisable graphics sudo systemctl isolate multi-user.targetRe-login as root and navigate back to the driver directory

cd /home/user/Downloads/NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5
  

Install the driver

./nvidia-installer --dkms
  

Add vgpu_unlock to service files.

nano /lib/systemd/system/nvidia-vgpud.service
  

Replace ExecStart with:

ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpud
  

nano /lib/systemd/system/nvidia-vgpu-mgr.service
  

Replace ExecStart with:

ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpu-mgr
  

systemctl daemon-reload
  

Add vgpu_unlock to source files.

nano /usr/src/nvidia-460.73.01/nvidia/os-interface.c
  

Under #include “nv-time.h” insert this line

#include "/root/vgpu_unlock/vgpu_unlock_hooks.c"
  

nano /usr/src/nvidia-460.73.01/nvidia/nvidia.Kbuild
  

Add this to the bottom of the file.

ldflags-y += -T /root/vgpu_unlock/kern.ld
  

Remove and reinstall the Nvidia dkms module

dkms remove -m nvidia -v 460.73.01 --all
dkms install -m nvidia -v 460.73.01
  

Make all of vgpu_unlock executable

chmod -R +x vgpu_unlock

Reboot

vGPU Profiles

• nvidia-48 GRID P40-3Q

How to installChoose profile

mdevctl types
  

I chose nvidia-47 (GRID P40-3Q) because it has 3Q at the end of its name. This means I can have two 3GB vGPU’s running.

Determine GPU PCI address

lspci | grep VGA
PCI_ADDRESS=$(lspci | grep VGA | grep -Po '\d\w:\d\d.\d')
echo $PCI_ADDRESS
  

Generate UUIDs for each vGPU

UUID1=$(uuidgen)
UUID2=$(uuidgen)
echo $UUID1 $UUID2
  

Create vGPUs

mdevctl start -u $UUID1 -p 0000:$PCI_ADDRESS -t nvidia-48
mdevctl start -u $UUID2 -p 0000:$PCI_ADDRESS -t nvidia-48

mdevctl define -a -u $UUID1
mdevctl define -a -u $UUID2
  

VM Image

The virtual machine configuration.

• Chipset: Q35
• Firmware: UEFI x86_64: /usr/share/edk2-ovmf/x64/OVMF_CODE.fd
• Hypervisor KVM
• Spice Server
• Video QXL
• MDEV
• shmem looking-glass

How to install

In virt-manager, create virtual machine.

Apply above settings minus “MDEV” and “shmem looking-glass”.

Start VM and install OS.

Shutdown guest.

Add looking-glass shmem device.

The size is determined like this.

<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>64</size>
</shmem>

Make a file /etc/tmpfiles.d/10-looking-glass.conf

#Type Path               Mode UID  GID Age Argument

f /dev/shm/looking-glass 0660 user kvm -

Remove tablet device

Add MDEV device.

Where $UUID1 is the UUID of one of the mdev vGPU’s created earlier.

<hostdev mode="subsystem" type="mdev" managed="no" model="vfio-pci" display="on">
  <source>
    <address uuid="$UUID1"/>
  </source>
</hostdev>

Install all guest software.

Shutdown guest.

Either set Video model to “none” or install guest Nvidia driver​.

How to run

1. Start VM. (May need to click run several times if error related to “vfio-<bus>” occurs.
2. Run looking-glass client

   export SDL_VIDEO_X11_VISUALID=
   looking-glass-client
       

Guest

Inside the VM.

Operating system

• Windows 10

To download (within guest)

• virtio guest tools
• spice guest tools
• looking glass host
• virtio PCI RAM controller driver
• Nvidia GRID driver
• vGPU_LicenseBypass
• Scream audio driver

Software

Looking Glass Host

How to install

1. Download components.
2. Update driver in Device Manager->System Devicess->PCI standard RAM Controller to virtio PCI RAM controller driver.
3. Install virtio guest tools and spice guest tools.
4. Install looking glass host by running looking-glass-host-setup.exe as administrator.
5. VM.

Nvidia GRID driver

How to install

1. Run Nvidia GRID driver installer.
2. Reboot.
3. In Nvidia Console set resolution. My monitor used 2560x1440.
4. Run vGPU_LicenseBypass.

Scream audio driver

How to install

1. Install driver.
2. Change sound source to Scream (WDM).

My files

• Final QEMU Windows 10 VM image
• mdev device creation script

Suggestions

Since there is a lot of messing with the kernel and graphics drivers, it’s important to have a plan for recovering your host OS.

• Use timeshift or other auto-snapshotting of the host.
• Have a recovery drive available.

Maintenance

Preventing accidental upgrades

In order to avoid accidentally upgrading these carefully chosen package versions, make sure to ignore updates to them:

IgnorePkg = linux linux-headers nvidia-dkms nvidia nvidia-utils nvidia-settings opencl-nvidia libxnvctrl