-
Notifications
You must be signed in to change notification settings - Fork 95
GVTd_Setup_Guide
- 1 Introduction
- 2 System Requirements
- 3 Host Environment Setup
- 4 Guest Setup
- 5 Basic usage
- 6 Features Supported
Intel GVT-d is one flavor of graphics virtualization approaches also names as Intel-Graphics-Device pass-through feature. It is based on Intel VT-d technology and some special graphics related configuration than NIC pass-through. This flavor allows direct assignment of an entire GPU’s prowess to a single user, passing the native driver capabilities through the hypervisor without any limitations.
Ubuntu 20.04 has been fully validated as host, other Linux operating system like RHEL with 4.X kernel is also OK.
For client platforms, from 5th to 11th is supported. For server platforms, E3_v4, E3_v5, or E3_v6 Xeon Processor Graphics is validated.
Use Ubuntu as an example, there are some software packages needed by host environment setup, as below:
apt-get install git libfdt-dev libpixman-1-dev libssl-dev vim socat libsdl1.2-dev libspice-server-dev autoconf libtool xtightvncviewer tightvncserver x11vnc libsdl1.2-dev uuid-runtime uuid uml-utilities bridge-utils python-dev liblzma-dev libc6-dev
Kernel: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
QEMU: git://git.qemu.org/qemu.git
Xen : git://xenbits.xen.org/xen.git
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-stable
git checkout master
With the default kernel, Linux VM will trigger GPU hang/GPU reset while running 2D /3D application, which is a known issue with hypervisor KVM, so we can add the disable stolen memory patch to resolve this problem (If you don't want to add this patch to kernel, you also can enable stolen memory patch in Qemu side, as section 3.3).
echo ""|make oldconfig
make -j8 && make modules_install && make install
Please follow chapter 3.3.1 for GVT-d KVM setup and chapter 3.3.2 for GVT-d Xen setup.
git clone git://git.qemu.org/qemu.git
cd qemu
git checkout master
git revert 93587e3af3a // enable stolen memory patch if you didn't do it in kernel side
./configure --prefix=/usr \
--target-list=x86_64-softmmu
make -j8
make install
git clone git://xenbits.xen.org/xen.git
cd xen
git checkout master
./configure
a. In order to run GVT-d well in 5th Gen or E3_v4 Xeon platform, we need to edit the rombios file with below changes:
tools/firmware/rombios/rombios.c
mov ax, #0xc780
call rom_scan
+ mov ax, #0x03
+ int #0x10
+
call _print_bios_banner;
make xen -j8
make tools –j8
b. In order to GVT-d run well in 6th / 7th Gen or E3_v5 / E3_v6 platform, we need to edit the qemu-xen-tradtional-dir file with below changes:
tools/qemu-xen-traditional-dir/hw/pt-graphics.c
+memset(bios, 0 ,bios_size);
cpu_physical_memory_rw(0xc0000, bios, bios_size, 1);
Then you need to build tools again.
make tools -j8
make install-xen
make install-tools
You need to add the GVT-d KVM and GVT-d Xen corresponding menu entry in your grub.cfg, or you can add it to /etc/default/grub is better. (Since grub.cfg will be auto regenerated by Linux distro version upgrade or kernel package update).
For GVT-d KVM, Make sure the "intel_iommu=on" exist in your menuentry, below is an example on ubuntu:
menuentry 'GVT-d KVM' --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 106518cd-fb74-4048-81de-3ecee7ad74d4
else
search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
fi
linux /boot/vmlinuz-4.17.0+ root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel log_buf_len=128M console=ttyS0,115200,8n1 kvm.ignore_msrs=1 intel_iommu=on
initrd /boot/initrd.img-4.17.0+
}
NOTICE: Since some Windows guest 3rd party applications/tools (like GPU-Z / Passmark9.0) will trigger MSR read/write directly if it accesses the unhandled MSR register, the guest will trigger BSOD soon. So we added the "kvm.ignore_msrs=1" into grub for workaround.
For GVT-d Xen, make sure the "modprobe.blacklist=i915 xen-pciback.passthrough=1 xen-pciback.hide=(00:02.0)" exist in your menuentry, below is an example on ubuntu:
menuentry 'GVT-d Xen' - --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 106518cd-fb74-4048-81de-3ecee7ad74d4
else
search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
fi
multiboot /boot/xen.gz dom0_max_vcpus=2 dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=all msi=1 conring_size=4M console=com1 com1=115200,8n1 sync_console
module /boot/vmlinuz-4.17.0+ root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel intel_iommu=on log_buf_len=128M console=hvc0 modprobe.blacklist=i915 xen-pciback.passthrough=1 xen-pciback.hide=(00:02.0)
module /boot/initrd.img-4.17.0+
}
You can prepare any version Ubuntu or any other Linux guest OS image manually. Then you can change the guest kernel to the same kernel as host, you also can download the stable version from https://www.kernel.org/ or use Ubuntu18.04 guest OS kernel directly.
-
We need to make the special X11 configuration change for Linux guests to use the Intel graphics card as a display card.
-
The emulated graphics card (like "QXL", "Cirrus")not support OpenCL / OpenGL hardware acceleration, we must make this change.
-
Here we use Ubuntu as an example, we save the following section at /etc/X11/xorg.conf file:
Section "Device" Identifier "intel" Driver "intel" BusID "PCI:0:4:0" # Sample: "PCI:0:2:0" EndSection Section "Screen" Identifier "intel" Device "intel" EndSection
BTW: The PCI ID was chosen by looking at the output of the "lspci" command.
root@gvtd-host:~# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
00:04.0 VGA compatible controller: Intel Corporation Broadwell-U Integrated Graphics (rev 0a)
NOTICE: After you change the display card to the Intel graphics card, Qemu Windows will show black screen, you only can see the guest desktop by a remote protocol.
We suggest using the X11VNC as the remote protocol to validation which has been validated by us.
Use Ubuntu as an example, to support X11VNC, you must install the xhost in guest first, the xhost program is used to add and delete host names or user names to the list allowed to make connections to the X server, you must make some configuration for lightdm, the example as below:
root@gvtd-host:~# cat /etc/profile.d/lightdm_display.sh
#!/bin/bash
export DISPLAY=:0
if [ ! -z "$DISPLAY" ]; then
xhost +local:
fi
root@gvtd-host:~# chmod +x /etc/profile.d/lightdm_display.sh
root@gvtd-host:~# cat /etc/lightdm/lightdm.conf
[SeatDefaults]
greeter-session=unity-greeter
user-session=ubuntu
greeter-setup-script=/etc/profile.d/lightdm_display.sh
session-setup-script=/etc/profile.d/lightdm_display.sh
After you complete the lightdm configuration, you can run the X11VNC service in guest by following commands or similar commands:
x11vnc -display :0 -forever -bg -repeat -nowf -o ~/.vnc/x11vnc.log -rfbport 5900
Then connect to the guest by the host with guest X support by typing the following command:
vncviewer $guestIP
NOTICE: $guestIP means guest IP address, you can get the guest IP address by "nmap" / "arp-scan" or any other network-related tools in the host.
Except for X11VNC, there are some other alternatives, which were not validated by us, you also can use it as a remote protocol for a try:
- NX
- X2Go and based on nx-libs (it seems QVD uses it too)
- QVD (open source version)
- NoMachine: Closed source.
- OpenNX. Client only. Not updated since 2016-10-01.
- VNC
- TurboVNC
Win7-32 / Win7-64 / Win8.1-64 /Win10-RS4-64 are validated.
-
After you boot up Windows guest, you can see two graphics card in "Device manager"
-
The virtual GFX card "QXL" or "Cirrus"
-
The Intel GFX card
-
Then you must install the Intel GFX driver for the Intel Graphics card to support OpenCL / OpenGL / DirectX9 / DirectX10/ DirectX11 / DirectX12 hardware acceleration .
-
You can get the Windows driver from graphics drivers section in the Intel website. Depending on your Intel processor, the latest available driver versions as below:
Kabylake & Skylake & Coffeelake & newer platform: Win10-64: 30.0.100.9667
Skylake Win7-32 / Win7-64 / Win8.1-64: 15.45.31.5127
Broadwell: 15.40.45.5126
NOTICE: After you install Windows GFX driver, you can see guest desktop in both QEMU and remote protocol. QEMU displays via the emulated driver and the remote protocol displays via the Intel GFX driver. BTW, There is a known issue that some 3D workload cannot run with hardware acceleration while both GFX card is available, so suggest to disable the emulated GFX card in "Device Manager" after you make sure the Intel GFX card is working.
We suggest using Tightvnc as the remote protocol to validation which has been validated by us.
Then connect to the guest by the host with guest X support by typing the following command:
vncviewer $guestIP
NOTICE: $guestIP means guest IP address, you can get the guest IP address by "nmap" / "arp-scan" or any other network-related tools in the host.
All Windows Remote protocols are supported, such as TightVNC, HP RGS, RDP.
Except for Tightvnc, there are some other alternatives, which were not validated by us, you also can use it as a remote protocol for a try:
- VNC (Virtual Network Computing)
- HP's RGS (Remote Graphics Software), which is an enterprise-level application.
- Microsoft's RDP (Remote Desktop Protocol).
- FreeRDP: The best Open Source RDP implementation.
- Remmina (GTK+ 3): Based on FreeRDP. It supports the RDP, VNC, NX, XDMCP and SSH protocols.
- WinConn: Promising but abandoned project. Just put here as a reference, not suggested to end-users (the readers of this document).
- PAC Manager
- KDE's KRDC (Remote Display Client): It supports both VNC and RDP protocols.
- Gnome's Vinagre
- NX
root@gvtd-host:~# modprobe vfio
root@gvtd-host:~# modprobe vfio_pci
root@gvtd-host:~# lspci -D -nn
0000:00:00.0 Host bridge [0600]: Intel Corporation Broadwell-U Host Bridge - DMI [8086:1614] (rev 06)
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1622] (rev 08)
Thus it can be seen, VGA device (0000:00:02.0,vendor & device ID 8086:1622)
root@gvtd-host:~# echo 0000:00:02.0 > /sys/bus/pci/devices/0000:00:02.0/driver/unbind
root@gvtd-host:~# echo 8086 1622 > /sys/bus/pci/drivers/vfio-pci/new_id
root@gvtd-host:~# xl pci-assignable-add 00:02.0
root@gvtd-host:~# xl pci-assignable-list 0000:00:02.0
For GVT-d KVM, in order to let guests using the same network segment as host, you must create the network bridge. Besides, in order to let Qemu using the network bridge you created, you also need to create a script as below (We saved it at /etc/qemu-ifup):
For details you can reference the the "Configuring Guest Networking" wiki page http://www.linux-kvm.org/page/Networking
For GVT-d Xen you can just create the network bridge by yourself.
We have two methods to create VM, one is UPT mode (guest cannot local display in the monitor), the other is legacy mode (guest can local display in monitor), the "legacy mode" was recommended.
The example of creating VM script as below :
UPT mode:
#! /bin/bash -x
/usr/bin/qemu-system-x86_64 \
-enable-kvm \
-m 2048 \
-smp 2 \
-cpu host \
-hda /home/img/win10-64.qcow2 \
-vga cirrus \
-device vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x6 \
-usb -usbdevice tablet \
-net nic,macaddr=00:16:3e:60:0a:50 -net tap,script=/etc/qemu-ifup \
-daemonize
Legacy mode:
#! /bin/bash -x
/usr/bin/qemu-system-x86_64 \
-enable-kvm \
-m 2048 \
-smp 2 \
-cpu host \
-hda /home/img/win10-64.qcow2 \
-vga none -nographic \
-device vfio-pci,host=00:02.0,x-igd-gms=2,id=hostdev0,bus=pci.0,addr=0x2,x-igd-opregion=on \
-usb -usbdevice tablet \
-vnc :2 \
-net nic,macaddr=00:16:3e:60:0a:50,addr=0x3 -net tap,script=/etc/qemu-ifup \
-daemonize
You can create hvm configuration file by modifying /etc/xen/xlexample.hvm like below:
builder = "hvm"
name = "HVM_IGD_PASSTHROUGH"
memory = 2048
vcpus=4
vif = [ 'type=ioemu, mac=00:16:3e:30:e0:6c, bridge=xenbr0' ]
disk = [ '/home/img/win10-red-stone.img,raw,hda,rw' ]
device_model_override = '/usr/local/lib/xen/bin/qemu-dm'
device_model_version = 'qemu-xen-traditional'
sdl=0
vnc=1
# default rdm_mem_boundary is 2G
rdm_mem_boundary=700
rdm="strategy=host,policy=strict"
pci=[ "00:02.0,rdm_policy=strict" ]
gfx_passthru=1
usb=1
usbdevice='tablet'
After that, you can create XenGT VM via the below command:
xl create xlexample.hvm
- General:
- Supported Linux distributions (64bit): Ubuntu, RHEL, SUSE, etc
- Supported Windows guest: Windows7 (64bit), Windows8.1 (64bit), Windows10 RedStone (64bit)
- Single display with resolution up to 1920x1080p
- 3D/2D:
- OpenGL4.5 for Linux guest
- OpenGL4.4 for Windows guest
- DirectX9, 10, 11, 12 for Windows guest
- Compute:
- OpenCL2.0 for Windows/Linux guest
- Media:
- Intel Media Service Studio (2015R5)
- Remote display protocol:
- X11VNC for Linux guest
- TightVNC, HP RGS, RDP for Windows guest
- 3D / 2D:
- 3Dmark06, 3Dmark11, WebGL, Passmark, Unigine-tropics, Unigine-heaven, Phoronix 3D, Cairo 2D, etc
- Media:
- Multiple thread decode/transcode for JPEG, MPEG2, H264, HEVC, VC1, VP8, etc
- OpenCL:
- LuxMark, Beignet