Guide VFIO Guide of 2020 Intel/Nvidia with All Major Distro Support

Status
Not open for further replies.

ech0

Administrator
Staff member
May 26, 2020
35
0
6
Disclaimer: This guide is my How-to for 2020. I will cover the most common Distros (Ubuntu/Fedora based) as best as I can but this guide was originally built for Arch in mind. With that out of the way lets get started



Introduction:

So first what is KVM/VFIO?

VFIO is way for Linux enthusiasts (or newbies alike) to run a Full Windows machine in a Virtual Machine for Productivity workloads or more common, gaming. This allows for near native/bare-metal performance (~95% performance),


Hardware Requirements:

* CPU MUST Support VT-d. No VT-D no go. You can check for your CPU here:
* Your motherboard must support IOMMU mapping as well as the same Intel VT-d or AMD equivalent.
* You need at least 16 GB of ram. Do not try this with less than 12 GB. You need at least 8 for the windows VM and enough for the host to run on.

* 2 separate GPUS is recommened. While you can do this with 1 GPU, the guide will NOT cover this and I really can not provide support at this time for 1 GPU setups. Don't forget the iGPU on your CPU DOES count. That can be used for the host.


Specs used in this Guide:

Gigabyte Z390 Aurous Master
i7-9700KF @5.0Ghz (No iGPU)
Nvidia GTX 2080 Super (for Win VM)
Nvidia GTX 1030 OC (for Linux VM)
32 GB Titec 3200MHz RAM
ADATA XPG SX8200 Pro 1TB (For Host)
Samsung 970 EVO Plus 1TB] (For Guest)



Gotchas:
1. It is suggested to have a OpenSSH server runnning and Port 22 open on your Machine befor because we will be playing with GPU drivers. In case something goes wrong and you boot up without a GUI you want to be able to ssh in and fix your drivers from another machine or even your phone).
2. Identical GPUs for both Host and Guest WILL bring some issues and requires a few additional steps. This guide does not cover this scenario atm but i can add it if the demand is there.
3. It is better to have an entire SSD dedicated to your VM for performance reasons specifically if you plan on gaming. Avoid Disk images if at all possible. If you must use a Disk image use RAW over QCOW (covered later).
4. VT-D must be supported by your motherboard.


Part 1: Getting Started:

Update your System:

Arch:
$ sudo pacman -Syu
Ubuntu:
$ sudo aptitude update && sudo aptitude upgrade
Fedora:
$ sudo dnf update


Check your IOMMU Groups:

Your IOMMU groups determine how you can route hardware (mainly your GPU) to your VM. Save this script to a file called iommu.sh

Bash:
[ech0@ech0-arch ~]$ nano ~/iommu.sh

...

#!/bin/bash

for d in /sys/kernel/iommu_groups/*/devices/*; do

  n=${d#*/iommu_groups/*}; n=${n%%/*}

  printf 'IOMMU Group %s ' "$n"

  lspci -nns "${d##*/}"

done
Run the scrip via
$ sh iommu.sh

You should get an output similar to the following:
Bash:
IOMMU Group 0:

    00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e30] (rev 0d)

IOMMU Group 1:

    00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0d)

IOMMU Group 10:

    00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)

IOMMU Group 11:

    00:1f.0 ISA bridge [0601]: Intel Corporation Z390 Chipset LPC/eSPI Controller [8086:a305] (rev 10)

    00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)

    00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)

    00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)

    00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)

IOMMU Group 12:

    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 SUPER] [10de:1e81] (rev a1)

IOMMU Group 13:

    01:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)

IOMMU Group 14:

    01:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)

IOMMU Group 15:

    01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)

IOMMU Group 16:

    02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [GeForce GT 1030] [10de:1d01] (rev a1)

IOMMU Group 17:

    02:00.1 Audio device [0403]: NVIDIA Corporation GP108 High Definition Audio Controller [10de:0fb8] (rev a1)

IOMMU Group 18:

    03:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)

IOMMU Group 19:

    04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]

IOMMU Group 2:

    00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 0d)

IOMMU Group 3:

    00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)

IOMMU Group 4:

    00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)

    00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)

IOMMU Group 5:

    00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)

IOMMU Group 6:

    00:17.0 RAID bus controller [0104]: Intel Corporation SATA Controller [RAID mode] [8086:2822] (rev 10)

IOMMU Group 7:

    00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)

IOMMU Group 8:

    00:1b.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0)

IOMMU Group 9:

    00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
If you have a low number of groups or you have several devices in the same IOUMMU group (like 2 of your GPUs sharing the same group) you have poor iommu grouping support. See below how to resolve.

Note: Write down the IDs for the GPU and GPU Audio you want to pass to your Windows VM. You'll need it later. You MUST write down the ID for the GPU audio as well! For example the ID for my 2080 Super is 10de:1e81 and 10de:10f8.
 
Last edited:

ech0

Administrator
Staff member
May 26, 2020
35
0
6
Part 2: Enabling IOMMU Support

What do I do if my iommu grouping is poor?

There are several routes you can take to fix this annoying problem:

AMD Users BIOS Settings:
Code:
Tweaker -> Advanced CPU Settings -> SVM Mode -> Enable
Settings -> Miscellaneous -> IOMMU -> Enable
Settings -> AMD CBS -> ACS Enable -> Enable
Settings -> AMD CBS -> Enable AER Cap -> Enable
If this doesn't fix the grouping issue try updating downgrading your BIOS. Otherwise use the ACS kernel patch Intel users have to use below
Intel Users: Z370/90 boards are fucked. You will have to to use the ACS Override kernel patch. Guide [here](https://forum.nixhaven.com/t/compiling-a-kernel-with-acs-patches-for-vfio/23)

We also need to enable IOMMU on the Linux side of things.
This is where it gets a bit tricky. On 18.04 you have grub by default (at least, my machine did). On 19.04, the boot system uses systemd by default in most scenarios.

For ease of use and simplicity sake for the guide I HIGHLY suggest you use grub2 for your bootloader if you are not and that is what I will cover in this guide.
his is where it gets a bit tricky. On 18.04 you have grub by default (at least, my machine did). On 19.04, the boot system uses systemd by default in most scenarios.
# apt install grub2


Once You have ACS enabled via BIOS or ACS Patches edit your grub line to enable IOMMU on the OS level like so:


Bash:
[ech0@ech0-arch ~]$ nano /etc/mkinitcpio.conf
...
MODULES="vfio_pci vfio vfio_iommu_type1 vfio_virqfd"
...
HOOKS="base udev autodetect modconf block keyboard keymap resume filesystems fsck"
Bash:
nano /etc/mkinitcpio.conf
[ech0@ech0-arch ~]$ nano /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt isolcpus=2-7 nohz_full=2-7 rcu_nocbs=2-7  transparent_hugepage=never rd.driver.pre=vfio-pci vfio-pci.ids=10de:1e81,10de:10f8 pcie_acs_override=downstream,multifunction>
Make sure you ADD this and don't just replace the line. Obviously you will need to use different IDs that your GPU that you want to pass to your VM is using.
AMD users replace intel_iommu=on with amd_iommu=on


Update your ramdisk. Be sure to update for the vfio kernel or the kernel you built the vfio patches for.
$ mkinitcpio -p linux-zen-vfio

Then update grub
Ubuntu:
$ update-grub
Fedora:
$ grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
Arch/Manjaro:
$ grub-mkconfig -o /boot/grub/grub.cfg


So what do each of these options in the Grub line do anyways?

  • isolcpus - Allow klernel to isolate CPU cores away from the host to use just for the VM
  • rcu_nocbs - allow the user to move all RCU offload threads to housekeeping cores
  • transparent_hugepage=never - Turns off transparent huge pages allowing for faster memory access for the VM
 
Last edited:

ech0

Administrator
Staff member
May 26, 2020
35
0
6
Part 3: Setting up QEMU/Permissions

Installing Prereqs
First we will need some software. Start by running the below commands
Ubuntu:
# apt-get install qemu-kvm qemu virt-manager virt-viewer libvirt-bin
Fedora:
# dnf -y install bridge-utils libvirt virt-install qemu-kvm virt-manager
$ pacman install qemu libvirt edk2-ovmf virt-manager

Enable Services:

Ubuntu/Fedora:
# systemctl start libvirtd && sudo systemctl enable libvirtd
Arch/Manjaro
# systemctl enable libvirtd.service virtlogd.socket && systemctl start libvirtd.service virtlogd.socket

Add user to libvirt group for authenication
Ubuntu:
# usermod -a -G libvirt user
Fedora:
# usermod -G libvirt user
Arch/Manjaro
# gpasswd -a user libvirt
 
Last edited:

ech0

Administrator
Staff member
May 26, 2020
35
0
6
Part 4: Setting up Windows VM in Virt-Manager

First a few points/notes before we start with this section of the guide
  • It is best to dedicate your Windows install to a separate External SSD/NVME. This will greatly improve performance. The difference is so enormous in the terms of Disk I/O performance and general latency of your VM it's not even funny. Even a cheap 128 GB SSD for $20-$30 makes all the difference!
  • If you must use a Disk image file then use the RAW format over QCOW format. The performance with QCOW is poor at best and downright unusable at worst.
  • My advice is to dedicate at least 12 GBs of memory to your VM minmum. Preferably 16Gbs. Remember your host OS depending on your Distro/DE will only use a few hundred MBs of RAM once you close all your apps on your host (which you should do anways unless you are isolateing your CPU cores)​

First start by enabling OVMF (UEFI support) for your VM
Bash:
$ sudo pacman -Syyu edk2-ovmf
Now look up where your Distro/system stores the OVMF files. For example on Arch:
Bash:
[user@archlocal]: /usr/share/ovmf/x64>$ ls
OVMF_CODE.fd OVMF_VARS.fd
Edit qemu.conf to point to your OVMF file
Screenshot_OVMFPath.png

Once you edit the file and save it restart libvirt
Bash:
$ sudo systemctl restart libvirtd
  1. Now Open Virt-Manager.
  2. Click New Virtual Machine. Give your VM a name
  3. Choose "Local install media (ISO image or CDROM)"
  4. Browse to your Windows ISO. The type "Windows" (without quotes) for the OS Type/Version fields
  5. Enter in Memory and CPU values. I recommend at least 2 CPU cores for host) (Do NOT allocate ALL cores for VM you'll have performance issues).
  6. Select "Enable storage for this virtual machine"
    • Select Create a disk image on the computer's hard drive
    • If you are passing through a SSD just create a 2 GB image for now to get past the wizard (We will remove it later)
    • Otherwise make your image file at least 30 Gbs is recommended (extra space for downloaded Windows updates is recommended).
    • Forward
  7. Now you should see the final screen. Don't Click Finish yet. For network select your Bridge or NAT Network.
  8. Before you hit Finish check the option "Customize Configuration before install"
  9. On the Overview Section change "Firmware" to "OVMF" and "Chipset" to "UEFI"
  10. Hit Finish
 
Last edited:

ech0

Administrator
Staff member
May 26, 2020
35
0
6
Part 5: Setting up Disk/USB/GPU Passthrough in Virt-Manager
Section A: Installing Windows
  1. Click Edit on your VM.
  2. Click "Add Hardware"
  3. Click Add disk"
  4. Browse to your NVME/SSD drive where you want Windows installed. For arch this was under /Dev for me,​
  5. Select VirtIO as the Disk type.​
  6. Remove Disk Image created during the VM Creation Wizard​
  7. Make sure ISO is mounted under CDROM and make sure it is IDE not VirtIO​
  8. 1591572913499.png
  9. Change the Network Adapter to VirtIO
    1591572863274.png
  10. Click Add Hardware and add a CDROM (IDE) device​
  11. Download the VirtIO drivers here
  12. Mount the ISO to your VM as a second CDROM device
  13. Under boot options check the CDROM where your WIndows ISO is mounted and enable Boot Menu
    1591572802401.png
  14. Now boot your VM and go through the Windows Installation process.​
  15. Under Custom Installation you should not see any hard drives. If you do verify steps 4-6 and make sure you selected VirtIO for the Hard drive type.​
  16. Click "Load Driver" and browse to your VirtIO ISO. (If you don't see it shutdown your VM verify step 11)|
    1591572965541.png
  17. Select the viostor folder/Win10 driver folder
    1591573004533.png
  18. Now repeat 15-16 for the Network Driver as well. The Driver is in the NetKVM folder
    1591573019689.png
  19. Proceed to Install Windows as Normal.​
  20. Once install is complete Windows should reboot in the Install.​
  21. Go through the User Creation Wizard of Windows.​
  22. Once you are on Desktop open File Explorer and open the VirtIO mounted Disk.​
  23. Install the Guest Agent service from the ISO​
  24. Open Device Manager for each device with a triangle right-click > Update Driver > Browse Computer > Browse Button > Select root of VirtIO disc.​
  25. Field should say E:\ or Similar. Make sure "Include Subfolders" is checked. Go through the prompts to install driver.
  26. Repeat for each unknown device,​

Section B: Final VM Config Changes
I will post my full VM XML file below but here are the important changes you need.
  1. First edit your VM. Go to the Overview tab and click XML tab
  2. I will post my full XML below. I will have a post soon breaking down all the different XML options to improve performance/optimizations.
  3. Enable Nvidia Support (Code 43)
    XML:
    <features>
    ...
    <kvm>
    <hidden state="on"/>
    </kvm>
    Code:
    <features>
    ...
    <hyperv>
    <vendor_id state="on" value="anything"/>
    </hyperv>

Full XML Config

XML:
<domain type="kvm">
  <name>win10</name>
  <uuid>2c9806d3-44a8-4239-9d08-4505b40c2440</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16482304</memory>
  <currentMemory unit="KiB">16482304</currentMemory>
  <vcpu placement="static">6</vcpu>
  <iothreads>6</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="2"/>
    <vcpupin vcpu="1" cpuset="3"/>
    <vcpupin vcpu="2" cpuset="4"/>
    <vcpupin vcpu="3" cpuset="5"/>
    <vcpupin vcpu="4" cpuset="6"/>
    <vcpupin vcpu="5" cpuset="7"/>
    <iothreadpin iothread="1" cpuset="0-1"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-4.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <bootmenu enable="yes"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="anything"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" cores="6" threads="1"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S59ANJ0N310683Z"/>
      <target dev="vda" bus="virtio"/>
      <boot order="2"/>
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0xb"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0xc"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0xd"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0xe"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </controller>
    <interface type="direct">
      <mac address="52:54:00:34:88:fc"/>
      <source dev="eno2" mode="bridge"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x0b05"/>
        <product id="0x184d"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x0d8c"/>
        <product id="0x0005"/>
      </source>
      <address type="usb" bus="0" port="2"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc539"/>
      </source>
      <address type="usb" bus="0" port="4"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1852"/>
        <product id="0x7022"/>
        <address bus="1" device="6"/>
      </source>
      <address type="usb" bus="0" port="3"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>
 
Last edited:
Status
Not open for further replies.