![Page 2: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/2.jpg)
The current state ofVGA assignment
![Page 3: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/3.jpg)
VGA assignment defined:Graphics card assigned as primary graphics for the VMUses VGA BIOS for initialization and runtime services
![Page 4: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/4.jpg)
Status:Works for discrete graphics cards
AMD Radeon (HD5xxx+)NVIDIA GeForce (8-series+)
Requires experimental vfio-pci optionx-vga=on
Requires working VGA arbitrationStill some device specific glitches
BSODs with AMD Catalyst + HDMI audioReset issues on some Radeon cardsCode 43 on GeForce
![Page 5: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/5.jpg)
Issues:Requires experimental vfio-pci option, x-vga=on
Not supported by libvirt because experimentalExperimental because...
Requires working VGA arbitrationDrivers don't participate (vgacon)Drivers lie (i915)Xorg DRI can't handle multiple arbitration participants
![Page 6: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/6.jpg)
Can't we fix these?Drivers don't participate
Drivers lie
Xorg DRI can't handle multiple arbitration participants
fixable
Once upon a time...i915 hardware allowed control of VGA resources
nobody noticed?!i915 continues to opt-out anyway
But if we fix that...
designers dropped this feature
![Page 7: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/7.jpg)
“WE DO NOT BREAK USERSPACE!”-Linus Torvalds
Start with fixing XorgDRI wants to mmap VGA MMIO spacePossible options...
Provide a new VGA arbiter interface allowing mmapsRemove the mmap requirement
But what about compatibility?
![Page 8: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/8.jpg)
VGA assignment status:
Stuck
![Page 9: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/9.jpg)
What else could we do?
![Page 10: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/10.jpg)
Do we really need VGA?
![Page 11: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/11.jpg)
Secondary graphicsEmulated VGA for VM primary displayAssigned graphics card for secondarySupported by NVIDIA
Requires K-series Quadro/GRID/TeslaSupported in RHEL7.0
Some reports of success with Radeon
Depends on guest driver support
![Page 12: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/12.jpg)
Legacy-free UEFI VMNo VGA required
No VGA arbitration required!
![Page 13: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/13.jpg)
Problem solved?VGA Issues:
Requires "experimental" vfio-pci optionNot supported by libvirt because experimentalExperimental because...
Requires working VGA arbitrationDrivers don't participate (vgacon)Drivers lie (i915)Xorg DRI cannot handle multiple arbitration participants
![Page 14: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/14.jpg)
No Blockers
![Page 15: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/15.jpg)
How do UEFI?
![Page 16: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/16.jpg)
Step 1: OVMFOpen Virtual Machine Firmware
An implementation of UEFI built on the EDK II code baseReplaces Seabios
![Page 17: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/17.jpg)
OVMF Setupvia command line:
via libvirt:
/usr/libexec/qemu-kvm ... \ -drive if=pflash,format=raw,readonly,file=/path/to/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/copy/of/OVMF_VARS.fd
<domain type='kvm'> ... <os> <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/> ... </os> </domain>
![Page 18: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/18.jpg)
Step 2: EFI graphics card ROMPCI ROMs can support multiple imagesMost newer cards already support EFI
![Page 19: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/19.jpg)
Step 3: UEFI compatible guestNewer versions of LinuxWindows 8, Server 2012
![Page 20: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/20.jpg)
That's it!
![Page 21: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/21.jpg)
Other GPU Topics
![Page 22: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/22.jpg)
440FX vs Q35Q35 originally recommended for VFIO-VGA
Largely FUDQ35 looks more like the host platformSome Linux drivers assume an upstream PCIe port
In realityWindows doesn't careLinux is fixable
440FX is likely the easier path for Windows guests
![Page 23: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/23.jpg)
NVIDIA: Code 43Driver detects KVM hypervisor, fails to initialize*
Nvidia - "Accidental" breakage, won't fix, unsupportedWe can't solve it, but we can work around it
Hide the hypervisorvia command line: via libvirt:
*NVIDIA driver version 338.77+
-cpu [type],kvm=off <domain type='kvm'> ... <features> <kvm> <hidden state='on'/> </kvm> ... </features> </domain>
![Page 24: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/24.jpg)
NVIDIA: Code 43 ReduxDriver detects Hyper-V extensions, fails to initialize*
Workaround: Disable use of Hyper-V extensionsPerformance hit for Windows VMs
Have we entered an arms race?
*NVIDIA driver version 344.11
![Page 25: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/25.jpg)
QuirksGeForce:
Radeon:
One still required*
Not needed!
*QEMU now enables this independent of x-vga (2.2)
![Page 26: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/26.jpg)
Unresolved IssuesRadeon device resets
Sea Island GPUs don't reset correctlySMC firmware issue?
Southern Island GPUs indicate PM reset capabilitydoesn't work
Radeon BSODTypically resolved by avoiding audio function
Intel Graphics supportIn progress? KvmGT?
Host/Guest suspendAnd ponies!
![Page 27: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/27.jpg)
ResourcesVFIO tips and tricks blog:
This presentation:http://vfio.blogspot.com
http://awilliam.github.io/presentations/KVM-Forum-2014
![Page 28: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/28.jpg)
Thank you
![Page 29: VFIO, OVMF, GPU, and You - Kernel-based Virtual … OVMF, GPU, and You The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com The current state of VGA](https://reader034.vdocuments.mx/reader034/viewer/2022051602/5aeebba67f8b9a8b4c8b5d01/html5/thumbnails/29.jpg)
Alex Williamson / [email protected]