![Page 1: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/1.jpg)
1
Towards Virtual Passthrough I/O
on Commodity Devices
Lei Xia, Jack Lange, Peter Dinda{lxia, jarusl, pdinda}@northwestern.edu
Department of Electrical Engineering and Computer Science
Northwestern University
http://v3vee.org/http://www.presciencelab.org/
![Page 2: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/2.jpg)
2
Overview• VPIO: A modeling-based approach to high
performance I/O virtualization for commodity devices– Models could be provided by HW vendors
• Intermediate option between passthrough I/O, self-virtualizing devices, and emulated I/O
• Promising initial results
• Work in progress
2
![Page 3: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/3.jpg)
3
Outline
I/O virtualization techniques Idea of VPIO Palacios VMM VPIO system Device Model Discussion Conclusion
3
![Page 4: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/4.jpg)
4
I/O virtualization – full emulated I/O
4
No guest software change
All New Device Drivers High performance
overhead
[Sugerman01]
![Page 5: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/5.jpg)
5
I/O virtualization – paravirtualized I/O
[Barham03, Levasseur04]
Performance Reuse Device Driver Change guest device
driver
![Page 6: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/6.jpg)
6
I/O virtualization – Passthrough I/O
Native performance Guest responsible for device
driver Specialized hardware
support Self-virtualizing devices
[Liu06, Raj07,Shafer07]
![Page 7: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/7.jpg)
7
I/O virtualization – Direct-Mapped I/O
Reusablity/multiplexing Security Issue
7
Guest OS
VMM
Device
Mapping
![Page 8: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/8.jpg)
8
VPIO Goals
• Achieve safe and secure direct-mapped I/O
• With reusability/multiplexing • To support commodity devices
– No self-virtualized features
• Without losing too much performance– Expect a little more performance
overhead compared to pass-through IO
![Page 9: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/9.jpg)
9
Two Requirements For VPIO
• Inexpensive formal model of device– Building model easier than building device
driver
– Inexpensive to drive such model
• Device can be context-switched
9
![Page 10: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/10.jpg)
10
Core Idea of VPIO
• VMM maintains a formal model of device – keeps track of the physical device status– driven by guest/device interactions– simpler than a device driver/virtual device
• Model determines– Reusable state – whether device is currently
serially reusable– DMA – whether a DMA is about to starting and
what memory locations will be used– Other interesting states
![Page 11: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/11.jpg)
11
Introducing Palacios• New VMM for HPC, architecture, systems, teaching,
and other uses– Fully Open Source, BSD License
• Type I VMM, embeddable into existing kernels• Operating System independent
– Kitten HPC OS (Sandia National Labs)– GeekOS (University of Maryland)
• Implemented using Hardware Virtualization Extensions
• Part of the V3VEE project– Collaboration between NU and UNM– http://www.v3vee.org
• Available at:– http://www.v3vee.org/download 11
![Page 12: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/12.jpg)
12
Palacios Overview• Supports 32 and 64 bit environments
– Hosts and Guests – Currently supports Linux Guests
• Currently uses AMD SVM extensions– partial Intel VT support
• Full hardware virtualization– Does not use paravirtualization– Includes complement of virtual devices
• More details:– J. Lange, P. Dinda, “An Introduction to the Palacios Virtual
Machine Monitor---Release 1.0”, Northwestern University EECS Technical Report NWU-EECS-08-11, November, 2008.
– See us for more info
![Page 13: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/13.jpg)
13
Palacios Details• Virtualization Interface
– Hook IO Ports– Hook Memory Regions– Hook interrupts– Handle VMExits
• Host Interface– Access standard OS facilities
• (debugging, memory allocation)
– Hook host events• Interrupts, timer, keyboard, etc…
• Can use shadow or nested paging
![Page 14: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/14.jpg)
14
Palacios People• Northwestern University
– Jack Lange (Lead Ph.D student)– Lei Xia (Ph.D student)– Peter Dinda (PI, Project Lead)
• University of New Mexico– Zheng Cui (Ph.D student)– Patrick Bridges (PI)
• Others– Trammell Hudson and Kevin Pedreti (Sandia)
![Page 15: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/15.jpg)
15
Guest OS
Unmodified Driver
Device Modeling Monitor (DMM)
Physical Device
Palacios VMM
Hooked I/OUnhooked I/OInterruptDMA
Guest OS
Unmodified Driver
VPIO In Context Of Palacios
![Page 16: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/16.jpg)
16
Current Status
• VPIO is a work in progress
• What is implemented in Palacios– hook I/O ports– hook memory address (byte)
• What is tested outside– Device Model embedded and tested on QEMU
PC emulator version 0.9.1
16
![Page 17: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/17.jpg)
17
Device requests and interrupts
• Guest talks to device by IN/OUT – Memory-mapped I/O will be similar– hooked I/O list, a list of I/O port operations for
read/write or both. VMM intercepts these I/Os– I/O port reads/writes drive model and HW– unhooked I/O list (ideally as large as possible)
• Interrupts– All physical interrupts are handled by VMM– Interrupts drive model and are also delivered to guest
17
![Page 18: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/18.jpg)
18
DMA• DMA is initiated directly on physical device by guest
– DMM is aware of guest’s DMA operations due to hooked ports
– DMA destination physical address is checked before the physical DMA operation is performed
– If validated, let DMA occur, otherwise, deny it.
• Challenge: Dealing with DMA failure– How to notify the guest?
– Ignore the DMA?– Machine Check Exception?
• Challenge: Dealing with physical address translations
18
![Page 19: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/19.jpg)
19
Device Multiplexing
• Context switch between guests– Switching when device in reusable state– Device context (related registers, model)– If not owning device, guest’s operation requests
are suspended
• Challenge: Device handoff on interrupt– Handle incoming packet for NIC– Coming back later
19
![Page 20: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/20.jpg)
20
Cost of VPIO – Experimental Setup
• The performance– Guests’ I/Os performance overhead– Palacios running on Qemu and HP system– Qemu PC emulator 0.9.1– HP ML115 1.8GHz, AMD Opteron 1210
20
![Page 21: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/21.jpg)
21
Issue: Cost of exits (Palacios / AMD SVM)
530
23976 22906
33582
1308 12180
5000
10000
15000
20000
25000
30000
35000
40000
1
Cyc
les
Unhooked I/O -QEMU
Hooked I/O - QEMU
VMM Exit - QEMUDevice Context Switch - QEMU
Hooked I/O - HP
VMM Exit - HP
21
![Page 22: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/22.jpg)
22
VPIO Issue 1/2 : Exit Costs
• Fundamental issue: O(1000) cycle exits.• Try to minimize number of hooked I/O ports
– Model is cheaper in terms of exits than an emulated device
– Not all I/O ports are needed to drive model
22
![Page 23: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/23.jpg)
23
VPIO Issue 2/2 : Model• Can we build such models?
– Is it feasible to build the model?– How easy to build such model, easier than device
driver?
• How expensive are they to run?
23
![Page 24: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/24.jpg)
24
Device Model
• Not for verification, not a complete behavioral model
• Only used to determine…– Whether and when the device is reusable– Whether DMA is to be initiated– What device requests are needed to update model
• State machine, with extra information– Events: device requests, interrupts– Checking functions, triggered by state transition– State transition approve or deny
24
![Page 25: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/25.jpg)
25
Experimental Setup
• Testing setup– Embedded model in QEMU PC emulator version
0.9.1
– Tested model on a set of network applications running in guest OS on Qemu
25
![Page 26: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/26.jpg)
26
Initiation
DmaRd DmaWr
Trans
TransDmaRd
TransDmaWr
Cmd: Stop
AbortDma
Cmd: DmaWr
Cmd: Transmit
Cmd: DmaRd
Cmd: DmaWr
Cmd: Transmit Cmd:
Transmit
INT: RDC
INT: RDC
INT: PTX, PXE
INT: PTX, PXE
INT: PTX, PXE
INT: RDC
Cmd: DmaRd
Cmd: Stop
AbortDmaCmd:
StopAbortD
ma
Cmd: Stop
INT: RDC
AbortDma
Initiate
Idle
NE2K NIC Example
Checking Function
26
Model is small
~700 LOC
![Page 27: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/27.jpg)
27
NE2K Model Performance
27
Only a small fraction of I/Os are needed to drive model
![Page 28: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/28.jpg)
28
Experience with NE2K Model
• Only about 1 in 30 I/O port reads/writes need to be intercepted to drive the model
• Average non-exit cost of updating the model is ~100 cycles– And could be done in parallel with device
execution
28
![Page 29: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/29.jpg)
29
Challenges
• VM exit performance is primary concern– Further reduce I/O operations intercepted– Move model into guest
– Either cooperatively or by code injection
• Handling incoming device input• Network card receive when incorrect guest has
control of NIC
• Hardware manufacturers could provide models along with device drivers
29
![Page 30: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/30.jpg)
30
Conclusions• VPIO: A modeling-based approach to high
performance I/O virtualization for commodity devices
• Intermediate option between passthrough I/O and emulated I/O
• Promising initial results• Work in progress
30
![Page 31: 1 Towards Virtual Passthrough I/O on Commodity Devices Lei Xia, Jack Lange, Peter Dinda {lxia, jarusl, pdinda}@northwestern.edu Department of Electrical](https://reader036.vdocuments.mx/reader036/viewer/2022062318/5518bddb550346881f8b5479/html5/thumbnails/31.jpg)
31
Thanks!
Questions??
[email protected]://www.cs.northwestern.edu/~lxi990/
http://v3vee.org/http://www.presciencelab.org/