updating xen for the client environment · 2011-02-28 · 11 12/3/07 •xen hypervisor loaded at...
TRANSCRIPT
Updating Xen for the ClientEnvironment
Junhong Jiang, Kevin Tian,Chris Wright, Don Dugger
12/3/072
Legal Content
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTIONWITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'STERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTELASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMSANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALEAND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY ORWARRANTIES RELATING TO FITNESS FOR A PARTICULARPURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANYPATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTYRIGHT.
Intel may make changes to specifications, product descriptions,and plans at any time, without notice.
Intel is a trademark of Intel Corporation in the U.S. and othercountries.
Copyright © 2007, Intel Corporation. All rights are protected.
12/3/073
Agenda
• Xen Client Overview
• Boot Scheme/Verified Launch
• Device Virtualization
• Power Management
12/3/074
Glossary
• VA Virtual Appliance
• Xen Client is a platform for implementing VAs
• UOS User Operating System/User OS/Primary User Partition
• SOS Service OS/VA Partition
• VM Virtual Machine/Guest Partition
• VMM Virtual Machine Monitor/Hypervisor
12/3/075
Xen Client
• Xen based virtual appliance infrastructure Limited device virtualization/isolation support
• Single SOS Linux kernel (Para-virtualized) No direct disk access at run time
• Permanent storage provided by UOS• Exception during boot process
•Dom0 provides disk access
• Target platform TXT support
• Measure and authenticated boot• Trust chain from BIOS
VT-d Support
• Virtualizes NIC TPM
12/3/076
Architecture Overview
• VMM Xen hypervisor
• 64 bits Dom0
• Para-virtualized Linux.• Runs out of RamFS• Minimal components
•Control Panel, device models, default power management policy, virtualappliance specific agents
• Headless•Capable of booting messages
• UP
• Service OS Paravirtualized XenLinux
• Fully virtualized possibility for future Headless
• Boot up display for SOS in Dom0 Survive even UOS hangs Support single or multiple SOS’s (At least one)
12/3/077
Architecture Overview (Cont.)
• SOS0 (LAN filtering) LAN Filtering
• Owns physical NIC• Provide VNIF BE driver
SMP capable
• SOS1 (VoIP) Own Audio SMP Capable
• User Operating System Near native performance Own rest physical resources (CPU, memory, devices) Virtual LAN
• VNIF front-end (FE) driver SMP Windows XP SP2 (32-bit PAE w/ XD support) Vista (32 and 64-bit) Windows PE Linux
12/3/078
Xen Client Architecture OverviewV
eri
fied
boot
vP
RO
Clie
nt C
fg./
Lau
nch
Pro
vis
ion
/In
stall
Ag
en
t P
rese
nce
Dom0Dom0
Vt-x LT VT-d EM64T Channel ACPI/PM
Shadow PT PIC/PIT Models
Xen hypervisorXen hypervisor
SOSSOS
Netw
ork
Dri
vers
VA
Serv
ices
Ag
en
ts
VA
Dri
vers
VA
Pro
vis
ion
ing
Ag
en
ts
VA
N/W
Rou
tin
g S
rvc.
VA
Sto
rag
e S
erv
ices
Oth
er
VA
Dri
vers
ISV Apps.ISV Apps.
UOSUOS
User Apps.User Apps.
VA
Dri
vers
VA
Serv
ices
Ag
en
ts
Nati
ve D
rivers
VA
Pro
vis
ion
ing
Ag
en
ts
VA
Sto
rag
e S
erv
ices
Oth
er
VA
Dri
vers
TPM HW LAN/WAN SATA, USB, etc.
vTP
M
12/3/079
Disk Layout
• Reserved SOS partition Type 0x71 Pointed to by MBR Contains boot/persistent storage file system
• Boot FS in SOS partition Contains grub loaded modules
• Xen kernel image• Dom0 vmlinuz• Dom0 initrd
SOS modules are not loaded by grub• Dom0 mounts the boot FS• SOS kernel/initrd loaded by domain builder in Dom0
12/3/0710
Disk layout
Part 3
Part 2
Part 1
Part 4 (type = 71)
UOS MBR
Boot Block
UOS Partition
SOS Boot Block
PartitionTableMaster Boot
RecordBoot code
PhysicalDisk
UOSusableDisk
SOSpartition
∞
0
SOS Partition
Part 3
Part 2
Part 1
Part 4 (type = 71)
12/3/0711
• Xen hypervisor loaded at fixed memory Affects minimum memory required
• Dom0 allocated at top of memory First partition loaded
• SOS’s loaded sequentially below Dom0
• UOS loaded at low memory Allows 1-1 guest physical to machine physical mapping
• Needed to execute platform BIOS, especially SMI’s Size of UOS dynamically determined
• Max memory minus fixed size for Xen, Dom0 & SOS’s
• Implications: UOS given contiguous machine physical address
• Hole in memory for Xen Xen hypervisor memory allocation change
Memory Layout
12/3/0712
Physical RAM layout – initial boot
0
640K
1M
256M
Max
BIOS
Grub Modules
Dom0 initrd
Dom0 image
Xen VMM image
12/3/0713
Physical RAM layout - runtime
0
640K
1M
256M
Max
BIOS
Xen VMM image
Dom0 RAM
SOS RAM
UOSgpfn == mpfn
UOSgpfn == mpfn
12/3/0714
Boot sequence – 4 phases
1. Grub loads Xen & Dom0 into memory
2. Dom0 starts
3. SOS starts
4. UOS starts
12/3/0715
Persistent Storage
• SOS runs out of RAM Needs access to persistent storage on disk
• SOS run a disk Front End driver Talks to disk Back End driver in Dom0 during boot Unmounts disk from Dom0 Shuts down disk FE from Dom0 Starts disk FE, now talking to UOS Remounts disk
12/3/0716
Verified Launch – Dom0Dom0 (Linux)
Standard Linux Drivers(VGA, Chipset NV, Disk)
Device Model(PCI Config,
Serial, ICH,..)
Domain Builder(xm/libxc
equivalent)
CMFM
InterVMcommunication driver
vTPM
VM PowerManager
Back-end diskdriver
Xen hypervisorXen hypervisor
SATA, USB, etc.Shared HWUser OptimizedHW
TPMDriver
Kern
elU
ser
12/3/0717
Verified Launch – SOS
SATA, USB, etc.Shared HWUser OptimizedHW
Dom0 (Linux)
Standard Linux Drivers(VGA, Chipset NV,
Disk)
Device Model(PCI Config,
Serial, ICH,..)
Domain Builder(xm/libxc
equivalent)
CMFMvTPM
VM PowerManager
BE diskdriver
Xen hypervisorXen hypervisor
TPMDriver
Linux NICDriver
SOS
Libxc (user levelevent channel)
Back-endNIC driver
vTPM FEDriver
CMFM agent(default)
InterVMcommunication
driver
Front-endDisk
Driver
VA services
InterVMCommunication
driverBE TPM
driver
12/3/0718
Verified Launch – UOS
SATA, USB, etc.Shared HWUser OptimizedHW
Dom0 (Linux)
Device Model(PCI Config,
Serial, ICH,..)
Domain Builder(xm/libxc
equivalent)
CMFMvTPM
VM PowerManager
Xen hypervisorXen hypervisor
TPMDriver
Linux NICDriver
SOS (Linux)
Libxc (user levelevent channel)
Back-endNIC driver
CMFM agent(default)
InterVMCommunication
driver
Front-endDisk
Driver
BE TPMdriver
UOS (Windows)
FE NICdriver
StandardWindows
DrivervTPM FE
Driver
VA services
InterVMCommunication
driver
VAservices
PS agent: Back-endDisk Driver
TPMDriver
InterVMCommunication
driver
Libxc (user levelevent channel)
DeviceModel
12/3/0719
I/O Device Handling
• Default is devices handled directly by UOS Near native performance DMA support
• VT-d provides protection between guests• Non-VT-d platforms will depend upon `well behaved’ guests
• Special devices paravirtualized through SOS LAN Disk
• Xen Client Currently•UOS owned
• Xen Client Future•PCI IOV w/ secure LBA allows dual ownership
Audio• Xen Client Future
•PCI IOV provides secure sharing
12/3/0720
PCI Config Space Virtualization
Dom0Dom0
Agent Integrity
Boot and Startup
ProvisioningVt-x LT VT-d EM64T Channel ACPI/PM
Shadow PT PIC/PIT Models
Xen hypervisorXen hypervisor
SOSSOS
Netw
ork
Dri
vers
VA
Serv
ices
Ag
en
ts
VA
Dri
vers
VA
Pro
vis
ion
ing
Ag
en
ts
VA
N/W
Rou
tin
g S
rvc.
VA
Sto
rag
e S
erv
ices
Oth
er
VA
Dri
vers
ISV Apps.ISV Apps.User Apps.User Apps.
VA
Dri
vers
VA
Serv
ices
Ag
en
ts
Nati
ve D
rivers
VA
Pro
vis
ion
ing
Ag
en
ts
VA
Sto
rag
e S
erv
ices
Oth
er
VA
Dri
vers
TPM HW LAN/WAN SATA, USB, etc.
TPMDriver
ICH(PowerControl)
PCI Configspace
Device ModelsDevice Models
vTPM
UOSUOS
12/3/0721
I/O Device virtualization - PCI Handling
• Dom0 owns PCI bus enumeration SOS/UOS PCI bus enumeration is virtualized in its host domain.
• SOS vPCI bus enumeration in dom0• UOS vPCI bus enumeration in Dom0
•Policy of device assignment to UOS, other SOSes setup by managementpartition
•Virtualized devices are inserted into PCI hierarchy
• PCI device physical BAR = vBAR in UOS Help for PCI hot plug support and thus docking
12/3/0722
Device assignment – PCI hierarchy
• PV (S0S) Virtual PCI front end/back end frame
• Qemu based partition (HVM SOS) Using current Piix4 in qemu as virtual PCI hierarchy Attach assigned physical device to virtual PCI bus in qemu, like current VT-
d effort• Qemu maintains vCONFIG_SPACE to pCONFIG_SPACE mapping• See PCI express, depends on virtual chipset in qemu
• Native based partition (UOS) See exactly same PCI hierarchy as physical
• FE becomes the placeholder of detached devices Need a stub to maintain vCONFIG_SPACE to pCONFIG_SPACE mapping
• HVM PCI config space interception Trapped by Xen, and emulated by vPCI Device Model
• Handlers has filter to decide which field can be written through• Handlers further issues io/mmio/interrupt assignment based on captured info
12/3/0723
Xen Client: Device Assignment
• “Hide” all PCI devices except for the Dom0 assigneddevices from Dom0 device drivers
• Assign PCI devices to guest
• Attach the assigned device to QEMU vPCI bus
• Assign corresponding VT-d context entry to guestdomain
• Intercept PCI config access in Xen and Qemuappropriately
• Turn on VMCS IOPORT bit to allow IOPORT access toassigned devices
• Install P2M entry for MMIO access of the assigneddevice
• Xen intercepts physical device interrupts and re-injectsto the target guest domain with vIRQ
12/3/0724
Platform ACPI Component
Xen Client PM Architecture Overview
DOM0 PM
OSPM Driver
DOM0
VMPM (VM Power Manager)
Semi-Virtual Platform
PlatformACPI Component
UOS PM
TPM Driver
TPM HW
Registers BIOS Tables
Virtual PM ops Real PM opsVM exit/entryXen event
Shared HWUser OptimizedHW
SegregatedHW
PlatformHardware
FEBE FE
ACPIDriver/AMLInterpreter
VA Agent
ACPI Virtualization
PM Event Coordination PCI PM Virtualization
PV SOS(No PM)
SOS Device PM
Use at boot time only
Device Driver
Event notification (via evtchn driver)
Primary UOS
12/3/0725
Power Management
• UOS OSPM policy virtualized
• Policy defined by UOS and enforced by Dom0 UOS controls native devices SOS controls everything else
• Shared devices• CPU• Platform
UOS initiates PM policy (Suspend, Hibernate, C-state or P-statechanges)
12/3/0726
PM - Suspend
• UOS owns physical PCI bridges It will suspend bridges before SOS can suspend it’s devices Implication is that UOS must see virtualized bridges
• One flat PCI space•Could be issue with drivers assuming bus limitations
• Virtual copy of actual topology•Preferable solution
• Suspend UOS suspends devices
• Including virtualized devices (as requested by device D3 request) UOS suspends PCI bridges
• Including any virtualized bridges and conforming to the policy setup inDom0 (VMPM – VM power management)
UOS completes suspend and request ACPI suspend request (ICHI/O port write)
• ICH PM suspend requests suspend of SOSes if needed•Should avoid keeping any state in SOS/DOM0 for restart of SOS/DOM0
• Wake from Suspend Restarts from Xen in real mode and execute
12/3/0727
Conclusion
• This is real, download the Alpha release at
http://eit.et.redhat.com
• Questions? Comments? Epithets?