1

Virtualization services

Virtual machines

2

referencesIntel Virtualization technology

IEEE xplorer , May 2005

Comparison of software and hardware techniques for x86 virtualization

ASPLOS 2006

Memory resource management in VMware ESX serverOSDI 2002

Virtualizing I/O devices on Vmware WorkstationUSENIX 2002

Xen and the art of virtualization SOSP 2003

3

Topics

1. What is a VM?2. Process vs System VMs3. Virtualizing the Processor4. Virtualizing Memory5. Virtualizing I/O6. VM Performance Issues7. Intel VT-x Technology8. Paravirtualization

4

VirtualizationWe have seen Vdisk, VIP, Vnode

An abstraction mapped to real resourcesTraditionally, a single OS virtualizes system resources among processes

Process VirtualizationWhat if we ran multiple OSes on the same physical hardware?

System VirtualizationVirtual Machine Monitor: A new layer of software (VMM) multiplexes the OSes

5

System Models

Non-virtual Machine Virtual Machine

6

Process Virtualization

MultiprogrammingMultiple programs resident in memorySame ISA (Instruction Set Architecture)

High level language (HLL) VMDifferent ISAEmulators

Interpreter or Dynamic BT

Java VM

7

System Virtualization

Support multiple guest operating systems simultaneouslyWhy?

8

Workload Isolation

Applications on older versions of the OS does not affect applications on a new OS

XP VISTA

XP

XP

newapp

XPXP

9

Workload consolidation

Instead of separate, maybe underutilized, vertical stacks (H/W, OS, APP) several OSes on the same H/W

App1Os1

App2Os2

App3OS3

VMMHW

20% 40% 20%

Mail server Web server Db server

10

Workload migration

A guest OS can migrate to new H/W running a VMM

11

Virtualization: Where is it going really!?

Hardware

Virtual Machine

OS1Windows Linux NT

Server Virtualization

Hardware

Virtual Machine

OS1Windows linux

Desktop Virtualization

Mobile Virtualization

2000

2005

2010

WHAT?WHY?

12

Mobile Virtualization: WHAT?

Hardware

Virtual Machine

OS1ServiceProvider OS

EnterpriseOS

ConsumerOS

Mobile Platform Virtualization

Blackberry OS and Apple OS on the same smart phone!!!

13

System Virtualization

Support multiple guest operating systems simultaneously on the same H/WWhy is it hard?

14

Even dilbert knows

15

System virtualization challenges

VMM is the new kernelJust run the OS as a user applicationBut OS accesses privileged state?Trap into VMMSome instructions behave differently in user modeKernel was written assuming it runs in privilege 0Guest OS must feel it is running on a real machine

16

Not every x86 instruction traps uniformly

Canonical example: popf instructionSame instruction behaves differently depending on execution modeUser Mode: changes ALU flagsKernel Mode: changes ALU and system flagsDoes not generate a trap in user mode

Could use additional ringsRing 0, VMM , ring 1 or 2 for guest OS, ring 3 for appToday only two rings used (0 for kernel, 3 for user mode)

17

System virtualization challenges

VMM is the new kernelHow to virtualize memory?Page tables within each OS. Traditionally OS context switches ProcessesPTE entries point to distinct physical frame numbers (unless shared)VMM needs to context switch entire VMMust support all features of memory management

Segments, pages, combined etc

Allocation, replacement?

18

System virtualization challenges

VMM is the new kernelHow to virtualize I/O?OS schedules I/O and CPU tasksMultiplexing an demultiplex I/O dataNeed to support real I/O devicesIn-bound and out-bound end pointsWhat about device drivers?Think about virtualizing nintendo or wii players

19

System VM models

Guest Apps

Hardware

Guest OS

VMM

Hardware

Host OS

VMM

Guest OS

Guest Apps

Native VMM Type IHosted VMM Type II

Guest Apps

Hardware

ModifiedGuest OS

VMM

Para Virtualization

Vmware Vmware XEN

20

Role of VMM

Virtual Machine Monitor (VMM)A new layer of software

Provides illusion of multiple isolated machines.Arbitrate access to hardware resources for multiple guest OSes.Layer between hardware and guest OS

VMM tasksManage state (privileged)Manage resources

21

Characteristics of VMM

Popek and Goldberg criteriaFidelity: software on the VMM executes identically to its execution on hardware, barring timing effectsPerformance: An overwhelming majority of guest instructions are executed by the h/w without the intervention of the VMMSafety: VMM manages all h/w resources

22

CPU scheduling

VMM schedules each guest OS Round robin or weighted round robinNeed to keep state of each guest OS when timer interruptsMeta level schedulerVMM schedules guest OSGuest OS schedules processesIssues?

23

VMM Execution cycle1. Meta Timer Interrupt in running VM.2. Context switch to VMM.3. VMM saves state of running VM.4. VMM determines next VM to execute.5. VMM sets meta timer interrupt.6. VMM restores state of next VM.7. VMM sets PC to timer interrupt handler of next

VM.8. Next VM active.

24

Resource Virtualization

1. Processor2. Memory3. I/O

25

Virtualizing ProcessorGuest OS should not access privilege stateAny trap should be handled by the trap handler of the VMMVMM should not execute the trap but just handle

Process Operating System VMM (user mode) (user mode) (kernel mode) trap trap from guest OS

JMP to guest OS trap handler BEGIN: OS trap handler decode trap and execute appropriate syscall routine RTT from Trap

intercept return from trapdo a real return from trap to user process

start running again

26

x86 obstacles to virtualization

Visibility of privileged stateCan read current privilege levelCpl read on %cs low bits indicate PL

Sensitive instructionsCertain instructions behave differently based on the CPLThis disrupts x86 binary distribution

27

CPU virtualization

Para virtualizationFor each critical instruction in the ISA (guest OS) replace with modified critical instructionDefine a replacement sequenceE.g., POPF{} to POPF’{}Rebuild OS to use POPF’ and VMM will support POPF’Need access to source code

28

CPU virtualization

InterpreterRun OS code inside a binary translator/InterpreterEvery instruction designed to behave appropriatelyToo much overheadDoes not satisfy P&G criterion

29

CPU virtualization

Binary translationTranslate critical instructions at runtimeCode is translated only when it is about to executeReplace critical instructions with acceptable instructionsCode cache to improve performanceLots of details

30

CPU virtualizationH/W support; new processors support virtualizationIntel and AMD VT-x architectureVT-x has a new operating mode; enabled with VMXON/VMXOFFProvides two new forms of operation: root operation (fully privileged, intended for VMM) and non-root operation (not fully privileged, intended for guest OS)guest OS runs at ring 0; apps running on guest OS run at ring 3 and VMM runs in new higher mode (VMX) --equivalently -1 privilege

31

Virtual memory

Each Process has its own page table that maps to physical frame that the VM is running inGuest OS should not map any Virtual page to any physical page No isolationOn the other hand need PTE to translate virtual addresses to physical addressesMore than one PTBR (one for each VM)

32

Shadow Page TablesGuest OS maintains its own page tables.

Virtual to real memory mapping.

VMM maintains shadow page tablesVirtual to physical memory mapping.Used by hardware to translate virtual addresses.VMM validates guest page table updates.Replicates guest changes in shadow page table.

Virtualize page table pointer register.VMM manages real page table pointer.Updates page table ptr when switching VMs.

33

Shadow page table

VMM

Physical memory

Real page table

Shadow page table(read only)

Trap to VMMProcessWrite real PTE

34

Shadow Page Tables

MMU

Guest OS

Hardware

Accessed &dirty bits

VMM

guest writes

guest reads Guest Page Table

Shadow Page Table

Updates

35

Reclaiming memoryIn non-virtualized system, OS uses page replacement algorithms to decide which page to evictWhat should the VMM do?Swap an entire VMSwap pages belonging to individual VMsVMM needs to decide the page as well as the which guest Oss pageWhat policy to use?What if it conflicts with guest OS page replacement policy

36

Double pagingSwap lists, free lists are maintained by each VM alsoVMM pages out, the guest OS selects the same pageVMM needs to bring back the page into memory and write back into virtual paging deviceSo the guest OS needs to decide what pages to swapWe need to induce the guest OS to start swapping

37

Ballooning

Create a balloon device ( a driver) inside each guest OSDriver can ask for physical pagesVMM inflates balloon when in need of pagesVMM deflates balloon when it does not need

Linux Windows XP Free BSD

Physical memory

38

Ballooning

VMM instructs balloon to inflateDriver can ask for physical pagesCauses memory pressure on guest OSGuest OS kicks in page swap processFreed pages can be allocated to another guest OS

Linux Windows XP Free BSD

Physical memory

SWAP

39

Ballooning limitations

Driver may be uninstalled, disabledUpper bounds on ballon sizes may need to imposedShould not penalize one guest OS over another

40

Page sharing

What if many guest OSes are identical copiesUnlike a guest OS, which knows text segments of processes (on fork, e.g.,), VMM has no ideaNice to find shared read-only pages, at leastIdea: scan physical pages and seek matchesSo identical pages point to the same PFNA Copy-on-write bit is set and if modified make a copy and make the pages different

41

Virtualizing I/O

All I/O operations are privilegedVMM must intercept all guest OS I/O operationsVMM must implement all device driversMany devices are chattyType II VM may be betterUses device drivers in the host OSNeed to switch between VMM world and host world

42

Hosted vs. Native

World switching is expensive

43

Connections…

44

Virtualizing a network card

Each Virtual NIC appears as a full fledged PCI controller with its own MAC addressAll Virtual NICs form a bridge with physical NICThe physical NIC in promiscuous mode picks up all packets and forwards it to all virtual NICsFor virtual networks, ethernet interface is not requiredHow does IP networking work?

45

Virtualizing a Network Card