private & confidential property of coseinc. who am i? senior security researcher at coseinc one...

The COSEINC Hypervisor Framework

Edgar BarbosaSyScan 2009 - Singapore

Private & Confidential Property of COSEINC

Who am I?

• Senior Security Researcher at COSEINC• One of the developers of BluePill, a

hardware-based virtualization rootkit.• Creator of one of the most effective

methods to detect virtualization rootkits.• Experience with systems programming

(kernel, device drivers) and reverse engineering for x86/x64 architectures.


Agenda

1. Review of the implementation methods for virtualization of the x86 architecture.

2. Show the complexity of using hardware supported virtualization instructions to implement virtual machines.

3. Present a framework that makes easy the task of creation of hypervisors.

4. Applications of the framework5. Security aspects


X86 VIRTUALIZATIONThe COSEINC Hypervisor Framework


System Virtual Machines

• System Virtual Machines: VM able to run multiple operating systems concurrently

• The code responsible for the virtualization is called Virtual Machine Monitor (VMM).

• Provides isolation between the guest OS• Physical hardware resources are shared

between the multiple virtual machines


IA-32 processor

VMM

Windows guest

Linuxguest

VMM - Type I

1. Type I (native)The VMM runs directly on the host’s hardware. Hardware resources controlled by the VMM.Examples: VMware ESX, Microsoft Hyper-V


IA-32 processor

VMM

Windows guest

Linuxguest

VMM - Type II

Operating System

• Type II – Hosted The VMM runs as an application.

Hardware resources controlled by the host OS. The COSEINC hypervisor framework creates a Type-II VMM.

Examples: VMware Workstation.Private & Confidential

Property of COSEINC

Execution of the guest instructions

• When the guest VM uses the same Instruction Set Architecture (ISA) of the host machine, the guest instructions can be executed in 2 ways:– Emulation– Direct native execution


Emulation

• The VMM must read and interpret each guest instruction

• Can be implemented using code interpretation or binary translation

• Performance penalty


Direct native execution

• The guest instructions are executed directly on the CPU.

• Great performance. • Some instructions still need to be

emulated.• How to decide which instructions can

be used for direct native execution?


Popek and Goldberg

• Popek and Goldberg published a paper which formally defines the requirements of an ISA for the implementation of efficient virtual machines.

• The VMMs must have 3 properties:– 1. Equivalence– 2. Resource control– 3. Efficiency


1. Equivalence

• “Implies that any program executing on a virtual machine must behave in a manner identical to the way it would have behaved when running directly on the native hardware”[1]

• Basically all the VMM detection methods are based on violations of the Equivalence property.


2. Resource Control

• “Implies that it should not be possible for guest software to directly change the configuration of any system resources”[1]

• Violation: VMM bug exploitation.


3. Efficiency

• “Implies that all instructions that are innocuous must be executed natively on the hardware, without no intervention or emulation by the VMM”[1]

• Depends on the features of the host ISA.

• How to implement efficient virtual machines on the x86 architecture?Private & Confidential

Property of COSEINC

x86 instruction types

INSTRUCTIONS

INNOCUOUS

SENSITIVE

PRIVILEGED

NON-PRIVILEGED


Innocuous instructions

• Innocuous instructions are instructions which doesn’t change or affect system configuration or resources.

• A efficient VMM allows the direct execution of innocuous instructions.

• Examples:– mov eax, 00204012h– shr ebx, 03– xor eax, eax– cmp ebx, ecx


Sensitive instructions

• Sensitive instructions affect system resources or behavior

• The VMM must prevent the direct execution of sensitive instructions!

• The IA-32 instruction set contains 17 sensitive instructions [2]

• Examples:– wrmsr– mov CR3, eax– out dx, eax


Privileged instructions

• All the VMM need now is a way to intercept the execution of the sensitive instructions.

• This is easy when the sensitive instruction is privileged!

• A sensitive instruction is privileged if it traps if the machine is executing in user mode and does not trap in system mode.

• In the x86 architecture, system mode = CPL zero (ring 0)


INNOCUOUS INSTRUCTIONS

Virtualization of sensitive instructions

KERNEL code

cmp eax, ebxjnz 8c0dab00xor edx, edxmov eax, 030h

wrmsr

cmp eax, 020Fhjnz 08000bc00shr eax, 8

SENSITIVE INSTRUCTION


Set CPL to RING 3 and execute the code directly on the cpu

Virtualization of sensitive instructions

KERNEL code

cmp eax, ebxjnz 8c0dab00xor edx, edxmov eax, 030h

wrmsr

cmp eax, 020Fhjnz 08000bc00shr eax, 8

#GENERAL PROTECTION

FAULT

VMM trap handler routine

(emulation)


Non-privileged instructions

• Virtualization of guest instruction would be very easy if all sensitive instructions generates a fault in ring 3.

• There are sensitive but non-privileged instructions in the x86 architecture!

• A sensitive non-privileged instruction will not generate an exception in ring 3!


POPFD example

• POPFD instruction writes a DWORD value in the EFLAGS register.

• It’s a sensitive instruction because it can be used to set the IF flag.

• The IF (Interrupt Flag) controls the hardware external interrupt mechanism.


POPFD instruction

• Problem: Executing POPFD in ring3 will not generate a fault! The CPU just ignores the IF flag modification attempt.

• How to virtualize sensitive non-privileged instructions?


Case study: VMware Player

• How VMware Player VMM is able to prevent direct execution of non-privileged instructions?

• VMware Player is a Type II VMM • The hypervisor is stored as a PE resource

inside the vmware-vmx.exe executable.• ELF executable loaded directly inside the

Windows kernel memory by the vmx86.sys device driver


VMware Player

vmplayer.exe vmware-

vmx.exe

USER MODEKERNEL MODE vmx86.sys

ntoskrnl.exe


Vmware Hypervisor

ELF executable stored as a PE resource

VMware Player

• Solution: Scan all the guest code instructions and search for non-privileged instructions.

• Replace the non-privileged instructions by a privileged instruction.

• VMM handles the faults and emulates the execution of the non-privileded instruction.


Agenda

1. Review x86 virtualization implementation methods.

2. Show how to use the Intel VT® to implement virtual machines.

3. Present a framework to make easy the task of creation of hypervisors.

4. Applications of the framework5. Security and detection discussion


HARDWARE ASSISTED VIRTUALIZATION

The COSEINC Hypervisor Framework


Hardware assisted virtualization

• Virtualizable ISA– If all sensitive instructions of some ISA

are privileged, the processor is considered to be ‘virtualizable’[3]

• IA-32 is obviously not-virtualizable.• New instruction sets created by Intel

and AMD– Intel Virtual Machine eXtensions (VMX)– AMD Secure Virtual Machine (SVM)


Intel VMX

• Presentation focus on Intel VMX. AMD SVM concepts are very similar.

• New form of processor operation: the ‘VMX operation mode’

• VMX mode – activated by the VMXON instruction.


Intel VMX

• VMXON fails if virtualization is locked.• Locked by default in the BIOS for

security reasons• Ring -1. • There’s no more need to move kernel

guest code from ring 0 to ring 3. Guest kernel code can run directly in ring 0.


VMX operation• 2 types of VMX operation:– VMX root operation– VMX non-root operation

• VMX root operation– New instructions available (VMX

instructions)– Used by the VMM (hypervisor)

• VMX non-root operation– Restricted mode of operation– Certain instructions and events are

intercepted to facilitate virtualization.Private & Confidential

Property of COSEINC

VMX transitions

• Transitions between VMX root operation and VMX non-root operation are called ‘VMX transitions’

• Transition from the VMM to the guest: VM-ENTRY.

• Transition from the Guest VM to the VMM: VMEXIT


VMX transitions

Hypervisor(vmx root operation)

VIRTUALMACHINE

(vmx non-root operation)

VM-ENTRY – vmresume/vmlaunch

VM-EXITevent interception


Creating a VMM with Intel VT® - first steps

• Detection of Intel VMX instruction support.– CPUID

• Enable VMX (CR4)– VMXE bit

• Check status of the Lock bit (rdmsr)– More about in the security section

• Setup of the VMXON region


Creating a VMM with Intel VT® - first steps

• Enable VMX instructions (VMXON)• Create and configure the VMCS

region of each guest VM.• Launch the guest VM with

VMLAUNCH instruction • Wait for VM-exit events


VMCS

• Virtual Machine Control Structure• Most important vmx data structure• One VMCS for each Virtual Machine

and for each CPU core.• It controls the behavior of VMX

transitions


VMCS

• VMM must not access the VMCS directly.

• Read and write access to the VMCS via VMREAD and VMWRITE instructions.

• Internal structure undocumented but reverse engineering it is easy.


VMXON and VMCS areasVMXON regionCPU A

VMLinux

VMWindows

VMXON regionCPU B

VMCS #1A

VMCS #2A

VMCS #1B

VMCS #2B

CPU A CPU BPrivate & Confidential

Property of COSEINC

VMCS logical groups

Guest-state area

Host-state area

VM-execution control fields

VM-exit control fields

VM-entry control fields

VM-exit information fields

4K-aligned physical address

6 logical areas


Guest-state area

• Area of the VMCS where guest context information is stored.

• On #VMEXIT, guest processor state is saved in this area.

• On VMENTRY this information is loaded.• Register state:

– Control Registers– Debug Registers– RSP, RIP, RFLAGS– LDTR, GDTR, IDTR– Segment selectors– Model Specific Registers


Guest-state area

• Non-register state– Activity State– Interruptibility state– VMCS link pointer• For future expansions


Host-state area

• Contains information about the host (VMM)

• Processor stated is loaded from this area after each #VMEXIT

• Registers:– RIP (Entry-point address of the

hypervisor routine responsible for handling #VMEXIT events)

– RSP, RFLAGS–MSR


VM-execution control fields

• Controls how the VM will be executed.• The instructions that the hypervisor wants to

intercept are specified in these control fields.– Example: HLT, INVLPG, MWAIT, RDPMC, RDTSC, MOV-DR

• Exception bitmap– Bitmap which controls interception of CPU interrupts like

page faults, debug exceptions, #GP, ...

• I/O bitmap– Can be used to control interception of I/O ports

• MSR bitmap– Interception of Model Specific Registers

• Some instructions wil unconditionally result in VMEXIT


VM-entry control fields

• Controls the behavior of VM entries.• Includes information about SMM,

debug registers and some MSRs.• Guest Event Injection:– It’s possible to inject virtual interrupt or

exception in the guest– Types of interrupts allowed:• External, NMI, Hardware exceptions,

software interrupt.


VM-exit fields

• `VM-exit control fields` which controls the behavior of VM exits.

• VM-exit information fields:– Read-only fields with information about

the most recent VM exit– Exit reason– Exit qualification


Interception

• After configuring the VMCS, the hypervisor can launch the virtual machine and wait for a VMEXIT event.

• When a instruction is intercepted in the guest, the processor will:– Save the VM-exit reason information in the

VMCS– Save guest context information– Load the host-state area– Transfer control to the hypervisor


VMX ROOT-MODERING 0

VMM

#VMEXIT event

handler

VMX NON-ROOTRING 0

mov eax, 23inc edx

xor ebx, edxsub ecx

cmp eax, 1jnz c080df00

retn

mov cr3, ebx

#VMEXIT

VMLAUNCH

#VMRESUME


Agenda






Hypervisor programming complexity

• Creating a VMM using these new hardware virtualization ISA is complex– More complex features always comming: EPT for

nested paging

• Very hard to find and to fix bugs• No debugger• Intel VT error codes not very useful– Code 33 = “VM-entry failure due to invalid guest

state”– What’s exactly invalid in the guest state?– More than 40 suspects!


The framework

• The COSEINC Hypervisor Framework, referred from now as just the ‘framework’, enables you to easily create a Hosted Virtual Machine Monitor (Type II VMM) using the Windows Operating System.

• Simple and easy-to-use API exported• Abstraction over the different

hardware virtualization instruction sets (VMX-SVM)


The framework

• 2 versions:– 32-bits Windows device driver– 64-bits Windows device driver

• API exported methods:– Export table– IOCTL codes for user-mode communication

• Initial version only for Windows, but porting to Mac/Linux should not be difficult.

• Release date: very soon! Private & Confidential

Property of COSEINC

Features

• Automatic detection of the virtualization instruction sets.

• SMP support• Evaluation of the lock bit• Detailed error-status codes• Plugin-like architecture


Architecture

Framework

Ring 0 Operating System

Kernel

Ring -1

User applications Ring 3

Framework Client


API

• The full documentation of the API will be released with the framework.

• Preliminary documentation. Subject to change.• Function categories:– Virtual Machine management functions

• Creation and deletion of Virtual Machines.• Executing and resuming a virtual machine.

– Interception Events functions• The framework call the registered client function

callbacks.

– Root guest VM.


Virtual Machine management

• VMSTATUSCreateVirtualMachine (

IN VMINFO *vminfo);

• This function creates a new virtual machine in the system.

• Fails if virtualization MSR is locked by the BIOS.


VMINFO data structure

• Most important framework data structure

• Contains all the information needed to create and control a VM:– all the GUEST context information– GDT, LDT, Page Tables, Control Registers, ...– Interception handler function callback

address.– Contains Event Injection information– VMEXIT information


GUEST_INFO

Registers

Segments

Descriptor Tables

Control

Debug

Model Specific

CONTROL_INFO

Interception

Event Injection

I/O

Interrupts

MSR

Virtual Machine

VMINFO data structure

VMEXIT info Extra info


Interception Event management

• VMSTATUSVirtualMachineExec (

IN VMINFO *vminfo);

• This function controls the execution of the virtual machine. It can be called after the creation of the VM and to resume the execution of the VM after an intercept event.

• If the VMM must inject some event in the guest VM, the information is provided in the VMINFO data structure.


x

VM creation and execution

Framework

Client(VMM

plugin)

Framework

CreateVirtualMachine( )

Intercept event

handler

Intercept Event Message

VirtualMachineExec( )

VirtualMachineExec( )


VM

VM Scheduler

VM message handler

Timer interrupt

VM Event Router

Virtual Machine

VM Event Manager

Hypervisor

Framework – Client communication

Root guest VM

• One of the best features of the framework:– Automatic conversion of the host

operating system into a virtual machine in runtime!

• This guest VM is called ‘root VM’• The creation of the root VM is

optional and controlled by the api.• Root VM is shared between all loaded

plugins.Private & Confidential

Property of COSEINC

Agenda






APPLICATIONSThe COSEINC Hypervisor Framework


Applications of the framework

• Specially useful for education and research purposes

• Can abe used to create any type of small and fast VM. Not only system VMs.

• The best features are available when using the root guest VM.


Process VM

• Whole virtualization of a process or a thread is possible with the framework.

• Normally achieved by interception of system calls.

• Additional functions will be added to the API for better memory virtualization.

• No support for EPT in the first version.


Syscall hooking

• A great number of system monitoring and security tools are implemented using system call hooking methods.

• Old Windows OS uses INT 2eh • Linux and newer Windows OS uses

SYSENTER instructions


Syscall mechanism - illustration

SYSENTER_EIP MSR

SYSENTER_CS MSR

mov edx, espsysenter

mov ecx, 23hpush 30hpop fs...

nt!KiFastCallEntry

Ntdll.dll


Windows OS syscall mechanism

Syscall hooking

• Syscall hooking methods includes:– Patching syscall handler– Patching of IDT table– Patching the SYSENTER Model Specific

registers


Syscall interception

• Syscall interception using the root guest VM• No need to hook SSDT• No need to patch/modify guest kernel code• Virtualization of the SYSENTER MSR• Plugin (framework)– VMINFO->ControlInfo->Interception->MSR

• Can also be applied to Linux guests• Virtualized IDTR for old guest operating

systems using INT xx instructions for syscall implementation.


Instrumentation

• Instrumentation is also easy to implement using the Interruptibility controls in the VMCS.

• Performance registers are also virtualizable

• Tools:– Optimization tools– System statistics


Nested virtualization

• The framework doesn’t provide support for nested virtualization

• But it is possible to add this feature via a VMM plugin.

• Also, a virtualization debugger could be implemented!


Agenda






64-bits

• The framework and the plugins must be digitally signed to run in 64-bit versions of Windows.


The LOCK bit

• MSR IA32_FEATURE_CONTROL (Index 3Ah)

• Controls:– SMX – Safer Mode eXtensions

• Disabled by default in the BIOS


Detection

• “There is no software-visible bit whose setting indicates whether a logical processor is in VMX non-root operation. This fact may allow a VMM to prevent guest software from determining that it is running in a virtual machine.” – Intel manual 3 – 19.3

• VMX transitions are cpu-expensive operations.

• Thousand of cycles just for a simple VMEXIT.• SyScan 2007 – Detecting BluePill


QUESTIONS?

THANK YOU FOR YOUR TIME!

Bibliography

1. John Scott Robin and Cynthia E. Irvine (2000). "Analysis of the Intel Pentium's Ability to Support a Secure Virtual Machine Monitor". Proc. 9th USENIX Security Symposium.

2. Virtual Machines: Versatile Platforms for System and Processes – Jim Smith, Ravi Nair – Morgan Kaufmann - 2005

3. Intel manuals (www.intel.com)

http://www.usenix.org/events/sec2000/robin.html



http://www.intel.com/

private & confidential property of coseinc. who am i? senior security researcher at coseinc one...

Documents

equivalence property

virtualization instructions

virtual machine monitor

guest vm

guest instructionswhen

guest software

guest instructioncan

nativethe vmm