a virtual instruction set interface for operating system kernels

A Virtual Instruction Set Interface for Operating System Kernels

John Criswell, Brent Monroe, Vikram AdveUniversity of Illinois at Urbana-Champaign

Outline

MotivationLLVA-OS Design

Hardware ControlState Manipulation

Preliminary Performance Results

Motivation

OS/Hardware interface is non-standard & often machine code Difficult to analyze OS

difficult to infer all control flow information some type information lost no ability to track virtual memory map changes

Difficult to adapt OS memory safety transforms (SAFECode) changes in processor instruction set

Difficult for hardware to infer OS behavior context switching

Motivation

Solution: decouple program representation (Virtual ISA) from hardware control (Native ISA)

Execution Engine translates between Virtual ISA and Native ISA

Virtual ISA design aids software analysis and transformation

Software

HardwareExecution Engine

Virtual ISA

Native ISA

LLVA Code Example [MICRO 03, CGO 04]

;; LLVA Translated Codeint %SumArray(int* %Array, int %Num){bb1:

%cond = setgt int %Num, 0br bool %cond, label %bb2, label %bb3

bb2:%sum0 = phi int [%tmp10, %bb2], [0, %bb1]%i0 = phi int [%inc, %bb2], [0, %bb1]%tmp7 = cast int %i0 to long%tmp8 = getelementptr int* %Array, long %tmp7%tmp9 = load int* %tmp8%tmp10 = add int %tmp9, %sum0%inc = add int %i0, 1%cond2 = setlt int %inc, %Numbr bool %cond2, label %bb2, label %bb3

bb3:%sum1 = phi int [0, %bb1], [%tmp10,%bb2]ret int %sum1

}

/* C Source Code */int SumArray(int Array[], int Num){ int i, sum = 0; for (i = 0; i < Num; ++i) sum += Array[i]; return sum;}

• Architecture-neutral• Low-level operations• SSA representation• Strictly-typed• High-level semantic info

LLVA-OS: Extend LLVA to OS Kernels

Kernels require new functionalityHardware Control

I/OMMU

State Manipulationcontext switching

Outline




Hardware Control

Registration functions void llva_register_syscall (int number, int (*f)(…)) void llva_register_interrupt (int number, int (*f)(void * icontext)) void llva_register_exception (int number, int (*f)(void * icontext))

I/O int llva_io_read (void * ioaddress) void llva_io_write (void * ioaddress, int value)

Memory Management void llva_load_pgtable (void * table) void * llva_save_pgtable ()

Outline




Virtual and Native State

Virtual State Virtual Registers

Program Counter Privilege Mode Interrupt Flag Stack Pointer

MMU State

Native State General Purpose

Registers

Control Registers

MMU Registers

Challenges with Virtual State

Mapping between virtual state and native state changes over short time intervals, requiring a large mapping structure

Manipulating virtual state is cumbersomeMany virtual registers per functionMany virtual registers are dead

State Saving/Restoring Instructions

Solution: Expose existence of native state Define native state based on correlation to

virtual state integer state floating point (FP) state

Instructions void llva_save_integer (void * buffer) void llva_load_integer (void * buffer) void llva_save_fp (void * buffer, bool save_always) void llva_load_fp (void * buffer)

Interrupted Program State

Execution Engine must save program state when entering the OS

Problem: Want to minimize the amount of state saved No need to save FP state How do we use low latency interrupt facilities

shadow registers (e.g. ARM) register windows (e.g. SPARC)

GPR 1:

0xBEEF0000

ControlReg 2:

0x4EF23465

ControlReg 1:

0xC025E525

Solution: Interrupt Context

Definition: Reserved space on kernel stack

Conceptually: the saved Integer State of the interrupted program

On interrupt, Execution Engine saves subset of Integer State on the kernel stack

Can leave state in registers if kernel does not overwrite it

Kernel can convert Interrupt Context to/from Integer State

Pointer to Interrupt Context passed to system call, interrupt, and trap handlers

Kernel StackProcessor

ControlReg 1:

0xC025E525

GPR 1:

0xBEEF0000

ControlReg 2:

0x4EF23465

GPR N:

0x00000000

GPR N:

0x00000000

Manipulating Interrupt Context

Push function framesvoid llva_ipush_function (void * icontext, void (*f)(…), …)

Interrupt Context Integer Statevoid llva_icontext_save (void * icontext, void * buffervoid llva_icontext_load (void * icontext, void * buffer)

Function 1

Example: Signal Handler Dispatch

Save program state with llva_icontext_save()

Save FP state with llva_save_fp()

Push new function frame on to program stack with llva_ipush_function()

Interrupt Context

SignalHandler

Stack

User Space Kernel SpaceStack

Heap

Processor

FP Registers

Other Registers

Trap Handler

Outline




LLVA-OS Prototype

LLVA-OS C and i386 assembly code for Pentium 3 Compiled to native code library ahead of time Some instructions inlined through header files

Linux 2.4.22 port to LLVA-OS Like a port to a new architecture Inline assembly replaced with LLVA-OS instructions Compiled with GCC and linked with LLVA-OS

library

Performance Evaluation

Nano- and micro-benchmarksBased on HBench-OS benchmark suiteRun for 100 iterations Identify overheads in key kernel operations

Macro-benchmarksDetermine impact of overheads on real

application loads

Nanobenchmarks

-20 -10 0 10 20 30 40 50

% Overhead

System Call Entry

Trap Entry

Read Page Fault

Kernel-User memcpy (1KB)

User-Kernel strcpy(1KB)

Overhead in Kernel Operations

Nanobenchmarks

-20 -10 0 10 20 30 40 50

% Overhead

System Call Entry

Trap Entry

Read Page Fault

Kernel-User memcpy (1KB)

User-Kernel strcpy(1KB)

Overhead in Kernel Operations

Absolute increase in page fault latency is very small User-Kernel strcpy overhead due to inefficient strcpy

routine Trap entry is faster (no VM86 mode)

Microbenchmarks

0 10 20 30 40 50

Overhead (%)

open/close

fork/exit

fork/exec

sighandler install

sighandler dispatch

Ben

chm

ark

Microbenchmark Overhead

Microbenchmarks

0 5 10 15 20 25 30 35 40 45 50

Overhead (%)

open/close

fork/exit

fork/exec

sighandler install

sighandler dispatch

Ben

chm

ark

Microbenchmark Overhead

Signal Handler Dispatch overhead due to extraneous FP state loading on sigreturn()

Open/Close has user to kernel strncpy() overhead

Microbenchmarks: Filesystem

0

50

100

150

200

250

300

Ban

dwid

th (M

B/s

)

4 16 64 256 1024 4096Buffer Size (KB)

File Read Bandwidth: 4MB (BW_FILE_RD)

i386 LLVA

Microbenchmarks: Filesystem

0

50

100

150

200

250

300

Ban

dwid

th (M

B/s

)

4 16 64 256 1024 4096

Buffer Size (KB)

File Read Bandwidth: 4MB (BW_FILE_RD)

i386 LLVA

Maximum overhead is 2%

Benchmark reads file using repeated read() calls

No I/O overhead (file in buffer cache)

Microbenchmarks: TCP

0

10

20

30

40

50

60

70

80

Ban

dwid

th (M

B/s

)

4 16 64 256 1024Buffer Size (KB)

TCP Bandwidth (BW_TCP)

i386 LLVA

Microbenchmarks: TCP

0

10

20

30

40

50

60

70

80

Ban

dwid

th (M

B/s

)

4 16 64 256 1024

Buffer Size (KB)

TCP Bandwidth (BW_TCP)

i386 LLVA

Maximum overhead is 21%

Server process reads at least 10 MB from client process

Performance: Macrobenchmarks

0 20 40 60 80 100Server Throughput (Mb/s)

1

4

# cl

ient

s

thttpd Bandwidth

i386 LLVA


0 20 40 60 80 100Server Throughput (Mb/s)

1

4

# clients

thttpdBandwidth

LLVAi386

WebStone: standard workload Less than 8% overhead to thttpd


0 50 100 150 200Wall Time (s)

OpenSSH Build LLVAi386

1.01% overhead Primarily CPU bound process

Future Work

Performance tuning of LLVA-OS implementation

Framework for providing additional securityMemory safety for OS kernelProtect application memory from kernelTranslator enforced system call policies Install time privilege bracketing

Acknowledgements

Pierre SalverdaDavid RailaLLVM Developers, past and presentThe ReviewersAnd all the others who gave us feedback

and input

Summary

LLVA-OS uses novel approaches to virtualize state manipulation

More tuning to the implementation is necessary but possible

Linux on LLVA-OSNo assembly codeMany compiler opportunities

a virtual instruction set interface for operating system kernels

Documents