automating mimicry attacks using static binary analysis · keywords: binary analysis, static...

Automating Mimicry Attacks Using Static Binary AnalysisChristopher Kruegel and Engin Kirda

Technical University [email protected], [email protected]

Darren Mutz, William Robertson, and Giovanni VignaReliable Software Group, University of California, Santa Barbara

{dhm,wkr,vigna}@cs.ucsb.edu

Abstract

Intrusion detection systems that monitor sequences of sys-tem calls have recently become more sophisticated indefining legitimate application behavior. In particular, ad-ditional information, such as the value of the programcounter and the configuration of the program’s call stackat each system call, has been used to achieve better char-acterization of program behavior. While there is com-mon agreement that this additional information compli-cates the task for the attacker, it is less clear to which ex-tent an intruder is constrained.

In this paper, we present a novel technique to evade the ex-tended detection features of state-of-the-art intrusion de-tection systems and reduce the task of the intruder to atraditional mimicry attack. Given a legitimate sequence ofsystem calls, our technique allows the attacker to executeeach system call in the correct execution context by ob-taining and relinquishing the control of the application’sexecution flow through manipulation of code pointers.

We have developed a static analysis tool for Intel x86 bi-naries that uses symbolic execution to automatically iden-tify instructions that can be used to redirect control flowand to compute the necessary modifications to the envi-ronment of the process. We used our tool to successfullyexploit three vulnerable programs and evade detection byexisting state-of-the-art system call monitors. In addition,we analyzed three real-world applications to verify thegeneral applicability of our techniques.

Keywords: Binary Analysis, Static Analysis, SymbolicExecution, Intrusion Detection, Evasion.

1 Introduction

One of the first host-based intrusion detection systems [5]identifies attacks by finding anomalies in the stream ofsystem calls issued by user programs. The technique is

based on the analysis of fixed-length sequences of sys-tem calls. The model of legitimate program behavior isbuilt by observing normal system call sequences in attack-free application runs. During detection, an alert is raisedwhenever a monitored program issues a sequence of sys-tem calls that is not part of the model.

A problem with this detection approach arises in situa-tions where an attack does not change the sequence of sys-tem calls. In particular, the authors of [17] observed thatthe intrusion detection system can be evaded by carefullycrafting an exploit that produces a legitimate sequence ofsystem calls while performing malicious actions. Suchattacks were named mimicry attacks.

To limit the vulnerability of the intrusion detection sys-tem to mimicry attacks, a number of improvements havebeen proposed [4, 9, 14]. These improvements are basedon additional information that is recorded with each sys-tem call. One example [14] of additional information isthe origin of the system call (i.e., the address of the in-struction that invokes the system call). In this case, theintrusion detection system examines the value of the pro-gram counter whenever a system call is performed andcompares it to a list of legitimate “call sites.” The ideawas extended in [4] by incorporating into the analysis in-formation about the call stack configuration at the time ofa system call invocation.

A call stack describes the current status and a partial his-tory of program execution by analyzing the return ad-dresses that are stored on the program’s run-time stack.To extract the return addresses, it is necessary to unwindthe stack, frame by frame. Figure 1 shows a number offrames on a program stack and the chain of frame (base)pointer references that are used for stack unwinding.

Checking the program counter and the call stack at eachsystem call invocation serves two purposes for the de-fender. First, the check makes sure that the system call

14th USENIX Security SymposiumUSENIX Association 161

14th USENIX Security Symposium

Function Parameters

Return Address N

Frame Pointer N

Local Variables

Function Parameters

Return Address N-1

Frame Pointer N-1

Local Variables

CurrentFrame Pointer

CurrentStack Pointer

Frame N

FrameN-1

Figure 1: Call stack and chain of frame pointers.

was made by the application code. This thwarts all codeinjection attacks in which the injected code directly in-vokes a system call. Second, after a system call has fin-ished, control is guaranteed to return to the original appli-cation code. This is because the return addresses on thestack have been previously verified by the intrusion detec-tion system to point to valid instructions in the applicationcode segment. This has an important implication. Even ifthe attacker can hijack control and force the applicationto perform a single system call that evades detection, con-trol would return to the original program code after thissystem call has finished.

The common agreement is that by including additionalinformation into the model, it is significantly more diffi-cult to mount mimicry attacks [3]. However, although ad-ditional information undoubtedly complicates the task ofthe intruder, the extent to which the attack becomes morecomplicated is less clear. System-call-based intrusion de-tection systems are not designed to prevent attacks (forexample, buffer overflows) from occurring. Instead, thesesystems rely on the assumption that any activity by theattacker appears as an anomaly that can be detected. Un-fortunately, using these detection techniques, the attackeris still granted full control of the running process. Whilethe ability to invoke system calls might be significantlylimited, arbitrary code can be executed. This includes thepossibility to access and modify all writable memory seg-ments.

The ability to modify program variables is in itself a sig-nificant threat. Consider, for example, an attacker thatalters variables that are subsequently used as open orexecv system call parameters. After the modification,the attacker lets the process continue. Eventually, a sys-tem call is invoked that uses values controlled by the at-

tacker. Because the system call is made by legitimate ap-plication code, the intrusion remains undetected.

In some cases, however, modifying program variables isnot sufficient to compromise a process and the attacker isrequired to perform system calls. Given the assumptionthat an attacker has complete knowledge about the detec-tion technique being used, it is relatively straightforwardto force the application to perform one undetected systemcall. To do so, the attacker first pushes the desired systemcall parameters on the stack and then jumps directly tothe address in the application program where the systemcall is done. Of course, it is also possible to jump to a li-brary function (e.g., fopen or execlp) that eventuallyperforms the system call. Because it is possible for theinjected code to write to the stack segment, one can injectarbitrary stack frames and spoof any desired function callhistory. Thus, even if the intrusion detection system fol-lows the chain of function return addresses (with the helpof the stored base pointers), detection can be evaded.

The problem from the point of view of the attacker is thatafter the system call finishes, the checked stack is used todetermine the return address. Therefore, program execu-tion can only continue at a legitimate program address andexecution cannot be diverted to the attacker code. As aconsequence, there is an implicit belief that the adversarycan at most invoke a single system call. This constitutesa severe limitation for the intruder, since most attacks re-quire multiple system calls to succeed (for example, a callto setuid followed by a call to execve). This limi-tation, however, could be overcome if the attacker wereable to somehow regain control after the first system callcompleted. In that case, another forged stack can be setup to invoke the next system call. The alternation of in-voking system calls and regaining control can be repeateduntil the desired sequence of system calls (with parame-ters chosen by the attacker) is executed.

For the purpose of this discussion, we assume that the at-tacker has found a vulnerability in the victim program thatallows the injection of malicious code. In addition, we as-sume that the attacker has identified a sequence of systemcalls s1, s2, . . . , sn that can be invoked after a successfulexploit without triggering the intrusion detection system(embedded within this sequence is the attack that the in-truder actually wants to execute). Such a sequence couldbe either extracted from the program model of the intru-sion detection system or learned by observing legitimateprogram executions. By repeatedly forcing the victim ap-plication to make a single undetected system call (of alegitimate sequence) and later regaining control, the pro-tection mechanisms offered by additional intrusion detec-tion features (such as checking return addresses or callhistories) are circumvented. Thus, the task of the intruder

USENIX Association162

is reduced to a traditional mimicry attack, where only theorder of system calls is of importance.

In this paper, we present techniques to regain control flowby modifying the execution environment (i.e., modifyingthe content of the data, heap, and/or stack areas) so thatthe application code is forced to return to the injected at-tack code at some point after a system call. To this end, wehave developed a static analysis tool that performs sym-bolic execution of x86 binaries to automatically determineinstructions that can be exploited to regain control. Upondetection of exploitable instructions, the code necessaryto appropriately set up the execution environment is gen-erated. Using our tool, we successfully exploited sampleapplications protected by the intrusion detection systemspresented in [4] and [14], and evaded their detection.

The paper makes the following primary contributions:

• We describe novel attack techniques against twowell-known intrusion detection systems [4, 14] thatevade the extended detection features and reduce thetask of the intruder to a traditional mimicry attack.

• We implemented a tool that allows the automatedapplication of our techniques by statically analyzingthe victim binary.

• We present experiments where our tool was usedto generate exploits against vulnerable sample pro-grams. In addition, our system was run on real-worldapplications to demonstrate the practical applicabil-ity of our techniques.

Although our main contributions focus on the automatedevasion of two specific intrusion detection systems, an im-portant point of our work is to demonstrate that, in gen-eral, allowing attackers to execute arbitrary code can havesevere security implications.

The paper is structured as follows. In Section 2, we re-view related work on systems that perform intrusion de-tection using system calls. In Section 3, we outline ourtechniques to regain control flow. Section 4 provides anin-depth description of our proposed static analysis andsymbolic execution techniques. In Section 5, we demon-strate that our tool can be used to successfully exploitsample programs without raising alarms. In addition, thesystem is run on three real-world applications to under-line the general applicability of our approach. Finally, inSection 6, we briefly conclude and outline future work.

2 Related Work

System calls have been used extensively to characterizethe normal behavior of applications. In [7], a classifi-cation is presented that divides system-call-based intru-sion detection systems into three categories: “black-box”,“gray-box”, and “white-box”. The classification is basedon the source of information that is used to build the sys-tem call profiles and to monitor the running processes.

Black-box approaches only analyze the system calls in-voked by the monitored application without consideringany additional information. The system presented in [5],which is based on the analysis of fixed-length sequencesof system calls, falls into this category. Alternative datamodels for this approach were presented in [18], whilethe work in [19] lifted the restriction of fixed-length se-quences and proposed the use of variable-length patterns.However, the basic means of detection have remained thesame.

Gray-box techniques extend the black-box approaches byincluding additional run-time information about the pro-cess’ execution state. This includes the origin of the sys-tem call [14] and the call stack [4], as described in theprevious section. Another system that uses context in-formation was introduced in [6]. Here, the call stack isused to generate an execution graph that corresponds tothe maximal subset of the program control flow graph thatone can construct given the observed runs of the program.

White-box techniques extract information directly fromthe monitored program. Systems in this class performstatic analysis of the application’s source code or binaryimage. In [16], legal system call sequences are rep-resented by a state machine that is extracted from thecontrol-flow graph of the application. Although the sys-tem is guaranteed to raise no false positives, it is vulner-able to traditional mimicry attacks. Another problem ofthis system is its run-time overhead, which turns out tobe prohibitively high for some programs, reaching sev-eral minutes per transaction. This problem was addressedin [8], using several optimizations (e.g., the insertion of“null” system calls), and later in [9], where a Dyck modelis used. For this approach, additional system calls need tobe inserted, which is implemented via binary rewriting.

An approach similar to the one described in the previousparagraph was introduced in [11]. In this work, systemcall inlining and “notify” calls are introduced instead ofthe “null” system calls. Also, source code is analyzed in-stead of binaries. Another system that uses static analysisto extract an automaton with call stack information waspresented in [3]. The work in this paper is based on thegray-box technique introduced in [4]. In [20], waypoints



are inserted into function prologues and epilogues to re-strict the types of system calls that they can invoke.

Black-box and gray-box techniques can only identifyanomalous program behavior on the basis of the previousexecution of attack-free runs. Therefore, it is possible thatincomplete training data or imprecise modeling lead tofalse positives. White-box approaches, on the other hand,extract their models directly from the application code.Thus, assuming that the program does not modify itself,anomalies are a definite indication of an attack. On thedownside, white-box techniques often require the analy-sis of source code, which is impossible in cases where thecode is not available. Moreover, an exhaustive static anal-ysis of binaries is often prohibitively complex [6].

This paper introduces attacks against two gray-box sys-tems. Thus, related work on attacking system-call-baseddetection approaches is relevant. As previously men-tioned, mimicry attacks against traditional black-box de-signs were introduced in [17]. A similar attack is dis-cussed in [15], which is based on modifying the exploitcode so that system calls are issued in a legitimate order.In [7], an attack was presented that targets gray-box intru-sion detection systems that use program counter and callstack information. This attack is similar to ours in that it isproposed to set up of a fake environment to regain controlafter the invocation of a system call. The differences withrespect to the attack techniques described in this paper aretwofold. First, the authors mention only one technique toregain control of the application’s executionflow. Second,the process of regaining control is performed completelymanually. In fact, although the possibility to regain con-trol flow by having the program overwrite a return addresson the stack is discussed, no example is provided that usesthis technique. In contrast, this paper demonstrates thatattacks of this nature can be successfully automated usingstatic analysis of binary code.

3 Regaining Control Flow

In this section, we discuss possibilities to regain controlafter the attacker has returned control to the application(e.g., to perform a system call). To regain control, theattacker has the option of preparing the execution envi-ronment (i.e., modifying the content of the data, heap,and stack areas) so that the application code is forced toreturn to the attacker code at some point after the sys-tem call. The task can be more formally described asfollows: Given a program p, an address s, and an ad-dress t, find a configuration C such that, when p is exe-cuted starting from address s, control flow will eventuallyreach the target address t. For our purposes, a configura-tion C comprises all values that the attacker can modify.

This includes the contents of all processor registers andall writable memory regions (in particular, the stack, theheap, and the data segment). However, the attacker can-not tamper with write-protected segments such as codesegments or read-only data.

Regaining control flow usually requires that a codepointer is modified appropriately. Two prominent classesof code pointers that an attacker can target are functionpointers and stack return addresses. Other exploitablecode pointers include longjmp buffers.

A function pointer can be modified directly by code in-jected by the attacker before control is returned to the ap-plication to make a system call. Should the applicationlater use this function pointer, control is returned to the at-tacker code. This paper focuses on binary code compiledfrom C source code, hence we analyze where functionpointers can appear in such executables. One instance iswhen the application developer explicitly declares pointervariables to functions at the C language level; whenevera function pointer is used by the application, control canbe recovered by changing the pointer variable to containthe address of malicious code. However, although func-tion pointers are a commonly used feature in many Cprograms, there might not be sufficient instances of suchfunction invocations to successfully perform a completeexploit.

A circumstance in which function pointers are used morefrequently is the invocation of shared library functions bydynamically linked ELF (executable and linking format)binaries. When creating dynamically linked executables,a special section (called procedure linkage table – PLT) iscreated. The PLT is used as an indirect invocation methodfor calls to globally defined functions. This mechanismallows for the delayed binding of a call to a globally de-fined function. At a high level, this means that the PLTstores the addresses of shared library functions. When-ever a shared function is invoked, an indirect jump is per-formed to the corresponding address. This provides theattacker with the opportunity to modify the address of alibrary call in the PLT to point to attacker code. Thus,whenever a library function call is made, the intruder canregain control.

The redirection of shared library calls is a very effectivemethod of regaining control, especially when one consid-ers the fact that applications usually do not invoke systemcalls directly. Instead, almost all system calls are invokedthrough shared library functions. Thus, it is very proba-ble that an application executes a call to a shared functionbefore every system call. However, this technique is onlyapplicable to dynamically linked binaries. For statically


linked binaries, alternative mechanisms to recover controlflow must be found.

One such mechanism is the modification of the functionreturn addresses on the stack. Unfortunately (from thepoint of view of the attacker), these addresses cannot bedirectly overwritten by the malicious code. The reason, asmentioned previously, is that these addresses are checkedat every system call. Thus, it is necessary to force the ap-plication to overwrite the return address after the attackerhas relinquished control (and the first system call has fin-ished). Also, because the stack is analyzed at every sys-tem call, no further system calls may be invoked betweenthe time when the return address is modified and the timewhen this forged address is used in the function epilogue(i.e., by the ret instruction).

In principle, every application instruction that writes adata value to memory can be potentially used to modifya function return address. In the Intel x86 instruction set,there is no explicit store instruction. Being based on aCISC architecture, many instructions can specify a mem-ory location as the destination where the result of an oper-ation is stored. The most prominent family of instructionsthat write data to memory are the data transfer instructions(using the mov mnemonic).

int global;

void f() {

global = 0;}

movl $0x0,0x8049578

(a) Direct variable access

int global;

void f() {

int *p = &global;

*p = 0;}

movl $0x8049578,0xfffffffc(%ebp)mov 0xfffffffc(%ebp),%eaxmovl $0x0,(%eax)

(b) Variable access via pointer

int index;int array[];

void f() {

array[index] = 0;}

mov 0x80495a0,%eaxmovl $0x0,0x80495c0(,%eax,4)

(c) Array access

Figure 2: Unsuitable store instructions.

Of course, not all instructions that write to memory canbe actually used to alter a return address. For example,consider the C code fragments and their correspondingmachine instructions shown in Figure 2. In the first ex-ample (a), the instruction writes to a particular address(0x8049578, the address of the variable global), whichis specified by an immediate operand of the instruction.This store instruction clearly cannot be forced to over-write an arbitrary memory address. In the other two cases((b) and (c)), the instruction writes to a location whose ad-dress is determined (or influenced) by a value in a regis-ter. However, in example (b), the involved register %eaxhas been previously loaded with a constant value (againthe address of the variable global) that cannot be influ-enced by the intruder. Finally, even if the attacker canchoose the destination address of the store instruction, itis also necessary to be able to control the content that iswritten to this address. In example (c), the attacker canchange the content of the index variable before returningcontrol to the application. When the application then per-forms the array access using the modified index variable,which is loaded into register %eax, the attacker can writeto an (almost) arbitrary location on the stack. However,the constant value 0 is written to this address, making theinstruction not suitable to set a return address to the mali-cious code.

The examples above highlight the fact that even if appli-cation code contains many store instructions, only a frac-tion of them might be suitable to modify return addresses.Even if the original program contains assignments thatdereference pointers (or access array elements), it mightnot be possible to control both the destination of the storeinstruction and its content. The possibility of using anassignment operation through a pointer to modify the re-turn address on the stack was previously discussed in [7].However, the authors did not address the problem that anassignment might not be suitable to perform the actualoverwrite. Moreover, if a suitable instruction is found,preparing the environment is often not a trivial task. Con-sider a situation where an application first performs anumber of operations on a set of variables and later storesonly the result. In this case, the attacker has to set up theenvironment so that the result of these operations exactlycorrespond to the desired value. In addition, one has toconsider the effects of modifications to the environmenton the control flow of the application.

A simple example is shown in Figure 3. Here, the attackerhas to modify the variable index to point to the (return) ad-dress on the stack that should be overwritten. The valuethat is written to this location (i.e., the new return address)is determined by calculating the sum of two variables aand b. Also, one has to ensure that a > 0 because other-wise the assignment instruction would not be executed.



int index, a, b;int array[];

void f() {

if (a > 0) array[index] = a + b;}

(a) Possible overwrite

Figure 3: Possibly vulnerable code.

The presented examples serve only as an indication of thechallenges that an attacker faces when attempting to man-ually comprehend and follow different threads of execu-tion through a binary program. To perform a successfulattack, it is necessary to take into account the effects ofoperations on the initial environment and consider differ-ent paths of execution (including loops). Also, one hasto find suitable store instructions or indirect function callsthat can be exploited to recover control. As a result, onemight argue that it is too difficult for an attacker to repeat-edly make system calls and recover control, which is nec-essary to perform a successful compromise. In the nextsection, we show that these difficulties can be overcome.

4 Symbolic Execution

This section describes in detail the static analysis tech-niques we use to identify and exploit possibilities for re-gaining control after the invocation of a system call. Asmentioned previously, control can be regained when aconfigurationC is found such that control flow will even-tually reach the target address t when program p is exe-cuted starting from address s.

Additional constraints are required to make sure that aprogram execution does not violate the application modelcreated by the intrusion detection system. In particular,system calls may only be invoked in a sequence that isconsidered legitimate by the intrusion detection system.Also, whenever a system call is invoked, the chain offunction return addresses on the stack has to be valid.In our current implementation, we enforce these restric-tions simply by requiring that the target address t must bereached from s without making any intermediate systemcalls. In this way, we ensure that no checks are made bythe intrusion detection system before the attacker gets achance to execute her code. At this point, it is straightfor-ward to have the attack code rearrange the stack to pro-duce a valid configuration (correct chain of function re-turn addresses) and to make system calls in the correctorder.

The key approach that we use to find a configuration Cfor a program p and the two addresses s and t is sym-bolic execution [10]. Symbolic execution is a techniquethat interpretatively executes a program, using symbolicexpressions instead of real values as input. In our case,we are less concerned about the input to the program. In-stead, we treat all values that can be modified by the at-tacker as variables. That is, the execution environment ofthe program (data, stack, and heap) is treated as the vari-able input to the code. Beginning from the start addresss, a symbolic execution engine interprets the sequence ofmachine instructions.

To perform symbolic execution of machine instructions(in our case, Intel x86 operations), it is necessary to ex-tend the semantics of these instructions so that operandsare not limited to real data objects but can also be sym-bolic expressions. The normal execution semantics of In-tel x86 assembly code describes how data objects are rep-resented, how statements and operations manipulate thesedata objects, and how control flows through the state-ments of a program. For symbolic execution, the defi-nitions for the basic operators of the language have to beextended to accept symbolic operands and produce sym-bolic formulas as output.

4.1 Execution State

We define the execution state S of program p as a snap-shot of the content of the processor registers (except theprogram counter) and all valid memory locations at a par-ticular instruction of p, which is denoted by the programcounter. Although it would be possible to treat the pro-gram counter like any other register, it is more intuitive tohandle the program counter separately and to require thatit contains a concrete value (i.e., it points to a certain in-struction). The content of all other registers and memorylocations can be described by symbolic expressions.

Before symbolic execution starts from address s, theexecution state S is initialized by assigning symbolicvariables to all processor registers (except the programcounter) and memory locations in writable segments.Thus, whenever a processor register or a memory locationis read for the first time, without any previous assignmentto it, a new symbol is supplied from the list of variables{υl, υ2, υ3, . . .}. Note that this is the only time when sym-bolic data objects are introduced.

In our current system, we do not support floating pointdata objects and operations, so all symbols (variables)represent integer values. Symbolic expressions are linearcombinations of these symbols (i.e., integer polynomialsover the symbols). A symbolic expression can be writtenas cn ∗ υn + cn−1 ∗ υn−1 + . . . + c1 ∗ υ1 + c0 where the


eax: v0edx: v1

8049588 (j) : v2804958c (k): v38049590 (i) : v4

PC: 8048364

eax: v0edx: v2

8049588: (j) : v2804958c: (k): v38049590: (i) : v4

PC: 804836a

eax: v2edx: v2

8049588 (j) : v2804958c (k): v38049590 (i) : v4

PC: 804836c

eax: 2*v2edx: v2

8049588 (j) : v2804958c (k): v38049590 (i) : v4

PC: 804836e

eax: 3*v2edx: v2

8049588 (j) : v2804958c (k): v38049590 (i) : v4

PC: 8048370

eax: 3*v2+v3edx: v2

8049588 (j) : v2804958c (k): v38049590 (i) : v4

PC: 8048376

eax: 3*v2+v3edx: v2

8049588 (j) : v2804958c (k): v38049590 (i) : 3*v2+v3

PC: 804837b

8048364: mov 0x8049588,%edx 804836a: mov %edx,%eax 804836c: add %eax,%eax 804836e: add %edx,%eax 8048370: add 0x804958c,%eax 8048376: mov %eax,0x8049590 804837b:

int i, j, k;

void f(){

i = 3*j + k;}

Step 1 Step 2

Step 3 Step 4 Step 5 Step 6 Step 7

Figure 4: Symbolic execution.

ci are constants. In addition, there is a special symbol ⊥that denotes that no information is known about the con-tent of a register or a memory location. Note that thisis very different from a symbolic expression. Althoughthere is no concrete value known for a symbolic expres-sion, its value can be evaluated when concrete values aresupplied for the initial execution state. For the symbol⊥, nothing can be asserted, even when the initial state iscompletely defined.

By allowing program variables to assume integer polyno-mials over the symbols υi, the symbolic execution of as-signment statements follows naturally. The expression onthe right-hand side of the statement is evaluated, substitut-ing symbolic expressions for source registers or memorylocations. The result is another symbolic expression (aninteger is the trivial case) that represents the new value ofthe left-hand side of the assignment statement. Becausesymbolic expressions are integer polynomials, it is pos-sible to evaluate addition and subtraction of two arbitraryexpressions. Also, it is possible to multiply or shift a sym-bolic expression by a constant value. Other instructions,such as the multiplication of two symbolic variables or alogic operation (e.g., and, or), result in the assignmentof the symbol ⊥ to the destination. This is because theresult of these operations cannot (always) be representedas integer polynomial. The reason for limiting symbolicformulas to linear expressions will become clear in Sec-tion 4.3.

Whenever an instruction is executed, the execution state ischanged. As mentioned previously, in case of an assign-ment, the content of the destination operand is replacedby the right-hand side of the statement. In addition, theprogram counter is advanced. In the case of an instructionthat does not change the control flow of a program (i.e.,an instruction that is not a jump or a conditional branch),

the program counter is simply advanced to the next in-struction. Also, an unconditional jump to a certain label(instruction) is performed exactly as in normal executionby transferring control from the current statement to thestatement associated with the corresponding label.

Figure 4 shows the symbolic execution of a sequence ofinstructions. In addition to the x86 machine instructions,a corresponding fragment of C source code is shown. Foreach step of the symbolic execution, the relevant partsof the execution state are presented. Changes betweenexecution states are shown in bold face. Note that thecompiler (gcc 3.3) converted the multiplication in theC program into an equivalent series of add machine in-structions.

4.2 Conditional Branches and Loops

To handle conditional branches, the execution state hasto be extended to include a set of constraints, called thepath constraints. In principle, a path constraint relates asymbolic expressionL to a constant. This can be used, forexample, to specify that the content of a register has to beequal to 0. More formally, a path constraint is a booleanexpression of the form L ≥ 0 or L = 0, in which L is aninteger polynomial over the symbols υ i. The set of pathconstraints forms a linear constraint system.

The symbolic execution of a conditional branch statementstarts in a fashion similar to its normal execution, by eval-uating the associated Boolean expression. The evalua-tion is done by replacing the operands with their corre-sponding symbolic expressions. Then, the inequality (orequality) is transformed and converted into the standardform introduced above. Let the resulting path constraintbe called q.



eax: v0edx: v1

8049588 (j): v2804958c (i): v3

PC: 804836b

Path Condition:

8048364: cmpl $0x2a,0x804958c 804836b: jle 8048379 804836d: movl $0x1,0x8049588 8048377: jmp 8048383 8048379: movl $0x0,0x8049588 8048383:

int i, j;

void f(){

if (i > 42) j = 1; else j = 0;}

Step 1

eax: v0edx: v1

8049588 (j): v2804958c (i): v3

PC: 804836d

Path Condition:(v3 - 42) > 0

Step 2a.

eax: v0edx: v1

8049588 (j): 1804958c (i): v3

PC: 8048377


Step 3a.

eax: v0edx: v1

8049588 (j): 1804958c (i): v3

PC: 8048383


Step 4a.

eax: v0edx: v1

8049588 (j): v2804958c (i): v3

PC: 8048379

Path Condition:(v3 - 42) � 0

Step 2b.

eax: v0edx: v1

8049588 (j): 0804958c (i): v3

PC: 8048383

Path Condition:(v3 - 42) 1 0

Step 3b.

Execution thread forks else continuationthen continuation

Figure 5: Handling conditional branches during symbolic execution.

To continue symbolic execution, both branches of thecontrol path need to be explored. The symbolic executionforks into two “parallel” execution threads: one threadfollows the then alternative, the other one follows theelse alternative. Both execution threads assume the ex-ecution state which existed immediately before the condi-tional statement but proceed independently thereafter. Be-cause the then alternative is only chosen if the conditionalbranch is taken, the corresponding path constraint q mustbe true. Therefore, we add q to the set of path constraintsof this execution thread. The situation is reversed for theelse alternative. In this case, the branch is not taken and qmust be false. Thus, ¬q is added to the path constraints inthis execution.

After q (or ¬q) is added to a set of path constraints,the corresponding linear constraint system is immediatelychecked for satisfiability. When the set of path con-straints has no solution, this implies that, independent ofthe choice of values for the initial configuration C, thispath of execution can never occur. This allows us to im-mediately terminate impossible execution threads.

Each fork of execution at a conditional statement con-tributes a condition over the variables υ i that must holdin this particular execution thread. Thus, the set of pathconstraints determines which conditions the initial execu-tion state must satisfy in order for an execution to followthe particular associated path. Each symbolic executionbegins with an empty set of path constraints. As assump-tions about the variables are made (in order to choose be-tween alternative paths through the program as presentedby conditional statements), those assumptions are added

to the set. An example of a fork into two symbolic execu-tion threads as the result of an if-statement and the cor-responding path constraints are shown in Figure 5. Notethat the if-statement was translated into two machine in-structions. Thus, special code is required to extract thecondition on which a branch statement depends.

Because a symbolic execution thread forks into twothreads at each conditional branch statement, loops rep-resent a problem. In particular, we have to make sure thatexecution threads “make progress” to achieve our objec-tive of eventually reaching the target address t. The prob-lem is addressed by requiring that a thread passes throughthe same loop at most three times. Before an executionthread enters the same loop for the forth time, its execu-tion is halted. Then, the effect of an arbitrary number ofiterations of this loop on the execution state is approxi-mated. This approximation is a standard static analysistechnique [2, 13] that aims at determining value rangesfor the variables that are modified in the loop body. Sincethe problem of finding exact ranges and relationships be-tween variables is undecidable in the general case, the ap-proximation naturally involves a certain loss of precision.After the effect of the loop on the execution thread wasapproximated, the thread can continue with the modifiedstate after the loop.

To approximate the effect of the loop body on an execu-tion state, a fixpoint for this loop is constructed. For ourpurposes, a fixpoint is an execution state F that, whenused as the initial state before entering the loop, is equiv-alent to the final execution state when the loop finishes. Inother words, after the operations of the loop body are ap-


int j, k;

void f() {

int i = 0; j = k = 0;

while (i < 100) { k = 1; if (i == 10) j = 2; i++; }

}

i = 1;j = 0;k = 1;

i = 2;j = 0;k = 1;

i = ;j = 0;k = 1;

i = ;j = 0;k = 1;

i = ;j = 2;k = 1;

S1 S2 S3

S4

S5

S6

S7

S8

i = ;j = ;k = 1;

i = ;j = ;k = 1;

i = ;j = ;k = 1;

Figure 6: Fixpoint calculation.

plied to the fixpoint state F , the resulting execution stateis again F . Clearly, if there are multiple paths through theloop, the resulting execution states at each loop exit mustbe the same (and identical to F ). Thus, whenever the ef-fect of a loop on an execution state must be determined,we transform this state into a fixpoint for this loop. Thistransformation is often called widening. Then, the threadcan continue after the loop using the fixpoint as its newexecution state.

The fixpoint for a loop is constructed in an iterative fash-ion as follows: Starting with the execution state S1 afterthe first execution of the loop body, we calculate the ex-ecution state S2 after a second iteration. Then, S1 andS2 are compared. For each register and each memory lo-cation that hold different values (i.e., different symbolicexpressions), we assign⊥ as the new value. The resultingstate is used as the new state and another iteration of theloop is performed. This is repeated until S i and S(i+1) areidentical. In case of multiple paths through the loop, thealgorithm is extended by collecting one exit state S i foreach path and then comparing all pairs of states. When-ever a difference between a register value or a memorylocation is found, this location is set to ⊥. The iterativealgorithm is guaranteed to terminate, because at each step,it is only possible to convert the content of a memory lo-cation or a register to ⊥. Thus, after each iteration, thestates are either identical or the content of some locationsis made unknown. This process can only be repeated untilall values are converted to unknown and no information isleft.

An example for a fixpoint calculation (using C code in-stead of x86 assembler) is presented in Figure 6. In thiscase, the execution state comprises of the values of thethree involved variables i, j, and k. After the first loopiteration, the execution state S1 is reached. Here, i hasbeen incremented once, k has been assigned the constant1, and j has not been modified. After a second iteration,S2 is reached. Because i has changed between S1 and S2,

its value is set to ⊥ in S3. Note that the execution hasnot modified j, because the value of i was known to bedifferent from 10 at the if-statement. Using S3 as thenew execution state, two paths are taken through the loop.In one case (S4), j is set to 2, in the other case (S5), thevariable j remains 0. The reason for the two different ex-ecution paths is the fact that i is no longer known at theif-statement and, thus, both paths have to be followed.Comparing S3 with S4 and S5, the difference betweenthe values of variable j leads to the new state S6 in whichj is set to ⊥. As before, the new state S6 is used for thenext loop iteration. Finally, the resulting states S7 and S8

are identical to S6, indicating that a fixpoint is reached.

In the example above, we quickly reach a fixpoint. In gen-eral, by considering all modified values as unknown (set-ting them to ⊥), the termination of the fixpoint algorithmis achieved very quickly. However, the approximationmight be unnecessarily imprecise. For our current pro-totype, we use this simple approximation technique [13].However, we plan to investigate more sophisticated fix-point algorithms in the future.

To determine loops in the control flow graph, we use thealgorithm by Lengauer-Tarjan [12], which is based ondominator trees. Note, however, that the control flowgraph does not take into account indirect jumps. Thus,whenever an indirect control flow transfer instruction isencountered during symbolic execution, we first checkwhether this instruction can be used to reach the targetaddress t. If this is not the case, the execution thread isterminated at this point.

4.3 Generating Configurations

As mentioned in Section 4, the aim of the symbolic ex-ecution is to identify code pointers that can be modifiedto point to the attacker code. To this end, indirect jumpand function call instructions, as well as data transfer in-structions (i.e., x86mov) that could overwrite function re-



80482a8: jmp *0x80495a4 <printf>...

8048394: cmpl $0x0,0x80495b4804839b: je 80483a9804839d: movl $0x80484a4,(%esp,1)80483a4: call 80482a880483a9:

int flag;

void f(){

if (flag) printf("flag\n");}

Path Constraint:

*0x80495b4 > 0

Jump Target Constraint:

*0x80495a4 == tU

=Linear Inequality System:

*0x80495b4 > 0*0x80495a4 == t

Figure 7: Deriving an appropriate configuration.

turn addresses, are of particular interest. Thus, wheneverthe symbolic execution engine encounters such an instruc-tion, it is checked whether it can be exploited.

An indirect jump (or call) can be exploited, if it is pos-sible for the attacker to control the jump (or call) target.In this case, it would be easy to overwrite the legitimatetarget with the address t of the attacker code. To deter-mine whether the target can be overwritten, the currentexecution state is examined. In particular, the symbolicexpression that represents the target of the control trans-fer instruction is analyzed. The reason is that if it werepossible to force this symbolic expression to evaluate to t,then the attacker could achieve her goal.

Let the symbolic expression of the target of the controltransfer instruction be called st. To check whether it ispossible to force the target address of this instruction tot, the constraint st = t is generated (this constraint sim-ply expresses the fact that st should evaluate to the targetaddress t). Now, we have to determine whether this con-straint can be satisfied, given the current path constraints.To this end, the constraint st = t is added to the pathconstraints, and the resulting linear inequality system issolved.

If the linear inequality system has a solution, then the at-tacker can find a configurationC (i.e., she can prepare theenvironment) so that the execution of the application codeusing this configuration leads to an indirect jump (or call)to address t. In fact, the solution to the linear inequal-ity system directly provides the desired configuration. Tosee this, recall that the execution state is a function of theinitial state. As a result, the symbolic expressions are in-teger polynomials over variables that describe the initialstate of the system, before execution has started from ad-dress s. Thus, a symbolic term expresses the current valueof a register or a memory location as a function of the ini-tial values. Therefore, the solution of the linear inequalitysystem denotes which variables of the initial state have tobe set, together with their appropriate values, to achieve

the desired result. Because the configuration fulfills thepath constraints of the current symbolic execution thread,the actual execution will follow the path of this thread.Moreover, the target value of the indirect control trans-fer instruction will be t. Variables that are not part of thelinear inequality system do not have an influence on thechoice of the path or on the target address of the controlflow instruction, thus, they do not need to be modified.

As an example, consider the sequence of machine instruc-tions (and corresponding C source code) shown in Fig-ure 7. In this example, the set of path constraints at theindirect jump consists of a single constraint that requiresflag (stored at address 0x80495b4) to be greater than0. After adding the constraint that requires the jump tar-get (the address of the shared library function printfstored at 0x80495a4) to be equal to t, the inequality sys-tem is solved. In this case, the solution is trivial: the con-tent of the memory location that holds the jump target isset to t and variable flag is set to 1. In fact, any valuegreater than 0 would be suitable for flag, but our con-straint solver returns 1 as the first solution.

The handling of data transfer instructions (store opera-tions) is similar to the handling of control transfer instruc-tions. The only difference is that, for a data transfer in-struction, it is necessary that the destination address ofthe operation be set to a function return address and thatthe source of the operation be set to t. If this is the case,the attacker can overwrite a function return address withthe address of the attacker code, and, on function return,control is recovered. For each data transfer instruction,two constraints are added to the linear inequation system.One constraint requires that the destination address of thestore operation is equal to the function return address. Theother constraint requires that the stored value is equal to t.Also, a check is required that makes sure that no systemcall is invoked between the modification of the functionreturn address and its use in the function epilogue (i.e., onfunction return). The reason is that the intrusion detec-tion system verifies the integrity of the call stack at each


system call. Note, however, that most applications do notinvoke system calls directly but indirectly using libraryfunctions, which are usually called indirectly via the PLT.To solve the linear constraint systems, we use the ParmaPolyhedral Library (PPL) [1]. In general, solving a lin-ear constraint system is exponential in the number of in-equalities. However, PPL uses a number of optimizationsto improve the run time in practice and the number of in-equalities is usually sufficiently small.

4.4 Memory Aliasing and Unknown Stores

In the previous discussion, two problems were ignoredthat considerably complicate the analysis for real pro-grams: memory aliasing and store operations to unknowndestination addresses.

Memory aliasing refers to the problem that two differentsymbolic expressions s1 and s2 point to the same address.That is, although s1 and s2 contain different variables,both expressions evaluate to the same value. In this case,the assignment of a value to an address that is specifiedby s1 has unexpected side effects. In particular, such anassignment simultaneously changes the content of the lo-cation pointed to by s2.

Memory aliasing is a typical problem in static analy-sis, which also affects high-level languages with pointers(such as C). Unfortunately, the problem is exacerbated atmachine code level. The reason is that, in a high-levellanguage, only a certain subset of variables can be ac-cessed via pointers. Also, it is often possible to performalias analysis that further reduces the set of variables thatmight be subject to aliasing. Thus, one can often guar-antee that certain variables are not modified by write op-erations through pointers. At machine level, the addressspace is uniformly treated as an array of storage locations.Thus, a write operation could potentiallymodify any othervariable.

In our prototype, we initially take an optimistic approachand assume that different symbolic expressions refer todifferent memory locations. This approach is motivatedby the fact that C compilers (we use gcc 3.3 for ourexperiments) address local and global variables so that adistinct expression is used for each access to a differentvariable. In the case of global variables, the address of thevariable is directly encoded in the instruction, making theidentification of the variable particularly easy. For eachlocal variable, the access is done by calculating a differentoffset to the value of the base pointer register (%ebp).

Of course, our optimistic assumption might turn out to beincorrect, and we assume the independence of two sym-bolic expressions when, in fact, they refer to the same

memory location. To address this problem, we introducean additional a posteriori check after a potentially ex-ploitable instruction was found. This check operates bysimulating the program execution with the new configu-ration that is derived from the solution of the constraintsystem.

In many cases, having a configuration in which sym-bolic variables have concrete numerical values allows oneto resolve symbolic expressions directly to unambiguousmemory locations. Also, it can be determined with cer-tainty which continuation of a conditional branch is taken.In such cases, we can guarantee that control flow willbe successfully regained. In other cases, however, notall symbolic expressions can be resolved and there is a(small) probability that aliasing effects interfere with ourgoal. In our current system, this problem is ignored. Thereason is that an attacker can simply run the attack tocheck whether it is successful or not. If the attack fails,one can manually determine the reason for failure andprovide the symbolic execution engine with aliasing in-formation (e.g., adding constraints to specify that two ex-pressions are identical). In the future, we will exploremechanisms to automatically derive constraints such thatall symbolic expressions can be resolved to a concretevalue.

A store operation to an unknown address is related to thealiasing problem as such an operation could potentiallymodify any memory location. Again, we follow an op-timistic approach and assume that such a store operationdoes not interfere with any variable that is part of the so-lution of the linear inequality system (and thus, part of theconfiguration) and use simulation to check the validity ofthis assumption.

5 Experimental Results

This section provides experimental results that demon-strate that our symbolic execution technique is capable ofgenerating configurationsC in which control is recoveredafter making a system call (and, in doing so, temporar-ily transferring control to the application program). Forall experiments, the programs were compiled using gcc3.3 on a x86 Linux host. Our experiments were carriedout on the binary representation of programs, without ac-cessing the source code.

For the first experiment, we attempted to exploit threesample programs that were protected by the intrusion de-tection systems presented in [4] and [14]. The first vulner-able program is shown in Figure 8. This program starts byreading a password from standard input. If the passwordis correct (identical to the hard-coded string “secret”), a



command is read from a file and then executed with supe-ruser privileges. Also, the program has a logging facilitythat can output the command and the identifier of the userthat has initially launched the program. The automaton inFigure 9 shows the relevant portion of the graph that de-termines the sequence of system calls that are permittedby the intrusion detection system. The first read systemcall corresponds to the reading of the password (on line23), while the execve call corresponds to the executionof the command obtained from the file (on line 30). Notethe two possible sequences that result because commandscan be either logged or not.

1: #define CMD_FILE "commands.txt" 2: 3: int enable_logging = 0; 4: 5: int check_pw(int uid, char *pass) 6: { 7: char buf[128]; 8: strcpy(buf, pass); 9: return !strcmp(buf, "secret");10: }11:12: int main(int argc, char **argv)13: {14: FILE *f;15: int uid;16: char passwd[256], cmd[128];17:18: if ((f = fopen(CMD_FILE, "r")) == NULL) {19: perror("error: fopen"); exit(1);20: }21: 22: uid = getuid();23: fgets(passwd, sizeof(passwd), stdin);24: 25: if (check_pw(uid, passwd)) {26: fgets(cmd, sizeof(cmd), f);27: if (enable_logging) 28: printf("uid [%d]: %s\n", uid, cmd);29: setuid(0);30: if (execl(cmd, cmd, 0) < 0) {31: perror("error: execl"); exit(1);32: }33: }34: }

Figure 8: First vulnerable program.

It can be seen that the program suffers from a simplebuffer overflow vulnerability in the check pw() func-tion (on line 8). This allows an attacker to inject code andto redirect the control flow to an arbitrary location. Onepossibility to redirect control would be directly after thecheck of the password, before the call to the fgets()function (on line 26). However, in doing so, the attackercan not modify the command that is being executed be-cause fgets() is used to retrieve the command. Onesolution to this problem could be to first modify the con-tent of the command buffer cmd, and then jump directly

to the setuid() function, bypassing the part that readsthe legitimate command from the file. By doing so, how-ever, an alarm is raised by the intrusion detection systemthat observes an invalid setuid system call while ex-pecting a read. To perform a classic mimicry attack, theintruder could simply issue a bogus read call, but, in ourcase, such a call would be identified as illegal as well. Thereason is that the source of the system call would not bethe expected instruction in the application code.

read

fstat64

setuid32

execve

write

mmap2

read

Figure 9: Fragment of automaton that captures permittedsystem call sequences.

To exploit this program such that an arbitrary commandcan be executed in spite of the checks performed by the in-trusion detection system, it is necessary to regain controlafter the call to fgets(). In this case, the attacker couldreplace the name of the command and then continue exe-cution with the setuid() library function. To this end,our symbolic execution engine is used to determine a con-figuration that allows the attacker to recover control afterthe fgets() call. For the first example, a simple config-uration is sufficient in which enable logging is setto 1 and the shared library call to printf() is replacedwith a jump to the attacker code. With this configuration,the conditional branch (on line 27) is taken, and insteadof calling printf(), control is passed to the attackercode. This code can then change the cmd parameter ofthe subsequent execve() call (on line 30) and contin-ues execution of the original program on line 29. Notethat the intrusion detection system is evaded because allsystem calls are issued by instructions in the applicationcode segment and appear in the correct order.

The buffer overflow in check pw() is used to inject theexploit code that is necessary to set up the environment.After the environment is prepared, control is returned tothe original application before fgets(), bypassing the


password check routine. Our system is generating actualexploit code that handles the setup of a proper configura-tion. Thus, this and the following example programs weresuccessfully exploited by regaining control and chang-ing the command that was executed by execl(). In allcases, the attacks remained undetected by the used intru-sion detection systems [4, 14].

As a second example, consider a modified version of theinitial program as shown in Figure 10. In this example,the call to printf() is replaced with the invocation ofthe custom audit function do log(), which records theidentifier of the last command issued by each user (withuid < 8192). To this end, a unique identifier calledcmd id is stored in a table that is indexed by uid (online 6).

1: int enable_logging = 0; 2: int cmd_id = 0; 3: int uid_table[8192];...

4: void do_log(int uid) 5: { 6: uid_table[uid] = cmd_id++; 7: }...

8: int main(int argc, char **argv)... 9: if (check_pw(uid, passwd)) {10: fgets(cmd, sizeof(cmd), f);11: if (enable_logging) 12: do_log(uid);13: setuid(0);14: if (execl(cmd, cmd, 0) < 0) {15: perror("error: execl"); exit(1);16: }17: }18: }

Figure 10: Second vulnerable program.

For this example, the application was statically linked sothat we cannot intercept any shared library calls. As forthe previous program, the task is to recover control afterthe fgets() call on line 10. Our symbolic executionengine successfully determined that the assignment to thearray on line 6 can be used to overwrite the return ad-dress of do log(). To do so, it is necessary to assign avalue to the local variable uid so that when this valueis added to the start address of the array uid table,the resulting address points to the location of the returnaddress of do log(). Note that our system is capableof tracking function calls together with the correspondingparameters. In this example, it is determined that the lo-cal variable uid is used as a parameter that is later usedfor the array access. In addition, it is necessary to storethe address of the attack code in the variable cmd id and

turn on auditing by setting enable logging to a value�= 0.

One might argue that it is not very realistic to store identi-fiers in a huge table when most entries are 0. Thus, for thethird program shown in Figure 11, we have replaced thearray with a list. In this example, the do log() functionscans a linked list for a record with a matching user iden-tification (on lines 12–14). When an appropriate recordalready exists, the cmd id field of the cmd entry struc-ture is overwritten with the global command identifiercmd id. When no suitable record can be found, a newone is allocated and inserted at the beginning of the list(on lines 16–21).

1: struct cmd_entry { 2: int cmd_id; unsigned int uid; 3: struct cmd_entry *next; 4: }; 5: int enable_logging = 0; 6: int cmd_id = 0; 7: struct cmd_entry *cmds = NULL;...

8: void do_log(int uid) 9: {10: struct cmd_entry *p;11: 12: for (p = cmds; p != NULL; p = p->next)13: if (p->uid == uid)14: break;15: 16: if (p == NULL) {17: p = (struct cmd_entry *) calloc(1, sizeof(struct cmd_entry));18: p->uid = uid;19: p->next = cmds;20: cmds = p;21: }22:23: p->cmd_id = cmd_id++;24: }...

25: int main(int argc, char **argv)...26: if (check_pw(uid, passwd)) {27: fgets(cmd, sizeof(cmd), f);28: if (enable_logging) 29: do_log(uid);30: setuid(0);31: if (execl(cmd, cmd, 0) < 0) {32: perror("error: execl"); exit(1);33: }34: }35: }

Figure 11: Third vulnerable program.

When attempting to find a suitable instruction to directcontrol flow back to the attacker code, the operation online 23 seems appropriate. The reason is that this state-ment assigns the global variable cmd id to the field of astructure that is referenced by the pointer variable p. Un-



fortunately, p is not under direct control of the attacker.This is because in the initialization part of the for-loopon line 12, the content of the pointer to the global list headcmds is assigned to p. In addition, the loop traversesthe list of command records until a record is found wherethe uid field is equivalent to the single parameter (uid)of the do log() function. If, at any point, the nextpointer of the record pointed to by p is NULL, the loopterminates. Then, a freshly allocated heap address is as-signed to p on line 17. When this occurs, the destinationof the assignment statement on line 23 cannot be forcedto point to the function return address anymore, which islocated on the stack.

Function Parameter(int uid)

Return Address

Old Frame Pointer

Local Variables

struct cmd_entry *next;

unsigned int uid;

int cmd_id;

p =p->cmd_id

p->uid

Stack struct cmd_entry pointed to by p

Figure 12: Successful return address overwrite via p.

The discussion above underlines that even if a pointer as-signment is found, it is not always clear whether this as-signment can be used to overwrite a return address. Forthis example, our symbolic execution engine discovered apossibility to overwrite the return address of do log().This is achieved by preparing a configuration in whichthe cmds variable points directly to the return address ofdo log(). After the content of cmds is assigned to p,p→uid is compared to the uid parameter on line 13.Because of the structure of the cmd entry record, thiscomparison always evaluates to true. To see why thisis the case, refer to Figure 12. The figure shows thatwhen p points to the function’s return address, p→uidpoints to the location that is directly “above” this ad-dress in memory. Because of the x86 procedure callingconvention, this happens to be the first argument of thedo log() function. In other words, p→uid and theparameter uid refer to the same memory location, there-fore, the comparison has to evaluate to true. As before,for a successful overwrite, it is necessary to set the value

of cmd id to t and enable auditing by assigning 1 toenable logging.

Without the automatic process of symbolic execution,such an opportunity to overwrite the return address isprobably very difficult to spot. Also, note that no knowl-edge about the x86 procedure calling convention is en-coded in the symbolic execution engine. The possibilityto overwrite the return address, as previously discussed,is found directly by (symbolically) executing the machineinstructions of the binary. If the compiler had arrangedthe fields of the cmd entry structure differently, or if adifferent calling convention was in use, this exploit wouldnot have been found.

For the second experiment, we used our symbolic execu-tion tool on three well-known applications: apache2,the netkit ftpd server, and imapd from the Univer-sity of Washington. The purpose of this experiment wasto analyze the chances of an attacker to recover controlflow in real-world programs. To this end, we randomly se-lected one hundred addresses for each program that wereevenly distributed over the code sections of the analyzedbinaries. From each address, we started the symbolic ex-ecution processes. The aim was to determine whether it ispossible to find a configuration and a sequence of instruc-tions such that control flow can be diverted to an arbitraryaddress. In the case of a real attack, malicious code couldbe placed at this address. Note that all applications weredynamically linked (which is the default on modern Unixmachines).

Program Instr. Success FailedReturn Exhaust

apache2 51,862 83 12 5ftpd 9,127 93 7 0imapd 133,427 88 11 1

Table 1: Symbolic execution results for real-world appli-cations.

Table 1 summarizes the results for this experiment. Foreach program, the number of code instructions (column“Instr.”) are given. In addition, the table lists the numberof test cases for which our program successfully found aconfiguration (column “Success”) and the number of testcases for which such a configuration could not be found(column “Failed”).

In all successful test cases, only a few memory locationshad to be modified to obtain a valid configuration. In fact,in most cases, only a single memory location (a functionaddress in the PLT) was changed. The code that is neces-sary to perform these modifications is in the order of 100


bytes and can be easily injected remotely by an attackerin most cases.

A closer examination of the failed test cases revealed thata significant fraction of these cases occurred when thesymbolic execution thread reached the end of the func-tion where the start address is located (column “Return”).In fact, in several cases, symbolic execution terminatedimmediately because the randomly chosen start addresshappened to be a ret instruction. Although the symbolicexecution engine simulates the run-time stack, and thuscan perform function calls and corresponding return op-erations, a return without a previous function call cannotbe handled without additional information. The reason isthat whenever a symbolic execution thread makes a func-tion call, the return address is pushed on the stack andcan be used later by the corresponding return operation.However, if symbolic execution begins in the middle of afunction, when this initial function completes, the returnaddress is unknown and the thread terminates.

When an intruder is launching an actual attack, she usu-ally possesses additional information that can be madeavailable to the analysis process. In particular, possiblefunction return addresses can be extracted from the pro-gram’s call graph or by examining (debugging) a runninginstance of the victim process. If this information is pro-vided, the symbolic evaluation process can continue at thegiven addresses. Therefore, the remaining test cases (col-umn “Exhaust”) are of more interest. These test instancesfailed because the symbolic execution process could notidentify a possibility to recover control flow. We set alimit of 1,000 execution steps for each thread. After that,a thread is considered to have exhausted the search spaceand it is stopped. The reason for this limit is twofold.First, we want to force the analysis to terminate. Second,when the step limit is reached, many memory locationsand registers already contain unknown values.

Our results indicate that only a small amount of test casesfailed because the analysis engine was not able to identifyappropriate configurations. This supports the claim thatour proposed evasion techniques can be successfully usedagainst real-world applications.

Program Steps TimeAvg. Max. Min. (in seconds)

apache2 24 131 0 12.4ftpd 7 62 0 0.3imapd 46 650 0 1.2

Table 2: Execution steps and time to find configurations.

Table 2 provides more details on the number of steps re-quired to successfully find a configuration. In this table,the average, maximum, and minimum number of stepsare given for the successful threads. The results showthat, in most cases, a configuration is found quickly, al-though there are a few outliers (for example, 650 steps forone imapd test case). Note that all programs containedat least one case for which the analysis was immediatelysuccessful. In these cases, the random start instructionwas usually an indirect jump or indirect call that could beeasily redirected.

The table also lists the time in seconds that the symbolicexecution engine needed to completely check all hundredstart addresses (successful and failed cases combined)for each program. The run time for each individual testcase varies significantly, depending on the amount of con-straints that are generated and the branching factor of theprogram. When a program contains many branches, thesymbolic execution process has to follow many differentthreads of execution, which can generate an exponentialpath explosion in the worst case. In general, however, therun time is not a primary concern for this tool and the re-sults demonstrate that the system operates efficiently onreal-world input programs.

6 Conclusions

In this paper, we have presented novel techniques to evadetwo well-known intrusion detection systems [4, 14] thatmonitor system calls. Our techniques are based on theidea that application control flow can be redirected to ma-licious code after the intruder has passed control to theapplication to make a system call. Control is regainedby modifying the process environment (data, heap, andstack segment) so that the program eventually follows aninvalid code pointer (a function return address or an indi-rect control transfer operation). To this end, we have de-veloped a static analysis tool for x86 binaries, which usessymbolic execution. This tool automatically identifies in-structions that can be used to redirect control flow. In ad-dition, the necessary modification to the environment arecomputed and appropriate code is generated. Using oursystem, we were able to successfully exploit three sam-ple programs, evading state-of-the-art system call moni-tors. In addition, we applied our tool to three real-worldprograms to demonstrate the general applicability of ourtechniques.

The static analysis mechanisms that we developed for thispaper could be used for a broader range of binary anal-ysis problems in the future. One possible application isthe identification of configurations for which the currentfunction’s return address is overwritten. This might allow



us to build a tool that can identify buffer overflow vulnera-bilities in executable code. Another application domain isthe search for viruses. Since malicious code is usually notavailable as source code, binary analysis is a promisingapproach to deal with this problem. In addition, we hopethat our work has brought to attention the intrinsic prob-lem of defense mechanisms that allow attackers a largeamount of freedom in their actions.

Acknowledgments

This research was supported by the National Sci-ence Foundation under grants CCR-0209065 and CCR-0238492.

References

[1] R. Bagnara, E. Ricci, E. Zaffanella, and P. M. Hill.Possibly not closed convex polyhedra and the ParmaPolyhedra Library. In 9th International Symposiumon Static Analysis, 2002.

[2] P. Cousot and R. Cousot. Abstract Interpretation:A Unified Lattice Model for Static Analysis of Pro-grams by Construction or Approximation of Fix-points. In 4th ACM Symposium on Principles of Pro-gramming Languages (POPL), 1977.

[3] H. Feng, J. Giffin, Y. Huang, S. Jha, W. Lee, andB. Miller. Formalizing sensitivity in static analysisfor intrusion detection. In IEEE Symposium on Se-curity and Privacy, 2004.

[4] H. Feng, O. Kolesnikov, P. Fogla, W. Lee, andW. Gong. Anomaly detection using call stack in-formation. In IEEE Symposium on Security and Pri-vacy, 2003.

[5] S. Forrest. A Sense of Self for UNIX Processes. InIEEE Symposium on Security and Privacy, 1996.

[6] D. Gao, M. Reiter, and D. Song. Gray-Box Extrac-tion of Execution Graphs for Anomaly Detection. In11th ACM Conference on Computer and Communi-cation Security (CCS), 2004.

[7] D. Gao, M. Reiter, and D. Song. On Gray-BoxProgram Tracking for Anomaly Detection. In 13thUsenix Security Symposium, 2004.

[8] J. Giffin, S. Jha, and B. Miller. Detecting Manipu-lated Remote Call Streams. In 11th Usenix SecuritySymposium, 2002.

[9] J. Giffin, S. Jha, and B.P. Miller. Efficient context-sensitive intrusion detection. In 11th Network andDistributed System Security Symposium (NDSS),2004.

[10] J. King. Symbolic Execution and Program Testing.Communications of the ACM, 19(7), 1976.

[11] L. Lam and T. Chiueh. Automatic Extraction of Ac-curate Application-Specific Sandboxing Policy. InSymposium on Recent Advances in Intrusion Detec-tion (RAID), 2004.

[12] T. Lengauer and R. Tarjan. A Fast Algorithm forFinding Dominators in a Flowgraph. ACM Trans-actions on Programming Languages and Systems,1(1), 1979.

[13] F. Nielson, H. Nielson, and C. Hankin. Principles ofProgram Analysis. Springer Verlag, 1999.

[14] R. Sekar, M. Bendre, D. Dhurjati, and P. Bolli-neni. A fast automaton-based method for detectinganomalous program behaviors. In IEEE Symposiumon Security and Privacy, 2001.

[15] K. Tan, K. Killourhy, and R. Maxion. Underminingan Anomaly-Based Intrusion Detection System Us-ing Common Exploits. In 5th Symposium on RecentAdvances in Intrusion Detection (RAID), 2002.

[16] D. Wagner and D. Dean. Intrusion Detection viaStatic Analysis. In IEEE Symposium on Security andPrivacy, 2001.

[17] D. Wagner and P. Soto. Mimicry Attacks on Host-Based Intrusion Detection Systems. In 9th ACMConference on Computer and Communications Se-curity (CCS), 2002.

[18] C. Warrender, S. Forrest, and B.A. Pearlmutter. De-tecting intrusions using system calls: Alternativedata models. In IEEE Symposium on Security andPrivacy, 1999.

[19] A. Wespi, M. Dacier, and H. Debar. Intrusion De-tection Using Variable-Length Audit Trail Patterns.In Recent Advances in Intrusion Detrection (RAID),2000.

[20] H. Xu, W. Du, and S. Chapin. Context SensitiveAnomaly Monitoring of Process Control Flow toDetect Mimicry Attacks and Impossible Paths. InSymposium on Recent Advances in Intrusion Detec-tion (RAID), 2004.


automating mimicry attacks using static binary analysis · keywords: binary analysis, static...

Documents