formalizing sensitivity in program models for intrusion detection

Formalizing Sensitivityin Program Models

for Intrusion Detection

Henry Hanping Feng

Yong Huang

University of Massachusetts

{hfeng,yhuang}@ecs.umass.edu

Jonathon T. GiffinSomesh Jha

Barton P. Miller

University of Wisconsin

{giffin,jha,bart}@cs.wisc.edu

Wenke Lee

Georgia Instituteof Technology

[email protected]

11 May 2004 Henry Feng & Jonathon Giffin 2

Important Ideas• Formalizing program models facilitates

understanding & comparison.

• Exposing additional program state improves monitoring speed & model accuracy.– VPStatic model: reads program’s call stack– Dyck model: instruments binary code


Model-Based Intrusion Detection

• Build model of correct program behavior

• Model: automaton specifying all valid system call sequences

• Runtime monitor ensures execution does not violate model

OperatingSystem

User Process


Model-Based Intrusion Detection

• Model must be fast to operate

• Model must accurately represent program– Context-sensitive

models restrict impossible pathsOperating

System

User Process


Code Examplechar *filename;

pid_t[2] pid;

int prepare (int index) {

char buf[20];

pid[index] = getpid();

strcpy(buf, filename);

return open(buf, O_RDWR);

}

getpid

open


Code Examplevoid action (void) { uid_t uid = getuid(); int handle;

if (uid != 0) { handle = prepare(1); read(handle, …); } else { handle = prepare(0); write(handle, …); }

close(handle);}

getuid

writeread

close

prepare prepare


NFA Model

getuid

writeread

close

prepare prepare

getpid

open

Function action Function prepare


NFA Model

getuid

writeread

close

getpid

open



Impossible Path Exploitvoid action (void) { uid_t uid = getuid(); int handle;

if (uid != 0) { handle = prepare(1); read(handle, …); } else { handle = prepare(0); write(handle, …); }

close(handle);}

getuid

writeread

close

prepare prepare


pop Y

push Y

pop X

PDA Model

getuid

writeread

close

getpid

open

push X



PDA Problems

• Impossible paths still exist– Non-determinism indicates missing execution

information

• PDA run-time state explosion– ε-edge identifiers maintained on a stack– Stack non-determinism is expensive– post* algorithm: cubic in automaton size

X

push Y

getuid getpid

push X


Determinize PDA• Expand the input alphabet by exposing

the stack operations and the target state of the transition– fa,p,z indicates consume input a, push z on the

stack, and transition to state p.– ga,p,z for pop operations.– ea,p for operations with no stack activity.

• Result in a Deterministic PDA (or DPDA).• Exposing only stack operations we get

PDA with deterministic stack operations (or sDPDA).


NFA

State non-determinism is cheap.

State non-determinism

unlink unlink


Non-Deterministic PDA

Stack non-determinism is expensive.


unlink unlinkpush Ypush X

Stack non-determinism


Deterministic PDA (DPDA)VPStatic Model

• Model exposes stack operations & target states

• Possible exponential increase in model Possible exponential increase in model size?size?





Stack-Deterministic PDA (sDPDA)Dyck Model

• Model exposes stack operations

• No increase in model size?No increase in model size?





Input Symbol Processing Complexity

• n is state count• m is transition count• k is PDA input alphabet size• r is PDA stack alphabet size

ModelTime

ComplexitySpace

ComplexityInput

Alphabet Size

PDA O(nm2) O(nm2) k

DPDA O(1) O(1) (knr)

sDPDA O(n) O(n) (kr)


VPStatic

• A variant of VtPath model.– DPDA.– Generated by static analysis of binary.

• Use Addr(state) to expose states.


Determinizing via Observation

• Extract return addresses from call stack into virtual stack list for each system call.

• Generate a bunch of input symbols for each system call.


Addr(C0)Addr(C1)

e(open,Addr(Sopen))

Determinizing via Observation

char *filename;

pid_t[2] pid;

int prepare (int index) {

char buf[20];

pid[index] = getpid();

strcpy(buf, filename);

return open(buf, O_RDWR);

}

…

Addr(C1)

…

e(none,Exit(prepare))g(none,Addr(C1),Addr(C1))e(none,Addr(C0)) f(none,Entry(prepare),Addr(C0))e(open,Addr(Sopen))

getpid open


pop Y]Y

[Y

]X

[X

push Y

pop X

Dyck Model

getuid

writeread

close

getpid

open

push X


Dyck Model

getuid [X getpid open ]X read close

getuid [Y getpid open ]Y write close

• Matching brackets are alphabet symbols– Expose stack operations to runtime monitor– Language of bracket symbols is a Dyck

language– Rewrite binary to generate bracket symbols


Determinizing via Binary Rewriting• Insert code to

generate bracket symbols around function call sites

• Notify monitor of stack activity

• Determinizes stack operations

void action (void) { uid_t uid = getuid(); int handle;

if (uid != 0) { precall(X); handle = prepare(1); postcall(X); read(handle, …); } else { precall(Y); handle = prepare(0); postcall(Y); write(handle, …); }

close(handle);}


push Ypush X[Y[X

Dyck ModelDyck model stack-determinizes PDA

Stack deterministism push Ypush X


Only one valid stack configuration


Test Programs

ProgramNumber of Instructions

htzipd 110,096

gzip 57,271

cat 52,601


Questions?• Henry Hanping Feng, Yong Huang

University of Massachusetts—Amherst {hfeng,yhuang}@ecs.umass.edu

• Jonathon T. Giffin, Somesh Jha, Barton P. MillerUniversity of Wisconsin—Madison{giffin,jha,miller}@cs.wisc.edu

• Wenke LeeGeorgia Institute of [email protected]

formalizing sensitivity in program models for intrusion detection

Documents