intrusion detection

Intrusion Detection

Somesh Jha University of Wisconsin

April 22, 2023 J. Giffin and S. Jha 2

Intrusion DetectionGoal: Discover attempts to maliciously

gain access to a system


Network Intrusion Detection Systems (NIDS)

• Inspects packets at certain vantage points– For example, behind the routers

• Look for malicious or anomalous behavior

• Much more fine-grained than firewalls– Example: drop a packet whose payload

“matches” a certain string


Nomenclature• Packet Classification

– Typically only look at packet headers– Enough to determine which firewall rules

apply• Deep packet inspection (DPI)

– Have to also inspect packet payloads– Needed for intrusion detection systems

(IDS) and intrusion prevention systems (IPS)


Classification of NIDS• Signature-based

– Also called misuse detection– Establish a database of malicious patterns– If a sequence of packets “matches” one of the

patterns, raise an alarm– Positives

• Good attack libraries• Easy to understand the results

– Negatives• Unable to detect new attacks or variants of old attacks

• Example– Cisco, Snort, Bro, Tippingpoint, NFR, …


Network Intrusion Prevention System (IPS)

• NIDS are generally “passive”– Raise alerts if something suspicious

happens• IPS are active

– Drop suspicious looking packages– Route certain packets for further

inspection– Main challenge: have to work at line

speeds


Classification of NIDS• Anomaly-based

– Establish a statistical profile of normal traffic– If monitored traffic deviates “sufficiently” from the

established profile, raise an alarm• Positives

– Can detect new attacks• Negatives

– High false alarm rate• High variability in normal traffic

– Intruder can go under the “radar”• Examples

– Mostly research systems


Classification of NIDS• Stateless

– Need to keep no state– Example: raise an alarm if you see a

packet that contains the pattern “mellissa”• Positives

– Very fast• Negatives

– For some attacks need to keep state


Evasion Attacks• Insertion, Evasion and Denial Of

Service: Eluding NIDS by T.H. Ptacek and T. Newsham

• Fragment the attack packet to break the keyword

• Reorder the packet• Any valid transformation that TCP

allows


Classification of NIDS• Stateful

– Keeps state– Sometime need to do reassembly

• Reassemble packets that belong to the same connection, e.g., packets that belong to the same ssh session

• Quite hard! (out-of-order delivery)– Positives

• Can detect more attacks– Negatives

• Requires too much memory


Typical Stages of a NIDS• Limited TCP reassembly

– Handle out-of-order packets and fragmentation

• Protocol Parsing and Normalization– Various fields of a protocol (e.g., HTTP)– Normalize inputs (e.g., convert all URL

names to a standard form)• Signature matching

– Simple keywords (fast algorithms exist)– Full regular expression matching (slow)


Snort

Filtered packet stream


libpcap• Takes the “raw” packet stream• Parses the packets and presents them

as a – Filtered packet stream

• Library for packet capture• Website for more details

– http://www-nrg.ee.lbl.gov/.


Malicious Pattern Examplealert tcp any any -> 10.1.1.0/24 80 (content: “/cgi-bin/phf”;

msg: “PHF probe!”;)

action pass log alert

protocolsource addresssource port

destination addressdestination port


Malicious Patterns Example• content: “/cgi-bin/phf”

– Matches any packet whose payload contains the string “/cgi-bin/phf”

– Look at http://www.cert.org/advisories/CA-1996-06.html

• msg: “PHF probe!”– Generate this message if a match happens

http://www.cert.org/advisories/CA-1996-06.html



More Examples

alert tcp any any -> 10.1.1.0/24 6000:6010 (msg: “X traffic”;)

alert tcp !10.1.1.0/24 any -> 10.1.1.0/24 6000:6010 (msg: “X traffic”;)


How to generate new patterns?• Buffer overrun found in Internet

Message Access Protocol (IMAP)– http://www.cert.org/advisories/CA-1997-09.

html• Run exploit in a test network and

record all traffic• Examine the content of the attack

packet




Notional "IMAP buffer overflow" packet

052499-22:27:58.403313 192.168.1.4:1034 -> 192.168.1.3:143TCP TTL:64 TOS:0x0 DF***PA* Seq: 0x5295B44E Ack: 0x1B4F8970 Win: 0x7D7890 90 90 90 90 90 90 90 90 90 90 90 90 90 EB 3B ...............;5E 89 76 08 31 ED 31 C9 31 C0 88 6E 07 89 6E 0C ^.v.1.1.1..n..n.B0 0B 89 F3 8D 6E 08 89 E9 8D 6E 0C 89 EA CD 80 .....n....n.....31 DB 89 D8 40 CD 80 90 90 90 90 90 90 90 90 90 [email protected] 90 90 90 90 90 90 90 90 90 90 E8 C0 FF FF FF ................2F 62 69 6E 2F 73 68 90 90 90 90 90 90 90 90 90 /bin/sh.........


Alert rule for the new buffer overflow

alert tcp any any -> 192.168.1.0/24 143 (content:"|E8C0 FFFF FF|/bin/sh"; msg:"New IMAP Buffer Overflow detected!";)

Can mix hex formatted bytecode and text


Advantages of Snort• Lightweight

– Small footprint– Focused monitoring: highly tuned Snort for

the SMTP server• Malicious patterns easy to develop

– Large user community– Consider the IRDP denial-of-service attack– Rule for this attack available on the same day

the attack was announced• Commercial company (Sourcefire) behind

it


Disadvantages• Does not perform full stream

reassembly– Attackers can use that to “fool” Snort– Break one attack packet into a stream

• Pattern matching is expensive– Matching patterns in payloads is expensive

(avoid it!)• Rule development methodology is

adhoc


Host-based ID• Monitor interaction between a specific

program and OS• Raise an alarm if suspicious “system

calls” are observed• Unlike NIDS monitoring happens at the

end hosts• Need to model

– Unusual behavior– Normal behavior


Goal: Discover attempts to maliciously gain access to a system

Specification-BasedMonitoring

•Specify constraints uponprogram behavior

•Ensure execution doesnot violate specification

•Our work; Ko, et. al.

•Specifications can becumbersome to create

Misuse Detection

•Specify patterns ofattack or misuse

•Ensure misuse patternsdo not arise at runtime

•Snort

•Rigid: cannot adaptto novel attacks

Anomaly Detection

•Learn typical behaviorof application

•Variations indicatepotential intrusions

• IDES

•High false alarm rate


Specification-Based Monitoring

• Two components:

– Specification: Indicates constraints upon program behavior

– Enforcement: How the specification is verified at runtime or from audit data


ExecutionObeys Static

Ruleset

Specification

Enforcement

ExecutionMatchesModel of

Application

Analyst orAdministrator

TrainingSets

StaticBinary Code

Analysis

StaticSource Code

Analysis


Representative Work by Ko, et al.

• Specification: Programmers or administrators specify correct program behavior

• Enforcement: At runtime, only allow actions that match the specified policy

PROGRAM fingerdread(X) :-

worldreadable(X);bind(79);write(“/etc/log”);exec(“/usr/ucb/finger”);

END



Ruleset

Specification

Enforcement


Application


TrainingSets

StaticBinary Code

Analysis

StaticSource Code

Analysis


Representative Work by Forrest, et al

• Specification: Learn correct program behavior with training– Record sequences of system calls

• Enforcement: Only accept behaviors similar to learned patterns

• Example system: STIDE


Training• Repeatedly run the program, varying

the input• For some n, record all sequences of n

system calls observed– n depends upon the program

• End result: database of n-tuples of system calls


cat (print file contents)geteuid, getuid, getegid, getgid, fstat, open, fstat, lseek, mmap, read, memcntl, write, lseek, munmap, lseek, close, close, exit

geteuid, getuidgetuid, getegidgetegid, getgidgetgid, fstatfstat, open / lseekopen, fstatlseek, mmap / munmap / closemmap, readread, memcntlmemcntl, writewrite, lseekmunmap, lseekclose, close / exit


Enforcement• Monitor system calls generated by

application• Ensure that the last n calls match a

sequence in the database• Option: Allow slight deviation from

database– Training set may have been incomplete


cat (print file contents)geteuid, getuid, getegid, getgid, fstat, open, fstat, lseek, mmap, read, memcntl, write, lseek, munmap, lseek, close, close, exit

Accepts incorrect system call sequences

geteuid, getuid, getegid, getgid, fstat, lseek, close, exit

geteuid, getuidgetuid, getegidgetegid, getgidgetgid, fstatfstat, open / lseekopen, fstatlseek, mmap / munmap / closemmap, readread, memcntlmemcntl, writewrite, lseekmunmap, lseekclose, close / exit


Drawbacks• Accepts incorrect call sequences

– Due to window-based approach with ambiguity– Opportunity for attack sequence to go

undetected• Only learn behaviors exercised in training

set– Not all execution paths followed

• Users must construct valid training sets• Users must determine window size


Drawbacks• Specification may over-fit the data• If training on real data, training set

may contain exploits– Learn exploit pattern as normal



Ruleset

Specification

Enforcement


Application


TrainingSets

StaticBinary Code

Analysis

StaticSource Code

Analysis

• Main idea: pre-determine control flow graph (CFG) of an application– Static analysis of source code– Static binary analysis CFI– Execution profiling– Explicit specification of security policy

• Execution must follow the pre-determined control flow graph

CFI: Control-Flow Integrity[Abadi et al.]

• Use binary rewriting to instrument code with runtime checks

• Inserted checks ensure that the execution always stays within the statically determined CFG– Whenever an instruction transfers

control, destination must be valid according to the CFG

CFI: Binary Instrumentation

CFI (Continued)• Goal: prevent injection of arbitrary

code and invalid control transfers (e.g., return-to-libc)– Secure even if the attacker has complete

control over the thread’s address space

April 22, 2023 Somesh Jha 38

CFG Example

• For each control transfer, determine statically its possible destination(s)

• Insert a unique bit pattern at every destination– Two destinations are equivalent if CFG

contains edges to each from the same source• This is imprecise (why?)

– Use same bit pattern for equivalent destinations

CFI: Control Flow Enforcement

CFI Enforcement• Insert binary code

– that at runtime will check whether the bit pattern of the target instruction

– matches the pattern of possible destinations


CFI: Example of Instrumentation

Original code

Instrumented code

Abuse an x86 assembly instruction toinsert “12345678” tag into the binaryJump to the destination only if

the tag is equal to “12345678”

• Unique IDs– Bit patterns chosen as destination IDs

must not appear anywhere else in the code memory except ID checks

• Non-writable code– Program should not modify code memory

at runtime• What about run-time code generation and

self-modification?

CFI: Preventing Circumvention

CFI: Preventing Circumvention

• Non-executable data– Program should not execute data as if it

were code• Enforcement: hardware support +

prohibit system calls that change protection state + verification at load-time


• Suppose a call from A goes to C, and a call from B goes to either C, or D (when can this happen?)– CFI will use the same tag for C and D, but this

allows an “invalid” call from A to D– Possible solution: duplicate code or inline– Possible solution: multiple tags

• Function F is called first from A, then from B; what’s a valid destination for its return?– CFI will use the same tag for both call sites, but this

allows F to return to B after being called from A– Solution: shadow call stack

Improving CFI Precision

CFI: Security Guarantees• Effective against attacks based on

illegitimate control-flow transfer– Stack-based buffer overflow, return-to-libc

exploits, pointer subterfuge• Does not protect against attacks that do

not violate the program’s original CFG– Incorrect arguments to system calls– Substitution of file names– Other data-only attacks


Questions?

intrusion detection

Documents

attack packet

packet payloadsneeded

packet headersenough

packet capturewebsite

order packets

filtered packet streamlibrary

raw packet streamparses

sequence of packets