Formal Methods for Intrusion Detection

Presented by Brian Kellogg

CSE 914: Formal Methods for Software DevelopmentMichigan State University

December 11th, 2002

Purpose and Method Find intrusion detection methods that

utilize formal methods Analyze strengths and weaknesses of

each method Compare the methods and see if they

can be combined in such a way to improve one another

Found three research papers on intrusion detection that used formal methods for different purposes

Intrusion Detection Quickie The SANS institute defines intrusion detection as “the

art of detecting inappropriate, incorrect, or anomalous activity”

Two types: Host-based: detects intrusions on a specific host Network-based: detects intrusions on a network

Two (main) methods: Knowledge-based

Determine vulnerabilities and attempts to detect vulnerabilities Low false alarm rate Attacks not specified are not detected

Behavior-based Determines normal system activity High false alarm rate Able to detect many intrusions (even ones not previously

known)

Intrusion Detection Continued Why use intrusion detection, why not just

prevent the attacks? Firewalls can prevent many attacks, but have no

power over the internal network Certain network activities that have legitimate uses

can also signify an attack (e.g. port scans) What should an intrusion system do when it

detects an attack? Responses range from e-mails to reconfiguring the

network Just because the system detects an intrusion, may

be legitimate Severe (or even simple) responses can be utilized by

attackers to create new attacks

Yasinsac Paper (Motivation) “An Environment for Security Protocol

Intrusion Detection” Traditional methods of protocol analysis

not fool proof or complete Different protocols running concurrently

can create new exploits Shift to “tunneling” paradigm in networks

Sensitive data sent over same links as non-sensitive data

Cryptographic techniques must be applied at higher layer (application layer)

Yasinsac Paper (Method) Take knowledge gained from formal

analysis of security protocols and make them in to intrusion signatures

Uses both knowledge-based and behavior-based intrusion detection Knowledge-based: signature an ordering of

activity traces Behavior-based: surveys taxonomies and

protocol principles to determine profile strategies and behavior recognition

State-based attack recognition

Yasinsac Paper (Method) IKE protocol:

AB: HDR1, SAA, KEA, NA, A BA: HDR2, SAB, KEB, NB, B, {prf(KAB, (KEB, KEA, KEB, KEA,

B))}KB Exploit:

AB: HDR1, SAA, KEA, NA, A IB: HDR1, SAA, KEA, NA, I BI: HDR2, SAB, KEB, NB, B, {prf(KAB, (KEB, KEA, KEB, KEA, B))}KB

Yasinsac Paper (Architecture) Central monitor, each principal communicates

with monitor through secure channel

Network

Principal A

Principal B

MonitorPrincipal C (Intruder)

Knowledgebase

Pouzol Paper Motivation:

Algorithm that detects attacks in a declarative IDS is a black box

Partial instances of attacks can choke an IDS Wants to give more power to security officer to

choose which attack instances are important Method:

Formally specify intrusion signatures and detection rules

Create a lattice used to define equivalence classes that defines a signature

Choose an equivalence relation that can reduce the number of instances reported

Pouzol LatticeТ

{U1, U2, T1, T2, T3}

{U1, U2, T3}

{U1, U2} {U2, T3}

{T3}{U1} {U2}

{ }

U1U2T3: In this equivalence class, every instance that has a unique pair of users and a third time stamp will be reported. This is an example of a good choice. This class will resist the choking attack, and will report all completed instances of an attack. Having the final timestamp means that the last part of the attack occurred, thus only a completed attack is being reported.

NetSTAT Paper (Motivation) “NetSTAT: A Network-based Intrusion

Detection Approach” Motivated by the increase of network

reliance and attacks Host-based intrusion detection fails to

detect these attacks Firewalls do an excellent job of

preventing external intrusions, but internal threats are left unchecked

NetSTAT Paper (Method) NetSTAT is a network-based intrusion

detection system Wants to solve:

Networks generate large amounts of data Some attacks occur only in a certain portion of a

network Too much communication between IDS components

can clog a network Networks can grow very large Able to work with host-based methods

Four components: A network fact base A state transition scenario database Many general purpose probes An analyzer

NetSTAT Paper (Method) Network fact base

Stand alone application that describes network topology and network services

Contains interfaces, hosts, and links Represented as a hypergraph Interfaces are nodes, hosts and links are

edges This is a formal model, adds benefits:

Well defined semantics Supports reasoning and automation Topological properties described in expressive way

NetSTAT Paper (Method) State transition scenario database

Contains signatures of attacks Attacks are sequences of states (snapshots) States are described by assertions that return Boolean

values Example: i.link.type==”ATM”;

Probes Sensors that are strategically placed in a network but

are also full blown intrusion detection system Made up of:

Filter that only collects data of interest Inference engine contains attack scenarios Decision engine issues response according to information

collected in the inference engine, or reports info to the analyzer

NetSTAT Paper (Method) Analyzer

Takes as input a network fact base and a state transition scenario

Tells security officer where probes are needed Sets up the probes It determines:

Events to be monitored, The network topology State information it requires to verify state

assertions

NetSTAT Paper (Architecture)

Gateway

Router

Internet

probe

probe

Network FactBase

ScenarioDatabase

Analyzer

probe

Security Officer

Analysis: Yacinsac Advantages

Able to find flaws in protocols that get past formal analysis

Able to detect flaws in concurrently running protocols Architecture is cheap and versatile

Disadvantages How do you choose the sources for signatures? How many signatures is too many? Architecture

Every single principal required to run software to report to central authority

Intruders can disable software Network attacks can still occur unnoticed

Analysis: Pouzol Advantages

Allows security officer to specify an equivalence relation to prevent choking attacks on the IDS

Formal specification of signatures and detection rules proven sound and complete

Disadvantages Has not been implemented in any IDS Complexity of algorithm may create choking

attacks Equivalence relations can be dangerous if

configured incorrectly

Analysis: NetSTAT Advantages:

Can detect intrusions on multiple sub-networks and total network

Scalable to large networks Formal methods allow expressiveness and

automation Disadvantages

Not yet fully implemented Analyzer does ad hoc configuring of probes

Combination

Pouzol’s technique to prevent choking attacks can be used by Yasinsac (and NetStat)

Two full intrusion detection architecture Which one is best? NetSTAT!

Yasinsac’s knowledge base can be used by NetSTAT (and all IDS)

Conclusion Formal methods and intrusion detection can

work together to make networks more secure

There are many different areas where formal methods can be applied

Neither is a silver bullet to network security Attackers are always evolving new

techniques to attack a network, and as security experts, so must we

Main References A. Yasinsac. An Environment for Security Protocol

Intrusion Detection. Special edition of the Journal of Computer Security, 2001

J. Pouzol and M. Ducassé. Formal Specification of Intrusion Signatures and Detection Rules. 15th IEEE Computer Security Foundations Workshop, June 2002

G. Vigna and R. Kemmerer. NetSTAT: A Network-based Intrusion Detection Approach. Computer Security Applications Conference, 1998