machine learning for network anomaly detection
DESCRIPTION
Machine Learning for Network Anomaly Detection. Matt Mahoney. Network Anomaly Detection. Network – Monitors traffic to protect connected hosts Anomaly – Models normal behavior to detect novel attacks (some false alarms) Detection – Was there an attack?. Host Based Methods. Virus Scanners - PowerPoint PPT PresentationTRANSCRIPT
Machine Learning for Network Anomaly Detection
Matt Mahoney
Network Anomaly Detection
• Network – Monitors traffic to protect connected hosts
• Anomaly – Models normal behavior to detect novel attacks (some false alarms)
• Detection – Was there an attack?
Host Based Methods
• Virus Scanners
• File System Integrity Checkers (Tripwire, DERBI)
• Audit Logs
• System Call Monitoring – Self/Nonself (Forrest)
Network Based Methods
• Firewalls
• Signature Detection (SNORT, Bro)
• Anomaly Detection (eBayes, NIDES, ADAM, SPADE)
User Modeling
• Source address – unauthorized users of authenticated services (telnet, ssh, pop3, imap)
• Destination address – IP scans
• Destination port – port scans
Frequency Based Models
• Used by SPADE, ADAM, NIDES, eBayes, etc.
• Anomaly score = 1/P(event)
• Event probabilities estimated by counting
Attacks on Public Services
PHF – exploits a CGI script bug on older Apache web servers
GET /cgi-bin/phf?Qalias=x%0a/usr
/bin/ypcat%20passwd
Buffer Overflows
• 1988 Morris Worm – fingerd
• 2003 SQL Sapphire Wormchar buf[100];
gets(buf);
buf stackExploit code
Return Address0 100
TCP/IP Denial of Service Attacks
• Teardrop – overlapping IP fragments
• Ping of Death – IP fragments reassemble to > 64K
• Dosnuke – urgent data in NetBIOS packet
• Land – identical source and destination addresses
Protocol Modeling
• Attacks exploit bugs
• Bugs are most common in the least tested code
• Most testing occurs after delivery
• Therefore unusual data is more likely to be hostile
Protocol Models
• PHAD, NETAD – Packet Headers (Ethernet, IP, TCP, UDP, ICMP)
• ALAD, LERAD – Client TCP application payloads (HTTP, SMTP, FTP, …)
Time Based Models
• Training and test phases
• Values never seen in training are suspicious
• Score = t/p = tn/r where– t = time since last anomaly– n = number of training examples– r = number of allowed values– p = r/n = fraction of values that are novel
Example tn/r
• Training: 0000111000 n/r = 10/2
• Testing: 01223– 0: no score– 1: no score– 2: tn/r = 6 x 10/2 = 30– 2: tn/r = 1 x 10/2 = 5– 3: tn/r = 1 x 10/2 = 5
PHAD – Fixed Rules
• 34 packet header fields– Ethernet (address, protocol)– IP (TOS, TTL, fragmentation, addresses)– TCP (options, flags, port numbers)– UDP (port numbers, checksum)– ICMP (type, code, checksum)
• Global model
LERAD – Learns conditional Rules
• Models inbound client TCP (addresses, ports, flags, 8 words in payload)
• Learns conditional rules
If port = 80 then word1 = GET, POST (n/r = 10000/2)
LERAD Rule Learning
• If word1 = GET then port = 80 (n/r = 2/1)• word1 = GET, HELO (n/r = 3/2)• If address = Marx then port = 80, 25 (n/r =
2/2)
Address Port Word1 Word2
Hume 80 GET /
Marx 80 GET /index.html
Marx 25 HELO Pascal
LERAD Rule Learning
• Randomly pick rules based on matching attributes
• Select nonoverlapping rules with high n/r on a sample
• Train on full training set (new n/r)
• Discard rules that discover novel values in last 10% of training (known false alarms)
DARPA/Lincoln Labs Evaluation
• 1 week of attack-free training data
• 2 weeks with 201 attacks
SunOS Solaris Linux NT
RouterInternet
SnifferAttacks
Attacks out of 201 Detected at 10 False Alarms per Day
0
20
40
60
80
100
120
140
PHAD ALAD LERAD NETAD
Problems with Synthetic Traffic
• Attributes are too predictable: TTL, TOS, TCP options, TCP window size, HTTP, SMTP command formatting
• Too few sources: Client addresses, HTTP user agents, ssh versions
• Too “clean”: no checksum errors, fragmentation, garbage data in reserved fields, malformed commands
Real Traffic is Less Predictable
r (Number ofvalues)
Time
Synthetic
Real
Mixed Traffic: Fewer Detections, but More are Legitimate
0
20
40
60
80
100
120
140
PHAD ALAD LERAD NETAD
Total
Legitimate
Project Status
• Philip K. Chan – Project Leader
• Gaurav Tandon – Applying LERAD to system call arguments
• Rachna Vargiya – Application payload tokenization
• Mohammad Arshad – Network traffic outlier analysis by clustering
Further Reading
• Learning Nonstationary Models of Normal Network Traffic for Detecting Novel Attacks by Matthew V. Mahoney and Philip K. Chan, Proc. KDD.
• Network Traffic Anomaly Detection Based on Packet Bytes by Matthew V. Mahoney, Proc. ACM-SAC.
• http://cs.fit.edu/~mmahoney/dist/