Network Intrusion Detectionand Prevention

Advances in Information Security

Sushil Jajodia Consulting Editor

Center for Secure Information Systems George Mason University Fairfax, VA 22030-4444 email: jajodia@gmu.edu

The goals of the Springer International Series on ADVANCES IN INFORMATION SECURITY are, one, to establish the state of the art of, and set the course for future re-search in information security and, two, to serve as a central reference source for advanced and timely topics in information security research and development. The scope of this se-ries includes all aspects of computer and network security and related areas such as fault tolerance and software assurance. ADVANCES IN INFORMATION SECURITY aims to publish thorough and cohesive overviews of specific topics in information security, as well as works that are larger in scope or that contain more detailed background information than can be accommodated in shorter survey articles. The series also serves as a forum for topics that may not have reached a level of maturity to warrant a comprehensive textbook treatment. Researchers, as well as developers, are encouraged to contact Professor Sushil Jajodia with ideas for books under this series. Additional titles in the series: CYBER SITUATIONAL AWARENESS: Issues and Research edited by Sushil Jajodia, Peng Liu, Vipin Swarup, Cliff Wang; ISBN: 978-1-4419-0139-2 SECURITY AND DEPENDABILITY FOR AMBIENT INTELLIGENCE edited by George Spanoudakis, Antonia Mana Gomez and Spyros Kokolakis; ISBN: 978-0-387-88774-6 IDENTIFYING MALICIOUS CODE THROUGH REVERSE ENGINEERING edited by Abhishek Singh; ISBN: 978-0-387-09824-1 SECURE MULTI-PARTY NON-REPUDIATION PROTOCOLS AND APPLICATIONS by José A. Onieva, Javier Lopez, Jianying Zhou; ISBN: 978-0-387-75629-5 GLOBAL INITIATIVES TO SECURE CYBERSPACE: An Emerging Langscape edited by Michael Portnoy and Seymour Goodman; ISBN: 978-0-387-09763-3 SECURE KEY ESTABLISHMENTS by Kim-Kwang Raymond Choo; ISBN: 978-0-387-87968-0 SECURITY FOR TELECOMMUNICATIONS NETWORKS by Patrick Traynor, Patrick McDaniel and Thomas La Porta; ISBN: 978-0-387-72441-6 INSIDER ATTACK AND CYBER SECURITY: Beyond the Hacker edited by Salvatore Stolfo, Steven M. Bellovin, Angelos D. Keromytis, Sara Sinclaire, Sean W. Smith; ISBN: 978-0-387-77321-6 For other titles published in this series, go to www.springer.com/series/5576

Ali A. Ghorbani • Wei Lu

Concepts and Techniques

Mahbod Tavallaee

Network Intrusion Detectionand Prevention

All rights reserved.

10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connectionwith any form of information storage and retrieval, electronic adaptation, computer software, or by similar

The use in this publication of trade names, trademarks, service marks, and similar terms, even if they arenot identified as such, is not to be taken as an expression of opinion as to whether or not they are subjectto proprietary rights.

Printed on acid-free paper

This work may not be translated or copied in whole or in part without the written

permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY

Springer is part of Springer Science+Business Media (www.springer.com)

Faculty in Computer ScienceUniversity of New BrunswickFredericton NB E3B 5A3

Ali A. Ghorbani

Canadaghorbani@unb.ca

Wei LuFaculty in Computer ScienceUniversity of New Brunswick

Canada

Faculty in Computer ScienceUniversity of New BrunswickFredericton NB E3B 5A3Canada

Mahbod Tavallaee

ISBN 978-0-387-88770-8 e-ISBN 978-0-387-88771-5

or dissimilar methodology now known or hereafter developed is forbidden.

DOI 10.1007/978-0-387-88771-5

Library of Congress Control Number: 2009934522

Fredericton NB E3B 5A3

© Springer Science+Business Media, LLC 2010

Springer New York Dordrecht Heidelberg London

Dedicated to all those who are helping tomake the world safer and more secure and toour families for their love and support.

Preface

Great changes are taking place is the area of information supply and demand dueto the wide spread application of computers and the exponential increase of com-puter networks such as the Internet. The Internet has become a popular mediumof commercial activities and this raised the stakes, both, for attackers and securitypersonnel. Trillions of dollars of transactions occur daily at each major financial in-stitution. For example, Visa processes 4,000 transactions per second, which meansthat if Visa’s system goes down for one minute because of a distributed denial ofservice attack (DDoS), and assuming only $100 per transaction, over $24 million intransactions is lost in one minute.

Today the world of business computing is faced with the ever-increasing likeli-hood of unplanned downtime due to various attacks and security breaches. In thisenvironment of uncertainty which is full of hackers and malicious threats, thosecompanies around the globe which are the best at maintaining the continuity oftheir services (i.e., survive the system) and retaining their computing power, enjoya significant competitive advantage.

Network downtime results in financial losses and more harms to the credibilityof commercial enterprises especially ISPs. Minimizing or possibly eliminating theunplanned downtime of the system establishes the continuity of the computing ser-vices. Minimizing unexpected and unplanned downtime can be done by identifying,prioritizing and defending against misuse, attacks and vulnerabilities. The challengeis to reduce the likelihood of catastrophic incidents by: a) using appropriate machineand statistical learning techniques to assess the relative danger of individual threatsand b) autonomously providing effective and appropriate response to the relevantthreats.

Intrusion Detection System (IDS) is a rapidly growing field that deals with de-tecting and responding to malicious network traffic and computer misuse. Intrusiondetection is the process of identifying and (possibly) responding to malicious activi-ties targetd at computing and network resources. Any hardware or software automa-tion that monitors, detects or responds to events occurring in a network or on a hostcomputer is considered relevant to the intrusion detection approach. Different IDSsprovide varying functionalities and benefits.

vii

viii Preface

An attempt to break or misuse a system is called “intrusion”. An intrusion nor-mally exploits a specific vulnerability and must be detected as quickly as possible.An intrusion detection system is a system for detecting such intrusions. Intrusiondetection systems are notable components in network security infrastructure. Theyexamine system or network activity to find possible intrusions or attacks and trig-ger security alerts for the malicious activities. They are generally categorized assignature-based and anomaly-based detection system. Other categories are network-based and host-based intrusion detection systems.

Network-based IDSs are placed at a strategic point or points within the networkto examine passing network traffic for signs of intrusion, whereas, host-based IDSsare run on individual hosts or devices on the network and look at user and processactivity for signs of malicious behavior.

In most networks, signature-based IDSs, which are very effective against knownattacks, are deployed. Signature-based systems (also know as misuse detection sys-tems) detect attacks based on known cases of misuse. Advantages of misuse detec-tion are high confidence in detection, low false positive rate and an unambiguousdetailed identification of attack. They are also more well understood and widely ap-plied. Disadvantages are inability to detect unknown attacks and the need for expertknowledge to create signatures.

For defense against unknown attacks, an anomaly detection scheme has to beused, which creates a model of normal bhaviour of the system and detects devia-tion from this model. Techniques used in detecting anomalies include data mining,clustering, and statistical signal processing. The main advantage of anomaly basedsystems is the ability to detect unknown attacks. The disadvantages are high falsepositive rate and difficulty in identification of attack type. Moreover, since what isconsidered normal could be different in different environments, a distinct model ofnormalcy need to be learned individually.

A more recent class of intrusion detectors is the specification-based detectors,which try to reach a common ground between misuse-based and anomaly-basedsystems. They are mainly based on specifications derived from protocols and detectdeviations from these specifications. Although they combine the benefits of anomalydetection and misuse detection, they suffer from the disadvantage that completespecifications are hard to create especially with most protocols being constantlyextended.

Due to the exponential growth in size, distribution, and complexity of communi-cation networks, current IDS technologies are not very effective against new attacksand have severe limitations as far as performance, scalability, and flexibility are con-cerned. Moreover, the improvements to the IDSs are often too slow and too little tokeep up with the innovations by the attackers.

The main drawbacks of the current IDSs are: 1) the large number of false pos-itives; 2) the inability to detect unknown attacks; and, 3) an inability to properlyassess the relative danger of the misuse and provide an appropriate response. Thereis a general consensus that the primary focus of the intrusion detection technologiesmust be: a) to reduce the rate of false positives; b) to develop non-signature-basedintrusion detection methods; and, c) work on prevention instead of detection.

Preface ix

There is a critical need to be able to deliver systems that can automatically de-tect intrusion patterns and performance bottlenecks, and dynamically defend them-selves. The main goal of an intrusion detection system is to survive the system andretain its essential services. Survivability is often defined by resistance, recognitionand recovery. Resistance deals with hardening a system to prevent a break-in orother malicious acts. The goal of recognition is to detect intrusive behavior fromnormal behavior. Recovery deals with ways of surviving malicious acts.

Any solution to the survivability problem must handle three basic criteria, in-cluding: dynamically changing network traffic patterns, occurrence of unpredictableevents (security breaches), and non-finite base of network traffic environment. Oneway to develop survivability tools capable of blending seamlessly into current dy-namic network environments is to design them as an agent-based distributed system.The key element of such a system is an intelligent agent, which is capable of analyz-ing a situation, making decisions and communicating with other agents and users.Multiagent systems together with the fuzzy systems can be used to establish a com-munity of intelligent agents with features such as autonomy, self

During the past number of years, machine learning and data mining techniqueshave received considerable attention among the intrusion detection researchers toaddress the weaknesses of knowledge-base detection techniques. This has led to theapplication of various supervised and unsupervised techniques for the purpose ofintrusion detection. This book provides a comprehensive review on current trends inintrusion detection systems and the corresponding technologies. We present a set ofexperiments which are carried out to analyze the performance of unsupervised andsupervised machine learning techniques considering their main design choices.

During the last decade, anomaly detection has attracted the attention of manyresearchers to overcome the weakness of signature-based IDSs in detecting novelattacks. However, having a relatively high false alarm rate, anomaly detection hasnot been wildly used in real networks. This book presents data driven approachesto automating network behavior modeling. One of the approaches is the techniquewe developed to create an ARX model of network signals and using it for detectingnetwork anomalies caused by intrusions. Network signals are nonstationary, highlyvolatile and hard to model using traditional methods. Our modeling technique usinga combination of system identification theory and wavelet approximation is veryeffective at addressing this issue.

Alert correlation is an important technique for managing large volume of intru-sion alerts that are raised by heterogeneous IDSs. The recent trend of research inthis area is towards extracting attack strategies from raw intrusion alerts. Know-ing the real security situation of a network and the strategies used by the attackersenables network administrators to launch appropriate response to stop attacks andprevent them from escalating. In this book we present alert management and corre-lation technique that can help to automatically extract attack strategies from a largevolume of intrusion alerts without specific prior knowledge about these alerts.

The intrusion detection books on the market are relatively unfocussed. They tendto leave out details of a variety of key techniques and models. Additionally, manybooks lack much detail on different types of attacks, theoretical foundation of attack

x Preface

detection approaches, implementation, data collection, evaluation, and intrusion re-sponse. In this book, our goal is to provide simple yet detailed and concise infor-mation on all these subjects. Additionally, we provide a detail overview of some ofthe commercially/publicly available intrusion detection and response systems. Onthe topic of intrusion detection system it is impossible to include everything thereis to say on all subjects. However, we have tried to cover the most important andcommon ones.

This book is divided into 8 chapters and one appendix as follows:

Chapter 1 (Network Attacks): In this chapter we first discuss different attack tax-onomies and then present the details of a large number of known vulnerabilitiesand strategies to lunch attacks.

Chapter 2 (Detection Approaches): Detection approach describes the attack anal-ysis method used in an IDS. In this chapter different detection approaches thatare currently available to detect intrusions and anomalous activities are given.Misuse, rule-based, model-based, anomaly and specification-based detection ap-proaches are explained in detail.

Chapter 3 (Data Collection): Intrusion detection systems collect their data fromvarious sources. Such sources include log files, network packets, system calls,or a running code itself. In this chapter, we provide detail information as to howthis very important step in the life of different intrusion detection systems (i.e.host-based, network-based and application-based) can be accomplished.

Chapter 4 (Theoretical Foundation of Detection): Understanding the strengths andweaknesses of the machine learning and data mining approaches helps to choosethe best approach to design and develop a detection system. Several approachesto the intrusion detection research area are introduced and analyzed in this chap-ter.

Chapter 5 (Architecture and Implementation): Intrusion detection systems can beclassified based on their architecture and implementation. This classification usu-ally refers to the locus of the data collection and analysis. This chapter introducesthe centralized, distributed and agent based intrusion detection systems.

Chapter 6 (Alert Management and Correlation):ger too many alerts that usually contain false alerts. Decreasing false positivesand improving the knowledge about attacks provides a more global view of whatis happening in a network. Alert management and correlation addresses the issueof managing large number of alerts by providing a condensed, yet more usefulview of the network from the intrusion standpoint. The correlation function canrelate different alerts to build a big picture of the attack. The correlated alertscan also be used for cooperative intrusion detection and tracing an attack to itssource.This chapter introduces different approaches to cluster, merge and corre-late alerts.

Chapter 7 (Evaluation Criteria): This chapter provides a number of approachesthat can be used to evaluate the potential intrusion detection systems for accu-racy, performance, completeness, timely response, cost and intrusion tolerance& attack resistance.

Intrusion Detection Systems trig-

Preface xi

Chapter 8 (Intrusion Response): The objective is to enable automated reasoningand decision-making to aid in human-mediated or automatic response. The cost-benefit analysis of response is the key for effective response. This chapter ex-pands on this and other approaches for providing effective and sensible re-sponses.

Appendix A (Examples of Commercial and Open Source IDSs): A brief introduc-tion to some of the current commercial and open source IDSs are given in Ap-pendix A.

Audience: This book provides students and security professionals with the nec-essary technical background to understand, develop and apply intrusion detectionsystems. Some background of machine learning and data mining is helpful to un-derstand the approaches presented in this book. We also assume that the readers ofthis book have a good command of data communication and networking. After read-ing this book, you should have a solid understanding of technical basics of intrusiondetection and response systems. In addition, you should know how to develop reli-able and effective detection systems and be able to install and manage commerciallyavailable IDSs.

Fredericton, Canada, Ali A. GhorbaniJuly 2009 Wei Lu

Mahbod Tavallaee

Acknowledgements

We would like to recognize the contribution of a number of people to the book.This book is basically the result of the work of research associates, postdoctoral fel-lows and graduate students at the Faculty of Computer Science, University of NewBrunswick, Canada, under the supervision of one of the authors (Ali A. Ghorbani).In particular, the authors would like to thank Drs. Peyman Kabiri, Mehdi Shajari,Mehran Nadjarbashi, Iosif-Viorel Onut and Mr. Bin Zhu for their effort and hardwork. They contributed a great deal to some of the original wording and editing ofwhat now became this book. Their work in putting together a comprehensive tech-nical report on intrusion detection and response systems, from which this book isinspired, has been invaluable. The authors truly appreciate their efforts. The authorsrecognize that the contents of the following chapters inspired and in some cases arepartially from this technical report.

• Chapter 1 (Network Attacks), Contributors: Dr. Mehdi Shajari, Dr. PeymanKabiri and Dr. Iosif-Viorel Onut.

• Chapter 2 (Detection Approaches), Contributors: Dr. Mehran Nadjarbashi, Dr.Iosif-Viorel Onut, andDr. Peyman Kabiri.

• Chapter 3 (Data Collection), Contributors: Mr. Bin Zhu, Dr. Iosif-Viorel Onut,Dr. Mehdi Shajari, Dr. Peyman Kabiri.

•Dr. Mehdi Shajari, Dr. Mehran Nadjarbashi, and Mr. Bin Zhu.

• Chapter 5 (Architecture and Implementation), Contributors: Dr. Mehdi Shajari,Dr. Peyman Kabiri, and Dr. Iosif-Viorel Onut.

• Chapter 6 (Alert Management and Correlation), Contributors: Mr. Bin Zhu, Dr.Mehdi Shajari, Dr. Peyman Kabiri.

• Chapter 7 (Evaluation Criteria), Contributors: Mr. Bin Zhu, Dr. Mehdi Shajari,Dr. Iosif-Viorel Onut, Dr. Peyman Kabiri.

• Chapter 8 (Intrusion Response), Contributors: Dr. Mehran Nadjarbashi, Dr. Iosif-Viorel Onut, Dr. Peyman Kabiri, and Dr. Mehdi Shajari.

• Appendix A (Examples of Commercial and Open Source IDSs), Contributors:Dr. Mehdi Shajari, Dr. Iosif-Viorel Onut, and Mr. Bin Zhu.

xiii

Chapter 4 (Theoretical Foundation of Detection) Contributors: Dr. Peyman Kabiri,

xiv Acknowledgements

We gratefully recognize the opportunity the Springer gave us to write this book.The authors would like to also recognize the financial support from Atlantic CanadaOpportunities Agency (ACOA) under the Atlantic Innovation Fund.

Contents

1 Network Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Attack Taxonomies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Probes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2.1 IPSweep and PortSweep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.2 NMap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.3 MScan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.4 SAINT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.5 Satan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.3 Privilege Escalation Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.3.1 Buffer Overflow Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.3.2 Misconfiguration Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.3.3 Race-condition Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.3.4 Man-in-the-Middle Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.3.5 Social Engineering Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.4 Denial of Service (DoS) and Distributed Denial of Service(DDoS) Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.4.1 Detection Approaches for DoS and DDoS Attacks . . . . . . . . 111.4.2 Prevention and Response for DoS and DDoS Attacks . . . . . . 131.4.3 Examples of DoS and DDoS Attacks . . . . . . . . . . . . . . . . . . . . 14

1.5 Worms Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161.5.1 Modeling and Analysis of Worm Behaviors . . . . . . . . . . . . . . 161.5.2 Detection and Monitoring of Worm Attacks . . . . . . . . . . . . . . 171.5.3 Worms Containment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181.5.4 Examples of Well Known Worm Attacks . . . . . . . . . . . . . . . . 19

1.6 Routing Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191.6.1 OSPF Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201.6.2 BGP Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

xv

xvi Contents

2 Detection Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.1 Misuse Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.1.1 Pattern Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282.1.2 Rule-based Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.1.3 State-based Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.1.4 Techniques based on Data Mining . . . . . . . . . . . . . . . . . . . . . . 34

2.2 Anomaly Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.2.1 Advanced Statistical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.2.2 Rule based Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.2.3 Biological Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392.2.4 Learning Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

2.3 Specification-based Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452.4 Hybrid Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553.1 Data Collection for Host-Based IDSs . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.1.1 Audit Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563.1.2 System Call Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

3.2 Data Collection for Network-Based IDSs . . . . . . . . . . . . . . . . . . . . . . . 613.2.1 SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613.2.2 Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.2.3 Limitations of Network-Based IDSs . . . . . . . . . . . . . . . . . . . . 66

3.3 Data Collection for Application-Based IDSs . . . . . . . . . . . . . . . . . . . . 673.4 Data Collection for Application-Integrated IDSs . . . . . . . . . . . . . . . . . 683.5 Hybrid Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4 Theoretical Foundation of Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 734.1 Taxonomy of Anomaly Detection Systems . . . . . . . . . . . . . . . . . . . . . 734.2 Fuzzy Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.2.1 Fuzzy Logic in Anomaly Detection . . . . . . . . . . . . . . . . . . . . . 774.3 Bayes Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.3.1 Naive Bayes Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 784.3.2 Bayes Theory in Anomaly Detection . . . . . . . . . . . . . . . . . . . . 78

4.4 Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.4.1 Processing Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.4.2 Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824.4.3 Network Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.4.4 Learning Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 844.4.5 Artificial Neural Networks in Anomaly Detection . . . . . . . . . 85

4.5 Support Vector Machine (SVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 864.5.1 Support Vector Machine in Anomaly Detection . . . . . . . . . . . 89

4.6 Evolutionary Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 894.6.1 Evolutionary Computation in Anomaly Detection . . . . . . . . . 91

Contents xvii

4.7 Association Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 924.7.1 The Apriori Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 934.7.2 Association Rules in Anomaly Detection . . . . . . . . . . . . . . . . 93

4.8 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 944.8.1 Taxonomy of Clustering Algorithms . . . . . . . . . . . . . . . . . . . . 954.8.2 K-Means Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 964.8.3 Y-Means Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 974.8.4 Maximum-Likelihood Estimates . . . . . . . . . . . . . . . . . . . . . . . 984.8.5 Unsupervised Learning of Gaussian Data . . . . . . . . . . . . . . . . 1004.8.6 Clustering Based on Density Distribution Functions . . . . . . . 1014.8.7 Clustering in Anomaly Detection . . . . . . . . . . . . . . . . . . . . . . . 102

4.9 Signal Processing Techniques Based Models . . . . . . . . . . . . . . . . . . . . 1044.10 Comparative Study of Anomaly Detection Techniques . . . . . . . . . . . . 109References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

5 Architecture and Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.1 Centralized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.2 Distributed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

5.2.1 Intelligent Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1165.2.2 Mobile Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

5.3 Cooperative Intrusion Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

6 Alert Management and Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1296.1 Data Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1296.2 Alert Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

6.2.1 Preprocess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1326.2.2 Correlation Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1396.2.3 Postprocess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1456.2.4 Alert Correlation Architectures . . . . . . . . . . . . . . . . . . . . . . . . . 1506.2.5 Validation of Alert Correlation Systems . . . . . . . . . . . . . . . . . 152

6.3 Cooperative Intrusion Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1536.3.1 Basic Principles of Information Sharing . . . . . . . . . . . . . . . . . 1536.3.2 Cooperation Based on Goal-tree Representation of Attack

Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1546.3.3 Cooperative Discovery of Intrusion Chain . . . . . . . . . . . . . . . . 1546.3.4 Abstraction-Based Intrusion Detection . . . . . . . . . . . . . . . . . . 1556.3.5 Interest-Based Communication and Cooperation . . . . . . . . . . 1556.3.6 Agent-Based Cooperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1566.3.7 Secure Communication Using Public-key Encryption . . . . . . 157

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

xviii Contents

7 Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1617.1 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

7.1.1 False Positive and Negative . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1627.1.2 Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1637.1.3 Precision, Recall, and F-Measure . . . . . . . . . . . . . . . . . . . . . . . 1647.1.4 ROC Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1667.1.5 The Base-Rate Fallacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

7.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1717.3 Completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1727.4 Timely Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1727.5 Adaptation and Cost-Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1757.6 Intrusion Tolerance and Attack Resistance . . . . . . . . . . . . . . . . . . . . . . 177

7.6.1 Redundant and Fault Tolerance Design . . . . . . . . . . . . . . . . . . 1777.6.2 Obstructing Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

7.7 Test, Evaluation and Data Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

8 Intrusion Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1858.1 Response Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

8.1.1 Passive Alerting and Manual Response . . . . . . . . . . . . . . . . . 1858.1.2 Active Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

8.2 Response Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1868.2.1 Decision Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1868.2.2 Control Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1898.2.3 Game theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1898.2.4 Fuzzy theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

8.3 Survivability and Intrusion Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . 194References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

A Examples of Commercial and Open Source IDSs . . . . . . . . . . . . . . . . . . 199A.1 Bro Intrusion Detection System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199A.2 Prelude Intrusion Detection System . . . . . . . . . . . . . . . . . . . . . . . . . . . 199A.3 Snort Intrusion Detection System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200A.4 Ethereal Application - Network Protocol Analyzer . . . . . . . . . . . . . . . 200A.5 Multi Router Traffic Grapher (MRTG) . . . . . . . . . . . . . . . . . . . . . . . . . 201A.6 Tamandua Network Intrusion Detection System . . . . . . . . . . . . . . . . . 202A.7 Other Commercial IDSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209