10/1/200710/1/2007 11

Automatic Evaluation of Automatic Evaluation of Intrusion Detection SystemsIntrusion Detection Systems

F. Massicotte, F. Gagnon, Y. Labich, L. Briand,F. Massicotte, F. Gagnon, Y. Labich, L. Briand,Computer Security Applications Conference,Computer Security Applications Conference,

ACSAC ’06, pp 361-370, 2006.ACSAC ’06, pp 361-370, 2006.

Presented by: Lei WEIPresented by: Lei WEI

10/1/200710/1/2007 22

SummarySummary

1.1. Proposed a strategy that is able to evaluate Proposed a strategy that is able to evaluate Intrusion Detection System (IDS) Intrusion Detection System (IDS) automatically and systematically automatically and systematically

2.2. Evaluated two famous IDS programs, Snort Evaluated two famous IDS programs, Snort 2.3.2 and Bro 0.9a9, by using this new 2.3.2 and Bro 0.9a9, by using this new proposed strategy.proposed strategy.

3.3. Proposed a 15-class taxonomy for test Proposed a 15-class taxonomy for test results.results.

10/1/200710/1/2007 33

Appreciative Comments: Appreciative Comments: AutomatizationAutomatization

This is an automatic IDS evaluation system. Because of This is an automatic IDS evaluation system. Because of automation, it is possible to efficiently and systematically automation, it is possible to efficiently and systematically create a large number of sample data .create a large number of sample data .

“ “ We use 124 VEP (covering a total of 92 vulnerabilities) and We use 124 VEP (covering a total of 92 vulnerabilities) and 108 different target system configurations” (Automatic 108 different target system configurations” (Automatic Evaluation of Intrusion Detection Systems)Evaluation of Intrusion Detection Systems)

““38 different attacks were launched against victim UNIX 38 different attacks were launched against victim UNIX hosts in seven weeks of training data and two weeks of test hosts in seven weeks of training data and two weeks of test data.” (Evaluation Intrusion Detection Systems: The 1998 data.” (Evaluation Intrusion Detection Systems: The 1998 DARPA Off-line Intrusion Detections Evaluation)DARPA Off-line Intrusion Detections Evaluation)

10/1/200710/1/2007 44

Critical Comment:Critical Comment:1. Complicated classification1. Complicated classification

Each of the collected traffic traces belongs to one of the type, TP, Each of the collected traffic traces belongs to one of the type, TP, TN, FP and FN.TN, FP and FN.According to types of all traces collected from IDS evaluation According to types of all traces collected from IDS evaluation tests, the authors suggested a 15-class taxonomy for IDSes, such tests, the authors suggested a 15-class taxonomy for IDSes, such as, as, alarmistalarmist, , quietquiet, , quietquiet and complete detectionand complete detection, , complete complete evasionevasion etc. etc.

This does make the evaluation complicated and confused.This does make the evaluation complicated and confused. Hard to remember all the class namesHard to remember all the class names Is Is quiet and complete detection quiet and complete detection a subclass of a subclass of quietquiet? No!? No!

I prefer a statistical way by calculating the following two ratios, I prefer a statistical way by calculating the following two ratios, ( , ) , from which we know the percentage of attack ( , ) , from which we know the percentage of attack being detected and the percentage about wrong alarms.being detected and the percentage about wrong alarms.

TNTPTP FNFP

FP

10/1/200710/1/2007 55

Critical Comment: Critical Comment: 2. Confused diagrams 2. Confused diagrams

In this paper, the two diagrams, Figure 5 and Figure 1, and In this paper, the two diagrams, Figure 5 and Figure 1, and relevant description used to represent the working process of the relevant description used to represent the working process of the whole system are not clear enough. whole system are not clear enough.

(a). A title should be “… an effective guide for scientists rapidly (a). A title should be “… an effective guide for scientists rapidly scanning lists of titles for information relevant to their interests.” scanning lists of titles for information relevant to their interests.” ((Scientific writing for graduate students: a manual on the teaching Scientific writing for graduate students: a manual on the teaching of scientific writingof scientific writing, , edited by F. Peter Woodford. New York: edited by F. Peter Woodford. New York: Rockefeller University Press, 1968.Rockefeller University Press, 1968. ) )

However, neither the title nor the content provides clear However, neither the title nor the content provides clear explanation to the meaning of numbers in Figure5. explanation to the meaning of numbers in Figure5.

10/1/200710/1/2007 66

Critical Comment: Critical Comment: 2. Confused diagrams (Continue)2. Confused diagrams (Continue)

(b). Although the article describes the steps listed in Figure1, the (b). Although the article describes the steps listed in Figure1, the provided diagram does confused us to understand the structure provided diagram does confused us to understand the structure and working process of the system. The title is Virtual network and working process of the system. The title is Virtual network infrastructure,infrastructure, but the figure actually covers more stuff than that. It but the figure actually covers more stuff than that. It does not only represent Virtual network infrastructure, but also does not only represent Virtual network infrastructure, but also shows the working process of the subsystem.shows the working process of the subsystem.

10/1/200710/1/2007 77

Working process of Automatic IDS Working process of Automatic IDS Evaluation systemEvaluation system

This system could be divided into two This system could be divided into two subsystems.subsystems.

The attack simulation and data collection The attack simulation and data collection systemsystem

The IDS Evaluation FrameworkThe IDS Evaluation Framework

10/1/200710/1/2007 88

1. Attack simulation and data collection system1. Attack simulation and data collection system

1. Choose Vulnerability Exploitation Program (VEP)

2. Choose Configuration of the target System (e.g. IDS)Script Generation

Set up Virtual Network

Set up Attack Script

Execute Attack

Data Set

Provide the virtual attacking machine the proper attack configuration (e.g. Whether apply IDS Evasion Tech.)

1. Capture attack traffic traces2. Document the traffic traces

Restore

1. Save the traffic traces and IDS alarms on the shared hard-drive

2. Restore the virtual attacker and target machines to their initial state

10/1/200710/1/2007 99

Data Set

IDS

IDS Evaluator

IDS Result Analyzer

Report

2. IDS Evaluation Framework2. IDS Evaluation Framework

IDS Evaluator takes documented traffic traces from the Data Set

IDS Evaluator provide traffic traces to each tested IDS

Compare the two groups of data sets and determine whether the IDS detection succeed

The collected IDS alarms are fetched by the IDS Results Analyser

Generate the evaluation report

10/1/200710/1/2007 1010

QuestionQuestion

This paper evaluated two open source IDSes This paper evaluated two open source IDSes by the new strategy. However, many IDSes by the new strategy. However, many IDSes have patent or copy right protection. Those have patent or copy right protection. Those creators would never reveal the weak points creators would never reveal the weak points of their products.of their products.

Is it ethical or illegal to publish the evaluations Is it ethical or illegal to publish the evaluations of IDS programs so that others can know the of IDS programs so that others can know the truth?truth?

10/1/200710/1/2007 1111

The EndThe End

10/1/200710/1/2007 1212

The 15-class taxonomy (Supplement)The 15-class taxonomy (Supplement)

10/1/200710/1/2007 1313

Document traffic traces (Supplement)Document traffic traces (Supplement)

Each traffic trace is documented by four Each traffic trace is documented by four characteristics:characteristics:

1.1. Target system configurationTarget system configuration

2.2. VEP configurationVEP configuration

3.3. Whether or not the VEP exploited the Whether or not the VEP exploited the vulnerability of the target systemvulnerability of the target system

4.4. Whether or not the attack is successfulWhether or not the attack is successful