PROJECT DESCRIPTION

1 Introduction

To gain back the trustworthiness of Internet infrastructures and network-centric computer systems, it is crit-ical to improve themeasurableperformance of network systems under cyber attacks and threats, and toprovide end users with timely information and more control. This project focuses on the design, devel-opment, and evaluation of a novel enterprise cyber-defense system (IMPACT), which integrates flexibleintrusion detection, information fusion for alert correlation, QoS-adaptive resource management for intru-sion mitigation, and user-centered multipath indirect rouing for intrusion tolerance components.

1.1 Motivations

The proliferation of Internet applications and network-centric mission-critical services is bringing networkand system security issues to the fore. The past few years have seen significant increase in cyber attackson the Internet, resulting in degraded confidence and trusts in the use of the Internet and computer systems.The cyber attacks, including email virus, worms, and DDoS, are getting more sophisticated, spreadingquicker, and causing more damage. Attacks originally exploited the weakness of the individual protocolsand operating systems but now have started to target the basic infrastructure of the Internet. To gain backthe trustworthiness of Internet infrastructures and computer systems, there is an urgent need to enhance theeffectiveness of the cyber defense systems, to provide end users with timely information and more control,and to improve the measurable performance of network systems when under attacks.

Improving the measurable performance against cyber threats requires a multi-faceted research programthat includes:

• improvement of core network and system Quality-of-Service (QoS) support models for intrusion mit-igation under uncertain threats,

• development of novel intrusion tolerance approaches to reduce the impact of the severe cyber attacksby making new network capabilities such as multipath indirect routing available and delivering timelynetwork/system information to end users,

• re-design of the intrusion detection and alert correlation considering the above QoS and intrusiontolerance capabilities, and

• development of evaluation tools to allow us to have at least statistical confidence in the experimentalresults.

There are, of course, many other research components that will also improve performance, but we envisionthat the four items above need to be a part of any serious solution to this growing problem. They are thefocus of this proposal.

As we more and more rely on the intrusion detection systems (IDS), the IDS systems also give us moreand more false alarms. This is due to its lack of flexibility under new cyber attacks and uncertain threats.Some cyber attacks are initially unobvious and disruptive. And, it is often too late when they are diagnosedas malicious attacks. Somemalware(malicious programs and code propagating on the Internet) [16, 30]can spread at exponential rates. They can quickly propagate through the network, infecting many machinesbefore the severity of the situation is recognized. Most today’s DDoS attacks aim to completely disable the

C—1

victim system’s service to its clients by consuming its available resources. Compared to the disruptive DDoSattacks, degrading DDoS attacks are new and emerging. The goal is to increasingly or periodically consumeportions of a victim system’s resources so as to result in denial of service to legitimate clients during highload periods. Some legitimate clients may also leave the victim system due to the poor QoS experience.Degrading DDoS attacks can remain undetected for a long time period since they do not lead to total servicedisruption, and therefore, it is difficult to identify the attackers. Thus, IDS systems have to be flexible underuncertain cyber threats. The existing on/off intrusion detection models are not sufficient.

A flexible intrusion detection model needs the support of QoS-adaptive resource management in networkrouters and end systems. On one hand, QoS is the target of cyber attacks. Cyber attacks, such as DDoS, aimto reduce QoS level provided by networks and systems and experienced by users; in the worst case, to noservice at all. On the other hand, QoS can be used as a means against cyber attacks. Under uncertain attackscenarios, a router or an end system can handle incoming traffic differently according to the confidencelevels about observed traffic behaviors provided by the flexible IDS systems. For example, the confidencelevel can be utilized to limit the propagation rate of a potentially malicious and suspective traffic. Thus,our proposal is to make the performance of networks and systems configurable and controllable by theadministrators and users, instead of by parameters and behaviors of uncertain attacks.

We further propose to build a bridge from distributed IDS systems to the QoS-adaptive resource manage-ment component in network systems. Today’s enterprise intrusion detection systems generate large volumesof data from distributed intrusion detection and traffic monitoring devices. It often takes a long time toanalyze the intrusion data, resulting in slow response time. Efficient information fusion techniques canhelp correlate distributed intrusion and traffic data and pass along urgent alerts. This enables early warningand intrusion handling. Furthermore, based on the brief network techniques, information fusion componentwill get intrusion data from IDS systems, execute novel alert correlation mechanisms, and provide controlparameters to the QoS-adaptive resource management component.

Recent advances in overlay and multihoming networks provides multiple alternate routes that can beused to increase aggregate bandwidth, improve network reliability, and tolerance DDoS attacks. To makethese new capabilities available to end users, it is required to enhance the Internet naming architecture andmodify the end system’s resolver library and routing module. To protect alternate routes, we propose tobuild a secure collective network defense framework utilizing a set of proxy servers. These proxy serversform the frontline of a wide area demilitarized zone to protect users from further DDoS attacks on thesealternate routes. To take advantage of multiple alternate routes at the end systems, we propose to enhancethe DNS to include the alternate route information in the DNS entries. To enhance the availability of theDNS system, peer-to-peer secure DNS query will be sent via the indirect routes since the main route could beblocked by DDoS attacks. To avoid stale DNS caching, we will investigate new secure information sharingand notification schemes to provide timely delivery of network status and naming information. We will alsoimprove the TCP/IP performance based on the availability of multiple indirect routes.

With the new QoS-aware intrusion mitigation and proxy server based intrusion tolerance capabilities,suspicious traffic can be put on special monitoring or classes with lower bandwidth/resource utilization.Further confirmation of the attacks can block those traffic and trigger the dynamic setup of multiple alternateroutes for legitimate clients. This implies early reporting of IP sweep and port scaning activities from IDSto the intrusion handling system component, and a much closer interaction among IDS, alert fusion andintrusion handling components. We will redesign the IDS taking into consideration these new adaptive QoScapabilities and evaluate these new techiques in an integrated enterprise cyber-defense prototype.

These proposed techniques detect and mitigate cyber attacks and threats. But equally important inimproving the predictability and trust of cyber infrastructure is the design and implementation of evaluationtools to allow the computation of (statistical) confidence intervals. It is not sufficient to know the IDSdetected, say 65% of the attacks in a particular test; we need to know the expected variation in that detection

C—2

Figure 1: The architecture of the IMPACT secure collective defense system.

rate over some range of experiments. In addition, as we move to adaptive detection and response techniques,we need to develop tools that allow efficient generation of online evaluations as well as offline datasets forfurther analysis.

1.2 Objectives

The goal of this project is to design, develop, and evaluate an integrated enterprise cyber-defense system(IMPACT) for improving measurable performance of network systems. The main objective is to make theperformance of network systems configurable and controllable by themselves, instead of by parameters andbehaviors of cyber attacks, and to provide end users with timely information and more control. Figure 1illustrates the architecture of the IMPACT system. We aim at achieving the following tasks:

1. Develop QoS-adaptive resource management mechanisms for service differentiation and traffic regu-lation in both network routers and end-point computer system, and investigate the impact of the servicedifferentiation techniques for limiting spreading rate of susceptible traffic and regulating traffic undercyber attacks.

2. Apply the user-centered paradigm to improve the cyber infrastructure by making new network capa-bilities such as multipath indirect routing available and delivering timely network/system informationto end users. Develop intrusion toleration techniques, based on autonomous establishment of securemultiple indirect routes for legitimate connections, to mitigate the impact of inevitable cyber attacks.DNS system will be enhanced to support multiple indirect routes. New secure information sharingand notification schemes will be developed to provide timely delivery of network status and nam-ing information. The resolver library and TCP/IP software at end systems will be improved to takeadvantage of these new capabilities.

3. Re-design the intrusion detection and alert correlation system components by providing early alertsand adaptive tracking feature. We will re-design the scan detection algorithm and behavior basedintrusion detection system to incorporate these changes. To support the tighter interaction such asadaptive monitoring and tracking, we would like to extend the current IDS API to accept requestsfrom intrusion handling systems. Efficient layer coding of cyber defense data will be investigated.

4. Develop effective cooperative intrusion detection and response systems and explore the use of multipleindirect routes for distributed intrusion detection and network reconfiguration.

C—3

5. Develop the IMPACT tool-set, which will be a distributed cyber-evaluation tool-set which will facili-tate the parallel dumping, sanitizing and scoring of network traffic and thus support generation of newdatasets as well as on-line evaluations. The tool-set will log with added time-stamping and indexingto support enhanced playback capabilities necessary for the resampling techniques we propose to sup-port “statistical” confidence testing. The tool chain will support mapping tables to “remap” users, IPaddresses and even some content, thus supporting enhanced resampling as well as addressing privacyissues. The result will be both a tool-set and new significant public datasets for evaluations.

2 Background & Related Work

2.1 QoS-adaptive Resource Management

QoS differentiation has been an active research topic in both network core and end-point computer systemsfor several years. In the network core, DiffServ architecture [14] was proposed by IETF to provide differ-entiated QoS levels with respect to delay and loss among classes of aggregated traffic flows. To receivedifferent levels of QoS, a packet can be assigned to different service types or traffic classes at the networkedges by setting its Differentiated Services code points; a DiffServ-compatible router performs differentforwarding operations, so-called “per-hop behaviors” (PHBs), to the classified packet. This architectureis scalable because it does not require a router to store “per-flow” state. Two different schemes exist forDiffServ: absolute DiffServ and relative DiffServ. Absolute DiffServ aims to guarantee a class’s receivedresource, such as bandwidth. Relative DiffServ is to quantify the quality spacings between different classes.

Recently, a proportional delay differentiation (PDD) model was proposed in support of relative Diff-Serv [26]. It ensures the quality spacing between classes of traffic to be proportional to certain pre-specifiedclass differentiation parameters. Due to its inherent differentiation predictability, controllability, and fair-ness, it has been accepted as an important DiffServ model, and many packet scheduling algorithms have beendeveloped for PDD provisioning. Representatives of the PDD algorithms include backlog-proportional rate(BPR) [26], joint buffer management and scheduling (JoBS) [45], proportional average delay (PAD) [27],waiting-time priority (WTP) [27], adaptive WTP [44], hybrid proportional delay (HPD) [27], mean-delayproportional (MDP) [55], and Little’s average delay (LAD) [76]. They demonstrated various characteristicsin support of the PDD model in different class load conditions and different time-scales. Most of them arecapable of achieving desired delay ratios, if the ratios are feasible, under heavy load conditions and in longtime-scales. However, for light load conditions and in short time-scales, they exhibit various limitations.

There are also efforts on providing QoS differentiation in the end-point computer systems. The effortsinclude content adaptation techniques for differentiation in multimedia servers [19, 81], priority schedulingwith admission control and feedback control for responsiveness differentiation in individual Internet serversand server clusters [1, 23, 43, 78, 79, 80, 82].

QoS has not been used for network security purpose until recently. In [22], the authors developed rate-limit algorithms to slow down the propagation of Internet worms. The work is based on the observation thata worm-infected host has a much higher connection-failure rate when it scans the Internet with randomlyselected addresses while a normal user deals mostly with valid addresses due to the use of DNS.

2.2 Proxy-based Multiple Path Routing

With the advent of inexpensive and high-bandwidth broadband connection technologies such as ADSL andcable modem for home and business users, multihoming has emerged a vital choice for large and smallbusinesses to ensure Internet connectivities even during network failures, congestion, or DDoS attacks

C—4

[36]. Recent studies on overlay networks such as Detour [64] and RON [13] showed that alternate indi-rect paths can offer much better performance in some cases. Akella et al [4] compared overlay routing andmultihoming route control using relay nodes on Akamai content delivery networks and observed that theperformance achieved by route control together with multihoming to that of three ISPs (3-multihoming) iswithin 5-15conjunction 3-multihoming, in terms of both end-to-end RTT and throughput. They showed thatwhile multihoming cannot offer the nearly perfect resilience of overlays, it can eliminate almost all failuresexperienced by a singly-homed end-network. Stoica et al [71]proposed a general overlay-based Internet In-direction Infrastructure (i3) that offers a rendezvous-based communication abstraction. Instead of explicitlysending a packet to a destination, each packet is associated with an identifier; this identifier is then used bythe receiver to obtain delivery of the packet. This level of indirection decouples the act of sending from theact of receiving, and allows i3 to efficiently support a wide variety of fundamental communication services.The Chord peer-to-peer lookup protocol [72] was used in i3 to map the identifier to the real IP address.

Secure Overlay Services (SOS) [40] was one of the first solutions to explore the idea of using overlaynetworks for pro-actively defending against DoS attacks. SOS protects end-hosts from flooding attacksby (i) installing filters at the ISP providing connectivity to the end-host and (ii) using an overlay networkto authenticate the users. Mayday [12] generalizes this SOS architecture and analyzes the implicationsof choosing different filtering techniques and overlay routing mechanisms.Adkin et al investigate how i3overlay network can be used to defense against DoS attacks [2].

Recently, TCP Westwood [31] improves TCP performance when packets are sent over multiple paths.TCP-PR [15] investigates the timer-based techniques for dealing persistent reordering of packets arrivedover multiple paths. The research results have demonstrated that the multiple paths pose new challengingproblems but it definitely can help improve TCP performance. Chen [20] proposed a multipath transportprotocol called MPTCP that opens multiple TCP connections over different paths and multiplexes dataamong the paths.

2.3 Intrusion Detection and Response

Recent intrusion detection research has been heading toward a distributed framework on monitors that dolocal detection and provide information for global detection of intrusions. These includes GrIDS [21],EMERALD [60] and AAFID [70]. They rely on some predefined hierarchical organization and most ofthem perform centralized intrusion analysis. Gopalakrishna and Spafford [33] present a framework fordoing distributed intrusion detection with no centralized analysis component. Ning et al [57] presents adecentralized method for autonomous but cooperative component systems to detect distributed attacks spec-ified by signatures. In Ning’s work the intrusion event sequences are analyzed and matched with a set ofattack scenarios. This leads to a more efficient alert aggregation and correlation technique. Besides provid-ing an abstraction for studying the attack strategies, It also allows the network administration to anticipatethe subsequent attacks and plan for the defense. However subtle changes in the signatures of the intrusiontraffic can lead to partial matches and require further monitoring or intensive analysis possibly with humaninvolvement.

In [77] a system architecture and mechanisms for protecting mobile ad hoc networks is proposed. Ex-periment results demonstrates that an anomaly detection approach works well on different mobile ad hocnetwork with route logic compromise and traffic pattern distortion. It will be interested to see how theirframework can be applied in SCOLD systems.

Julisch propose a novel alarm clustering method [38] that remove redundant alarms and support thehuman analysis in identifying root causes. Experiments show significant reduction in alert load. Effectivegeneralization hierarchies for IP addresses, ports, and time duration are used in the alarm clustering.

C—5

Feedback Controller

Router

process allocation module

process pool 1

process pool n Apache

port 1

port n

traffic generator 1

traffic generator n

Management module

process pool 2 port

2 traffic

classifier

Figure 2:The implementation of a service differentiation Web server.

3 Proposed Work

3.1 Predictable QoS Differentiation and Regulation

Today’s IDS systems often give us false alarms due to the lack of flexibility under new cyber attacks anduncertain threats. Some cyber attacks are initially unobvious and disruptive, while it is often too late whenthey are known to be malicious attacks. Malware can spread at exponential rates, infecting many machinesbefore the severity of the situation can be recognized by IDS systems. The emerging degrading DDoS attacksaim to increasingly or periodically consume portions of a victim system’s resources so as to result in denialof service to legitimate clients during high load periods. Traditional threshold-based firewall techniques maynot be able to recognize the attacks. Thus, the existing on/off intrusion detection models are not sufficient.Novel IDS systems have to be flexible under uncertain cyber attacks.

We propose to expand the capability of resource management in network systems so as to enable flexibleintrusion detection. If a network system is able to manage its resources and provide QoS levels differentlyto various incoming traffic, the front-end IDS systems can be flexible in reporting intrusion scenarios. Evenif the confidence about some network traffic is below a certain threshold for raising an alert, an IDS systemcan report the confidence level to the QoS-aware intrusion mitigation component which handles the networktraffic accordingly. Thus, the QoS-aware resource management has Dual effect: it enables IDS systems tobe flexible in raising alerts and enables network systems to behave differently to different traffic.

Preliminary Results: We propose to develop QoS differentiation and regulation techniques to support theflexible intrusion detection and adaptive intrusion mitigation purposes. We designed resource managementapproaches to bandwidth differentiation on streaming servers [81], two-dimensional service differentiationon e-Commerce servers [79], slowdown differentiation on Internet servers [80], and delay differentiationon network routers [76]. Recently, we designed and implemented a dynamic process allocation approach,integrated with a proportional integral derivation (PID) feedback controller, on an Apache Web server forproportional response time differentiation [78]. Figure 2 depicts the architecture of the integrated processallocation implementation. The dynamic project allocation is guided by a queueing-theoretical approach andcorrected by the PID controller according to the errors between expected QoS spacings and the experiencedQoS spacings of the traffic classes.

Figure 3 depicts the achieved response time ratio with its 95% confidence interval of two classes dueto the queueing-theoretical approach alone and the integrated approach, respectively. The pre-specified dif-ferentiation weight ratio of two classes was set to be 1:2. In the integrated approach, two different sets ofcontrol parameter were adopted. We observe that the integrated approach with both feedback parametersettings significantly outperforms the queueing-theoretical approach with respect to the response time dif-

C—6

Figure 3: 2-class responsiveness differentiation. Figure 4: 3-class responsiveness differentiation.

ferentiation ratio. When the system load is close to system capacity, say at 90%, the queueing-theoreticalapproach generates very poor proportionality. This can be explained by the fact that as the system load isclose to its capacity, the impact of the variance of interarrival times on queueing delay dominates and thusqueueing delay in all traffic queues increase significantly. This affects the controllability of the queueing-theoretical process allocation approach. On the other hand, the integrated approach with feedback controlis able to maintain desirable differentiation proportionality. Figure 4 depicts the differentiation results ofthree-class workload due to the approaches. The pre-specified differentiation weight ratio of the classeswas set to be 1:2:3. The results demonstrate the feasibility and practice of the integrated process allocationapproach for providing QoS differentiation.

Research Plans:The preliminary results have demonstrated the feasibility of providing predictable QoSdifferentiation by adaptive resource management. They advanced our understanding of QoS differentiationtechniques to the level where further studies along the line would lead to a breakthrough in the integration offlexible intrusion detection and QoS-adaptive resource management for mitigating the impact of uncertaincyber attacks and threats. We plan to conduct further studies along the line in the following aspects:

1. Develop generalized resource management mechanisms for QoS differentiation and regulation in bothnetwork routers and end-point computer system. Current QoS differentiation mechanisms lack ofgenerality, which means their performance depends upon specific assumptions of distributions andcharacteristic of traffic patterns, i.e., inter-arrivals and job sizes. For example, most differentiatedpacket forwarding mechanisms assumed Pareto arrival pattern and fixed-size packets on the Inter-net [27, 37, 44, 45]. This may not reflect the true traffic scenarios on the Internet. A robust cyber-defense mechanism should be of generality.

2. Investigate the impact of service differentiation techniques based on QoS-aware resource manage-ment for limiting spreading rate of susceptible traffic and regulating traffic under cyber attacks. Theobjective is to make the performance of network systems configurable and controllable by themselves,instead of by parameters and behaviors of cyber attacks. The techniques should minimize the perfor-mance impact on normal routers and end-point computer systems.

3.2 Improve Service Availability with Novel User-centered Paradigm

Most of the recent overlay and multihoming work, such Internet indirection infrastructure [71] and CoDNS [75],focus on improving network core or ISP routing and naming infrastructure. Many of them still emphasize

C—7

Figure 5: The Scold DDoS defense system. Figure 6: Peer-to-peer secure DNS update.

making the changes transparent to the end users by hiding them at the network edges or gateways. Fewemphasize making the new capabilities such as the multi-path and indirect routing available to and con-trolled by the end users or end systems. Our SCOLD probably is the first system that delivers the multi-pathand indirect routing capabilities all the way to the end users through the secure peer-to-peer DNS updatewith indirect routing entries and the enhanced resolver library on end users machine. Following the sameuser-enabling theme, we propose to make network system status information and new network capabilitiesavailable to the end users as much as possible, and as fast and secure as we can. This include the tasks ofdesigning new subscription/notification APIs to receive network connection status and new interface to pro-vide users more control on the resource in their end machines, their fair share of network bandwidth withinthe organization, and their internet connectivity. Provide timely notification and giving user more controlare essential to gain back the trust on cyber infrastructure.

Preliminary Results: In [24], we designed and implemented a secure collective DDoS defense system(SCOLD) based on indirect routing. The key idea is to follow intrusion tolerance paradigm and providealternate routes via a set of proxy servers and alternate gateways when the normal route is unavailable orunstable due to network failures, congestion, or DDoS attacks. The BIND9 DNS server and its DNS updateutilities are enhanced to support new DNS entries with indirect routing. Protocol software for supportingthe establishment of indirect routes based on the new DNS entries is developed for Linux systems.

Figure 5 shows when DDoS attacks were detected by the IDS attached to the main gateway. It sends thedistress call to the SCOLD Reroute Coordinator, which computes the proper subset of proxy servers for aclient subnet and sends the reroute command with the new DNS entry to update the client DNS server via aselected proxy server. The new DNS entry contains the domain name, IP address of the victim and a set ofdesignated proxy servers for indirect routing. Further attacks on the new routes via those proxy servers willbe detected and blocked at the proxy servers farther away from the alternate gateways. The IP addressesof alternate gateways are hidden from the clients. The reroute coordinator informs a proxy server whichalternate gateway(s) to use.

Figure 6 shows that after the client DNS server is updated with the new DNS entry, it can utilize thesame indirect route to query the target DNS. This in effect provides a secure peer-to-peer DNS query viaindirect route. Since the main gateway is being attacked by DDoS attacks, DNS queries via normal routewill not be answered. IP tunnels are established among the client DNS server, the proxy, and the alternategateway for indirect routing. We modified Bind9 server and DNS update utilities. The client resolver libraryof Linux system was modified to receive the new DNS entries and to set up the IP tunnel routing entry.

Figure 7 shows the benefit of SCOLD indirect routing defense against DDoS and its processing over-

C—8

Figure 7: Indirect routing performance result. Figure 8: Multi-path routing performance.

head, which comes from the IP tunneling overhead and more hops involved in the indirect route. It isobserved that the overhead of the indirect route in term of the response time is about 70 percent. Furtherexperiments shows the overhead varies from 30 to 200 percent. However, under DDoS attack, the responsetime with direct route increases dramatically (15 times to infinity), while the response time with indirectroute keeps the same. Therefore, compared with the serious impact of DDoS attacks on the direct route,indirect routing can improve the network security, availability and performance with acceptable initial setupoverhead and processing overhead.

We have modified the networking modules in a 2.4.24 Linux kernel with a thin layer between TCP andIP implementing double buffer to alleviate persistent ordering. It reads the configuration in aproc file forspreading the traffic over multiple paths established by the SCOLD system. Figure 8 shows the aggregateweb access bandwidth improves proportionally as the number of proxy servers increases in our testbed. Theconfiguration parameters include the buffer size, the number of multiple paths, and their weights.

Proposed Work: The above preliminary work has demonstrated the feasibility of a secure collective defensesystem based on peer-to-peer secure DNS update via indirect routes and proxy based multiple path indirectrouting. It gives us insight on key issues related the improvement of such secure collective defense system.In this project, we plan to conduct the following studies:

1. Even though our secure peer-to-peer DNS update can deliver the new routing info to the client subnets,the existing connections will not be aware of the changes. To make the new multipath indirect routingand other network capabilities available to the end users, we will develop new secure informationsharing and notification system with new API to facilitate the creation of new network services.

2. Develop new efficient proxy server selection algorithms for selecting an optimal or semi-optimalsubset of proxy servers between a victim site and a client site. Even though similar cache serverselection algorithms exist for content delivery networks, here the metrics for evaluating the algorithmswill include both the geographical diversity for security concerns, the aggregated bandwidth, and QoSof connnections.

3. Improve the reliability of the SCOLD protocol and server modules. Port our Linux implementationsto work on windows systems to make it widely available to end systems. Propose changes to IETFto include the enhancement on DNS for supporting proxy-based indirect routing and secure indirectDNS update.

4. We propose to develop new thin layer between transport protocol and IP layers that takes advantageof the available multiple paths and provide API to inform the status of these connections and allow

C—9

Figure 9: A2D2 Multilevel Rate Limiting.

users finer control. We will explore the effective of using intelligent buffering with dynamic feedbackand evaluate their performance in the testbed. It is important to investigate how to distribute the dataover multiple routes based on desirable requirements such as security, performance, and reliability.

3.3 Cooperative Intrusion Detection and Response (Cooperative-IDR)

Existing intrusion detection systems are plagued by too many false positives. Techniques are needed toclustering the reports, and remove the redundant alerts generated by the same root cause. Allowing dis-tributed coordination and direct communications among the IDR devices will help track down the intrusionsources, push back intrusion traffic, detect compromised or malfunction nodes, and provide alternate routesfor intrusion tolerance. It will be also of interest to investigate how the collection of proxy servers and theavailability of the multiple path indirect routes can be used to improve the security of the network system.

Preliminary Results: In [17], we have developed an Autonomous Anti-DDoS system, called A2D2, wherean enhanced SNORT IDS with subnet flooding plug-in is integrated with a multiple level adaptive ratelimiting firewall. Figure 9 shows that alerts which are generated by the enhanced SNORT system with subnetflooding detection, automatically trigger the insertion of the firewall rules. Users of A2D2 can specify themultiple level of rate limiting. The system keeps history records, and adaptively blocks potential intrusion,or puts those suspicious traffic in queues with restricted packet rates. Preliminary experiment results showsthat A2D2 can tolerate various DDoS attacks. A subset of Intrusion Detection and Isolation Protocol [56]was developed and is used in cooperative intrusion push back experiments.

Figure 10 shows the histogram of RealPlayer traffic during an ICMP Stacheldraht attack on our A2D2testbed: a total of 7,127 packets are received by the RealPlayer instead of the 23,000 packets normally trans-mitted for the entire 10-minute video clip. Only four packets were recovered out of the 2,105 retransmissionrequests sent. Figure 11 shows the histogram of RealPlayer traffic when A2D2 multiple level rate limitingdefense is turned on. It improves QoS during DDoS attacks. A total of 23,444 packets were received by theA2D2 RealPlayer client, similar to those received by the baseline test scenario. There was no observableservice degrades.

Proposed Work: The above preliminary work has demonstrated the feasibility of an automated cyber de-fense system based on intrusion tolerance and QoS based intrusion mitigation techniques. To take fulladvantage of these techniques, it requires us to re-examine the IDS and intrusion fusion techniques for

C—10

Figure 10: Histogram of a DDoS Attack. Figure 11: Histogram of an A2D2 Defense.

providing early warnings and to enable the intrusion handling system to take early proper actions. In thisproject, we plan to conduct the following studies:

1. Re-design the intrusion detection and alert correlation system components by providing early alertsand adaptive tracking feature. We will re-design the scan detection algorithm and behavior basedintrusion detection system to incorporate these changes. To support the tighter interaction such asadaptive monitoring and tracking, we would like to extend the current IDS API to accept requestsfrom the intrusion handling system. Efficient layer coding of cyber defense data will be investigated.

2. Develop secure information sharing framework and related API so that all cyber defense system com-ponents can communicate using the secure service-oriented architecture (SE-SOA). We will investi-gate how our existing privilege management infrastructure based on attribute certificate and LDAPsystems can be integrated to provide secure access control. We will explore the implementation ofsecure service-oriented architecture where IDMEF format is used to exchange cyber-defense data.The web service interface will be enhanced with an access control decision engine that queries LDAPsystems for proper authorization.

3. Integrate enhanced IDS and adaptive firewall for distributed intrusion detection and handling with SE-SOA. Design efficient techniques for tracing the intrusion routes. Develop specification language forspecifying the secure collective defense architecture and the related rules for reconfiguring network,server cluster, and IDS rule updates.

4. Develop an adaptive intrusion handling system that utilizes the QoS-based intrusion mitigation andproxy based multipath intrusion tolerance techniques. Evaluate the effectiveness of these techniqueson their effectiveness in blocking DDoS attacks and potential worms from spreading.

3.4 Evaluation of the Integrated Cyber-defense System

In this section we address issues in evaluation of the integrated cyber-defense system with particular em-phasis on predictability issues. This is inter-disciplinary work drawing from the experience of Dr. Boult inthe areas of video-based physical security systems as well as his work in evaluation of biometric systems.The section begins with a discussion on the background of evaluation, reviews the state of the art, and thendiscusses the proposed work and timeline.

C—11

In the 70s and 80s, the term “intrusion detection system” was unknown by people and would have meant,to almost any security personnel, a system to detect the breach of their physical security perimeter. As wehave moved to the network centric world, the more feared attacks on these same facilities have moved tothe cyber world. Unfortunately, it seems the engineering and scientific knowledge developed with yearsof experience with physical systems, has gone largely unnoticed in the cyber security arena. In particular,while the “sensors” and attacks may be different, the evaluations and methodologies apply in both. Whilethe recent evaluation work in cyber IDS systems such as [61, 6, 46, 48, 5, 18, 63, 74, 7, 10, 9, 11] have madeconsiderable progress, they have yet to consider one of the most basic issues of performance evaluation –confidencein the results. Before we can discuss these works in context, we present a bit more backgroundon detection system evaluation.

In the evaluation of a physical intrusion detection system (hereafter called Physical Security Equipment(PSE) to help avoid confusion), the approaches to measure/insure confidence in evaluations have long beenstudied. The ROC curves, common in the sensor community can, if the measurements are appropriate,allow efficient performance trade-offs, including confidence measures. But the measurements must be ofthe appropriate type (and numbers) before that can be used. A good example is [8], which discusses the useof statistical process control (SPC) [34], for both initial testing and ongoing assessments of PSE to insurethe levels of detectionand confidencerequired by DOE regulations for nuclear site security. In [8] bothsimple (e.g. metal detector) and complex (e.g. hand-geometry based biometric) sensors were discussed.

The SPC approach allows one to approach the confidence estimates in many different ways. [8] high-lights some of the differences between testing with binary data (detect/fail), which they call attribute data,and using value data (e.g. event confidence), which is then thresholded to produce a decision (i.e. appropri-ate ROC type curves). With pure binary data, validating a requirement of 85% probability of detection and95% confidence level with a single-phase testing attribute testing strategy and allowing no missed alarmsrequired 19 “identical” tests for every “set of conditions” to be considered. Allowing a single miss in the test(so there is a known miss-detection) requires 30 tests per setting to validate a 85% PD with 95% confidence.With value data,and a stable process, the number of tests can be significantly reduced down to 5.

This model of performance, detection probability with confidence levels, is so standard within the physi-cal intrusion detection community, that government physical security procurement programs routinely writetheir requirements/specifications in that form without even bothering to provide references to the definitions.To those that work with (proper) ROC curves, their ability to help characterize performance is important.However, as was discussed in [49], the “ROC” curves that have been produced for IDS systems are not trueROC curves. The values are not actually related to trueprobabilitiesof detection or false alarms used forstandard ROC curves (or even the likelihood ratio test or maximal Bayesian estimator use in other fields).Hence they provide little relation to “confidence”. Rather the ROCs used in IDS are detection counts versusfalse alarms (or worse, false alarm percentages), where the “units” are not well defined. They are closer tothe Cumulative Match Score curves used in earlier biometric evaluations [59]. While they can provide somemeans for comparison, the lack of association with true probability and confidence means no “staticallysignificant” difference can even be discussed.

In the biometric community, many of these issues have not only been addressed in their evaluations,years of research have realized subtle issues that are of particular relevance to IDS. The underlying as-sumption in statistical process control or other simple statistical evaluation approaches is that test trials (andimplicitly detection failures) are independent Bernoulli trials. This works well for simple sensors; however,when used for more complex “detectors,” such as “biometric,” it has had difficulties in actually predictingperformance.

The SPCA theory is valid, and will be tried, but it is not clear that the assumptions will hold for IDS. Forface-based biometric systems, our earlier research demonstrated the underlying assumptions of Bernoulliwere not statistically valid for even moderately large (1000’s of trials) sets of biometric data [51]. In partic-

C—12

ular, we showed that some sets of “individuals” were inherently more difficult to detect/recognize than othersand showed a statistically significant “gallery” effect [50]. This fact means that the standard Bernoulli-basedanalysis was not appropriate for producing confidence intervals for this type of data. We also proposed anapproach, based on the advanced statistical concepts of balanced repeated replication [68, 65, 42], that, atleast over the sample data, allowed us to compute standard errors without assuming independent errors.While at first resisted by others in the biometrics community, within two years other researchers began to re-alize that giving up the independence assumptions, while requiring more complex data analysis techniques,allowed more detailed analysis which could focus on what common factors across subsets may make somegroups more difficult than others [32]. In the end, we significantly generalized our sampling work to addressgeneral sampling design in large scale evaluations, including an approach to detect “unknown co-factors”in biometric evaluation data [53]. This was evaluated on a face-based biometric dataset collection over 6months that contained over .75TB of data, over 1 million facial images and 1 billion biometric signaturecomparisons. Also discussed in [53], is how any properly normalized ROC, with sufficiently many samplesand proper units, can be used to estimate confidence intervals.

Our push to improve the computation of confidence intervals for biometrics has already been adoptedby other groups and has been used by NIST for the analysis of FingerPrint systems and the recent FaceRecognition Vendor Test 2003 [58, 35, 52].

There have been a few other “evaluation” metrics proposed. Stolfo, [73] proposed a cost based approach.Unfortunately quantifying the actual “costs” is impractical. Even if using a cost model, the uncertainty in the“measurements” are still critical to assessing the “cost risk”. In a similar manner, [29] suggests a cost model.That paper mixes costs and ROC analysis, but requires the ROC to be actual probabilities and explicitly statesit ignores how to compute an actual (probabilistic valid) ROC. In [74], an evaluation metric, that includes theimpact of the response, is considered. Again it is not clear the metric generalizes nor is there any discussionof how statistical variations in the detection would be accommodated in the optimization models.

Having briefly introduced some background from Physical security systems and biometric evaluation,let us now revisit the state of evaluation in IDS and more general network/cyber threat detection systems.The majority of the work on IDS testing and evaluation has focused on the use of simulated attacks, howto effectively generate the attacks and tools for automating the attack process and evaluation, [61, 41, 25,39, 3]. Use of live data is also common, clearly realistic, but not reproducible. The efficient generationof “test data” is, without question, an important topic. While there will always be arguments about therepresentativeness of such attack data [49], it is impractical to have large scale testing that does not useat least moderate amounts of simulated or at least replayed data. We don’t expect to add significant newresearch in “generation” components of these areas. However, to reach meaningful statistical conclusions,we will examine both the proper subdivision of categories of data, and the sampling of that data to achievestatistical confidence (within the range of what can be simulated). From a research perspective this willinvolve obtaining (or if necessary reimplementing) many of these generation tools, and then developingtechniques to more easily reintegrate different mixes of data and then automatically score the results.

We agree with the statements in [10], that new public datasets are needed. While much has been statedabout the needs of such a dataset, few have addressed statistical characterization. It is known that differentenvironments have vastly different background behavior, and even the same environment has significanttemporal variations. Realistically, it is not a single dataset that is needed, but a collection of them.

For datasets that are already existing and labeled, we will look at two modifications. The first will be togenerate “randomizers” that take the original data and perturb the data in various ways to simulate different“samplings” of the original data streams and variations in the timing and order of events. These randomizedvariations will allow a very restricted form of “confidence” computations from the resulting analysis. Whilethis is less ideal than new generation with proper sampling, many researchers have already used this data, aswell as its use in evaluating commercial systems [69]. While it would be good to proceed by extending the

C—13

best existing protocols/software such as [63], they are, unfortunately, not available for public use.

More “realistic” data has been used in tests conducted by Nophais Labs, [67, 66, 54], with backgroundtraffic created by replaying traffic captured from a DePaul University lab. This was useful for high loadtesting (30-99Mbits/seconds), but no detailed analysis was made of detection rates and it is unclear whatground truth is available. Interestingly, the open-source Snort IDS[62], which we use in our labs, wasrated third in their evaluation, outperforming 7 of the 9 commercial products tested. For the more complexdatasets, which were a mix of real and/or sanitized traffic and simulated attacks, such as [28, 54], we willexplore the variational approach as well, if we can obtain the data.

Given the simulation tools and datasets we will develop hierarchical distributional models, and then testthe resampled data with the available network-based anomaly detection system to insure they do not findany anomalies. This is necessary because the statistics characteristics may change during re-sampling andwe need to find re-samplings that do no change the background “detection” of these systems. In [47], theiranalysis shows that even the existing (synthetic) IDEVAL data has simulation artifacts that may producemeasurable artifacts in anomaly detection systems.

We are unaware of any evaluation that attempts to quantify the impact of mitigation, i.e. evaluating theimpact of techniques designed to reduce or eliminate the impact of the attack. The closest would be thework of [74] which presents a network model and an “algorithm” to evaluate the impact of the responseusing that model. But that paper presents no significant evaluation or even evaluation methodology. Thoughit is phrased differently the “cost” models of [73] address some of these issues for standard IDS and anomalydetection system, but not for complex response models. Furthermore, that work suggested a “cost-based’criterion of evaluation but not an evaluation methodology. How do we effectively evaluate techniques likethose proposed in other sections of the proposal that seek to limit the impact with a distributed “response”,e.g. one that uses network wide QoS to limit the DDOS, or that uses proxy servers to redirect traffic? Weneed measures that make sure the responses not only improve local responsiveness, but reduce the overallimpact of the attack across the full network. These are research questions that will be addressed, in partby instrumented toolsets and in part, we expect, by designing special experiments to measure the requestedQoS or routing changes and then using quantitative network models to estimate the impact of such changes.

In summary, to support the dataset generation and analysis we will be developing the IMPACT toolset,a distributed cyber-evaluation toolset which will facilitate the parallel dumping, sanitizing and scoring ofnetwork traffic, supporting generation of new datasets as well as on-line evaluations. It will work as a coor-dinated distributed systems to allow processing/collecting data at closer to backbone speeds, but with addedtime-stamping and indexing to support enhanced playback capabilities necessary for randomization to sup-port “statistical” confidence testing. The tool chain will contain components to allow it to programmatically“remap” users, IP addresses and even some content.

4 Broader Impact of the Project

In addition to the technical contributions, this proposal will have more significant broader impact. The first isa mixture of education and societal. As part of our program’s outreach to the community, we have arrangedto develop a local cable-television show focused on cyber security issues within our community. The show,minimally 30 min per month and possibly more depending on costs/sponsorship and viewer feedback, willseek to educate, and actively involve the local IT workforce in cyber security issues. The Colorado Springsarea has almost 200,000 white-collar and military employees. The surrounding areas, some of which areserved by that same cable company and would receive the broadcasts, nearly doubles that level. The areais home to multiple military groups including US Northern Command (in charge of US Military HomelandDefense), Cheyenne Mounting Complex (US Strategic Command), Fort Carson, Peterson AFB, Shriver

C—14

AFB, and the Air Force Academy. There are significant installations of major corporations including Intel,Amtel, LockHeed-Martin, Raytheon, ITT, HP/Compaq, Boeing, MCIworldcom, Quantum, Oracle, FederalExpress Data Systems, Adelphia Cable, Allied Signal, Qwest Communications, Comp.Sci.Corp., TRW,DRS, and Gateway 2000, as well as three major hospitals and a significant school system. By making itlocal and related to things they know, we expect this TV show to help to keep this high-tech workforcemore up to date and significantly improve our local impact. This effort will be pursued in conjunction withour Networking Information and Space Security Center in Colorado Springs, which already has significantties with Northern Command and the local military that, given their missions, are very interested in cybersecurity issues. In conjunction with NISSC, the UCCS CS Department provides a certificate program inInformation Assurance, some of the courses might be used to add more detailed technical content to thebroadcasts, if there is sufficient demand for the increased frequency and increased depth.

The second impact will be on the university itself, where it will be engaging both graduate and un-dergraduate students. In its 2004 college rankings edition, “America’s Best Colleges,” US News’s editorsranked CU-Colorado Springs 5th among public master’s universities in the West. While the school’s historyis one of undergraduate and MS level education with pockets of research, it is on the road to becoming aregional research university. It has had doctoral programs in Engineering for over a decade, but only smallamounts of funded research – limiting the growth of the doctoral program. The proposal will help fund newdoctoral students and aid in the transformation of UCCS to a regional research university. UCCS has a non-traditional undergraduate population with a mostly commuting student body, a median undergraduate ageapproaching 30. A majority of the UCCS students are self-supporting, and opportunities to work on campuswill improve their chances of success and potential to have research or advanced development careers. Thisfunding will provide unique opportunities to these students.

5 Research Schedule and Deliverables

The research team will include the three PIs, two graduate research assistants, and two undergraduate re-search assistants. Dr. Chow has extensive experience in network and protocol design, network restoration,content switching, and network security. Dr. Zhou’s expertise is in QoS-adaptive scheduling and resourcemanagement on distributed systems. Dr. Boult is an expert in physical security and network systems. We areteaming up to investigate cost-effective solutions for establishing trustworthy network system performance.

One GRA will be working on the QoS-aware adaptive resource management and proxy-based multipleindirect routing techniques. The other GRA will improve intrusion detection, alert fusion, and intrusion han-dling systems that utilize the new QoS-aware intrusion mitigation techniques, and evaluate the performanceof the integrated cyber-defense system. Two undergraduate students will work part-time and participate inthe research and development effort. In keeping with past experience we also expect 4-6 MS level studentsto work on the related network and system areas. These unpaid MS students are still expected to produceconference/workshop papers and their travel expenses may come from the grant. The students working onIMPACT are expected to interact closely, and may be contributing to the other aspects of the project, es-pecially the distributed detection components. The proposed research is planned to be carried out on theexperimental testbeds at the Networking and Systems Laboratory, with which the PIs are affiliated.

The deliverables include the publications of research results, the software packages developed for en-hanced secure DNS update, multiple indirect routing, and cooperative IDR system, a library of the inte-gration of admission control, feedback control, and resource management algorithms, and technical reportsafter each milestone. The reports will be published in ACM/IEEE sponsored leading technical conferencesand journals. Any software package resulted from this project will be released through the project homepagefor the public use free of charge.

C—15

References

[1] T. F. Abdelzaher, K. G. Shin, and N. Bhatti. Performance guarantees for Web server end-systems: acontrol-theoretical approach.IEEE Trans. on Parallel and Distributed Systems, 13(1):80–96, 2002.

[2] D. Adkins, K. Lakshminarayanan, A. Perrig, and I. Stoica. Toward a more functional and securenetwork infrastructure. Technical report, Univ. California, Berkeley, UCB Tech. Rep. UCB/CSD-03-1242, http://sahara.cs.berkeley.edu/jun2003-retreat/TR-CSD-03-1242.pdf, Internet Draft, 2003.

[3] S.J. Aguirre and W.H.Hill. Intrusion detection fly-off: Implications for the united states navy. Technicalreport, MITRE, 1997.

[4] A. Akella, P. Pang, B. Maggs, S. Seshan, and A. Shaikh. A comparison of overlay routing and multi-homing route control. InProc. of ACM SIGCOMM, pages 93–106, 2004.

[5] D. Alessandri. Using rule-based activity descriptions to evaluate intrusion detection systems. In H. De-bar, L. Me, and S. F. Wu, editors,Int. Workshop on Recent Advances in Intrusion Detection, volume1907 ofLectures in CS, pages 183–196. Springer Verlag, 2000.

[6] E. Amoroso and R. Kwapniewski. A selection criteria for intrusion detection systems. InProc. 14thAnnualComputer Security Applications Conference, pages 280 – 288. IEEE, Dec 1998.

[7] G.A. Fink and B.L. Chappell and T.G. Turner and K.F. O’Donoghue. A metrics-based approach tointrusion detection system evaluation for distributed real-time systems. InProc. Int. Parallel andDistributed Processing Symposium, pages 93–100. IEEE, 2002.

[8] D.W. Murray and D.D. Spencer. Statistical process control testing of electronic security equipment. InIEEE Int. Carnahan Conf on Security Technology, pages 53–59. IEEE, Oct 1994.

[9] W.H. Allen and G.A. Marin. On the self-similarity of synthetic traffic for the evaluation of intrusiondetection systems. InProc. Symp. on Applications and the Internet, pages 242–248. IEEE, Jan 2003.

[10] N. Athanasiades and R. Abler and J. Levine and H. Owen and G. Riley. Intrusion detection testing andbenchmarking methodologies. InFirst IEEE Int. Workshop on Informatoin Assurance (IWIAS), pages63–72. IEEE, March 2003.

[11] K.M.C. Tan and R.A. Maxion. Determining the operational limits of an anomaly-based intrusiondetector.IEEE Journal on Selected Areas in Communications, 21(1):96–110, Jan 2003.

[12] D. Andersen. Mayday: Distributed filtering for internet services. InUSENIX Symposium on InternetTechnologies and Systems, pages 20–30, 2003.

[13] D. Andersen, H. Balakrishnan, M. Kaashoek, and R. Morris. Resilient overlay networks. InProc. ofthe 18th Symposium on Operating System Principles, pages 131–145, 2002.

[14] S. Blake, D. Black, M. Carlson, E. Davies, Wang Z., and W. Weiss. An architecture for differentiatedservices.IETF RFC 2475, 1998.

[15] S. Bohacek, J. Hespanha, J. Lee, C. Lim, and K. Obraczka. Tcp-pr: Tcp for persistent packet re-ordering. InProc. of the IEEE 23rd Int. Conf. on Distributed Computing Systems, pages 222–231,2003.

[16] L. Briesemerister, P. Lincoln, and P. Porras. Epidemic profiles and defense of scale-free networks. InProc. of ACM WORM, 2003.

D–1

[17] A. Cearns and C. E. Chow. A2d2: Design of an autonomous anti-ddos (a2d2) network. InProc. ofIASTED Conf. on Applied Informatic, 2003.

[18] T. Champion and M. Denz. A benchmark evaluation of network intrusion detection systems. InProc.of IEEE Conf. on Aerospace Systems, 2001.

[19] S. Chandra, C. S. Ellis, and A. Vahdat. Application-level differentiated multimedia Web services usingquality aware transcoding.IEEE J. on Selected Areas in Communications, 18(12):2544–2265, 2000.

[20] J. Chen. New approaches to routing for large scale data networks. Technical report, Ph.D. Dissertation,Rice University, 1999.

[21] S. Chen, S. Cheung, R. Crawford, and M. Dilger. GrIDS-a graph based intrusion detection system forlarge networks. InIn Proc. of the 19th National Information Systems Security Conference, 1996.

[22] S. Chen and Y. Tang. Slowing down Internet worms. InProc. of IEEE ICDCS, 2004.

[23] X. Chen and P. Mohapatra. Performance evaluation of service differentiating Internet servers.IEEETrans. on Computers, 51(11):1,368–1,375, 2002.

[24] C. E. Chow, P. J. Fong, and G. Godavari. An exercise in constructing secure mobile Ad Hoc networks.In Proc. of Int’l Conf. on Advanced Information Networking and Applications, 2004.

[25] K. Das. The development of stealthy attacks to evaluate intrusion detection systems. Master’s thesis,MIT EECS, June 2000.

[26] C. Dovrolis, D. Stiliadis, and P. Ramanathan. Proportional differentiated services: Delay differentiationand packet scheduling. InProc. ACM SIGCOMM, 1999.

[27] C. Dovrolis, D. Stiliadis, and P. Ramanathan. Proportional differentiated services: Delay differentiationand packet scheduling.IEEE/ACM Trans. on Networking, 10(1):12–26, 2002.

[28] National Laboratory for Applied Network Research. Nlar network traffic packet header traces, 2002.http://pma.nlanr.net/Traces/.

[29] J.E. Gaffney and J.W. Ulvila. Evaluation of intrusion detectors: A decision theory approach. InIEEESymp. on Security and Privacy, Oakland, CA, May 2001. IEEE.

[30] M. Garetto, W. Gong, and D. Towsley. Modeling malware spreading dynamics. InProc. of IEEEINFOCOM, 2003.

[31] M. Gerla, S. S. Lee, and G. Pau. Tcp westwood simulation studies in multiple-path cases. InProc.International Symposium on Performance Evaluation of Computer and Telecommunication Systems(SPECTS), 2002.

[32] G. Givens, J.R. Beveridge, B.A. Draper, and D. Bolme. A statistical assessment of subject factors inthe pca recognition of human faces. In”IEEE Workshop on Statistical Analysis in Computer Vision.IEEE, June 2003.

[33] R. Gopalakrishna and E. Spafford. A framework for distributed intrusion detection using interest-driven cooperating agents. InFourth International Symposium on Recent Advances in Intrusion De-tection (RAID)1, 2004.

[34] E.L. Grant and R.S Leavenworth.Statistical Quality Control. McGraw-Hill, 1972.

D–2

[35] P.J. Grother, R.J. Micheals, and P. J. Phillips. Face recognition vendor test 2002 performance metrics.In Proceedings 4th International Conference on Audio Visual Based Person Authentication, June 2003.

[36] F. Guo, J. Chen, W. Li, and T. Chiueh. Experiences in building a multihoming load balancing system.In Proc. of INFOCOM 2004, pages 1241 – 1251, 2004.

[37] Y. Huang and R. Gu. A simple fifo-based scheme for differentiated loss guarantees. InProc. IWQoS,2004.

[38] Klaus Julisch. Clustering intrusion detection alarms to support root cause analysis.ACM Trans. onInformation and System Security, 6(4):443–471, 2003.

[39] K. Kendall. A database of computer attacks for the evaluation of intrusion detection systems. Master’sthesis, MIT EECS, 1999.

[40] A. D. Keromytis, V. Misra, and D. Rubenstein. Sos: Secure overlay services. InProc. of ACMSIGCOMM, pages 20–30, 2002.

[41] J. Korba. Windows nt attacks for the evaluation of intrusion detection systems. Master’s thesis, MITEECS, June 2000.

[42] D. Krewski and J. N. K. Rao. Inference from statified samples: Properties of the linearization, jackknifeand balanced repeated replication methods.The Annals of Statistics, 9(5):1010–1019, 1981.

[43] S. C. M. Lee, J. C. S. Lui, and D. K. Y. Yau. A proportional-delay diffserv-enabled Web server: admis-sion control and dynamic adaptation.IEEE Trans. on Parallel and Distributed Systems, 15(5):385–400,2004.

[44] M. K. H. Leung, J. C. S. Lui, and D. K. Y. Yau. Adaptive proportional delay differentiated services:Characterization and performance evaluation.IEEE/ACM Trans. on Networking, 9(6):908–817, 2001.

[45] J. Liebeherr and N. Christin. JoBS: Joint buffer management and scheduling for differentiated services.In Proc. of the Int’l Workshop on Quality of Service (IWQoS), pages 404–418, June 2001.

[46] R. Lippmann, J. Haines, D. Fried, J. Korba, and K. Das. The 1999 darpa off-line intrusion detectionevaluation.Computer Networks, 34:579–595, 2000.

[47] M.V. Mahoney and P.K. Chan. An analysis of the 1999 darpa/lincond laboratory evaluation data fornetwork anomaly detection. InProc. Recent Advances in Intrusion Detection, volume 2820 ofLecturesin CS, pages 220–237. Springer Verlag, November 2003.

[48] R.A. Maxion and K.M.C. Tan. Benchmarking anomaly-based detection systems. InIEEE Proc Int.Conf on Dependable Systems and Networks, pages 623–630, 2000.

[49] J. McHugh. Testing intrusion detection systems: A critique of the 1998 and 1999 darpa off-line intru-sion detection system evaluation as performed by lincoln laboratory.ACM Transactions on Informationand System Security, 3(4), November 2000.

[50] R. J. Micheals and T. E. Boult. Efficient evaluation of classification and recognition systems. InPro-ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2001), Hawaii,December 11–13 2001.

[51] R. J. Micheals and T. E. Boult. A stratified methodology for classifier and recognizer evaluation. InIEEE Workshop on Emperical Evaluation Methods in Computer Vision, Kauai, Hawaii, Dec 2001.

D–3

[52] R. J. Micheals, P. Grother, and P.J. Phillips. The nist human id evaluation framework. InProc. of the4th Int. Conference on Audio and Video-based Biometric Person Authentication, June 2003.

[53] R.J. Micheals.Biometric systems evaluation. PhD thesis, Lehigh University, 2003.

[54] P. Mueller and G. Shipley. Dragon claws its way to the top.Network Computing, August 2001.http://www.networkcomputing.com/1217/1217f2.html.

[55] Thyagarajan Nandagopal, Narayanan Venkitaraman, Raghupathy Sivakumar, and Vaduvur Bhargha-van. Delay differentiation and adaptation in core stateless networks. InProceedings of IEEE INFO-COM, pages 421–430, April 2000.

[56] Boeing Phantom Works. Network Associates Labs. Intrusion detection and isolation protocol, IDIP.Technical report, 2002.

[57] P. Ning, S. Jajodia, and S. Wang. Abstraction-based intrusion detection in distributed environments.ACM Trans. on Information and System Security (TISSEC), 4:407–452, 2001.

[58] P.J. Phillips, P. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi, and M. Bone. Face recognitionvendor test 2002. evaluation report. Technical Report IR 6965, National Institute of Standards andTechnology, March 2003. www.itl.nist.gov/iad/894.03/face/face.html.

[59] P. J. Phillips, H. Moon, S. A. Rizvi, and P. J. Rauss. The FERET evaluation methodology forface-recognition algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence,22(10):1090–1104, October 2000.

[60] P. A. Porras and P. G. Neumann. Emerald: event monitoring enabling responses to anomalous livedisturbances. InIn 1997 National Information Systems Security Conference, 1997.

[61] N. Puketza, M. Chung, R. A. Olsson, and B. Mukherjee. A software platform for testing intrusiondetection systems.IEEE Software, pages 43–51, September/October 1997.

[62] M. Roesch. Snort - lightweight intrusion detection for networks. InUSENIX 13th Systems Adminis-tration Conference - LISA ’99, Seattle, Washington, 1999. usenix. see also www.snort.org.

[63] L.M. Rossey, R.K. Cunningham, D.J. Fried, J.C. Rabek, R.P. Lippmann, J.W. Haines, and M.A.Zissman. Lariat: Lincoln adaptable real-time information assurance testbed. InIEEE Proc. AerospaceConference, volume 6, pages 2671–2682, March 2002.

[64] S. Savage, A. Collins, E. Hoffman, J. Snell, and T. Anderson. The end-to-end effects of internet pathselection. InProc. of ACM SIGCOMM, pages 289–299, 1999.

[65] J. Shao and C. F. J. Wu. Asymptotic properties of the balanced repeated replication method for samplequantiles.Annals of Statistics, 20(3):1571–1593, September 1992.

[66] G. Shipley. Intrusion detection, take two. Network Computing, November 1999.http://www.networkcomputing.com/1023/1023f1.html.

[67] G. Shipley. Iss realsecure pushes past newer ids players.Network Computing, May 1999.http://www.networkcomputing.com/1010/1010r1.html.

[68] R. R. Sitter. Balanced repeated replications based on orthogonal multi-arrays.Biometrika, 80(1):211–221, March 1993.

D–4

[69] D. Song, G. Shaffer, and M. Undy. Nidsbench - a network intrusion detection test suite. InRecent Ad-vances in Intrusion Detection, Second International Workshop, West Lafayette, 1999. http://www.raid-symposium.org/raid99/PAPERS/Song.pdf.

[70] E. H. Spafford and D. Zamboni. Intrusion detection using autonomous agents.Computer Networks,34(4):547–570, 2000.

[71] I. Stoica, D. Adkins, S. Zhuang, S. Shenker, and S. Surana. Internet indirection infrastructure.IEEE/ACM Trans. on Networking, 12(2):205–218, 2004.

[72] I. Stoica, R. Morris, D. Karger, M.F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peerlookup service for internet applications. InProc. of ACM SIGCOMM, pages 149–160, 2001.

[73] S. Stolfo, W. Fan, W. Lee, A. Prodromidis, and P. Chan. Cost-based modeling for fraud and intrusiondetection: Results from the jam project. Inn Proceedings of the 2000 DARPA Information SurvivabilityConference and Exposition (DISCEX ’00), 2000.

[74] T. Toth and C. Kruegel. Evaluating the impact of automated intrusion response mechanisms. In18thComputer Security Applications Conference, pages 301–310. IEEE, December 2002.

[75] R. ”Venugopalan and E”. Sirer. ”the design and implementation of a next generation name service forthe internet”. In”Proc. of the ACM SIGCOMM04”, 2004.

[76] J. Wei, C.-Z. Xu, and X. Zhou. A robust packet scheduling algorithm for proportional delay differen-tiation services. InProc. of IEEE Globecom, 2004.

[77] Y. Zhang, W. Lee, and Y. Huang. Intrusion detection techniques for mo-bile wireless networks.Wire-less Network, 9:545–556, 2003.

[78] X. Zhou, Y. Cai, G. K. Godavari, and C. E. Chow. An adaptive process allocation strategy for propor-tional responsiveness differentiation on Web servers. InProc. IEEE 2nd Int’l Conf. on Web Services(ICWS), July 2004.

[79] X. Zhou, J. Wei, and C.-Z. Xu. Modeling and analysis of 2D service differentiation on e-Commerceservers. InProc. of IEEE 24th Int’l Conf. on Distributed Computing Systems (ICDCS), pages 740–747,March 2004.

[80] X. Zhou, J. Wei, and C.-Z. Xu. Processing rate allocation for proportional slowdown differentiation onInternet servers. InProc. IEEE 18th Int’l Parallel and Distributed Processing Symp. (IPDPS), pages88–97, April 2004.

[81] X. Zhou and C.-Z. Xu. Harmonic proportional bandwidth allocation and scheduling for service dif-ferentiation on streaming servers.IEEE Trans. on Parallel and Distributed Systems, 15(9):835–848,2004.

[82] H. Zhu, H. Tang, and T. Yang. Demand-driven service differentiation for cluster-based network servers.In Proc. IEEE INFOCOM, pages 679–688, 2001.

D–5