virtual machine monitor-based lightweight intrusion detection

Download Virtual machine monitor-based lightweight intrusion detection

Post on 24-Dec-2016




7 download

Embed Size (px)


  • Virtual Machine Monitor-Based Lightweight IntrusionDetection

    Fatemeh AzmandianNortheastern University

    Micha MofeNortheastern University

    Malak AlshawabkehNortheastern University

    malshawa@ece.neu.eduJennifer Dy


    Javed AslamNortheastern

    David KaeliNortheastern

    ABSTRACTAs virtualization technology gains in popularity, so do at-tempts to compromise the security and integrity of virtual-ized computing resources. Anti-virus software and rewallprograms are typically deployed in the guest virtual machineto detect malicious software. These security measures are ef-fective in detecting known malware, but do little to protectagainst new variants of intrusions. Intrusion detection sys-tems (IDSs) can be used to detect malicious behavior. Mostintrusion detection systems for virtual execution environ-ments track behavior at the application or operating systemlevel, using virtualization as a means to isolate themselvesfrom a compromised virtual machine.

    In this paper, we present a novel approach to intrusion de-tection of virtual server environments which utilizes onlyinformation available from the perspective of the virtual ma-chine monitor (VMM). Such an IDS can harness the abilityof the VMM to isolate and manage several virtual machines(VMs), making it possible to provide monitoring of intru-sions at a common level across VMs. It also oers uniqueadvantages over recent advances in intrusion detection forvirtual machine environments. By working purely at theVMM-level, the IDS does not depend on structures or ab-stractions visible to the OS (e.g., le systems), which aresusceptible to attacks and can be modied by malware tocontain corrupted information (e.g., the Windows registry).In addition, being situated within the VMM provides ease ofdeployment as the IDS is not tied to a specic OS and can bedeployed transparently below dierent operating systems.

    Due to the semantic gap between the information availableto the VMM and the actual application behavior, we em-ploy the power of data mining techniques to extract usefulnuggets of knowledge from the raw, low-level architecturaldata. We show in this paper that by working entirely atthe VMM-level, we are able to capture enough information

    to characterize normal executions and identify the presenceof abnormal malicious behavior. Our experiments on over300 real-world malware and exploits illustrate that there issucient information embedded within the VMM-level datato allow accurate detection of malicious attacks, with an ac-ceptable false alarm rate.

    Categories and Subject DescriptorsD.4.6 [Security and Protection]

    General TermsVirtualization, Security, Data Mining, Intrusion Detection

    KeywordsVirtual Machine, Virtual Machine Monitor, Intrusion De-tection System, Data Mining

    1. INTRODUCTIONVirtual execution environments provide many advantagesover traditional computing environments, such as server con-solidation, increased reliability and availability, and enhancedsecurity through isolation of virtual machines (VMs) [29].Anti-virus programs and rewalls can guard a system againstknown exploits, but these mechanisms provide little protec-tion against new classes of attacks and insider threats. Vir-tualization can provide us the ability to isolate and inspectVM-based execution. Virtual machines themselves are notcompletely immune to viruses and malicious attacks. Toprotect the guest OS running inside a virtual machine andguard against the existence of malicious software, or mal-ware, there needs to be an intrusion detection system (IDS)in place.

    Traditionally, an IDS can be categorized as one of two types:a host-based intrusion detection system (HIDS) or a network-based intrusion detection system (NIDS). An HIDS resideson the system that is being monitored and thus has theadvantage of a rich view of the internal workings of the sys-tem. The disadvantage with this approach is that a malwarecan determine the existence of the HIDS and subsequentlycompromise it or attempt to evade detection. An NIDS, onthe other hand, performs intrusion detection from outsidethe target system, using information from the network ow.This makes it more resistant to attacks and evasion, but atthe cost of poor visibility of the system.


  • In a virtualized execution environment, the virtual machinemonitor (VMM) is a software layer that allows the multi-plexing of the underlying physical machine between dierentvirtual machines, each running its own operating system. Inthis paper we propose a VMM-based IDS, a variant of host-based intrusion detection systems wherein the IDS resideson the physical host machine, yet remains outside of the vir-tual machine being monitored. As such, a VMM IDS is ableto enjoy the advantages oered by both HIDSs and NIDSs:a rich view of the target system (the VM) combined witha greater resistance to attacks and evasion by the malware.The latter is one of the benets of isolation provided by theVMM.

    The VMM IDS only uses information available at the VMM-level to detect intrusions. There exists a large semantic gapbetween this low-level architectural data and the actual pro-gram behavior. Consequently, we utilize sophisticated datamining algorithms to extract meaningful and useful informa-tion to distinguish normal (non-malicious) from abnormal(malicious) behavior.

    There are two main approaches to intrusion detection: mis-use detection and anomaly detection. In misuse detection,the behavior of the system is compared to patterns of knownmalicious behavior, or attack signatures. A weakness of thisapproach is its inability to detect new and previously unseenattacks, known as zero-day attacks. In anomaly detection,a prole of normal behavior is built and any deviations fromthis normal prole is agged as a potential attack. Whileanomaly detection has the ability to detect zero-day attacks,it is also prone to false alarms, i.e., previously unseen nor-mal behavior may incorrectly be identied as an attack. Asvirtualization and the information available to the VMM fa-cilitate the proling of normal behavior, in our VMM IDS wetake the second approach to intrusion detection. We use sys-tem events visible to the VMM and incorporate data miningalgorithms to help characterize normal execution patternsand distinguish deviating anomalous behavior, while tryingto balance the trade-o between true detections and falsealarms.

    A key advantage that a pure VMM-level IDS provides isease of deployment. Only the VMM needs to be modied toextract low-level architectural events during runtime. Thisties the IDS to a particular VMM and instruction set archi-tecture (ISA). No modication to the operating system isrequired. Hence, it can be deployed in any virtualized com-puting environment with minimal eort. In our work, wefocus on virtualized server applications [38]. These applica-tions are combined with a customized commodity operatingsystem to run optimally in a virtual environment. As thereare no login operations and typical execution consists of onemain process running alongside background processes, weexpect the normal behavior of these workloads to be fairlystable in time and space. Our IDS uses data mining algo-rithms to characterize the normal behavior of the workload.A malicious attack would introduce deviations from the nor-mal behavior, which should be identied by the data min-ing algorithms and agged by the IDS. Along these lines, aVMM IDS has the advantage of being able to detect zero-dayattacks, in addition to previously known malware.

    As part of our contributions, we have implemented a proto-type of a pure VMM-level intrusion detection system usingVirtualBox [15], an open-source full-virtualization VMM. Tothe best of our knowledge, this is the rst work to utilize onlythe low-level architectural information visible to the VMMfor detecting the existence of malware. Our IDS consists oftwo key components:

    A front-end, whose duties include: Event Extraction - Capturing the low-level archi-

    tectural data available to the VMM such as diskand network IO accesses, page faults, translationlook-aside buer (TLB) ushes, and control reg-ister updates.

    Feature Construction - Using statistical techniquesto transform the raw data into features, which areused by the data mining algorithms.

    A back-end, whose duties include: Feature Reduction - Reducing the large space of

    possible features, which improves both the timecomplexity and the accuracy of data mining algo-rithms.

    Normal Model Creation - Proling the normal ex-ecution of the workloads and build a model ofnormal behavior.

    Anomaly Detection - Identifying anomalous be-havior as deviations from the model of normalbehavior.

    Raising an Alarm - Flagging behavior that devi-ates from the norm as a possible threat.




    Figure 1: High-level design of our VMM IDS

    A high-level overview of our VMM IDS design is presentedin Figure 1. There are two main phases of the VMM IDS op-eration: a calibration phase and a testing phase. In the cali-bration phase, the front-end extracts the VMM-level eventsand constructs all the possible features (using methods de-scribed in section 3.2). These features are passed on to theback-end where feature reduction takes place. The reducedset of features are provided to the data mining algorithms tobuild a model of normal execution behavior. Next, anomaly


  • detection is performed on a set of both normal and abnormaldata points, assigning a score to each based on how muchthey deviate from the normal model. The scores are thenpassed through a lter to remove noise and determine whento raise an alarm.1 During the calibration phase, we evalu-ate th


View more >