mutantx-s: scalable malware clustering based … association 2013 usenix annual technical conference...

USENIX Association 2013 USENIX Annual Technical Conference (USENIX ATC ’13) 187

MutantX-S: Scalable Malware Clustering Based on Static Features

Xin Hu1 Sandeep Bhatkar2 Kent Griffin2 Kang G. Shin3

1 IBM T.J. Waston Research Center 2 Symantec Research Labs 3 University of Michigan, Ann Arbor

AbstractThe current lack of automatic and speedy labeling of alarge number (thousands) of malware samples seen ev-eryday delays the generation of malware signatures andhas become a major challenge for anti-virus industries.In this paper, we design, implement and evaluate a novel,scalable framework, called MutantX-S, that can effi-ciently cluster a large number of samples into familiesbased on programs’ static features, i.e., code instructionsequences. MutantX-S is a unique combination of sev-eral novel techniques to address the practical challengesof malware clustering. Specifically, it exploits the in-struction format of x86 architecture and represents a pro-gram as a sequence of opcodes, facilitating the extrac-tion of N-gram features. It also exploits the hashingtrick recently developed in the machine learning com-munity to reduce the dimensionality of extracted featurevectors, thus significantly lowering the memory require-ment and computation costs. Our comprehensive eval-uation on a MutantX-S prototype using a database ofmore than 130,000 malware samples has shown its abil-ity to correctly cluster over 80% of samples within 2hours, achieving a good balance between accuracy andscalability. Applying MutantX-S on malware sam-ples created at different times, we also demonstrate thatMutantX-S achieves high accuracy in predicting labelsfor previously unknown malware.

1 IntroductionAccording to the Symantec’s latest Internet Threat Re-port, 403 million new variants of malware were createdin 2011, a 41% increase from 2010. This exponentialgrowth of malware samples has created a major challengefor anti-virus (AV) companies: how to efficiently processthis huge influx of new samples and accurately labelsthem? It is practically impossible to manually analyzeseveral thousands of suspicious samples received everyday. As a result, a large fraction of samples are left un-labeled, which delays the signature generation. One pos-sible solution is to automatically cluster malware sam-ples and assign them labels according to their similarities.The intuition is that malware programs bearing signifi-cant similarities are likely to have been derived from thesame code base, and hence from the same malware fam-ily. One can thus group similar malware and label them

with high accuracy by analyzing only a few representa-tive samples. Moreover, the label of a new sample can beautomatically derived and previous mitigation techniquescan be re-used if it is determined to belong to an exist-ing family. Therefore, accurate clustering plays a crucialrole in helping AV companies categorize large amountof incoming samples by avoiding duplicate work and en-abling malware analysts to prioritize limited resources onnovel and representative samples [17, 12, 7]. In this pa-per, we design, implement and evaluate MutantX-S, anovel and scalable system, that can efficiently cluster alarge number of malware samples into families based ontheir static features, i.e., code instruction sequences.

Many existing malware clustering/classification sys-tems are based on dynamic behavioral features such asruntime API or system call traces [6, 7, 24]. The majorbenefit of using dynamic behavioral features is that theyare less susceptible to mutation schemes frequently em-ployed by malware writers to evade binary analysis, e.g.,packing or obfuscation. However dynamic-feature-basedapproaches also suffer from several limitations. First,they may have only limited coverage of an application’sbehavior, failing to reveal the entire capabilities of a givenmalware program. This is because a dynamic analysiscan only capture API or system call traces correspondingto the code path that was taken during a particular exe-cution. Different code paths may be taken in differentruns, depending on the program’s internal logics and/orexternal environments. Also, malware often include trig-gers in their programs and exhibit an interesting behav-ior only when certain conditions are met. For examplebot programs wait for commands from botmasters andsome malware are designed to launch attacks on a certaintime. Although there exists work that forces a programto run all code paths [21], they are too expensive to an-alyze large amount of malware. Second, dynamic analy-sis is inherently resource-intensive and doesn’t scale well.With limited resource and the sheer number of malware,a dynamic-analysis system can execute and monitor eachsample only for a short period of time, e.g., a couple ofminutes. Unfortunately, this time is often too short fortypical malware to reveal all their true behavior.

In this paper, we present MutantX-S, a new and prac-tical system that exploits static features of code instruc-tion sequences for efficient and automatic malware clus-

188 2013 USENIX Annual Technical Conference (USENIX ATC ’13) USENIX Association

tering and labeling. MutantX-S is motivated by thecommon observation that majority of today’s maliciousprograms are variations of a relative small number ofmalware families and thus share similar instruction se-quences. Analyzing static features of malware offers sev-eral unique benefits. First, it has the potential to cover allpossible code paths, yielding more accurate representa-tions of the entire functionalities of the program. More-over, approaches based on static features are much morescalable than their dynamic counterparts, as they do notrequire resource-intensive and time-consuming monitor-ing of program behavior. This is particularly importantfor AV companies to process a rapidly-increasing numberof new malware samples. Unfortunately, static analysis iswell-known to suffer from run-time packing and obfus-cation techniques. Therefore, the goal of MutantX-S isnot to replace existing dynamic-behavior-based systems,but to complement them to achieve higher clustering ac-curacy and better coverage of malware programs.MutantX-S features a unique combination of tech-

niques to address the deficiencies of static malware anal-ysis. First, it tailors a generic unpacking technique tohandle run-time packers without needs to know its spe-cific packing algorithm. Second, it employs an efficientencoding mechanism that exploits the IA32 instructionformat to encode a program into opcode sequences thatare resilient to low level mutations. In addition, it appliesa hashing-trick and a close-to-linear clustering algorithmto allow MutantX-S to efficiently handle large num-ber of malware with very high dimensional features. Wehave successfully implemented a fully-automated proto-type of MutantX-S and evaluated its performance usingover 130,000 distinct malicious programs. Our evalua-tion demonstrates MutantX-S’ efficiency and efficacyof creating clusters corresponding to malware familiesand accurately predicting labels for new malware.

The rest of the paper is organized as follows. Section 2surveys related work of malware analysis. Section 3describes the architecture of MutantX-S followed byelaboration of all subcomponents including unpacking(Section 4), feature extraction (Section 5) and clustering(Section 6). The performance evaluation is presented inSection 7. Section 8 discusses the limitation and potentialimprovement, and Section 9 concludes the paper.

2 Related WorkMalware pose one of the severest threats to computer sys-tems and the Internet. Various schemes have been pro-posed to automatically cluster/classify malware based oneither dynamic behavior or features.

Dynamic-analysis approaches have the major benefitof handling obfuscated malware samples based on theirruntime system or API calls. Lee and Mody [18] useda sequence of runtime events (e.g., registry and file sys-

tem modifications) to cluster similar malware programs.Rieck et al. [23] applied SVM (Support Vector Machine)to learn the frequency of run-time behavior, and classi-fied unknown samples to their closest kin. Later, Bai-ley et al. [6] applied a hierarchical clustering algorithmto group similarly-behaving malware samples. Unfor-tunately, the complexity of this clustering algorithm isO(n2), limiting its applicability only to a small numberof samples. To address this problem, Bayer et al. [7] andRieck et al. [24] developed different methods to scale theclustering. Bayer et al. [7] applies locality-sensitive hash-ing (LSH) to efficiently compute an approximate hierar-chical clustering with a significantly smaller number ofdistance computations. By contrast, Rieck et al. [24] ap-plied a prototype-based clustering algorithm that reducesthe runtime complexity by performing clustering only onrepresentative samples. Comparing to LSH clustering, aprototype-based algorithm facilitates the analysis of be-havior groups because each prototype represents a partic-ular malware group [24]. In MutantX-S, we adopt thesame prototype-based algorithm as in [24] because of itsefficiency and explicit expression of malware features.

Static analysis, on the other hand, uses features ex-tracted directly from malware binaries. Christodorescuet al. [8] discovered malicious patterns from disassem-bled malware that are resilient to obfuscation. Wich-erski [30] utilizes static features from PE headers, e.g.,entry point, import table, etc., to group malware pro-grams. Karim et al. [13] demonstrates the effectivenessof N-gram and N-perm on assembly instructions by usingthem to study the malware evolution. Similar featureshave also been used in [15] to validate various learningmethods. MutantX-S falls into the static-analysis cat-egory since it relies on features extracted from the mal-ware instructions. MutantX-S differs from previousapproaches in its unique combination of novel techniquesto improve its scalability in handling very large malwaredatasets. Another independently developed system sim-ilar to MutantX-S is BitShred [12] which also focuseson malware comparison and triage on a large scale. How-ever, BitShred compares malware using their byte se-quences which is susceptible to binary level obfuscation.

3 ArchitectureFigure 1 shows an overview of MutantX-S. At a highlevel, MutantX-S takes a set of malicious or suspicioussamples as input and extracts their features using staticanalysis to avoid the computational overhead and maxi-mize code coverage. Specifically, MutantX-S first usesexisting tools (e.g., PeID1 [3]) to identify malware filesthat are likely processed by packing tools such as UPX

1a popular packer detection tool that currently detect more than 470different packer signatures in executables


[28], ASPack [5]. These files will be unpacked with ageneric unpacking technique tailored for MutantX-S.Together with samples that are in their original binary(not packed), they are disassembled to code instructions.These pre-processing steps ensure that features inherentto malware families can be successfully extracted withoutinfluence of encryption or compression. Then, all mal-ware samples are processed with three steps to extractrepresentative features: (1) Instruction Encoding for con-verting each instruction to a sequence of operation codesthat capture the underlying semantics of the programs,(2) N-gram analysis for constructing feature vectors usedto compute program similarities, and (3) Hashing Trickfor compressing the feature vectors, which significantlyimproves the speed of similarity computation with onlya small penalty in accuracy. Finally, a prototype-basedclustering algorithm is applied on compressed featurevectors and partitions samples into clusters, each repre-senting a group of similar malware programs.

Figure 1: A system overview of MutantX-S

4 Generic Unpacking AlgorithmRun-time packing is arguably the most popular tech-niques used by malware writers to circumvent anti-virusdetection. More than 80% of malware programs are es-timated to be packed [10]. A typical packer like UPXworks as follows. UPX first compresses all the code anddata sections of a portable executable (PE) binary2 into asingle section. Then, it creates a new PE binary contain-ing the compressed data followed by the unpacker code.The entry point in the new PE header is altered to pointto the unpacker code such that when the packed programruns the unpacker will first be executed . The unpackerdecompresses the original program codes into memoryand then jump to the first instruction of the restored codes(i.e., the original entry point) to resume execution. Thispacking process enables malware programs to disguisetheir malicious instructions as random-looking data whilekeeping the original functionality intact. Since all staticanalysis tools including MutantX-S rely on featuresextracted from original instructions, it is imperative forthem to handle packing correctly and efficiently.

While there exist unpacking tools such as UPX itself,ArmaGedoon, etc., they are often targeted specifically at

2PE is the executable file format used by Windows OS

one or a few packers. As more packers appear in thewild, the cost of manually reverse-engineering packersand continually updating unpacking tools is expected togrow over time. In addition, unpacking tools often haveto perform expensive processing to ensure that the un-packed program can be successfully executed (e.g., thePE headers and imported tables must be correctly recon-structed), making them too slow for large scale process-ing. MutantX-S, on the other hand, need not guaran-tee the executability of unpacked programs as long asthe original instructions can be inspected and features ex-tracted. MutantX-S thus exploits this advantage andtailors a generic unpacking mechanism to meet the partic-ular need for efficient malware clustering. The basic ideais to use the inherent property of the unpacking proce-dure, i.e., a packed binary has to write the unpacked codeinto some memory space and transfer control to the mod-ified memory locations to continue execution. By contin-uously monitoring memory accesses, we can learn the oc-currence of some form of unpacking, self-modification oron-the-fly code generation if the program executes codeat a memory address after writing into it. These written-then-executed memory pages likely contain the originalinstructions and thus are the targets of MutantX-S.

The unpacking component of MutantX-S exploitsthe physical non-execution (NX) support in modern x86CPUs to track memory page status. It consists of a kerneldriver responsible for tracking system calls and a user-level component that is injected as a remote thread intoa program’s address space. The unpacking componentdoes two things: (1) runs the packed binary and dumpsthe memory image of the running process at an appro-priate time when the binary is likely to finish unpack-ing, and (2) determines the correct original entry point(OEP) for the dumped image. Finding the correct OEPis critical for correct program disassembling and featureextraction. A wrong entry point may cause a disassem-bler to miss all the instructions between the original andthe misidentified entry points (if there is no other refer-ence to this portion of codes). In addition, if the entrypoint is incorrectly set in the middle of an instruction, thedisassembler will fail or generate completely wrong as-sembly codes. The unpacking process is summarized inAlgorithm 1 and elaborated below:1. MutantX-S loads the packed program, suspends itsexecution and injects the user-level hooking DLL into theprocess’ memory space. It marks all the memory pages asexecutable but non-writable, and resumes its execution.2. During the execution, when the unpacker attemptsto write unpacked codes into memory (which has beenmarked as non-writable), a write exception will occur.MutantX-Smarks the page as dirty and changes its per-mission to writable but non-executable.3. When the unpacker jumps to the the newly-generated


code for execution (e.g., after finishing unpacking), thenon-executable permission on these pages triggers an ex-ecution exception. MutantX-S intercepts the exceptionand records the memory address where it occurred. Forsimple packers (e.g. UPX) that first unpack the entireprogram and then jump to the restored codes for execu-tion, the memory address where the exception occurs isthe OEP (original entry point). However, this is not nec-essarily true for more sophisticated packers (e.g., self-modifying code that rewrite to the same memory loca-tion). Hence, MutantX-S removes the write permissionfrom these memory pages again, grants execution privi-lege and continues execution. MutantX-S also moni-tors dynamic allocation of memory pages and removestheir write permission to track unpacking on these pages.4. MutantX-S dumps the process memory image eitherat the end of program execution or after certain time. Therationale is that after the program has been running for asufficient amount of time (e.g., 1 minute), it is fairly safeto assume that the program has finished unpacking andthe original codes are placed in the memory.

Algorithm 1 MutantX-S unpacking algorithm1: Input: A packed binary program B2: Output: A unpacked PE file with original program codes3:4: Load the packed program into memory5: for all p in the program’s memory pages do6: Permission(p)|= W //remove write permission7: end for8:9: while B is running and Truntime < Tthresh do

10: a: The address of the page fault11: t: The page fault type t ∈ {WRITE, EXECUTE}12: p ← Page(a)13: if t = WRITE then14: Permission(p)| = (W |X) // Writable but non-

executable15: last written(p)← current time16: end if17: if t = EXECUTE then18: Permission(p)| = (W |X) //non-writable but exe-

cutable19: last exec(p)← current time20: addr exec(k)← a21: end if22: end while23:24: Dump process memory25: reconstruct B′ by setting OEP to be addr exec(k) where:26: k = argmink(last exec(k)> max(last written(i))27: return B′

With the dumped memory image, MutantX-S cre-ates a valid PE file from which a standard disassemblercan disassemble instructions. As mentioned earlier, the

major challenge in creating a valid PE file is to identifythe correct entry point in the PE header. For simple pack-ers like UPX, the entry point is simply the start addressof the dirty memory page where the first execution ex-ception occurs. Unfortunately, as adversaries become in-creasingly sophisticated, various evasion schemes havebeen developed to obfuscate OEP. A typical method is tofake end-of-unpacking by writing rouge instructions intoa reserved memory page, transfer control to it, and jumpback to the unpacker code. More advanced packers useincremental unpacking that decrypts only part of the pay-load and executes them before decrypting more instruc-tions. In such cases, detecting the first execution excep-tion is not enough because only rouge instructions or partof the original program is visible. To address these prob-lems, we developed a new heuristic called LMFE (LastModification First Execution).

The idea is to keep track of time when the last writeexception and a subsequent execution exception occur oneach memory page, so MutantX-S can identify the un-packer’s attempts to write to the same memory page mul-tiple times, in which case, the previous modification andexecution on the page are likely to be spurious. Morespecifically, for each memory page, MutantX-S keepsa record of: 1) last modification time (i.e., a write excep-tion occurred), denoted as last written; 2) last executionexception time, denoted as last exec; and 3) the addressaddr exec where the exception had occurred. At anypoint of execution, there are 3 types of memory pages:Type I: memory pages that have valid last written andlast exec, i.e., pages that have been both modified andexecuted.Type II: memory pages that have valid last written butnot last exec, i.e., pages that have been modified but notexecuted. They could either be page containing tempo-rary data or code pages that have not yet been executed.Type III: memory pages that have neither validlast written nor valid last exec. These could be initial-ized data-section pages or unpacker-code pages.

Essentially, type-I memory pages are those that holdthe unpacked instructions and thus contain the OEP.When dumping the process memory, MutantX-S usesthe following algorithm to pinpoint the correct OEP.Let P(i), i = 1..n represent all type-I memory pagesand last written(i), last exec(i) and addr exec(i) re-spectively represent the time of the last write exception,last execution exception and address where the exceptionoccurred for page P(i). Then, the OEP is addr exec(k) inthe memory page P(k) where

k = argmink

(last exec(k)> max(last written(i)) (1)

where i = 1..n. In other words, P(k) is the first memorypage that is executed after all type-I memory pages havebeen written. Below we show that the heuristic is able


to find the correct OEP (i.e. addr exec(k)) even whenthe packers try to fool the MutantX-S using spuriouswrite-and-execute sequences or multi-layer packing.

Proposition. For k satisfying Eq. (1), addr exec(k) is thecorrect OEP of the original program no matter whetherthe program is packed with simple packers or more ad-vanced packers that fake the end of unpacking.Proof. First, for simple packers like UPX that restorethe entire program into memory before executing it, letP( j), j = 1..m denote memory pages where the unpackerwrites the original program codes. Without loss of gener-ality, we can assume that the contents are written sequen-tially from P(1) to P(k), meaning that last written(1) <last written(2) < .. . < last written(m). When thepacker finishes unpacking and starts executing the re-stored program by jumping to the OEP, an execu-tion exception will first occur in the page P(k) thatcontains the OEP, i.e., last exec(k) > last written(m)and last exec(k) < last exec( j)∀ j �= k. As a result,addr exec(k) is the correct OEP.

Second, assume a more advanced packer with thefollowing spurious unpacking sequence: it writes arbi-trary instructions into some memory page, executes themand, at the end of execution, returns to the unpackercode. Such a routine may be called multiple times dur-ing the entire unpacking process. As a result, an un-packing tool cannot assumes that unpacking ends at thefirst (or first few) execution exception. MutantX-S isresilient to this type of evasion by enforcing the invari-ant that the execution exception on the OEP must suc-ceed all the write exceptions. For example, when thespurious unpacking routing touches memory page P(s),MutantX-S records last exec(s) and marks P(s) as ex-ecutable but un-writable. Then, the unpacker resumesthe normal unpacking and writes more decrypted instruc-tions to memory page P(t) (t could be any page includings). This creates a new write exception on P(t) at times-tamp last written(t). Note that because last exec(s) <last written(t), the heuristic determines s to not containthe OEP. In contrast, after the packer finishes unpackingand transfers control to the real OEP, the execution excep-tion satisfies Eq. (1). By keeping addr exec up-to-dateand pointing to a valid instruction, MutantX-S is ableto keep track of the real OEP accurately. Same argumentshold for multi-layer packing because the write exceptionsof code pages will always precede the executable excep-tions caused by jumping to the OEP.

5 Feature ExtractionWith the correct OEP identified, MutantX-S recon-structs a new PE file with dumped memory images. Thecorrect OEP ensures a proper starting point to disassem-

Figure 2: x86 instruction format

ble instructions and MutantX-S uses the IDA Pro3 todisassemble a malware program into a sequence of ma-chine instructions that are then used for feature extrac-tion. The key step in MutantX-S is the similaritycomparison between malware samples based on the dis-assembled instruction sequences, e.g., move eax, ebx;cmp eax, 1h. The main challenge in similarity compar-ison lies in handling the variations of machine instruc-tions. Malware often undergo changes for many reasons,such as mutation, polymorphism, and obfuscation wheresemantically-equivalent instruction sequences are used toreplace each other. Hence, ensuring exactness in compar-ing instructions will not tolerate any variation in the syn-tax. At the other extreme, correctness is compromised ifall forms of variation are tolerated. MutantX-S strikesa balance between these two extremes by exploiting thex86 instruction format (Fig. 2) and uses the opcode as asuccinct representation of the instruction semantics.

Using opcodes—instead of other features used be-fore, such as control flow graphs, binary sequences ormnemonic sequences—offers several benefits. First, op-codes generalize well to represent variants of a malwarefamily. Malware in the same family are often derivedfrom the same code base and thus share similarities intheir instructions. However, due to relinking, rebind-ing and rebasing, the operands (e.g., registers, mem-ory addresses) of instructions tend to change across thevariants. Using opcodes and ignoring the operands (i)make MutantX-S more resilient to low-level mutationswhile providing a meaningful characterization of seman-tics and (ii) reflect the functionality of the malware pro-grams. Second, previous approaches often use mnemonicsequences (e.g., mov, push) to represent the instructionfunctionalities and address the variability of operands.From the evaluation of MutantX-S, we discover that theopcode sequence offers a better representation of instruc-tion semantics. Mnemonics sometimes overly general-ize the underlying CPU operations, causing instructionswith distinct semantics to appear similar. To illustratethis, consider all the instructions in Table 1. Althoughall of them have the same mnemonic (i.e., mov), the un-derlying functionalities are drastically different. For in-stance, moving a value to a control or debug register of-

3the de-facto disassembler for the analysis of hostile code


ten indicates a critical OS operation, such as interruptcontrol, switch addressing mode or enable/disable debug-ging, etc., which should not be treated the same as mov-ing a value between registers. Ideally, moving data frommemory to a register (memory load operation) shouldalso be considered as a distinct operation from that ofmoving from a register to memory (memory store oper-ation). Unfortunately, using mnemonics would cause allthese distinct instructions to be represented with a singlefeature (i.e., mov), which may lead to an accidental sim-ilarity between code sequences. As illustrated in Fig. 3,however, features based on opcode provide higher distin-guishability between semantically different instructions,thus yielding better clustering accuracy.

Op Instruction Description89 MOV r/m32, r32 Move from reg to mem/reg8B MOV r32, r/m32 Move from mem/reg to regB8 MOV r32, imm32 Move immediate val to reg0F 20 MOV r32, CR0-CR4 Move from control reg to reg0F 22 MOV CR0-CR4,r32 Move from reg to control reg0F 21 MOV r32, DR0-DR7 Move from debug reg to reg0F 23 MOV DR0-DR7,r32 Move from reg to debug reg

Table 1: Op (Opcodes) provides fine-grain representa-tions of instruction semantics

Figure 3: Two code pieces with different semantics sharesame mnemonic representation (i.e., move, cmp, jnz).However, they can be differentiated by their opcode rep-resentation: ”0F 21 3D 75” vs ”8B 3D 75”

.With this encoding scheme, a program is represented

as a sequence of opcodes (Fig. 4). MutantX-S usesthe standard N-gram analysis to characterize the contentof a malware program, i.e., moving a fixed-length win-dow over the sequence and considering a subsequenceof length N at each position. The resulting N-gram ofopcodes reflects short instruction patterns and implic-itly captures the underlying program semantics. Then,MutantX-S constructs a feature vector V in an |S|-dimensional space (|S| = |O|N where O is the set of allpossible opcodes). Each dimension of V is the numberof occurrences of a particular opcode N-gram. Then,MutantX-S can geometrically calculate the similaritybetween two malware programs (m, v) as the Euclideandistance between their feature vectors in the vector space:

d(m,n) = ‖Vm −Vn‖ =

√∑|S|

i=1(Vm(i)−Vn(i))2. Com-pared to the other similarity metrics (e.g., locality-basedhashing), geometric calculation of similarity in the vector

space provides explicit feature representation [24] wherethe importance or contribution of each N-gram in clus-tering malware can be traced back to its original codepatterns. For N-grams that may correspond to inherentcharacteristics of a malware family (e.g., those that ap-pear frequently within a family but rarely in others), theiroriginal code segments can be traced back and used assignatures to detect variants.

Figure 4: Encoding a function into a feature vector

6 Clustering AlgorithmThe next step in MutantX-S is clustering malware sam-ples into groups that share common traits. Consideringthe enormous amount of malware in the wild, the goal ofMutantX-S is to process hundreds of thousands mal-ware files sufficiently fast. Unfortunately, classic clus-tering algorithms such as hierarchical and partitioning-based clustering, e.g., K-Means or K-Medoids—althoughthey have been successfully applied to cluster malwarebehaviors and programs [6, 13]—incur a time complex-ity at least quadratic in the number of samples, whichin practice, does not scale to the MutantX-S’ target.MutantX-S exploits two approaches to address the scal-ability issue: (1) a hash kernel that compresses the highdimensional feature vector into a low dimensional space,and (2) a prototype-based clustering algorithm that hasclose-to-linear runtime complexity.

6.1 Hashing Kernel

Kernel methods [25] are powerful tools used in machinelearning to allow operation in the high-dimensional fea-ture space without having to compute the coordinates ofthe data in that space. This is particularly useful whenthe input data has a non-linear decision boundary butcan be linearly separated in a high dimensional featurespace. In MutantX-S, however, we have encounteredthe opposite problem: the original space is very high-dimensional4. The number of dimensions D determinesthe complexity when computing the vector distance andD increases exponentially with N in N-gram (i.e. D =|O|N where |O| is the number of different opcodes and inpractice |O|> 200). Therefore even a small N like 3 willresult in a (very sparse) feature vector with more than 8million dimensions, which is computationally prohibitivewhen calculating similarities for large amount of malware

4thus, the input data are likely already linearly separable


samples. Unfortunately, N has to be at least 3 or 4 to bedescriptive enough for capturing the program semantics.MutantX-S addresses the problem by exploiting the

hashing-trick recently developed in the machine learningcommunity[26], which hashes the high dimensional inputvector x ∈ Rn into a lower dimensional feature space Rm

with the mapping function φ : X → Rm. Since m � n,the hashing trick reduces a feature vector to a more com-pact representation, allowing the clustering algorithm tohandle a large volume of data, and save both computationand memory requirements. Previous research has shownthat the hash kernel approximately preserves the vectordistance and the penalty incurred from using a hash forreducing dimensionality only grows logarithmically withthe number of samples and groups [26, 14].

Specifically, MutantX-S applies a uniform hashfunction H : {N-gram} → [1..m] that hashes N-gram di-rectly into a position in the feature vector of length m. Incase of a collision where two or more N-grams map tothe same position, the sum of their counts is used as thevalue in the new vector. More formally, for malware Mand M′, let v and v′ represent their original feature vectorextracted from the encoded opcode sequences and ξ de-note the mapping from the N-gram (o1,o2, . . . ,oN)∈ S tothe index in v. We define the hash feature map φ as

φi(v) = ∑l:H(o)=i,l∈S

v(ξ (o))

and the distance between M and M′ as

dφ (M,M′) = ‖v− v′‖φ = ‖φ(v),φ(v′)‖.

The choice of m, the length of the low dimensional vec-tor, is a trade-off between clustering accuracy and stor-age overhead plus computation complexity. Choosing asmaller m means shorter vector length, thus, faster dis-tance computation and smaller memory footprint to storemalware features. However, decreasing m increases thecollision possibility, leading to over-compression of fea-tures and negative impact of the clustering accuracy.

6.2 Prototype-Based ClusteringClassic clustering algorithms typically incur a complex-ity that is super-linear in the size of the input data.For example, the running time for two most widely-used clustering algorithms k-means and hierarchical clus-tering are O(nkd) [4] and O(n2logn) [19], resulting inthe computation time that is prohibitively large for thenumber of malware we have to deal with. Instead,MutantX-S adopts the prototype-based close-to-linearclustering algorithm[24].

Despite their simplicity, Prototype-based algorithmshave been empirically shown to be very effective andoften one of the best performers in real data [11].

Prototype-based clustering extracts a set of prototypeseach of which serves as the representative for a smallgroup. The remaining data points are associated withtheir closest prototype in the feature space. The key ideaof Prototype-based algorithm is to perform computation(e.g. clustering) only on the prototypes which are a smallsubset of original data points, thus reducing the computa-tion time significantly. The algorithm comprises 2 steps.

Prototype extraction: The quality of final clusters de-pends on the choice of the prototypes. Well-positionedprototypes can accurately capture the distribution of in-put data and allows creating accurate class boundaries inthe feature space. Unfortunately, determining the optimalnumber and positions of prototypes is NP-hard and anapproximate algorithm by Gonzalez [9] was commonlyused to iteratively select prototypes. During each itera-tion, the data point with the largest distance to existentprototypes is selected as the next prototype (the first pro-totype is selected randomly). The process repeats untilthe distance from all the data points to their nearest pro-totype is smaller than a predefined threshold Pmax, i.e., allthe data points are located within a certain radius fromtheir closest prototypes. The run-time complexity of thisalgorithm is O(kn) where k is the number of prototypesselected. Since k only depends on the distribution of thedata (in this case, k is proportional to the amount of mal-ware families), with a reasonable choice of Pmax the algo-rithm is linear in the number of input data n.

Clustering with prototypes. Instead of working on thehuge number of original data, the algorithm performsagglomerative hierarchical clustering only on the proto-types. Specifically, the algorithm starts with individualprototypes as singleton clusters, successively merges twoclosest clusters, and terminates when the distance be-tween the closest clusters is larger than a predefined dis-tance threshold Mind . Then, prototypes within the samecluster are assigned the same cluster label and subse-quently propagate the label to their associated data points.Because each prototype is a good representation of its as-sociated data points (all within a radius of Pmax), the al-gorithm avoids expensive distance computation betweenthe original data points without too much loss in theoverall accuracy. The respective run-time complexitiesof clustering and propagation steps are O(k2log k) andO(n). Compared to the O(n2log n) complexity of ap-plying an hierarchical clustering algorithm on the origi-nal data points, this algorithm achieves a speed-up with afactor of at least (n/k)2.

7 Experimental EvaluationWe now evaluate MutantX-S’ efficiency and accuracyusing two data sets: (1) a reference data set containing4821 malware files whose labels are generated by se-curity experts from a large anti-virus company and thus


more reliable; and (2) a large malware data set collectedfrom an online malware archive [29] which comprises132,234 malware samples with unreliable labels derivedfrom AV scanners. The reference data set includes mal-ware samples from 20 different families and their distri-butions are given in Table 2. Considering its reliable la-beling, the reference set is used to evaluate and fine-tunethe empirical parameters for the MutantX-S’ clusteringengine while the larger set is used to assess its scalability.

Family # Family # Family #Pilleuz 500 Bredolab 301 Tidserv 59Koobface 496 Vundo 249 Waledac 34Silly 489 Almanahe 241 Ackantta 32Fakeav 489 Sasfis 199 Mebroot 26Zbot 459 Graybird 166 Hotbar 21Banker 449 Gammima 126 Qakbot 17Virut 361 Mabezat 107

Table 2: Malware families of the reference data set

7.1 Effectiveness of Unpacking EngineTo evaluate the effectiveness of MutantX-S’s unpack-ing component, we select and then pack a malware pro-gram with 8 popular packers. We then unpack them withMutantX-S and compare unpacked files with the origi-nal version. Ideally, the unpacked binary should be byte-to-byte identical to the original file. However, this is nei-ther possible (MutantX-S does not reconstruct the im-port table, and the unpacker code is also dumped from thememory), nor necessary for the purpose of malware clus-tering. As a result, we compared the unpacked files withthe original one using two metrics: (i) the difference intheir instruction count (IC), and (ii) the distance betweentheir N-gram feature vectors (NG), because they are di-rectly related to clustering accuracy. Table 3 summa-rized the results. For most packers, the MutantX-S suc-cessfully recovered their original binaries with only a 1–6% increase of ICs which is often due to the inclusionof unpacker routines in the dumped memory. Besides,the feature vectors of unpacked binaries are very sim-ilar to those of the original binary with most normal-ized distance measurements below 0.1, where 0 meansidentical and 1 means completely different. However,MutantX-S also failed on certain packers. In partic-ular, the memory dump of Armadillo-packed malwaresample still contains a packed version of the binary. Afurther investigation showed that Armadillo works by un-packing an intermediate executable on disk and creatinganother process to run this executable [20]. Therefore,memory dump of an Armadillo-packed file does not con-tain original instructions. After running MutantX-S onthe larger data set, we have also observed other causesof unsuccessful unpacking, such as malware samples re-

fusing to run in a virtual machine or the time requiredfor unpacking is longer than the threshold. Nevertheless.MutantX-S’ generic unpacking technique is still effec-tive against popular packers without requiring any spe-cialized knolwedge of packing algorithms.

Packer Diff in NG Packer Diff in NGIC (%) Dist IC (%) Dist

PEcompact 0.88% 0.068 ASprotect 6.70% 0.133EXECryptor 3.20% 0.176 UPX 0.88% 0.068EXEStealth 0.88% 0.071 NSPack 0.87% 0.069VMprotect 2.50% 0.10 Armadillo - -

Table 3: Unpacking effectiveness (IC: Instruction Count;NG Dist: N-gram Difference)

7.2 Malware Clustering AccuracyWe first evaluate and calibrate MutantX-S against thereference data set. All of our evaluations were doneon a Ubuntu 10.4 machine with Core i7 3.0G CPU and12GB memory. We use precision and recall as the mainmetrics to assess the accuracy of MutantX-S. Sup-pose that with respect to the original labels (i.e., fam-ily names in Table 2), n input malware samples canbe grouped into a set of clusters O = {O1,O2, . . . ,Oo}.Assume MutantX-S outputs a set of clusters C ={C1,C2, . . . ,Cc}. Then, precision P measures how wellindividual clusters agree with the original classes (i.e., theexactness of clusters), and recall R measures how muchthe malware classes are scattered across the clusters (i.e.,the completeness of each cluster). Formally, we define

P =1n

c

∑i=1

max(|Ci ∩O1|, |Ci ∩O2|, . . . , |Ci ∩Oo|)

R =1n

o

∑j=1

max(|O j ∩C1|, |O j ∩C2|, . . . , |Oi ∩Cc|)

P will be 1 if all the samples in every cluster Ci arefrom the same family and R will be 1 if all malware sam-ples from the same family fall into a single cluster (butnot necessarily the only family in this cluster). Fig. 5shows the precision and recall of MutantX-S’s cluster-ing with varying thresholds Pmax and Mind (defined inSection 6). The experiment uses 4-gram and 12 hash bits(i.e., the 4-gram is mapped into 212 hash bins).

From the figure, we observe that MutantX-S is ableto cluster the samples with the precision ranging from0.72 to 0.89 (average=0.80). The precision number issmaller than those reported in previous dynamic-behaviorapproaches, e.g., 0.996 in [24] and 0.984 in [7]. We con-jecture that this difference may be due to different mal-ware sets (and possibly incorrect labeling) used in ourexperiments and the reason for the higher accuracy ofdynamic-behavior approaches is also likely due to theirhigh-level generalization of behavior at the cost of longer


0.20.4

0.60.8

0.2 0.4 0.6 0.8 1

70

75

80

85

90

Mind

Clustering Precision

Pmax

Prec

isio

n

0

0.5

1 0.2 0.4 0.6 0.8 1

0.1

0.2

0.3

0.4

0.5

Mind

Clustering Recall

Pmax

Rec

all

0.20.4

0.60.8

0.20.4

0.60.8

1

0

10

20

30

40

Pmax

Running Time of Clutering

Mind

Seco

nds

Figure 5: Precision, recall and running time of MutantX-S

running time and limited coverage as discussed in Sec-tion 1. Therefore, MutantX-S can provide an alter-native way of categorizing malware and is complemen-tary to the behavior-based analysis with better scalabil-ity while maintaining reasonably good accuracy. In-deed, Fig. 5 shows that it takes only less than 30 sec-onds to complete the clustering for the entire referencedataset (we also ran the K-mean and hierarchical clus-tering on the same dataset which respectively took 32.3seconds with precision 0.75 and 51.3 seconds with pre-cision 0.82). In addition, we observe that the recall ofMutantX-S is around 0.3 and 0.4. However, this lowvalue of recall is not surprising, because there often ex-ists significant diversity across malware variants. Forinstance, we observed that one variant in Vundo fam-ily is 10 times larger in terms of file size than the otherVundo variant. This is possibly due to mislabeled sam-ples, unidentified packers or heavily-obfuscated binaries.Because of the highly diverse variants, MutantX-S of-ten breaks one family into several sub-families, resultingin a low recall, e.g., MutantX-S creates more than 50clusters for the reference dataset which contains 20 fam-ilies according to the labels. Albeit less ideal, a break-down into sub-families is acceptable in practice, e.g., pre-dicting labels for unknown samples as we will show later.

Another observation from these results is that Pmax (thethreshold for distances from data points to their near-est prototypes) has a greater influence on the clusteringspeed, since a smaller Pmax forces the algorithm to findmore prototypes to cover all the data points, thus requir-ing more computation. On the other hand, Mind has amajor impact on the clustering accuracy. Increasing Mindreduces the precision, because a smaller inter-cluster dis-tance threshold will stop the prototype-merging processearlier which reduces the probability of combining unre-lated prototypes into a larger cluster. However, the pricefor this is the over-fitting of clustering, i.e., the algorithmtends to create several small clusters. Hence, a trade-offhas to be empirically made, as in our later experiments.

7.3 Validity of the Hashing TrickThe main concern in using the hashing trick is the pos-sible loss of information due to the compression of highdimensional features into a lower dimensional space. Toevaluate the efficacy of hashing trick, we use differentnumber of hash bins to cluster the reference data set.The hash function used in MutantX-S is MurmurHash2.0 [1], a simple hash implementation with uniform valuedistribution, high throughput, and good collision resis-tance. As comparison, we also ran MutantX-S on theoriginal feature vectors without the hashing trick, whichserves as the baseline benchmark and best-possible resultachievable without information loss.

Figure 6 compares the precision, clustering time andpeak memory requirements with different hash sizes (thenumber of hash bins ranging from 28 to 216 and no hash).Different bars represent the results generated by differentparameter combinations. From the left figure, we findthat as the hash size increases, the precision improvesbecause the collision probability reduces. In fact, whenthe hash size is large enough, the probability of colli-sion becomes so negligible that the hashed features vec-tor perform the same as the original ones. For instance,with more than 212 hash bins, the clustering achieves al-most the same precision, 0.864, as the original featuresP = 0.868. However, as the hash size becomes smaller,the impact of collision starts to surface. When the num-ber of hash bins reduces to 28 = 256, the precision dropssignificantly (less than 0.5) for some parameter combi-nations, due to collision of many critical features (e.g.,features indicative of different families are now mappedto the same hash bin) In this regard, a larger hash size ispreferable. On the other hand, the middle and right fig-ures in Figures 6 show that a small hash size is very effec-tive in reducing the algorithm’s running time and memoryfootprints. This is because smaller number of hash binsmeans shorter feature vectors which require less mem-ory for storage and fewer CPU cycles to compute the dis-tance. For instance, as the hash size decreases from 16bits to 8 bits, the required running time drops from al-most 2 minutes to less than 10 seconds and memory re-quirement from 800 Mbytes to less than 100 Mbytes, at


Figure 6: Precision, time, and peak memory with hash bin number ranging from 28 to216 and with no hash trick .

0 .3 0 .4 0 .5 0 .6 0 .70 .7

0 .75

0 .8

0 .85

0 .9

3 gram4 gram5 gram6 gram

Figure 7: Precision of clus-tering with different N.

the cost of precision. In practice, a 12-bit hash functionis found to be a good compromise, reducing the time andmemory requirements by over 80% while still keepinggood accuracy5. Figure 6 also shows that as the numberof malware increases, the hashing trick becomes critical.Without it, the memory requirement could quickly be-come prohibitively high.

7.4 Impact of N-gram on PerformanceIntuitively, as N increases, N-gram becomes more de-scriptive, providing better distinguishability. However,this comes at the cost of exponential increase in the di-mensionality of the resulting feature vectors (mN wherem is total number of different opcodes), as well as therequired storage and computation time. Therefore, previ-ous work that uses N-gram based approaches commonlychose small N (3 or 4). Fortunately, the hashing trick en-ables us to compress the feature vectors and evaluate theperformance of large N. Figure 7 summarizes the result.

From Figure 7, one can observe that use of a larger Nvalue indeed improves the precision, e.g., 4- and 5-gramsachieve better precision than 3-gram since larger gramscan better capture the underlying instruction semantics.However, the figure also shows that 6-gram performs theworst. This is because the number different 6-grams (i.e.,over 6.4 ∗ 1012) is too large for the 12-bit hash function(4096 hash bins), leading to a large number of collisionsbetween irrelevant features. In MutantX-S, we havechosen 4-gram, because the improvement provided by 5-gram is not large enough to warrant the additional storageand computation overheads.

7.5 Scalability of MutantX-S

In this subsection, we evaluate the scalability and accu-racy of MutantX-S on the large malware data set withover 130,000 samples. We ran MutantX-S on the en-tire set with different parameters and plotted the resultsin Fig. 8. The right figure shows the amount of time forclustering the entire set. the value Pmax seems to have a

5Hence, unless specified otherwise, throughout the paper, the exper-iments are performed with a 12-bit hash function

more significant impact on the running time. For exam-ple, when Pmax is set to 0.5, the clustering takes less than1 hour which is almost half of the time when Pmax is setto 0.2. As mentioned before, Pmax determines the numberof prototypes extracted from the input data which deter-mines the total number of distance computations requiredfor clustering. Although a larger Pmax leads to a shorterrunning time, the left plot in Fig. 8 illustrates the cor-relation between a large Pmax and the reduced clusteringprecision, i.e., increasing Pmax from 0.2 to 0.5 reducesthe precision by almost 10%. This can be explained asfollows: a large Pmax allows each prototype to cover alarge portion of the space, thus increasing the possibil-ity of including samples from irrelevant families. Witha reasonable setting (e.g., Mind = 0.5 and Pmax = 0.4),MutantX-S is able to complete the clustering of over130K malware in less than 1.5 hours with the precisionclose to 0.82.6 The peak memory usage is around 3.6GB.These results indicate that MutantX-S is very efficientin handling a large number of samples and thus has thepotential to keep up with the huge influx of malware vari-ants received nowadays.

Figure 8: Precision and running time of MutantX-S’sclustering over 130K samples

7.6 Predicting Labels of Unknown Mal-ware

So far, we have evaluated MutantX-S using the dataset of known malware families. In a realistic scenario,e.g., in AV companies, MutantX-S is more likely tobe used to analyze new incoming malware and predicttheir family labels. In such a scenario, incoming malware

6The recall for the large data set is around 0.25 because of breakingthe samples from large malware families into relatively small groups.


are analyzed and labeled according to their associationwith the closest kin in the previously-analyzed samples.To simulate this situation, we need a chronological or-der of malware samples according to their creation time.We extract the creation time for each malware from theirIMAGE FILE HEADER. IMAGE FILE HEADER is astandard header in the PE file and contains a timestampfield that is set by the compiler at the compilation time.We use this timestamp to bucket malware programs intomonths and select one year worth of malware (more than40,000 unique samples). Fig. 9 shows the distribution ofthe number of new malware samples across all months.Next, we use these malware to simulate the process ofdetermining the labels for new incoming samples.

0 2000 4000 6000123456789

101112

Number of new malware

Mon

th

Figure 9: Number of newsamples in each month usedto evaluate prediction capa-bility

7 8 9 10 11 120

0 .2

0 .4

0 .6

0 .8

1

Test Month

Acc

urac

y

Predict with all previous monthsPredict with previous 6 monthsPredict with first 6 month

Figure 10: Accuracy in ap-plying MutantX-S to pre-dict family labels for un-known malware

Specifically, we separate the malware program intotraining set and testing set based on their creation timein order to simulate the scenario where AV companieshave analyzed the malware from the training set and tryto predict the labels for newly-received malware (testset). The test set consists of malware samples from eachmonth between July and December (these months are“test months”). Then, we choose 3 different training sets.For the first training set, we use samples from all monthsfrom January to one month before the test month. For in-stance, if the test month is September, the training monthsare January through August. For the second training set,we use 6 months prior to the test month, i.e., if Septem-ber is the test month, March through August will be thetraining months. Finally, as a controlled experiment, wekeep the the first 6 months (i.e., January through June)as the training month regardless of test months. Givenany training set, MutantX-S creates a set of clusters Ci(i= 0,1, . . . ,n). Each cluster has a label L(Ci) determinedby the majority family labels of the constituent malwaresamples. Then MutantX-S determines the family labelL(x j) of the new sample x j in the test month based on thelabel of the cluster that is closest to x j, i.e., L(x j) = L(Ci)where d(x j,Ci) =min(d(x j,Ck))∀k = 0,1, . . . ,n. We thencompare this predicted family label with the original la-bel of x j, and plot the percentage of correctly predictedsamples in Fig. 10. The first observation from the figure

is that malware are constantly evolving and the informa-tion obtained from previous clustering can become obso-lete quickly, as shown by the bottom green line where wekept on using the same first 6 months as the training dataand the prediction accuracy degraded rapidly from 0.7 inJuly to below 0.4 in December. In contrast, if we use thefull history as the training data, the accuracy stays con-sistently in the 0.7–0.8 range (the top red line in Fig. 10),thanks to the up-to-date information from the recent mal-ware. However, in reality, due to the resource (e.g., stor-age) constraints, it may not be possible to keep the entirehistory of previous malware samples. It is more efficientto use only the most recent history, e.g., 6 months as inthe middle blue line. From Fig. 10, one can see that theresult is very close to that of using the full history, withonly a small decrease, about 2 to 3%. These results implythat there exists a strong temporal correlation among mal-ware variants which can be exploited by MutantX-S inpredicting the labels for unknown malware samples.

8 Limitations and Improvements

Here we discuss limitations of the current prototypeof MutantX-S that could be exploited by adver-saries to degrade its effectiveness in clustering. As astatic-analysis approach, MutantX-S is vulnerable tobinary/instruction-level obfuscation. First, even with ageneric unpacking algorithm, MutantX-S is less ef-fective against advanced packers that employ sophisti-cated protection mechanisms, e.g., driver-level protec-tion, anti-debug, anti-emulation, etc. Specialized unpack-ing tools [2] have been developed for these packers andthey can be incorporated into MutantX-S to combat so-phisticated packers. Second, MutantX-S extracts fea-tures from disassembled malware code. Unfortunately,producing correct disassembly is often very challengingand many anti-disassembly tricks [31] can be used to con-fuse a disassembler, such as mixture of code and data,making an infeasible conditional jump to the middle ofnext instruction, etc. Although the current prototype doesnot handle these types of obfuscation for simplicity, thereare a variety of techniques [16] proposed to mitigate theseproblems. Third, MutantX-S relies on the similarity ofcode instructions to cluster malware samples. It is possi-ble to create syntactically distinct but semantically sim-ilar variants through heavy instruction-level obfuscation.To address these problems, MutantX-S could incorpo-rate more advanced de-obfuscation techniques [27, 22]and normalize the malware codes before clustering them.Note that dynamic-behavior-based approaches do not suf-fer from this limitation, but they come with their owndeficiencies—limited coverage, scalability and specificevasion techniques. Therefore, MutantX-S’ goal is notto replace the dynamic approaches, but to complement


them and collaboratively mitigate their weaknesses (e.g.apply dynamic analysis only on representative samplesor outliers of static analysis). Finally, MutantX-S can-not handle file infector or parasitic malware types whichinject themselves into host executables. This is a limita-tion for any similarity based clustering, regardless staticor dynamic approaches, because most features are fromthe host executables rather than the maleware. Such par-asitic malware are a matter of our future inquiry.

9 Conclusion

In this paper, we have presented the design, implementa-tion and evaluation of a malware clustering system basedon static features, called MutantX-S. MutantX-S canaccurately and efficiently group malware variants accord-ing to the similarity in their code instructions. It convertseach malware program into a compact but effective op-code representation and performs prototype-based clus-tering on the corresponding N-gram feature vectors. Italso incorporates a generic unpacking technique to max-imize the capability of analyzing the malware’s originalinstructions. To ensure the scalability, MutantX-S usesa combination of a hashing kernel that reduces the di-mensionality of feature vectors and a close-to-linear timeprototype-based clustering that uses a small set of repre-sentative samples for fast data organization. Equippedwith these techniques, MutantX-S is experimentallyshown to be able to process more than 130,000 malwaresamples within a few hours. As a static-analysis ap-proach, MutantX-S is expected to be very effective andcan be combined with existing dynamic-behavior-basedsystem to provide the level of accuracy and coverage re-quired to pace with the current malware sample submis-sion rate.

References[1] Murmurhash 2.0. http://sites.google.com/site/murmurhash/.

[2] Unpackers. http://www.exetools.com/unpackers.htm.

[3] Peid 0.95. http://www.peid.info/, 2008.

[4] ARTHUR, D., AND VASSILVITSKII, S. How slow is the k-meansmethod? In Proceedings of the twenty-second annual symposiumon Computational geometry (2006).

[5] ASPACK SOFTWARE. Aspack. http://www.aspack.com/.

[6] BAILEY, M., ANDERSEN, J., MAO, Z. M., AND JAHANIAN, F.Automated classification and analysis of internet malware. Tech.rep., Proceedings of RAID, 2007.

[7] BAYER, U., COMPARETTI, P., HLAUSCHEK, C., KRUEGEL, C.,AND KIRDA, E. Scalable, behavior-based malware clustering. InProc. of the 16th NDSS (2009).

[8] CHRISTODORESCU, M., AND JHA, S. Static analysis of executa-bles to detect malicious patterns. In In Proceedings of the 12thUSENIX Security Symposium (2003).

[9] GONZALEZ, T. Clustering to minimize the maximum interclus-ter distance. In Theoretical Computer Science (1985), vol. 38,pp. 293–306.

[10] GUO, F., FERRIE, P., AND CHIUEH, T.-C. A study of the packerproblem and its solutions. In Proceedings of RAID (2008).

[11] HASTIE, T., TIBSHIRANI, R., AND FRIEDMAN, J. The Elementsof Statistical Learning: Data Mining, Inference, and Prediction.Springer-Verlag, 2009.

[12] JANG, J., BRUMLEY, D., AND VENKATARAMAN, S. Bitshred:feature hashing malware for scalable triage and semantic analysis.In Proceedings of CCS’11 (2011).

[13] KARIM, M. E., WALENSTEIN, A., LAKHOTIA, A., ANDPARIDA, L. Malware phylogeny generation using permutationsof code. JOURNAL IN COMPUTER VIROLOGY 1 (2005), 13–23.

[14] KILIANWEINBERGER, DASGUPTA, A., LANGFORD, J.,SMOLA, A., AND ATTENBERG, J. Feature hashing for large scalemultitask learning. In Proceedings of the 26th ICML (2009).

[15] KOLTER, J. Z., AND MALOOF, M. A. Learning to detect andclassify malicious executables in the wild. Journal of MachineLearning Research 7 (2006), 2006.

[16] KRUEGEL, C., ROBERTSON, W., VALEUR, F., AND VIGNA, G.Static disassembly of obfuscated binaries. In Proceedings of the13th conference on USENIX Security Symposium (2004).

[17] LABS, A. R. New toy in the avast research lab.https://blog.avast.com/2012/12/03/new-toy-research-lab/.

[18] LEE, T., AND J.MODY, J. An automated virus classification sys-tem. In Proceedings of VIRUS BULLETIN CONFERENCE OC-TOBER 2005 (2005).

[19] MANNING, C. D., RAGHAVAN, P., AND SCHUTZE, H. Introduc-tion to Information Retrieval. Cambridge University Press, 2008.

[20] MARTIGNONI, L., CHRISTODORESCU, M., AND JHA, S. Om-niunpack: Fast, generic, and safe unpacking of malware. In Pro-ceedings of ACSAC (2007).

[21] MOSER, A., KRUEGEL, C., AND KIRDA, E. Exploring multipleexecution paths for malware analysis. In Proceedings of the 2007IEEE Symposium on Security and Privacy (2007).

[22] RABER, J., AND LASPE, E. Deobfuscator: An automated ap-proach to the identification and removal of code obfuscation. Re-verse Engineering, Working Conference on (2007).

[23] RIECK, K., HOLZ, T., WILLEMS, C., DUSSEL, P., ANDLASKOV, P. Learning and classification of malware behavior. InProc. of the DIMVA’08 (2008).

[24] RIECK, K., TRINIUS, P., WILLEMS, C., AND HOLZ, T. Auto-matic analysis of malware behavior using machine learning. techreport, Berlin Institute of Technology, 2009.

[25] SHAWE-TAYLOR, J., AND CRISTIANINI., N. Kernel Methodsfor Pattern Analysis. Cambridge University Press, 2004.

[26] SHI, Q., PETTERSON, J., DROR, G., LANGFORD, J., SMOLA,A., STREHL, A., AND VISHWANATHAN, V. Hash kernels. InProc. of the 12th International Conference on Artificial Intelli-gence and Statistics (2009).

[27] UDUPA, S. K., DEBRAY, S. K., AND MADOU, M. Deobfusca-tion: Reverse engineering obfuscated code. Reverse Engineering,Working Conference on (2005).

[28] UPX. http://upx.sourceforge.net/.[29] VXHEAVEN. Vxheaven virus collection. http://vx.netlux.org/,

2010.[30] WICHERSKI, G. pehash: A novel approach to fast malware clus-

tering. In 2nd Usenix LEET Workshop (2009).[31] YASON, M. The art of unpacking, https://www.blackhat.com/pre

sentations/bh-usa-07/yason/whitepaper/bh-usa-07-yason-wp.pdf.

mutantx-s: scalable malware clustering based … association 2013 usenix annual technical conference...

Documents