Upload File

behavioral clustering of http-based malware and …behavioral clustering of http-based malware and...

  • Home
  • Documents
  • Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke
31
Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci (1,2) , Wenke Lee (1,2) , Nick Feamster (1) USENIX NSDI 2010 (1) (2)

Upload: others

Post on 28-Mar-2020

8 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Behavioral Clustering of HTTP-based Malwareand Signature Generation using

Malicious Network Traces

Roberto Perdisci(1,2), Wenke Lee(1,2), Nick Feamster(1)

USENIX NSDI 2010

(1) (2)

Page 2: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Malware = Malicious Software● Most modern cyber crimes are carried out using

malicious software

– Spam, Identity Theft, DDoS...

● Many different types of malware– Trojans

– Bots

– Spyware

– Adware

– Scareware ...

Page 3: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Traditional AVs are not enough!AV scan Malware

Benign

OriginalMalware

HiddenMalware

ExecutablePacking

(obfuscation)

.exe

Page 4: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

What can we do to detect malware?● Most malware need a network connection to perpetrate

malicious activities

– Bots need to contact C&C server, send spam, etc...

– Spyware need to exfiltrate private info

– Trojan droppers need to download further malicious software ...

● Variants of the same malware can evade AVs

– When executed they generate similar malicious behavior

No AVdetection

Honeypot

GET /in.php?affid=101POST /jump2/?affiliate=boo1

obfuscationengine

GET /in.php?affid=132POST /jump2/?affiliate=boo3

GET /in.php?affid=123POST /jump2/?affiliate=boo2

Similar network behavior

Page 5: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Our Approach● Detect the Network Behavior of Malware

–Complement existing host-based detection systems

–Improve “coverage”

IDS

Alarm

Admin

Page 6: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Web-based Malware(2009 – source: Team Cymru)

HTTP-C&C

IRC-C&C

● Use HTTP protocol

● Bypass existing network defenses

–Firewalls

● Web kits for malware control available

Web-ProxyFW

Enterprise Network

Page 7: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Detecting Web-based Malware

Web-ProxyFW

NetworkAdmin

Enterprise Network

IDS

BehavioralAnalysis

Malware detection models

Malware Collection

Page 8: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Malware Detection Signature:

GET /in\.php\?affid=.*&url=5&win=Windows%20XP\+2\.0&sts=.*

System Overview

Malware Traffic:

GET /in.php?affid=94901&url=5&win=Windows%20XP+2.0&sts=|US|1|6|4|1|284|0

GET /in.php?affid=43403&url=5&win=Windows%20XP+2.0&sts=

GET /in.php?affid=94924&url=5&win=Windows%20XP+2.0&sts=|US|1|6|8|1|184|0

BehavioralClustering

Malware Families

1

23

1

2

3

3

2

1

Page 9: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Behavioral Malware Clustering● Related Work (host-level behavior)

– Automated analysis of Internet malware [Bailey et al., RAID 2007]

– Scalable malware clustering [Bayer et al., NDSS 2009]

– Malware indexing using function-call graphs [Hu et al., CCS 2009]

● Our approach

– Focus on network-level behavior

we want network signatures

– Better malware detection signatures than using host-level behavior

Page 10: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Network Behavioral ClusteringMalware Traces Coarse-grained Fine-grained Meta-clusters

● Three-steps clustering refinement process

● Good trade-off between efficiency and accuracy

Page 11: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Network Behavioral ClusteringMalware Traces Coarse-grained Fine-grained Meta-clusters

Honeypot

GET /bins/int/9kgen_up.int?fxp=6d HTTP/1.1User-Agent: DownloadHost: X1569.nb.host192-168-1-2.comCache-Control: no-cache

HTTP/1.1 200 OKConnection: closeServer: Yaws/1.68 Yet Another Web ServerDate: Mon, 15 Mar 2010 11:47:11 GMTContent-Length: 573444Content-Type: application/octet-stream

Page 12: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Malware Traces Coarse-grained Fine-grained Meta-clusters

Network-level Clustering

# GET req# POST reqavg(len(url))avg(len(data_sent))avg(len(response))...

HierarchicalClustering

Statistical Features

Page 13: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Malware Traces Coarse-grained Fine-grained Meta-clusters

Network-level Clustering

GET /in.php?affid=94900GET /bins/int/9kgen_up.int?fxp=6dc23POST /jump2/?affiliate=boo1POST /trf?q=Keyword1&bd=-5%236

HierarchicalClustering

Structural Features

GET /in.php?affid=94900GET /bins/int/9kgen_up.int?fxp=6dc23POST /jump2/?affiliate=boo1POST /trf?q=Keyword1&bd=-5%236

GET /index.php?v=1.3&os=WinXPGET /kgen/config.txtPOST /bots/command.php?a=6.6.6.6POST /attack.php?ip=10.0.1.2&c=dos

d(M1,M2)

Malware Trace M1 Malware Trace M2

Page 14: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Malware Traces Coarse-grained Fine-grained Meta-clusters

Network-level Clustering

● Meta-clustering recovers from possible mistakes made in previous steps

● Improves overall quality of malware clusters and malware detection models

Page 15: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Malware Traces Coarse-grained Fine-grained Meta-clusters

Network-level Clustering

HierarchicalClustering

ComputeCentroids

MeasureDistance

d(C1,C2)

GET /in.php?affid=234GET /bins/in\.int?fxp=02POST /j?affiliate=boo1POST /trf?q=bd=-1%236

GET /in\.php\?affid=.*GET /bins/in\.int\?fxp=.*POST /j\?affiliate=boo.*POST /trf\?q=bd=.*%23.*

Centroid

TokenSubsequences

Algorithm

Page 16: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Signature Generation

GET /in\.php\?affid=.*GET /bins/int/9kgen_up\.int\?fxp=.*POST /jump2/\?affiliate=boo.*POST /trf\?q=Keyword.*&bd=.*%23.*

Signature Set

Enterprise Network

TokenSubsequences

Algorithm

Malware Families

Polygraph IEEE S&P 2005

Page 17: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Experimental Results● Malware Dataset

– 6 months of malware collection (Feb-Jul 2009)

– ~25k distinct real-world malware samples

● Clustering Results

Dataset Samples Malware Families

Modeled Samples

Signatures Time

Feb-2009 4,758 234 3,494 446 ~8h

Compact and well Separated Clusters

Cluster ValidityAnalysis

Page 18: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Experimental Results

GET /in\.php\?affid=.*GET /bins/int/9kgen_up\.int\?fxp=.*POST /jump2/\?affiliate=boo.*POST /trf\?q=Keyword.*&bd=.*%23.*

IDS

Signature Set

Malware Set

Malware ClustersHoneypot

DetectionResults

Feb09 Mar09 Apr09 May09 Jun09 Jul09

Sig. Feb09 85.9% 50.4% 47.8% 27.0% 21.7% 23.8%

Detection Test on All Samples

Feb09 Mar09 Apr09 May09 Jun09 Jul09

Sig. Feb09 54.8% 52.8% 29.4% 6.1% 3.6% 4.0%

Detection Test on Malware undetected by commercial AVs

Sig. Feb09 No False Alerts → Tested on 12M legitimate HTTP queries

Page 19: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Comparison with other approaches

Malware Set Coarse-grained Fine-grained Meta-clusters

Feb09 Mar09

78.6% 48.9%

Malware Set Fine-grained

Feb09 Mar09

60.1% 35.1%

Signature extracted from reduced malware set of ~2k malware samples

Using only fine-grained clustering

Using approach proposed in [Bayer et al. NDSS 2009]

Host-basedBehavioralClustering

Malware Set

Feb09 Mar09

56.9% 33.9%

Page 20: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Conclusion

● Novel behavioral malware clustering system● Focus on network-level behavior● Find malware families● Trade-off between efficiency and accuracy● Better detection models compared to using

host-level behavioral clustering approaches● Malware signatures complement existing host-

level malware detection approaches

Page 21: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

"If I haven't said this enough, this tool is so badass Roberto...It does an awesome job correlating and clustering these samples"

Sean M. Bodmer, CISSP CEHSenior Research Analyst

Damballa, Inc.

Page 22: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Thank You!

Q&A?

[email protected]

Page 23: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Appendix

Page 24: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

AV malware detection statsSource: Oberheide et al., USENIX Security 2008

Page 25: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Real-World Deployment

● Deployed in large enterprise network

– ~ 2k-3k active nodes

– 4 days of testing● Findings

– 25 machines infected by spyware

– 19 machines infected by scareware (fake AVs)

– 1 bot-compromised machine

– 1 machine compromised by banker trojan

Page 26: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Cluster Validity Analysis

M1 : W32/Virut.gen WORM/Rbot.50176.5 PE_VIRUT.D-1M2 : W32/Virut.gen WORM/Rbot.50176.5 PE_VIRUT.D-2M3 : W32/Virut.gen W32/Virut.Gen PE_VIRUT.D-4M4 : W32/Virut.gen W32/Virut.X PE_VIRUT.XO-2M5 : W32/Virut.gen WORM/Rbot.50176.5 PE_VIRUT.D-2M6 : W32/Virut.gen W32/Virut.H PE_VIRUT.NS-2M7 : W32/Virut.gen WORM/Rbot.50176.5 PE_VIRUT.D-2M8 : W32/Virut.gen WORM/Rbot.50176.5 PE_VIRUT.D-1

McAfeeM1

M8M5

M6M7

M2M3

M4

MalwareCluster Avira Trend Micro

M_W32/Virut

A_WORM/RbotA_W32/Virut

T_PE_VIRUT

0

58

1-

58

1-38

1-

38

1-

AV-Label Graph

Cohesion Index

Separation Index

Page 27: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Experimental Results

Cluster Validity Analysis

Compact and well Separated Clusters

6 months malware collection → over 25k distinct samples

Page 28: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Signature Generation and Pruning

GET /in\.php\?affid=.*GET /bins/int/9kgen_up\.int\?fxp=.*GET /img/logo.jpgPOST /jump2/\?affiliate=boo.*POST /trf\?q=Keyword.*&bd=.*%23.*GET /index\.asp\?version=.*

GET /in\.php\?affid=.*GET /bins/int/9kgen_up\.int\?fxp=.*POST /jump2/\?affiliate=boo.*POST /trf\?q=Keyword.*&bd=.*%23.*

IDS

Original Signature Set Pruned Signature Set

FinalMalwareClusters

GET /in\.php\?affid=.*GET /bins/int/9kgen_up\.int\?fxp=.*GET /img/logo.jpgPOST /jump2/\?affiliate=boo.*POST /trf\?q=Keyword.*&bd=.*%23.*GET /index\.asp\?version=.*

IDS

Original Signature Set

FinalMalwareClusters

Enterprise Network

LegitimateTraffic

Page 29: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Experimental Results

False Positives as measured on 12M legitimate HTTP requests from 2,010 clients

Malware Detection rate (all samples)

“Zero-Day” Malware Detection rate

Complements traditional AV detection systems

Detects significant fractionof current and futuremalware variants

Page 30: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Comparison with other approaches

Reduced dataset of ~4k malware samples

net-clusters = our three-step clustering approachnet-fg-clusters = only fine-grained clusteringsys-clusters = using approach proposed in [Bayer et al. NDSS 2009]

Malware Traces Coarse-grained Fine-grained Meta-clusters

SignatureGeneration

Page 31: Behavioral Clustering of HTTP-based Malware and …Behavioral Clustering of HTTP-based Malware and Signature Generation using Malicious Network Traces Roberto Perdisci(1,2), Wenke

Challenges

● Detecting malware traffic is hard– Many different types of malware

– Different communication protocols

– Malware can use legitimate protocols to communicate (e.g., HTTP)

Find a needle in haystack!

– Identify malware traffic among very large volumes of legitimate traffic


Mr201311 behavioral-based malware clustering (English)

Http://linc.ucy.ac.cy Andreas Papadopoulos - [email protected] [DEXA 2015] Clustering Attributed Multi-graphs with Information Ranking 26th International

Scalable, Behavior-Based Malware Clustering

Apache Tomcat Clusteringpeople.apache.org/.../presentations/2013-02-Apache-Tomcat-Clustering.… · Clustering •Has lots of meanings / uses •In Tomcat, we mean replicating HTTP

AVCLASS: A Tool for Massive Malware Labelingjuanca/papers/avclass_raid16.pdf · Keywords: Malware Labeling, AV labels, Classification, Clustering 1 Introduction Labeling a malicious

Http://creativecommons.org/licens es/by-sa/2.0/. Principal Component Analysis & Clustering Prof:Rui Alves [email protected] 973702406 Dept Ciencies Mediques

Intrusion Detection and Malware Analysis - Malware collection134.2.173.140/lehre/ws10/ids-malware/12-malware-collection.pdf · Intrusion Detection and Malware Analysis Malware collection

Malware Intelligence - index-of.co.ukindex-of.co.uk/Malware/pek-analysis-es.pdfde botnets a través del protocolo HTTP. Su primera versión, la primera beta (1.0beta), se remonta a

Introduction to Information Retrieval http ...hs/teach/14s/ir/pdf/16flat.flat.pdf · 2 Clustering: Introduction 3 Clustering in IR 4 K-means 5 Evaluation 6 How many clusters? 11/86

TOWARDS HIERARCHICAL CLUSTERING Mark Sh. Levin Inst. for Inform. Transm. Problems, Russian Acad. of Sci. Email: [email protected] Http:

Malware Fails Best Bugs in Malware Felix Leder [Malware ... · 1 Malware Fails Best Bugs in Malware Felix Leder [Malware Detection Team] [email protected] 5. desember 2011 malware

MutantX-S: Scalable Malware Clustering Based … Association 2013 USENIX Annual Technical Conference (USENIX ATC ’13) 187 MutantX-S: Scalable Malware Clustering Based on Static Features

Behavioral Clustering of HTTP-based Malware and … · and Signature Generation using Malicious Network Traces ... Honeypot GET /in.php?affid ... Detecting Web-based Malware FW Web-Proxy

Malware clustering and classification Peng Li [email protected]

(Un)Supervised Learning for Malware with quickSpan · Seeland Clustering Algorithm (Un)Supervised Learning for Malware with quickSpan 13 Find clusters in size ordered graphs using

Clustering for hybrid malware analysis and multi-path execution...Manual malware analysis is not ffit due to the huge number of such cases. As a result automated yet ffit malware analysis

On Challenges in Evaluating Malware Clustering

Agents in Smartphone 2. SMS Clustering Signature ... · Malware Analysis Signature Generation Phone Number Blacklist Response Action ... The clustering technique can increase the

HTTP-Based APT Malware Infection Detection Using URL

LNCS 7967 - Exploring Discriminatory Features for ...ghyan/papers/dimva13.pdf · ware clustering automatically identifies multiple classes of malware that share similar features

Understanding Malware - · PDF file•HTTP Bot •Banking Trojan Understanding malware ... - Packer info - Anti-virus detection info Activity of - File system - Registry - Process

Malware-multzokatze Teknikak (Malware Clustering ...joxeankoret.com/download/hack_beers_malware_multzokatzea.pdf · Malware Eboluzioa: 2000an 2000. urtean, astean behin ikusten zen

Android Malware Clustering through Malicious Payload Mining

Malware & Anti-Malware

Simseer.com - Malware Similarity and Clustering Made Easy

Behavioral Clustering of HTTP-Based Malware and Signature ...€¦ · The battle against malicious software (a.k.a. malware) is becoming more difficult. Today’s malware writers

SHELLTEA + POSLURP MALWARE - root9B Malware... · The source code to the Carberp banking trojan was leaked a few years ago ... performs HTTP requests with a user-agent created with

Machine Learning for Malware Classification and Clustering

Malware Nilgün Kablan. Malware …………………………………….…….……………………………………………… Geschichte der Malware ……………….…………………………………………………

Wesley Shields & MuradKhan - … and Control Analysis • Malware protocols change • Protocols hide in plain sight – So much HTTP based malware! • Need to understand layer 7

Selecting Features to Classify Malware€¦ · HTTP Redirection and Malware Served Redirection to . Routed to “fast-flux” networks Served key-logger (or other)

Http://. Text Clustering David Kauchak cs160 Fall 2009 adapted from:

Familial Clustering For Weakly-labeled Android Malware Using … · 2020-07-15 · Android malware is a major source of cyberattacks and is a serious threat to smartphone users. Labeling

Behavioral Clustering of HTTP-based Malware and Signature ...cobweb.cs.uga.edu/~perdisci/CSCI6900-F10/HTTPrecon.pdf · – (semi-)manual reference clustering – Precision and Recall

Automated Malware Analysis Report for http