detection of ascii malware

33
Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen

Upload: benard

Post on 13-Jan-2016

55 views

Category:

Documents


3 download

DESCRIPTION

Detection of ASCII Malware. Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen. Internet Worm and Malware. Huge damage potential Infects hundreds of thousands of computers Costs millions of dollars in damage Melissa, ILOVEYOU, Code Red, Nimda, Slammer, SoBig, MyDoom - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Detection of ASCII Malware

Detection of ASCII Malware

Parbati Kumar Manna

Dr. Sanjay Ranka

Dr. Shigang Chen

Page 2: Detection of ASCII Malware

2

Internet Worm and Malware

• Huge damage potential Infects hundreds of thousands of

computers Costs millions of dollars in damage Melissa, ILOVEYOU, Code Red,

Nimda, Slammer, SoBig, MyDoom

• Mostly uses Buffer Overflow

• Propagation is automatic (mostly)

Page 3: Detection of ASCII Malware

3

Recent Trends

• Shift in hacker’s mindset

• Malware becoming increasingly evasive and obfuscative

• Emergence of Zero-day worms

• Arrival of Script Kiddies

Page 4: Detection of ASCII Malware

4

Motivation for ASCII Attacks

• Prevalence of servers expecting text-only input

• Text-based protocols

• Presumption of text being benign

• Deployment of ASCII filter for bypassing text

Page 5: Detection of ASCII Malware

5

IDS Detecting ASCII Attack?

• Disassembly-based IDS

All jump instructions are ASCII

Higher proportion of branches

Exponential disassembly cost

High processing overhead for IDS

• Frequency-based IDS

PAYL evaded by ASCII worm

Page 6: Detection of ASCII Malware

6

Buffer Overflow

Page 7: Detection of ASCII Malware

7

• Opcode Unavailability Shellcode requires binary opcodes Here only xor, and, sub, cmp etc. Must generate opcodes dynamically

• Difficulty in Encryption No backward jump Can’t use same decrypter routine

for each encrypted block No one-to-one correspondence

between ASCII and binary

Constraints of ASCII Malware

0 m a y v a r y

ASCII binary

Page 8: Detection of ASCII Malware

8

Creation of ASCII Malware

Page 9: Detection of ASCII Malware

9

Buffer Overflow using ASCII

Overflowing a buffer using an ASCII string:

Page 10: Detection of ASCII Malware

10

• Opcode Unavailability Dynamic generation of opcodes

needs more ASCII instructions for each binary instruction

• Difficulty in Encryption No backward jump means

decrypter block for each encrypted block must be hardcoded

Long sequence of contiguous valid instructions likely high MEL

Detection of ASCII Malware

What is this MEL?

Page 11: Detection of ASCII Malware

11

• Indicates maximum length of an execution path

Need to disassemble (and execute) from all possible entry points

All branching must be considered• Abstract payload execution

Used for binary worms with sled Effectiveness dwindled presently

Maximum Executable Length

Page 12: Detection of ASCII Malware

12

Benign Text has Low MEL

• Contains characters that correspond to invalid instructions

Privileged Instruction (I/O) Arbitrary Segment Selector More Memory-accessing

instructions – may use uninitialized registers

Long sequence of contiguous valid instructions unlikely low MEL

Page 13: Detection of ASCII Malware

13

Proposed Solution

Question:

• How long is “long”?

• Find out the maximum length of valid instruction sequence

• If it is long enough, the stream contains a malware

Page 14: Detection of ASCII Malware

14

• Toss a coin n times

• What is the probability that the max distance between two consecutive heads is ?

Probabilistic Analysis

Head (H) Invalid Instruction (I)

Tail (T) Valid Instruction (v)

T H T T H T T T T T H T T TV I V V I V V V V V I V V V

Page 15: Detection of ASCII Malware

15

Probabilistic Analysis

n = number of coin tosses p = probability of a head Xi = R.V.s for inter-head

distancesXmax = Max inter-head distanceC.D.F of Xmax = Prob [Xmax ≤ x]

= [1 – p(1-p)x ]n

F.P. rate = 1 - Prob [Xmax ≤ τ] = 1 - [1 – p(1-p)τ ]n

Page 16: Detection of ASCII Malware

16

Probabilistic Analysis

For a fixed N = k (exactly k invalid instructions)

Page 17: Detection of ASCII Malware

17

Probabilistic Analysis

For all possible values of N:

Page 18: Detection of ASCII Malware

18

Threshold Calculation

n , p , (false positive rate)

(max inter-head distance)

Known

Unknown

)1log(

log))1(1log(1

p

pn

Threshold

Page 19: Detection of ASCII Malware

19

Independence Assumption

2 test contingency table

Observed Expected

I2 is valid

I2 is invali

d

I1 is valid

I2 is invalid

I1 is valid 8960 2797 8922 2835

I1 is invalid 2797 938 2835 900

• Validity of an instruction is an independent event

• All the Xi’s are independent (while Xi = n)

Page 20: Detection of ASCII Malware

20

Threshold Calculation

With increasing n, we must choose a larger to keep the same rate of false positive

Page 21: Detection of ASCII Malware

21

Threshold Calculation

With decreasing p, we must choose a larger to keep the same rate of false positive

Page 22: Detection of ASCII Malware

22

Determine n

size)n instructio (average

)charactersinput ofnumber (

I

Cn

E[I] = E[Prefix chain length] + E[core instruction length]

Obtained from character frequency of input data

Page 23: Detection of ASCII Malware

23

1.Privileged instructions

2.Wrong Segment Prefix Selector

3.Un-initialized memory access

Determine p

Invalid Instructions

Only 1. and 2. can be determined on a standalone basis

Page 24: Detection of ASCII Malware

24

Experimental Setup

Page 25: Detection of ASCII Malware

25

Implementation

Page 26: Detection of ASCII Malware

26

Experimental Setup

• Benign data setup ASCII stream captured from live CISE network

using Ethereal

• Malicious data setup Existing framework used to generate ASCII worm

by converting binary worms

• Promising experimental results for max valid instruction length Benign: all max values all below threshold Malicious: values significantly higher than

Page 27: Detection of ASCII Malware

27

Experimental Results (DAWN)

Page 28: Detection of ASCII Malware

28

Experimental Results (APE-L)

Page 29: Detection of ASCII Malware

29

Contrasting with APE

• Full content examination

• Threshold calculation

• Sled Vs. malware

• Exploiting text-specific properties

Page 30: Detection of ASCII Malware

30

Multilevel Encryption

Encryption

Decryption

binary ASCII ASCII

ASCII ASCII binary

Only Visible decrypter

Page 31: Detection of ASCII Malware

31

Multilevel Encryption

Text0x20 – 0x3F

Text0x40 – 0x5F

Text0x60 – 0x7E

Binary

Binary

Page 32: Detection of ASCII Malware

32

Questions

Page 33: Detection of ASCII Malware

33

Thank you