Program Obfuscation:A Quantitative Approach

Presented by: Mariusz Jakubowski Microsoft Research

Third Workshop on Quality of Protection October 29 th, 2007

Bertrand Anckaert, Matias Madou, Bjorn De Sutter, Bruno De Bus, Koen De Bosschere, and Bart Preneel

Ghent University and K.U.Leuven, Belgium

2

Obfuscation has many applications

0101110 00111001010 00101011001000110001110110010111011011001011101010110100010110111111110001010110110011111001010111001110010111 1 11111111111111110

3

There is a large gap between theoretical results

- On the (Im)possibility of Obfuscating Programs – Barak et al. (2001)- On the Impossibility of Obfuscation with Auxiliary Input – Goldwasser et al. (2005)

- Positive Results and Techniques for Obfuscation – Lynn et al. (2004)- Towards Realizing Random Oracles: Hash Functions that Hide All Partial Information Canetti et al. (1997)

+

-

Large gapIntuitively, obfuscation does help

4

We need a practical system for evaluating obfuscating transformations

• It should be easy to evaluate existing and future transformations=> Automated

• The evaluation should convey difficulty of reverse-engineering=> Build upon experience from complexity metrics

5

o Introo Metrics

o Instruction Counto Cyclomatic Numbero Knot Count

o (De)Obfuscating transformations

Outline

6

Four axes based on typical reverse-engineering scenario

Disassemble Flow graph construction

Analyse Data Flow

Interpret Data

Code

Control flow

Data flow

Data

7

+ No uncertainty about executed code+ Always availabe- Only about covered part of the code

Evaluated Complexity Metrics

Code

Control flow

Data flow

Data

Instruction Count

Cyclomatic NumberKnot Count

Metrics are collected by a run-time instrumentation framework

8

Cyclomatic number and knot count

• Cyclomatic number: – #edges – #nodes + 2– Intuitively: the number of decision points

• Knot count: – #crossings– Intuitively: the unstructuredness

9

o Introo Metricso (De)Obfuscating transformations

o Jump redirection [Linn et al. 2003]

o Control flow flattening [Chenxi Wang et al. 2001]

o Opaque predicates [Collberg et al. 1998]

Outline

10

Jump redirection

• Redirect branches to function

1

Jmp 2

2

1call branch

Branch Function

2

garbage

assumed return site

11

Impact of Jump Redirectiongz

ip vpr

cc1

mcf

craft

ypa

rser

perlb

mk

gap

vorte

xbz

ip2

twol

fav

erag

e

0

50

100

150

200

250

Instruction CountCyclomatic NumberKnot Count

Incr

ease

(%)

12

Jump redirection - deobfuscation

• Identify Branch Function– signature based– run-time behavior

• Record (call,return) pairs under debugger

• Overwrite calls

1call branch

Branch Function

2

garbage

assumed return site

(1,2)(4,7)(9,5)… jmp 2

13

Success of De-obfuscationgz

ip vpr

cc1

mcf

craft

ypa

rser

perlb

mk

gap

vorte

xbz

ip2

twol

fav

erag

e

0

50

100

150

200

250

Instruction CountCyclomatic NumberKnot Count

Incr

ease

(%)

gzip vpr

cc1

mcf

craft

ypa

rser

perlb

mk

gap

vorte

xbz

ip2

twol

fav

erag

e

0

50

100

150

200

250

Instruction CountCyclomatic NumberKnot Count

Incr

ease

(%)

14

Control flow flattening

All original basic blocks have the same predecessor and successor

1

4

32

1 432

switch

15

Control flow flattening significantly increases the complexity metrics

gzip vpr

cc1

mcf

craft

ypa

rser

perlb

mk

gap

vorte

xbz

ip2

twol

fav

erag

e

0

50

100

150

200

250

300

350

400

450

Instruction CountCyclomatic NumberKnot Count

Incr

ease

(%)

16

gzip vpr

cc1

mcf

craft

ypa

rser

perlb

mk

gap

vorte

xbz

ip2

twol

fav

erag

e

0

50

100

150

200

250

300

350

400

450

Instruction CountCyclomatic NumberKnot Count

Incr

ease

(%)

gzip vpr

cc1

mcf

craft

ypa

rser

perlb

mk

gap

vorte

xbz

ip2

twol

fav

erag

e

0

50

100

150

200

250

300

350

400

450

Instruction CountCyclomatic NumberKnot Count

Incr

ease

(%)

Success of De-obfuscation

17

Opaque predicates

1

Jmp 2

2

1

Jmp if (2==2)

2fake

• Add fake decision statements

18

Impact of Opaque Predicationgz

ip vpr

cc1

mcf

craft

ypa

rser

perlb

mk

gap

vorte

xbz

ip2

twol

fav

erag

e

0

2

4

6

8

10

12

14

16

18

Instruction CountCyclomatic NumberKnot Count

Incr

ease

(%)

19

Conclusion

• A first step towards a unified quantitative evaluation of– obfuscating transformations– deobfuscating transformations

• Which leverages experience from the established field of complexity metrics

Program Obfuscation:A Quantitative Approach

Presented by: Mariusz Jakubowski Microsoft Research

Third Workshop on Quality of Protection October 29 th, 2007

Bertrand Anckaert, Matias Madou, Bjorn De Sutter, Bruno De Bus, Koen De Bosschere, and Bart Preneel

Ghent University and K.U.Leuven, Belgium