adaboost derek hoiem march 31, 2004

46
Adaboost Derek Hoiem March 31, 2004

Upload: others

Post on 07-Apr-2022

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t

Der

ek H

oiem

Mar

ch 3

1, 2

004

Page 2: Adaboost Derek Hoiem March 31, 2004

Out

line

Bac

kgro

und

Ada

boos

t Alg

orith

m

Theo

ry/In

terp

reta

tions

Pra

ctic

al Is

sues

Face

det

ectio

n ex

perim

ents

Page 3: Adaboost Derek Hoiem March 31, 2004

Wha

t’s S

o G

ood

Abo

ut A

dabo

ost

Impr

oves

cla

ssifi

catio

n ac

cura

cy

Can

be

used

with

man

y di

ffere

nt c

lass

ifier

s

Com

mon

ly u

sed

in m

any

area

s

Sim

ple

to im

plem

ent

Not

pro

ne to

ove

rfitti

ng

Page 4: Adaboost Derek Hoiem March 31, 2004

A B

rief H

isto

ry

Boo

tstra

ppin

g

Bag

ging

Boo

stin

g (S

chap

ire19

89)

Ada

boos

t (S

chap

ire19

95)

Page 5: Adaboost Derek Hoiem March 31, 2004

Boo

tstra

p E

stim

atio

n

Rep

eate

dly

draw

nsa

mpl

es fr

om D

For e

ach

set o

f sam

ples

, est

imat

e a

stat

istic

The

boot

stra

p es

timat

e is

the

mea

n of

the

indi

vidu

al e

stim

ates

Use

d to

est

imat

e a

stat

istic

(par

amet

er)

and

its v

aria

nce

Page 6: Adaboost Derek Hoiem March 31, 2004

Bag

ging

-A

ggre

gate

Boo

tstra

ppin

g

For i

= 1

.. M

Dra

w n

* <n

sam

ples

from

D w

ith re

plac

emen

tLe

arn

clas

sifie

r Ci

Fina

l cla

ssifi

er is

a v

ote

of C

1 ..

CM

Incr

ease

s cl

assi

fier s

tabi

lity/

redu

ces

varia

nce

Page 7: Adaboost Derek Hoiem March 31, 2004

Boo

stin

g (S

chap

ire19

89)

Ran

dom

ly s

elec

t n1 <

nsa

mpl

es fr

om D

with

out r

epla

cem

ent t

o ob

tain

D1

Trai

n w

eak

lear

ner C

1

Sel

ect n

2 <

nsa

mpl

es fr

om D

with

hal

f of t

he s

ampl

es m

iscl

assi

fied

by C

1 to

obta

in D

2Tr

ain

wea

k le

arne

r C2

Sel

ect a

llsa

mpl

es fr

om D

that

C1

and

C2

disa

gree

on

Trai

n w

eak

lear

ner C

3

Fina

l cla

ssifi

er is

vot

e of

wea

k le

arne

rs

Page 8: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t -A

dapt

ive

Boo

stin

g

Inst

ead

of s

ampl

ing,

re-w

eigh

tP

revi

ous

wea

k le

arne

r has

onl

y 50

% a

ccur

acy

over

ne

w d

istri

butio

n

Can

be

used

to le

arn

wea

k cl

assi

fiers

Fina

l cla

ssifi

catio

n ba

sed

on w

eigh

ted

vote

of

wea

k cl

assi

fiers

Page 9: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t Ter

ms

Lear

ner =

Hyp

othe

sis

= C

lass

ifier

Wea

k Le

arne

r: <

50%

erro

r ove

r any

di

strib

utio

n

Stro

ng C

lass

ifier

: thr

esho

lded

linea

r co

mbi

natio

n of

wea

k le

arne

r out

puts

Page 10: Adaboost Derek Hoiem March 31, 2004

Dis

cret

e A

dabo

ost (

Dis

cret

eAB

)(F

riedm

an’s

wor

ding

)

Page 11: Adaboost Derek Hoiem March 31, 2004

Dis

cret

e A

dabo

ost (

Dis

cret

eAB

)(F

reun

d an

d S

chap

ire’s

wor

ding

)

Page 12: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t with

Con

fiden

ce

Wei

ghte

d P

redi

ctio

ns (R

ealA

B)

Page 13: Adaboost Derek Hoiem March 31, 2004

Com

paris

on2

Nod

e Tr

ees

Page 14: Adaboost Derek Hoiem March 31, 2004

Bou

nd o

n Tr

aini

ng E

rror

(Sch

apire

)

Page 15: Adaboost Derek Hoiem March 31, 2004

Find

ing

a w

eak

hypo

thes

is

Trai

n cl

assi

fier (

as u

sual

) on

wei

ghte

d tra

inin

g da

ta

Som

e w

eak

lear

ners

can

min

imiz

e Z

by g

radi

ent d

esce

nt

Som

etim

es w

e ca

n ig

nore

alp

ha (w

hen

the

wea

k le

arne

r ou

tput

can

be

freel

y sc

aled

)

Page 16: Adaboost Derek Hoiem March 31, 2004

Cho

osin

g A

lpha

Cho

ose

alph

a to

min

imiz

e Z

Res

ults

in 5

0% e

rror

rate

in la

test

wea

k le

arne

r

In g

ener

al, c

ompu

te n

umer

ical

ly

Page 17: Adaboost Derek Hoiem March 31, 2004

Spe

cial

Cas

e

Dom

ain-

parti

tioni

ng w

eak

hypo

thes

is (e

.g. d

ecis

ion

trees

, hi

stog

ram

s)

x x xx

x xo o

oo o

ox

P1

P3

P2

Wx1

= 5/

13, W

o1=

0/13

Wx2

= 1/

13, W

o2=

2/13

Wx3

= 1/

13, W

o3=

4/13

Z =

2 (0

+ s

qrt(2

)/13

+ 2/

13) =

.525

Page 18: Adaboost Derek Hoiem March 31, 2004

Sm

ooth

ing

Pre

dict

ions

Equ

ival

ent t

o ad

ding

prio

r in

parti

tioni

ng c

ase

Con

fiden

ce b

ound

by

Page 19: Adaboost Derek Hoiem March 31, 2004

Just

ifica

tion

for t

he E

rror

Fun

ctio

n

Ada

boos

t min

imiz

es:

Pro

vide

s a

diffe

rent

iabl

e up

per b

ound

on

train

ing

erro

r (S

hapi

re)

Min

imiz

ing

J(F)

is e

quiv

alen

t up

to 2

ndor

der T

aylo

r ex

pans

ion

abou

t F =

0 to

max

imiz

ing

expe

cted

bin

omia

l lo

g-lik

elih

ood

(Frie

dman

)

Page 20: Adaboost Derek Hoiem March 31, 2004

Pro

babi

listic

Inte

rpre

tatio

n (F

riedm

an)

Proo

f:

Page 21: Adaboost Derek Hoiem March 31, 2004

Mis

inte

rpre

tatio

ns o

f the

P

roba

bilis

tic In

terp

reta

tion

Lem

ma

1 ap

plie

s to

the

true

dist

ribut

ion

Onl

y ap

plie

s to

the

train

ing

set

Not

e th

at P

(y|x

) = {0

, 1} f

or th

e tra

inin

g se

t in

mos

t ca

ses

Ada

boos

t will

con

verg

e to

the

glob

al m

inim

umG

reed

y pr

oces

s lo

cal m

inim

umC

ompl

exity

of s

trong

lear

ner l

imite

d by

com

plex

ities

of

wea

k le

arne

rs

Page 22: Adaboost Derek Hoiem March 31, 2004

Exa

mpl

e of

Joi

nt D

istri

butio

n th

at C

anno

t Be

Lear

ned

with

Naï

ve W

eak

Lear

ners

x2

x1

0, 0

0, 1

1, 0

1, 1

x

xo

o

P(o

| 0,0

) = 1

P(x

| 0,1

) = 1

P(x

| 1,0

) = 1

P(o

| 1,1

) = 1

H(x

1,x2

) = a

*x1

+ b*

(1-x

1) +

c*x

2 +

d*(1

-x2)

=

(a-b

)*x1

+ (c

-d)*

x2 +

(b+d

)

Line

ar d

ecis

ion

for n

on-li

near

pro

blem

whe

n us

ing

naïv

e w

eak

lear

ners

Page 23: Adaboost Derek Hoiem March 31, 2004

Imm

unity

to O

verfi

tting

?

Page 24: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t as

Logi

stic

Reg

ress

ion

(Frie

dman

)A

dditi

ve L

ogis

tic R

egre

ssio

n: F

ittin

g cl

ass

cond

ition

al p

roba

bilit

y lo

g ra

tio w

ith a

dditi

ve te

rms

Dis

cret

eAB

build

s an

add

itive

logi

stic

regr

essi

on m

odel

via

New

ton-

like

upda

tes

for m

inim

izin

g

Rea

lAB

fits

an a

dditi

ve lo

gist

ic re

gres

sion

mod

el b

y st

age-

wis

e an

d ap

prox

imat

e op

timiz

atio

n of

Eve

n af

ter t

rain

ing

erro

r rea

ches

zer

o, A

B p

rodu

ces

a “p

urer

” so

lutio

n pr

obab

ilist

ical

ly

Page 25: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t as

Mar

gin

Max

imiz

atio

n (S

chap

ire)

Page 26: Adaboost Derek Hoiem March 31, 2004

Bou

nd o

n G

ener

aliz

atio

n E

rror

Num

ber o

f tra

inin

g ex

ampl

es

VC

dim

ensi

on

1/20

1/8

1/4

1/2

Con

fiden

ce o

f cor

rect

dec

isio

n

Con

fiden

ce M

argi

nB

ound

con

fiden

ce te

rm

To lo

ose

to b

e of

pr

actic

al v

alue

:

Page 27: Adaboost Derek Hoiem March 31, 2004

Max

imiz

ing

the

mar

gin…

But

Ada

boos

t doe

sn’t

nece

ssar

ily m

axim

ize

the

mar

gin

on th

e te

st s

et (R

atsc

h)R

atsc

hpr

opos

es a

n al

gorit

hm (A

dabo

ost*

) tha

t doe

s so

Page 28: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t and

Noi

sy D

ata

Exa

mpl

es w

ith th

e la

rges

t gap

bet

wee

n th

e la

bel a

nd th

e cl

assi

fier p

redi

ctio

n ge

t the

mos

t wei

ght

Wha

t if t

he la

bel i

s w

rong

?

Opi

tz: S

ynth

etic

dat

a w

ith 2

0% o

ne-s

ided

noi

sy la

belin

g

Num

ber o

f Net

wor

ks in

Ens

embl

e

Page 29: Adaboost Derek Hoiem March 31, 2004

Wei

ghtB

oost

(Jin

)

Use

s in

put-d

epen

dent

wei

ghtin

g fa

ctor

s fo

r wea

k le

arne

rsTe

xt C

ateg

oriz

atio

n

Page 30: Adaboost Derek Hoiem March 31, 2004

Bro

wnB

oost

(Fre

und)

Non

-mon

oton

ic w

eigh

ting

func

tion

Exa

mpl

es fa

r fro

m b

ound

ary

decr

ease

in w

eigh

t

Set

a ta

rget

err

or ra

te –

algo

rithm

atte

mpt

s to

ach

ieve

th

at e

rror

rate

No

resu

lts p

oste

d (e

ver)

Page 31: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t Var

iant

s P

ropo

sed

By

Frie

dman

Logi

tBoo

stS

olve

s R

equi

res

care

to a

void

num

eric

al p

robl

ems

Gen

tleB

oost

Upd

ate

is f m

(x) =

P(y

=1 |

x) –

P(y

=0 |

x) in

stea

d of

Bou

nded

[0 1

]

Page 32: Adaboost Derek Hoiem March 31, 2004

Com

paris

on (F

riedm

an)

Syn

thet

ic, n

on-a

dditi

ve d

ecis

ion

boun

dary

Page 33: Adaboost Derek Hoiem March 31, 2004

Com

plex

ity o

f Wea

k Le

arne

r

Com

plex

ity o

f stro

ng c

lass

ifier

lim

ited

by

com

plex

ities

of w

eak

lear

ners

Exa

mpl

e:W

eak

Lear

ner:

stub

s W

L D

imV

C=

2In

put s

pace

: RN

Stro

ng D

imV

C=

N+1

N+1

par

titio

ns c

an h

ave

arbi

trary

con

fiden

ces

assi

gned

Page 34: Adaboost Derek Hoiem March 31, 2004

Com

plex

ity o

f the

Wea

k Le

arne

r

Page 35: Adaboost Derek Hoiem March 31, 2004

Com

plex

ity o

f the

Wea

k Le

arne

r

Non

-Add

itive

Dec

isio

n Bo

unda

ry

Page 36: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t in

Face

Det

ectio

n

One

Gro

up o

f

N-D

His

togr

ams

Rea

lAB

Sch

neid

erm

an

1-D

His

togr

ams

KLB

oost

KLB

oost

1-D

His

togr

ams

Floa

tBoo

stFl

oatB

oost

Stu

bsD

iscr

eteA

BV

iola

-Jon

es

Wea

k Le

arne

rA

dabo

ost

Var

iant

Det

ecto

r

Page 37: Adaboost Derek Hoiem March 31, 2004

Boo

sted

Tre

es –

Face

Det

ectio

n

Stu

bs (2

Nod

es)

8 N

odes

Page 38: Adaboost Derek Hoiem March 31, 2004

Boo

sted

Tre

es –

Face

Det

ectio

n

Ten

Stu

bs v

s. O

ne 2

0 N

ode

2, 8

, and

20

Nod

es

Page 39: Adaboost Derek Hoiem March 31, 2004

Boo

sted

Tre

es v

s. B

ayes

Net

Page 40: Adaboost Derek Hoiem March 31, 2004

Floa

tBoo

st(L

i)FP

Rat

e at

95.

5% D

et R

ate

35.0

%

40.0

%

45.0

%

50.0

%

55.0

%

60.0

%

65.0

%

70.0

%

75.0

%

80.0

%

510

1520

2530

Num

Iter

atio

ns

Tree

s - 2

0 bi

ns(L

1+L2

)Tr

ees

- 20

bins

(L2)

Tree

s - 8

bin

s (L

2)

Tree

s - 2

bin

s (L

2)

1. L

ess

gree

dy a

ppro

ach

(Flo

atB

oost

) yie

lds

bette

r res

ults

2. S

tubs

are

not

suf

ficie

nt fo

r vis

ion

Page 41: Adaboost Derek Hoiem March 31, 2004

Whe

n to

Use

Ada

boos

t

Giv

e it

a try

for a

ny c

lass

ifica

tion

prob

lem

Be

war

y if

usin

g no

isy/

unla

bele

d da

ta

Page 42: Adaboost Derek Hoiem March 31, 2004

How

to U

se A

dabo

ost

Sta

rt w

ith R

ealA

B(e

asy

to im

plem

ent)

Pos

sibl

y try

a v

aria

nt, s

uch

as F

loat

Boo

st, W

eigh

tBoo

st,

or L

ogitB

oost

Try

vary

ing

the

com

plex

ity o

f the

wea

k le

arne

r

Try

form

ing

the

wea

k le

arne

r to

min

imiz

e Z

(or s

ome

sim

ilar g

oal)

Page 43: Adaboost Derek Hoiem March 31, 2004

Con

clus

ion

Ada

boos

t can

impr

ove

clas

sifie

r acc

urac

y fo

r man

y pr

oble

ms

Com

plex

ity o

f wea

k le

arne

r is

impo

rtant

Alte

rnat

ive

boos

ting

algo

rithm

s ex

ist t

o de

al w

ith m

any

of A

dabo

ost’s

prob

lem

s

Page 44: Adaboost Derek Hoiem March 31, 2004

Ref

eren

ces

Dud

a, H

art,

ect–

Pat

tern

Cla

ssifi

catio

n

Freu

nd –

“An

adap

tive

vers

ion

of th

e bo

ost b

y m

ajor

ity a

lgor

ithm

Freu

nd –

“Exp

erim

ents

with

a n

ew b

oost

ing

algo

rithm

Freu

nd, S

chap

ire–

“A d

ecis

ion-

theo

retic

gen

eral

izat

ion

of o

n-lin

e le

arni

ng a

nd a

n ap

plic

atio

n to

boo

stin

g”

Frie

dman

, Has

tie, e

tc –

“Add

itive

Log

istic

Reg

ress

ion:

A S

tatis

tical

Vie

w o

f Boo

stin

g”

Jin,

Liu

, etc

(CM

U) –

“A N

ew B

oost

ing

Alg

orith

m U

sing

Inpu

t-Dep

ende

nt R

egul

ariz

er”

Li, Z

hang

, etc

–“F

loat

boos

tLea

rnin

g fo

r Cla

ssifi

catio

n”

Opi

tz, M

aclin

–“P

opul

ar E

nsem

ble

Met

hods

: An

Em

piric

al S

tudy

Rat

sch,

War

mut

h–

“Effi

cien

t Mar

gin

Max

imiz

atio

n w

ith B

oost

ing”

Sch

apire

, Fre

und,

etc

–“B

oost

ing

the

Mar

gin:

A N

ew E

xpla

natio

n fo

r the

Effe

ctiv

enes

s of

Vot

ing

Met

hods

Sch

apire

, Sin

ger –

“Impr

oved

Boo

stin

g A

lgor

ithm

s U

sing

Con

fiden

ce-W

eigh

ted

Pre

dict

ions

Sch

apire

–“T

he B

oost

ing

App

roac

h to

Mac

hine

Lea

rnin

g: A

n ov

ervi

ew”

Zhan

g, L

i, et

c –

“Mul

ti-vi

ew F

ace

Det

ectio

n w

ith F

loat

boos

t”

Page 45: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t Var

iant

s P

ropo

sed

By

Frie

dman

Logi

tBoo

st

Page 46: Adaboost Derek Hoiem March 31, 2004

Ada

boos

t Var

iant

s P

ropo

sed

By

Frie

dman

Gen

tleB

oost