Nonnegative Matrix Factorization via Nonnegative Matrix Factorization via Rank-one DowndateRank-one Downdate

Nonnegative Matrix Nonnegative Matrix FactorizationFactorization

560 by 1965

560 by 2

2 by 1965

20 by 28 20 by 28

-2.19

-0.02

-3.19

1.02

2 by 12 by 1

Singular Value Decomposition Singular Value Decomposition (SVD)(SVD)

HistoryHistory

HistoryHistory

HistoryHistory

HistoryHistory

History (Algorithms)History (Algorithms)

History (Algorithms)History (Algorithms)

First observationFirst observation

Power methodPower method

• Computes the leading singular vectors/value (or eigenvector/value) of a matrix

:)d(powermetho,, Avu

1

2 while not converged

uA

uA

uAv

AvAv

u

T

T

T

ones of vectorv

3

4

5

6 end

ExampleExample

• Computes the leading singular vectors/value (or eigenvector/value) of a matrix

:)d(powermetho,, Avu

2 while not converged

uA

uA

uAv

AvAv

u

T

T

T

ones of vectorv

3

4

5

6 end

1

2

3

4

5 for all set

6 end for

Naive approach to NMF using this Naive approach to NMF using this observationobservation

Without step 5, this will simply compute the SVD (Jordan's algorithm, Camille Jordan 1874. )

:)nmf(, AHW

ki : for 1)d(powermetho],,[ Avu

TTii vHuW ,

TvuAA 0jiA ,0jiA ,

Second observationSecond observation

Modified power iterationModified power iteration

:)d(powermethomodified,, Avu

1

2 while not converged

uA

uA

uAv

AvAv

u

T

T

T

method power using initial find v

3

4

5

6

7

8 end

mean smaller withcluster the remove and cluster u

mean smaller withcluster the remove and cluster v

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

A =

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

0.14 0.07 0.64 0.41 0.55v:

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

0.0 0.0 0.64 0.41 0.55v:

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

v: 0.0 0.0 0.64 0.41 0.55

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

0.160.210.220.440.740.20

v: 0.0 0.0 0.64 0.41 0.55

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v: 0.0 0.0 0.64 0.41 0.55

0.00.00.0

0.440.740.20

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v: 0.0 0.0 0.64 0.41 0.55

0.00.00.0

0.440.740.20

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v: 0.0 0.0 0.60 0.28 0.59

0.00.00.0

0.440.740.20

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v:

0.00.00.0

0.440.740.20

0.0 0.0 0.60 0.28 0.59

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v:

0.00.00.0

0.440.740.20

0.0 0.0 0.60 0.28 0.59

Zero-out!Zero-out!

Modified power iteration: DemoModified power iteration: Demo

Anew =

Rank-1submatrix

Rank-one Downdata (R1D)Rank-one Downdata (R1D)

1

2

3

4

5 end for

:)nmf(, AHW

ki : for 1

)wermethod(modifiedpo],,[ Avu TT

ii vHuW ,

000 yxyx vuA and if Let ,

Objective functionObjective function

ApproxRankOneSubmatrix(A)ApproxRankOneSubmatrix(A)

Rank-one Downdata (R1D)Rank-one Downdata (R1D)

Rank-one Downdata (R1D)Rank-one Downdata (R1D)

A simple model for textA simple model for text

Generating a corpus in the Generating a corpus in the modelmodel

Theorem about textTheorem about text

LSILSI

R1DR1D

Experimental resultsExperimental results

• Information retrieval task– TDT corpus (pilot study, v. 1.3, 1997): news

articles

Identified topics (first few columns of ):W

Topic: president, clinton, house, white

Topic: bosnian, serbs, bosnia, serb, nato, sarajevo, air, bihac

Topic: haiti, military, aristide, haitian, troops, port, invasion, …

Topic: simpson, defense, judge, case, jury, trial, angeles, los,

court, …

Topic: bill, today, senate, republicans, house, congress,

republican, …

Experimental resultsExperimental results• For topic: “OJ Simpson trial” (simpson, defense, judge,

case, jury, trial)

Document 1Allen

Simpson

Judge

Defence

Marcus

Court

Witness

Testify

Nicole

gloves

Document 2Simpson

Judge

Defense

Statements

Opening

Jury

Legal

Yesterday

Prosecution

Marc

los

Document 3Simpson

Gloves

Case

Prosecution

Defense

Roger

Put

Fit

Today

Problems

jury

Document 4Simpson

Kaelin

Police

Defense

Chicago

Testimony

Night

Knapsack

Cnn

Blood

Witness

Theorem about imagesTheorem about images

Experimental resultsExperimental results

LSILSI

NMF-DIVNMF-DIV

R1DR1D

LSI

NMF_DIV

NMF_SC

R1D