latent factor models

Latent Factor Models

Geoff GordonJoint work w/ Ajit Singh, Byron Boots,

Sajid Siddiqi, Nick Roy

Motivation

A key component of a cognitive tutor: student cognitive model

Tracks what skills student currently knows—latent factors

circle-area

rectangle-area

decompose-area

right-answer

Motivation

Student models are a key bottleneck in cognitive tutor authoring and performance

rough estimate: 20-80 hrs to hand-code model for 1 hr of content

result may be too simple, not rigorously verified

But, demonstrated improvements in learning from better models

E.g., Cen et al [2007]:12% less time to learn 6 geometry units (same retention) using tutor w/ more accurate model

This talk: automatic discovery of new models and data-driven revision of existing models via (latent) factor analysis

SCORE: STDNT I, ITEM J

Simple case: snapshot, no side information

1 2 3 4 5 6 …

A 1 1 0 0 1 0 …

B 0 1 1 0 0 0 …

C 1 1 0 1 1 0 …

D 1 0 0 1 1 0 …

… … … … … … … …

ITEMS

STUDENTS

Missing data

1 2 3 4 5 6 …

A 1 ? ? ? 1 0 …

B 0 ? 1 0 ? ? …

C 1 1 ? ? ? 0 …

D 1 0 0 1 ? ? …

… … … … … … … …

ITEMS

STUDENTS

Data matrix X

xx11

xx22

xx33

..

..

..xxnn

STUDENTS

ITEMS

Simple case: model

XX

VV

UU

U: student latent factorsV: item latent factorsX: observed performance

n students

m items k latent factors

k latent factors

observed

unobserved

Linear-Gaussian version

student factoritem factor

XX

VV

UU

n students

m items k latent factors

k latent factors

U: Gaussian (0 mean, fixed var)V: Gaussian (0 mean, fixed var)X: Gaussian (fixed var, mean at left)

Matrix form: Principal Components Analysis

xx11

xx22

xx33

..

..

..xxnn

DATA MATRIX X

≈

COMPRESSED MATRIX U

uu11

uu22

uu33

..

..

..uunn

vv11 …… vvkk

BASIS MATRIX VT

PCA: the picture

PCA: matrix form

xx11

xx22

xx33

..

..

..xxnn

DATA MATRIX X

≈

COMPRESSED MATRIX U

uu11

uu22

uu33

..

..

..uunn

vv11 …… vvkk

BASIS MATRIX VT

COLS OF V SPAN THE LOW-RANK SPACE

Interpretation of factors

uu11

uu22

uu33

..

..

..uunn

vv11 …… vvkk

STUDENTS

ITEMSBASIS WEIGHTS

BASIS VECTORS

BASIS VECTORS ARE CANDIDATE “SKILLS” OR “KNOWLEDGE COMPONENTS”

WEIGHTS ARE STUDENTS’ KNOWLEDGE LEVELS

PCA is a widely successful model

FACE IMAGES FROM Groundhog Day, EXTRACTED BY CAMBRIDGE FACE DB PROJECT

Data matrix: face images

xx11

xx22

xx33

..

..

..xxnn

IMAGES

PIXELS

Result of factoring

uu11

uu22

uu33

..

..

..uunn

vv11 …… vvkk

IMAGES

PIXELSBASIS WEIGHTS

BASIS VECTORS

BASIS VECTORS ARE OFTEN CALLED “EIGENFACES”

Eigenfaces

IMAGE CREDIT: AT&T LABS CAMBRIDGE

PCA: the good

Unsupervised: need no human labels of latent state!

No worry about “expert blind spot”

Of course, labels helpful if available

Post-hoc human interpretation of latents is nice too—e.g., intervention design

PCA: the bad

Linear, Gaussian

PCA assumes E(X) is linear in UV

PCA assumes (X–E(X)) is i.i.d. Gaussian

Nonlinearity: conjunctive skills

P(CORRECT)

SKILL 1SKILL 2

Nonlinearity: disjunctive skills

P(CORRECT)

SKILL 1SKILL 2

Nonlinearity: “other”P(CORRECT)

SKILL 1SKILL 2

Non-Gaussianity

Typical hand-developed skill-by-item matrix

1 2 3 4 5 6 …

1 1 0 0 1 1 …

0 0 1 1 0 1 …

SKILLS

ITEMS

Result of Gaussian assumption

true recovered

rows of true and recovered V matrices

Result of Gaussian assumption

true recovered

rows of true and recovered V matrices

The ugly: MLE only

PCA yields maximum-likelihood estimate

Good, right?

sadly, the usual reasons to want the MLE don’t apply here

e.g., consistency: variance and bias of estimates of U and V do not approach 0 (unless #items/student and #students/item )

Result: MLE is typically far too confident of itself

Too certain: example

Learned coefficients

(e.g., a row of U)

Predictions

Result: “fold-in problem”

Nonsensical results when trying to apply learned model to a new student or item

Similar to overfitting problem in supervised learning: confident-but-wrong parameters do not generalize to new examples

Unlike overfitting, fold-in problem doesn’t necessarily go away with more data

Summary: 3 problems w/ PCA

Can’t handle nonlinearity

Can’t handle non-Gaussian distributions

Uses MLE only (==> fold-in problem)

Let’s look at each problem in turn

Nonlinearity

In PCA, had Xij ≈ Ui ⋅ Vj

What if

Xij ≈ exp(Ui ⋅ Vj)

Xij ≈ logit(Ui ⋅ Vj)

…

Non-Gaussianity

In PCA, had Xij ∼ Normal(μ), μ = Ui ⋅ Vj

What if

Xij ∼ Poisson(μ)

Xij ∼ Binomial(p)

…

Exponential family review

Exponential family of distributions:

P(X | θ) = P0(X) exp(X⋅θ – G(θ))

G(θ) is always strictly convex, differentiable on interior of domain

• means G’ is strictly monotone (strictly generalized monotone in 2D or higher)

Exponential family review

Exponential family PDF:

P(X | θ) = P0(X) exp(X⋅θ – G(θ))

• Surprising result: G’(θ) = g(θ) = E(X | θ)

• g & g–1 = “link function”

• θ = “natural parameter”

• E(X | θ) = “expectation parameter”

Examples

Normal(mean)

g = identity

Poisson(log rate)

g = exp

Binomial(log odds)

g = sigmoid

Nonlinear & non-Gaussian

Let P(X | θ) be an exponential family with natural parameter θ

Predict Xij ∼ P(X | θij), where θij = Ui ⋅ Vj

e.g., in Poisson, E(Xij) = exp(θij)

e.g., in Binomial, E(Xij) = logit(θij)

Optimization problem

max ∑ log P(Xij | θij)

s.t. θij = Ui ⋅ Vj

• “Generalized linear” or “exponential family” PCA

• all P(…) terms are exponential families

• analogy to GLMs

+ log P(U) + log P(V)U,V

[Collins et al, 2001][Gordon, 2002][Roy & Gordon, 2005]

Special cases

PCA, probabilistic PCA

Poisson PCA

k-means clustering

Max-margin matrix factorization (MMMF)

Almost: pLSI, pHITS, NMF

Comparison to AFM

p = probability correct

θ = student overall performance

β = skill difficulty

Q = item x skill matrix

γ = skill practice slope

T = number of practice opportunities

TTikik γkkθ

β0

QQ

11

xx

Theorem

• In GL PCA, finding U which maximizes likelihood (holding V fixed) is a convex optimization problem

• And, finding best V (holding U fixed) is a convex problem

• Further, Hessian is block diagonal

So, an efficient and effective optimization algorithm: alternately improve U and V

Example: compressing

histograms w/ Poisson PCA

Points: observed frequencies in ℝ3

Hidden manifold: a 1-parameter family of multinomials

A

B C

Example

ITERATION 1

Example

ITERATION 2

Example

ITERATION 3

Example

ITERATION 4

Example

ITERATION 5

Example

ITERATION 9

Remaining problem: MLE

Well-known rule of thumb: if MLE gets you in trouble due to overfitting, move to fully-Bayesian inference

Typical problem: computation

In our case, the computation is just fine if we’re a little clever

Additional wrinkle: switch to hierarchical model

Bayesian hierarchical exponential-family PCA

XX

VV

UU

U: student latent factorsV: item latent factorsX: observed performanceR: shared prior for student latentsS: shared prior for item latents

n students

m items

k latent factors

k latent factors

observed

unobserved

RR

SS

student factoritem factor

A little clever: MCMC

Z P(X)

Experimental comparisonGeometry Area 1996-1997

data

Geometry tutor: 139 items presented to 59 students

On average, each student tested on 60 items

Results: hold-out error

Embedding dimension for *EPCA is K = 15

credit: Ajit Singh

Extensions

Relational models

Temporal models

Relational models

1 2 3 4 5 6john

1 1 0 0 1 0

sue 0 1 1 0 0 0

tom 1 1 0 1 1 0

ITEMS

STUDENTS

1 2 3 4 5 6trig 1 1 0 0 1 0

story

0 1 1 0 0 0

hard 1 1 0 1 1 0

ITEMS

TAGS

Relational hierarchical Bayesian exponential-family

PCA

XX

VV

UU

X, Y: observed dataU: student latent factorsV: item latent factorsZ: tag latent factorsR, S, T: shared priors

n students

m items

k latent factors

k latent factors

observed

unobserved

RR

SS

p tags

YY

ZZk latent factors

TT

X ≈ f(UVT) Y ≈ g(VZT)

Example: brain imaging

2000 dictionary words

60 stimulus words

500 brain voxels

X = co-occurrence of (dictionary word, stimulus word) on web

Y = activation of voxel when presented with stimulus

Task: predict X

HB-EPCA

H-EPCA

EPCA

Relational versions

Mean squared error

credit: Ajit Singh

Temporal models

So far: latent factors of students and content

e.g., knowledge components

for student: skill at KC

for problem: need for KC

e.g., student affect

But limited idea of evolution through time

e.g., fixed-structure models: proficiency = a + B X, where x = # practice opportunities, A = initial skill level, b = skill learning rate

Temporal models

For evolving factors, we expect far better results if we learn about time explicitly

learning curves, gaming state, affective state, motivational state, self-efficacy, …

XX11XX11XX11LATENT STATE

PROPERTIES OF TRANSACTION

X1X1X1X1YY11

INSTRUCTIONAL DECISIONS X1X1X1X1UU11

TRANS. 1 TRANS. 2 TRANS. 3

XX11XX11XX22

X1X1X1X1YY22

X1X1X1X1UU22

XX11XX11XX33

X1X1X1X1YY33

X1X1X1X1UU33

Example: Bayesian Evaluation & Assessment

[BECK ET AL., 2008]

PROPERTIES OF TRANSACTIONS

LATENT STATE

INSTRUCTIONAL DECISIONS

The hope

Fit a temporal model

Examine learned parameters and latent states

Discover important evolving factors which affect performance

learning curve, affective state, gaming state, …

Discover how they evolve

The hope

Reduce assumptions about what the factors are

Explore a wider variety of models

Model search guided by data

⇒ discover factors we might otherwise have missed

Walking: original data

QuickTime™ and a decompressor

are needed to see this picture.

THANKS: BYRON BOOTS, SAJID

SIDDIQI

Walking: original data

THANKS: BYRON BOOTS, SAJID

SIDDIQI

XX11XX11XX11LATENT STATE

JOINT ANGLESX1X1X1X1YY11

DESIREDDIRECTION X1X1X1X1UU11

TRANS. 1 TRANS. 2 TRANS. 3

XX11XX11XX22

X1X1X1X1YY22

X1X1X1X1UU22

XX11XX11XX33

X1X1X1X1YY33

X1X1X1X1UU33

Walking: learned model

QuickTime™ and a decompressor

are needed to see this picture.

Steam: original data

QuickTime™ and a decompressor

are needed to see this picture.

Steam: original data

XX11XX11XX11LATENT STATE

PIXELSX1X1X1X1YY11

(EMPTY) X1X1X1X1UU11

TRANS. 1 TRANS. 2 TRANS. 3

XX11XX11XX22

X1X1X1X1YY22

X1X1X1X1UU22

XX11XX11XX33

X1X1X1X1YY33

X1X1X1X1UU33

Steam: learned model

QuickTime™ and a decompressor

are needed to see this picture.