Latent Variable / Hierarchical Models in Computational

Neural Science

Ying Nian WuUCLA Department of Statistics

March 30, 2011

Outline• Latent variable models in statistics• Primary visual cortex (V1)• Modeling and learning in V1 • Layered hierarchical models• Joint work with Song-Chun Zhu and Zhangzhang Si

Latent variable models

Hidden

Observed

Learning: Examples

Inference:

Latent variable models Mixture model

Factor analysis

Latent variable models

Hidden

Observed

Learning: Examples

Maximum likelihood

EM/gradient

Inference / explaining away

E-step / imputation

Computational neural science

Z: Internal representation by neurons

Y: Sensory data from outside environment

Hidden

Observed

Connection weights

Hierarchical extension: modeling Z by another layer of hidden variables explaining Y instead of Z

Inference / explaining away

Source: Scientific American, 1999

Visual cortex: layered hierarchical architecture

V1: primary visual cortex simple cells complex cells

bottom-up/top-down

1]}[2

1exp{)(

22

22

21

21 ixe

xxxG

Simple V1 cells Daugman, 1985

Gabor wavelets: localized sine and cosine waves

Transation, rotation, dilation of the above function

)'()'(,'

,,,, xBxIBIx

sxsx

image pixels

V1 simple cells

,,sxB

respond to edges

Complex V1 cells Riesenhuber and Poggio,1999

2,,)(),( |,|max sxxAx BI

Image pixels

V1 simple cells

V1 complex cells

Local max

Local sum

•Larger receptive field •Less sensitive to deformation

Independent Component Analysis Bell and Sejnowski, 1996

CBcBcI NN B ...11

Nicpci ,...,1tly independen )(~

)dim(IN

IIC AB 1

mNNmmm CBcBcI B ,11, ...

mmm IIC AB 1

Laplacian/Cauchy

Hyvarinen, 2000

Sparse coding Olshausen and Field, 1996

Laplacian/Cauchy/mixture Gaussians

Nicpci ,...,1tly independen )(~

NNBcBcI ...11

mNNmmm BcBcI ,11, ...)dim(IN

Inference: sparsification, non-linear lasso/basis pursuit/matching pursuit mode and uncertainty of p(C|I) explaining-away, lateral inhibition

Nicpci ,...,1tly independen )(~

Sparse coding / variable selection

Learning: mNNmmm BcBcI ,11, ...

)dim(IN

A dictionary of representational elements (regressors)

NNBcBcI ...11

Olshausen and Field, 1996

}exp{)(

1),(

,, j

jiiji vhW

WZVHp

Nihi ,...,1 ,

V

Restricted Boltzmann Machine Hinton, Osindero and Teh, 2006

P(V|H)P(H|V): factorized no-explaining away

hidden, binary

visible

Energy-based model Teh, Welling, Osindero and Hinton, 2003

)},(exp{),(

1)(

iiBIZ

Ip B

Features, no explaining-away

Maximum entropy with marginalsExponential family with sufficient stat

)},(exp{)(

1)(

,,,,,

sxsxs BI

ZIp

Zhu, Wu, and Mumford, 1997Wu, Liu, and Zhu, 2000

Markov random field/Gibbs distribution

Zhu, Wu, and Mumford, 1997Wu, Liu, and Zhu, 2000

Source: Scientific American, 1999

Visual cortex: layered hierarchical architecture

bottom-up/top-down

What is beyond V1?Hierarchical model?

Hierchical ICA/Energy-based model?

Larger featuresMust introduce nonlinearitiesPurely bottom-up

P(V,H) = P(H)P(V|H) P(H) P(V’,H)

I

H

V

V’

Discriminative correction by back-propagation

Unfolding, untying, re-learning

Hierarchical RBM Hinton, Osindero and Teh, 2006

Hierarchical sparse coding

NNBcBcI ...11

,,sxB

Attributed sparse coding elements transformation group topological neighborhood system

UBcIii sx

n

ii

,,

1

Layer above : further coding of the attributes of selected sparse coding elements

Active basis modelWu, Si, Gong, Zhu, 10Zhu, Guo, Wang, Xu, 05

n-stroke templaten = 40 to 60, box= 100x100

Active basis model Wu, Si, Gong, Zhu, 10Zhu, et al., 05

Yuille, Hallinan, Cohen, 92

n-stroke templaten = 40 to 60, box= 100x100

•Simplest AND-OR graph (Pearl, 84; Zhu, Mumford 06) AND composition and OR perturbations or variations of basis elements

•Simplest shape model: average + residual•Simplest modification of Olshausen-Field model•Further sparse coding of attributes of sparse coding elements

Simplicity

Bottom layer: sketch against texture

Only need to pool a marginal q(c) as null hypothesis • natural images explicit q(I) of Zhu, Mumford, 97• this image explicit q(I) of Zhu, Wu, Mumford, 97

Maximum entropy (Della Pietra, Della Pietra, Lafferty, 97; Zhu, Wu, Mumford, 97; Jin, S. Geman, 06; Wu, Guo, Zhu, 08) Special case: density substitution (Friedman, 87; Jin, S. Geman, 06)

p(C, U) = p(C) p(U|C) = p(C) q(U|C) = p(C) q(U,C)/q(C)

Shared sketch algorithm: maximum likelihood learning

Prototype: shared matching pursuit (closed-form computation)

Step 1: two max to explain images by maximum likelihood no early decision on edge detection Step 2: arg-max for inferring hidden variablesStep 3: arg-max explains away, thus inhibits (matching pursuit, Mallat, Zhang, 93)

Finding n strokes to sketch M images simultaneouslyn = 60, M = 9

Bottom-up sum-max scoring (no early edge decision)

Top-down arg-max sketching

1. Reinterpreting MAX1: OR-node of AND-OR, MAX for ARG-MAX in max-product algorithm2. Stick to Olshausen-Field sparse top-down model : AND-node of AND-OR Active basis, SUM2 layer, “neurons” memorize shapes by sparse connections to MAX1 layer Hierarchical, recursive AND-OR/ SUM-MAX Architecture: more top-down than bottom-up Neurons: more representational than operational (OR-neurons/AND-neurons)

Cortex-like sum-max maps: maximum likelihood inference

SUM1 layer: simple V1 cells of Olshausen, Field, 96MAX1 layer: complex V1 cells of Riesenhuber, Poggio, 99

Scan over multiple resolutions

Bottom-up detectionTop-down sketching

SUM1

MAX1

SUM2

arg MAX1

Sparse selective connection as a result of learningExplaining-away in learning but not in inference

Bottom-up scoring and top-down sketching

Scan over multiple resolutions and orientations (rotating template)

Classification based on log likelihood ratio score

Freund, Schapire, 95; Viola, Jones, 04

Adjusting Active Basis Model by L2 Regularized Logistic RegressionBy Ruixun Zhang

•Exponential family model, q(I) negatives Logistic regression for p(class | image), partial likelihood•Generative learning without negative examples basis elements and hidden variables•Discriminative adjustment with hugely reduced dimensionality correcting conditional independence assumption

L2 regularized logistic regressionre-estimated lambda’s

Conditional on: (1) selected basis elements (2) inferred hidden variables (1) and (2) generative learning

Active basis templates

Adaboost templates

# of negatives: 10556 7510 4552 1493 12217

• Arg-max inference and explaining away, no reweighting, • Residual images neutralize existing elements, same set of training examples

• No arg-max inference or explaining away inhibition• Reweighted examples neutralize existing classifiers, changing set of examples

double # elements

same # elements

Mixture model of active basis templates fitted by EM/maximum likelihood with random initialization

MNIST500 total

Learning active basis models from non-aligned imageEM-type maximum likelihood learning, Initialized by single image learning

Learning active basis models from non-aligned image

Learning active basis models from non-aligned image

Hierarchical active basis by Zhangzhang Si et al. •And-OR graph: Pearl, 84; Zhu, Mumford, 06•Compositionality and reusability: Geman, Potter, Chi, 02; L.Zhu, Lin, Huang, Chen,Yuille, 08•Part-based method: everyone et al. •Latent SVM: Felzenszwalb, McAllester, Ramanan, 08•Constellation model: Weber, Welling, Perona, 00

Lowlog-likelihood

Highlog-like

Simplicity

•Simplest and purest recursive two-layer AND-OR graph•Simplest generalization of active basis model

AND-OR graph and SUM-MAX mapsmaximum likelihood inference

Cortex-like, related to Riesenhuber, Poggio, 99•Bottom-up sum-max scoring•Top-down arg-max sketching

Hierarchical active basis by Zhangzhang Si et al.

Shape script by composing active basis shape motifsRepresenting elementary geometric shapes (shape motifs) by active bases (Si, Wu, 10) Geometry = sketch that can be parametrized

UBcIii sx

n

ii

,,

1

),...,1,,(motif shape),...,1,,( nixnix iik

kii

Bottom-layer: Olshausen-Field (foreground) + Zhu-Wu-Mumford (background) Maximum entropy tilting (Della Pietra, Della Pietra, Lafferty, 97) white noise texture (high entropy) sketch (low and mid entropy) (reverse the central limit theorem effect of information scaling)

Build up layers: (1) AND-OR, SUM-MAX (top-down arg-MAX) (2) Perpetual sparse coding: further coding of attributes of the current sparse coding elements (a) residuals of attributes continuous OR-nodes (b) mixture model discrete OR-nodes

Summary