ICA

Alphan Altinok

Outline

PCA ICA

Foundation Ambiguities Algorithms Examples Papers

PCA & ICA

PCA Projects d-dimensional data onto a lower dimensional subspace

in a way that is optimal in Σ|x0 – x|2.

ICA Seek directions in feature space such that resulting signals show

independence.

PCA

Compute d-dimensional μ (mean). Compute d x d covariance matrix. Compute eigenvectors and eigenvalues. Choose k largest eigenvalues.

k is the inherent dimensionality of the subspace governing the signal and (d – k) dimensions generally contain noise.

Form a d x k matrix A with k columns of eigenvalues. The representation of data by principal components

consists of projecting data into k-dimensional subspace by x = At (x – μ).

PCA A simple 3-layer neural network can form such a

representation when trained.

ICA

While PCA seeks directions that represents data best in a Σ|x0 – x|2 sense, ICA seeks such directions that are most independent from each other.

Used primarily for separating unknown source signals from their observed linear mixtures.

Typically used in Blind Source Separation problems. ICA is also used in feature extraction.

ICA – Foundation q source signals s1(k), s2(k), …, sq(k)

with 0 means k is the discrete time index or pixels in images scalar valued mutually independent for each value of k

h measured mixture signals x1(k), x2(k), …, xh(k)

Statistical independence for source signals p[s1(k), s2(k), …, sq(k)] = П p[si(k)]

ICA – Foundation The measured signals will be given by

xj(k) = Σsi(k)aij + nj(k)

For j = 1, 2, …, h, the elements aij are unknown.

Define vectors x(k) and s(k), and matrix A Observed: x(k) = [x1(k), x2(k), …, xh(k)]

Source: s(k) = [s1(k), s2(k), …, sq(k)]

Mixing matrix: A = [a1, a2, …, aq]

The equation above can be stated in vector-matrix form x(k) = As(k) + n(k) = Σsi(k)ai + n(k)

Ambiguities with ICA

The ICA expansion x(k) = As(k) + n(k) = Σsi(k)ai + n(k)

Amplitudes of separated signals cannot be determined.

There is a sign ambiguity associated with separated signals.

The order of separated signals cannot be determined.

ICA – Using NNs

Prewhitening – transform input vectors x(k) by v(k) = V x(k) Whitening matrix V can be obtained by NN or PCA

Separation (NN or contrast approximation) Estimation of ICA basis vectors (NN or batch approach)

ICA – Fast Fixed Point Algorithm

FFPA converges rapidly to the most accurate solution allowed by the data structure.

ICA – Example

ICA – Example

ICA – Example

ICA – Example

ICA – Example

ICA – Example BSS of recorded speech and music signals.

speech / music speech / speechspeech / speech in difficult environment

mic1 mic1 mic1

mic2 mic2 mic2

separated1 separated1 separated1

separated2 separated2 separated2

http://www.cnl.salk.edu/~tewon/ica_cnl.html

ICA – Example

Source images

separation demo

http://www.open.brain.riken.go.jp/demos/researchBSRed.html

ICA – Papers Hinton – A New View of ICA

Interprets ICA as a probability density model. Overcomplete, undercomplete, and multi-layer non-linear ICA

becomes simpler.

Cardoso – Blind Signal Separation, Statistical Principles Modelling identifiability. Contrast functions. Estimating functions. Adaptive algorithms. Performance issues.

ICA – Papers Hyvarinen – ICA Applied to Feature Extraction from

Color and Stereo Images Seeks to extend ICA by contrasting it to the processing done in

neural receptive fields.

Hyvarinen – Survey on ICA

Lawrence – Face Recognition, A Convolutional Neural Network Approach Combines local image sampling, a SOM, and a convolutional

NN that provides partial invariance to translation, rotation, scaling, and deformations.

ICA – Papers Sejnowski – Independent Component Representations for

Face Recognition

Sejnowski – A Comparison of Local vs Global Image Decompositions for Visual Speechreading

Bartlett – Viewpoint Invariant Face Recognition Using ICA and Attractor Networks

Bartlett – Image Representations for Facial Expression Coding

ICA – Links

http://sig.enst.fr/~cardoso/

http://www.cnl.salk.edu/~tewon/ica_cnl.html

http://nucleus.hut.fi/~aapo/

http://www.salk.edu/faculty/sejnowski.html