factor analysis and i-vectorsmwmak/papers/fa-beamer.pdf · wiki: \factor analysis is a statistical...

$: Factor Analysis and I-Vectorsmwmak/papers/FA-beamer.pdf · Wiki: \Factor analysis is a statistical method used to describe variability among observed, correlated variables in terms$
Factor Analysis and I-Vectors

Man-Wai MAK

Dept. of Electronic and Information Engineering,The Hong Kong Polytechnic University

[email protected]://www.eie.polyu.edu.hk/∼mwmak

References:

M.W. Mak and J.T. Chien, Machine Learning for Speaker Recognition, CambridgeUniversity Press, 2020.

S.J.D. Prince,Computer Vision: Models Learning and Inference, Cambridge UniversityPress, 2012

C. Bishop, ”Pattern Recognition and Machine Learning”, Appendix E, Springer, 2006.

http://www.eie.polyu.edu.hk/∼mwmak/papers/FA-Ivector.pdf

June 3, 2019

Man-Wai MAK (EIE) FA and I-Vectors June 3, 2019 1 / 42

Overview

1 Factor AnalysisWhat is Factor AnalysisFA versus PCAGenerative ModelEM Formulation

2 Applications of FA: I-VectorsGMM I-VectorsFactor Analysis ModelI-Vectors for Speaker and ECG Recognition


What is Factor Analysis?

Wiki: “Factor analysis is a statistical method used to describevariability among observed, correlated variables in terms of apotentially lower number of unobserved variables called factors.”

It was originally introduced by psychologists to find the underlyinglatent factors that account for the correlations among a set ofobservations or measures.

Example: The exam scores of 10 different subjects of 1,000 studentsmay be explained by two latent factors (also called common factors):

Language abilityMath ability

Each student has their own values for these two factors across all ofthe 10 subjects. His/her exam score for each subject is a linearweighted sum of these two factors plus the score mean and an error.



For each subject, all students share the same weights, which arereferred to as the factor loadings, for this subject.

.

.

.

.

.

.

.

.

.

.

.

.

Scoreof10subjectsofone

student

FactorLoadingMatrix Factors

Residue

.

.

.

Globalmean

60 5 1 2.5

1.0

73 0.5LanguageabilityMathability

History

Physics 1 66572 -1.5



Denote xi = [xi,1 · · · xi,10]T and zi = [zi,1 zi,2]T as the exam scores

and common factors of the i-th student, respectively. Also, denotem = [m1 · · · m10]

T as the mean exam scores of these 10 subjects.Then, we have

xi,j = mj+vj,1zi,1+vj,2zi,2+εi,j , i = 1, . . . , 1000 and j = 1, . . . , 10,(1)

where vj,1 and vj,2 are the factor loadings for the j-th subject and εi,jis an error term.Eq. 1 can also be written as:

xi =

m1

...m10

+

v1,1 v1,2

......

v10,1 v10,2

[zi,1zi,2

]+

εi,1

...εi,10

= m + Vzi + εi, i = 1, . . . , 1000,

(2)

where V is a 10× 2 matrix comprising the factor loadings of the 10subjects.Man-Wai MAK (EIE) FA and I-Vectors June 3, 2019 5 / 42

Example Usage of FA

For the exam score example, we may apply clustering on the factorszi, for i = 1, . . . , 1000.

We may identify the students who are weak in both math andlanguage.


FA versus PCA

Both PCA and FA are dimension reduction techniques. But they arefundamentally different.

Components in PCA are orthogonal to each other, whereas factoranalysis does not require the columns of the loading matrix to beorthogonal, i.e., the correlation between the factors could be non-zero.

PCA finds a linear combination of the observed variables to preservethe variance of the data, whereas FA predicts observed variables fromlatent factors.

PCA : yi = WT(xi −m) and x̂i = m + Wyi

FA : xi = m + Vzi + εi

where xi’s are observed vectors, yi’s are PCA-projected vectors oflower dimension, and zi’s are factors.


Why is FA Important?

Finance: Financial analysts use FA to find “what really driveperformance of underling assets”, allowing them to make betterinvestment decisions.

Social Science: FA reduces the number of variables to a smaller setof factors that facilitates our understanding of social problems.

Marketing: How change in price (factor) affect the change in thesales (observations)

Engineering: Many engineering applications need to determine thehidden factors that lead to the observations.


Generative Model of FA

Denote X = {x1, . . . ,xN} as a set of R-dimensional vectors. Infactor analysis, xi’s are assumed to follow a linear model:

xi = m + Vzi + εi i = 1, . . . , N (3)

where m is the global mean, V is a low-rank R×D matrix, zicomprises D latent factors with prior p(zi) = N (zi|0, I), and εi is theresidual following a Gaussian density with zero mean and covariancematrix Σ.

The marginal distribution of x is given by

p(x) =

∫p(x, z)dz =

∫p(x|z)p(z)dz

=

∫N (x|m + Vz,Σ)N (z|0, I)dz

= N (x|m,VVT + Σ)

(4)



Eq. 4 can be obtained by noting that p(x) is a Gaussian.

We take the expectation of x in Eq. 3 to obtain:

E{x} = m.

We take the expectation of (x−m)(x−m)T in Eq. 3 to obtain:

E{(x−m)(x−m)T} = E{(Vz + ε)(Vz + ε)T}= VE{zzT}VT + E{εεT}= VIVT + Σ

= VVT + Σ.

(5)

Eq. 4 and Eq. 5 suggest that x’s vary in a subspace of RD withvariability explained by the covariance matrix VVT. Any deviationsaway from this subspace are explained by Σ.



Showing p(x|z) = N (x|m + Vz,Σ)

The mean can be obtained by considering z deterministic (known):

E{x|z} = E{m + Vz + ε|z}= m + Vz + E{ε}= m + Vz

Given that the mean is m + Vz when we know z, the covariancematrix of p(x|z) is

E{(x−m−Vz)(x−m−Vz)T} = E{εεT}= Σ



xi zi

N

V

Σ

m

FA

✏i<latexit sha1_base64="6aqIrt8TgZI/OwoF7MkelEzD4Nc=">AAACunicfVFNbxMxEHW2fJTy1ZYjlxUREuIQ7SIkOHCogEMviCI1baXsKrK9s4kVf2HPto2s/R29lp/Fv8Gb7IGkiJFsP8280Ty/YVYKj1n2e5Ds3Lv/4OHuo73HT54+e75/cHjmTeM4jLmRxl0w6kEKDWMUKOHCOqCKSThniy9d/fwSnBdGn+LSQqnoTItacIoxVRaXUID1Qho9FdP9YTbKVpHeBXkPhqSPk+nB4KaoDG8UaOSSej/JM4tloA4Fl9DuFY0HS/mCzmASoaYKfBlWqtv0dcxUaW1cPBrTVfbvjkCV90vFIlNRnPvtWpf8V23SYP2xDELbBkHz9aC6kSmatLMgrYQDjnIZAeVORK0pn1NHOUajNqYwFf+g4YobpaiuQrSrneRlKJiRVafNyFB0AlgdhnnbbrH93FzVYrZqiaWvEE1y8C3yv1twFI17GwrqZopet6F//0cTek2L7+YgtG2IN1yjr8Np1BFXmW8v7i44ezfKs1H+4/3w6HO/1F3ykrwib0hOPpAjckxOyJhw8pPckFvyK/mUsEQkizU1GfQ9L8hGJPgHF3vf8w==</latexit><latexit sha1_base64="6aqIrt8TgZI/OwoF7MkelEzD4Nc=">AAACunicfVFNbxMxEHW2fJTy1ZYjlxUREuIQ7SIkOHCogEMviCI1baXsKrK9s4kVf2HPto2s/R29lp/Fv8Gb7IGkiJFsP8280Ty/YVYKj1n2e5Ds3Lv/4OHuo73HT54+e75/cHjmTeM4jLmRxl0w6kEKDWMUKOHCOqCKSThniy9d/fwSnBdGn+LSQqnoTItacIoxVRaXUID1Qho9FdP9YTbKVpHeBXkPhqSPk+nB4KaoDG8UaOSSej/JM4tloA4Fl9DuFY0HS/mCzmASoaYKfBlWqtv0dcxUaW1cPBrTVfbvjkCV90vFIlNRnPvtWpf8V23SYP2xDELbBkHz9aC6kSmatLMgrYQDjnIZAeVORK0pn1NHOUajNqYwFf+g4YobpaiuQrSrneRlKJiRVafNyFB0AlgdhnnbbrH93FzVYrZqiaWvEE1y8C3yv1twFI17GwrqZopet6F//0cTek2L7+YgtG2IN1yjr8Np1BFXmW8v7i44ezfKs1H+4/3w6HO/1F3ykrwib0hOPpAjckxOyJhw8pPckFvyK/mUsEQkizU1GfQ9L8hGJPgHF3vf8w==</latexit><latexit sha1_base64="6aqIrt8TgZI/OwoF7MkelEzD4Nc=">AAACunicfVFNbxMxEHW2fJTy1ZYjlxUREuIQ7SIkOHCogEMviCI1baXsKrK9s4kVf2HPto2s/R29lp/Fv8Gb7IGkiJFsP8280Ty/YVYKj1n2e5Ds3Lv/4OHuo73HT54+e75/cHjmTeM4jLmRxl0w6kEKDWMUKOHCOqCKSThniy9d/fwSnBdGn+LSQqnoTItacIoxVRaXUID1Qho9FdP9YTbKVpHeBXkPhqSPk+nB4KaoDG8UaOSSej/JM4tloA4Fl9DuFY0HS/mCzmASoaYKfBlWqtv0dcxUaW1cPBrTVfbvjkCV90vFIlNRnPvtWpf8V23SYP2xDELbBkHz9aC6kSmatLMgrYQDjnIZAeVORK0pn1NHOUajNqYwFf+g4YobpaiuQrSrneRlKJiRVafNyFB0AlgdhnnbbrH93FzVYrZqiaWvEE1y8C3yv1twFI17GwrqZopet6F//0cTek2L7+YgtG2IN1yjr8Np1BFXmW8v7i44ezfKs1H+4/3w6HO/1F3ykrwib0hOPpAjckxOyJhw8pPckFvyK/mUsEQkizU1GfQ9L8hGJPgHF3vf8w==</latexit><latexit sha1_base64="6aqIrt8TgZI/OwoF7MkelEzD4Nc=">AAACunicfVFNbxMxEHW2fJTy1ZYjlxUREuIQ7SIkOHCogEMviCI1baXsKrK9s4kVf2HPto2s/R29lp/Fv8Gb7IGkiJFsP8280Ty/YVYKj1n2e5Ds3Lv/4OHuo73HT54+e75/cHjmTeM4jLmRxl0w6kEKDWMUKOHCOqCKSThniy9d/fwSnBdGn+LSQqnoTItacIoxVRaXUID1Qho9FdP9YTbKVpHeBXkPhqSPk+nB4KaoDG8UaOSSej/JM4tloA4Fl9DuFY0HS/mCzmASoaYKfBlWqtv0dcxUaW1cPBrTVfbvjkCV90vFIlNRnPvtWpf8V23SYP2xDELbBkHz9aC6kSmatLMgrYQDjnIZAeVORK0pn1NHOUajNqYwFf+g4YobpaiuQrSrneRlKJiRVafNyFB0AlgdhnnbbrH93FzVYrZqiaWvEE1y8C3yv1twFI17GwrqZopet6F//0cTek2L7+YgtG2IN1yjr8Np1BFXmW8v7i44ezfKs1H+4/3w6HO/1F3ykrwib0hOPpAjckxOyJhw8pPckFvyK/mUsEQkizU1GfQ9L8hGJPgHF3vf8w==</latexit>

Figure: Graphical model of factor analysis.

xi = m + Vzi + εi i = 1, . . . , N



V

m

z

Cov matrix:

The column vectors in V define the directions (subspace) in whichthe data are most correlated.

The covariance matrix Σ defines the remaining variation that cannotbe captured by VVT.


Posterior Density of Latent Factor

V = [v1 v2]<latexit sha1_base64="kUt/k8ZrtUBMqP/OkfLwxn7SG2w=">AAADX3icfVFdb9MwFHVaYCOwrYMnxItFValDaGoGEjxQaQIeeOBjiLWbVIfKcZzWmh1HtlNajPdbeOM38cg/wWmDtK6IK8X35J5z77V8koIzbXq9X0GjeePmra3t2+Gduzu7e639e0MtS0XogEgu1XmCNeUspwPDDKfnhaJYJJyeJRevK/5sRpVmMj81i4LGAk9yljGCjS+NWz876J1Xpxj2IeI0M8iigo3JF9tNDtwTNKNIlGu/n9lE4L8VpNhkapALO56C375X5xxpJiwimMMPrltPsFf45RxYD1ojDsIqD/ujKs3G0eXlChzF41a7d9hbBtwEUQ3aoI6T8X7wA6WSlILmhnCs9SjqFSa2WBlGOHUhKjUtMLnAEzryMMeC6tguX9TBjq+kMJPKf7mBy+rVDouF1guReKXAZqqvc1XxX9yoNNmL2LK8KA3NyWpRVnJoJKzsgSlTlBi+8AATxfxdIZlihYnxJoYop1+JFALnafVubhTFFiWSp9VdJLeoWphkth0559bVpnDWn3RudGZPN2hRDZivBmq/uDDaLLh3qepQYjkxRG+of0pF3/stHwuqsJHqsUVYeSPnztb5fzKWr2Q+h6H3NLru4CYYHh1GHn961j5+Vbu7DR6CR6ALIvAcHIO34AQMAAl2gqfBy6Df+N3cau42WytpI6h77oO1aD74A8uYGDk=</latexit><latexit sha1_base64="kUt/k8ZrtUBMqP/OkfLwxn7SG2w=">AAADX3icfVFdb9MwFHVaYCOwrYMnxItFValDaGoGEjxQaQIeeOBjiLWbVIfKcZzWmh1HtlNajPdbeOM38cg/wWmDtK6IK8X35J5z77V8koIzbXq9X0GjeePmra3t2+Gduzu7e639e0MtS0XogEgu1XmCNeUspwPDDKfnhaJYJJyeJRevK/5sRpVmMj81i4LGAk9yljGCjS+NWz876J1Xpxj2IeI0M8iigo3JF9tNDtwTNKNIlGu/n9lE4L8VpNhkapALO56C375X5xxpJiwimMMPrltPsFf45RxYD1ojDsIqD/ujKs3G0eXlChzF41a7d9hbBtwEUQ3aoI6T8X7wA6WSlILmhnCs9SjqFSa2WBlGOHUhKjUtMLnAEzryMMeC6tguX9TBjq+kMJPKf7mBy+rVDouF1guReKXAZqqvc1XxX9yoNNmL2LK8KA3NyWpRVnJoJKzsgSlTlBi+8AATxfxdIZlihYnxJoYop1+JFALnafVubhTFFiWSp9VdJLeoWphkth0559bVpnDWn3RudGZPN2hRDZivBmq/uDDaLLh3qepQYjkxRG+of0pF3/stHwuqsJHqsUVYeSPnztb5fzKWr2Q+h6H3NLru4CYYHh1GHn961j5+Vbu7DR6CR6ALIvAcHIO34AQMAAl2gqfBy6Df+N3cau42WytpI6h77oO1aD74A8uYGDk=</latexit><latexit sha1_base64="kUt/k8ZrtUBMqP/OkfLwxn7SG2w=">AAADX3icfVFdb9MwFHVaYCOwrYMnxItFValDaGoGEjxQaQIeeOBjiLWbVIfKcZzWmh1HtlNajPdbeOM38cg/wWmDtK6IK8X35J5z77V8koIzbXq9X0GjeePmra3t2+Gduzu7e639e0MtS0XogEgu1XmCNeUspwPDDKfnhaJYJJyeJRevK/5sRpVmMj81i4LGAk9yljGCjS+NWz876J1Xpxj2IeI0M8iigo3JF9tNDtwTNKNIlGu/n9lE4L8VpNhkapALO56C375X5xxpJiwimMMPrltPsFf45RxYD1ojDsIqD/ujKs3G0eXlChzF41a7d9hbBtwEUQ3aoI6T8X7wA6WSlILmhnCs9SjqFSa2WBlGOHUhKjUtMLnAEzryMMeC6tguX9TBjq+kMJPKf7mBy+rVDouF1guReKXAZqqvc1XxX9yoNNmL2LK8KA3NyWpRVnJoJKzsgSlTlBi+8AATxfxdIZlihYnxJoYop1+JFALnafVubhTFFiWSp9VdJLeoWphkth0559bVpnDWn3RudGZPN2hRDZivBmq/uDDaLLh3qepQYjkxRG+of0pF3/stHwuqsJHqsUVYeSPnztb5fzKWr2Q+h6H3NLru4CYYHh1GHn961j5+Vbu7DR6CR6ALIvAcHIO34AQMAAl2gqfBy6Df+N3cau42WytpI6h77oO1aD74A8uYGDk=</latexit><latexit sha1_base64="kUt/k8ZrtUBMqP/OkfLwxn7SG2w=">AAADX3icfVFdb9MwFHVaYCOwrYMnxItFValDaGoGEjxQaQIeeOBjiLWbVIfKcZzWmh1HtlNajPdbeOM38cg/wWmDtK6IK8X35J5z77V8koIzbXq9X0GjeePmra3t2+Gduzu7e639e0MtS0XogEgu1XmCNeUspwPDDKfnhaJYJJyeJRevK/5sRpVmMj81i4LGAk9yljGCjS+NWz876J1Xpxj2IeI0M8iigo3JF9tNDtwTNKNIlGu/n9lE4L8VpNhkapALO56C375X5xxpJiwimMMPrltPsFf45RxYD1ojDsIqD/ujKs3G0eXlChzF41a7d9hbBtwEUQ3aoI6T8X7wA6WSlILmhnCs9SjqFSa2WBlGOHUhKjUtMLnAEzryMMeC6tguX9TBjq+kMJPKf7mBy+rVDouF1guReKXAZqqvc1XxX9yoNNmL2LK8KA3NyWpRVnJoJKzsgSlTlBi+8AATxfxdIZlihYnxJoYop1+JFALnafVubhTFFiWSp9VdJLeoWphkth0559bVpnDWn3RudGZPN2hRDZivBmq/uDDaLLh3qepQYjkxRG+of0pF3/stHwuqsJHqsUVYeSPnztb5fzKWr2Q+h6H3NLru4CYYHh1GHn961j5+Vbu7DR6CR6ALIvAcHIO34AQMAAl2gqfBy6Df+N3cau42WytpI6h77oO1aD74A8uYGDk=</latexit>

v1<latexit sha1_base64="xTX6u2XN05yTaRFRJqcoSnK6ZFs=">AAADaHicfVHrbtMwFHZWLiPcOpBAiD/WqkodQlMzIcEPJk3cxA8uQ6zdpDpUjuO01uw4sp3SYrxn4RF4HV6Bp8BpgrSuiCPF58v5zvmO5S8pONOm3/8VbLQuXb5ydfNaeP3GzVu321t3hlqWitABkVyqkwRryllOB4YZTk8KRbFIOD1OTl9W/PGMKs1kfmQWBY0FnuQsYwQbXxq3f3bRO9+dYrgPEaeZQRYVbEy+2F6y4x6jGUWiXPn9zCYC/60gxSZTg1zY9RT89r0650gzYRHBHH5wvUbBnuOXOrARWiF2ap3h/qhKs3F0dlaDvThsKuN2p7/bXwZcB1EDOqCJw/FW8AOlkpSC5oZwrPUo6hcmtlgZRjh1ISo1LTA5xRM68jDHgurYLp/Wwa6vpDCTyn+5gcvq+QmLhdYLkfhOgc1UX+Sq4r+4UWmyZ7FleVEampN6UVZyaCSsfIIpU5QYvvAAE8X8XSGZYoWJ8W6GKKdfiRQC52n1gG4UxRYlkqfVXSS3qFqYZLYTOedWu03hrD/p3OjMHq3RohKY14LaLy6MNgvu7aomlFgqhugV9U+p6Hu/5WNBFTZSPbIIK+/o3Nkm/6+N5XWbz2HoPY0uOrgOhnu7kcefnnQOXjTuboKHYBv0QASeggPwFhyCASDBveB58Dp4s/G71W7dbz2oWzeCZuYuWInW9h85Ahsf</latexit><latexit sha1_base64="xTX6u2XN05yTaRFRJqcoSnK6ZFs=">AAADaHicfVHrbtMwFHZWLiPcOpBAiD/WqkodQlMzIcEPJk3cxA8uQ6zdpDpUjuO01uw4sp3SYrxn4RF4HV6Bp8BpgrSuiCPF58v5zvmO5S8pONOm3/8VbLQuXb5ydfNaeP3GzVu321t3hlqWitABkVyqkwRryllOB4YZTk8KRbFIOD1OTl9W/PGMKs1kfmQWBY0FnuQsYwQbXxq3f3bRO9+dYrgPEaeZQRYVbEy+2F6y4x6jGUWiXPn9zCYC/60gxSZTg1zY9RT89r0650gzYRHBHH5wvUbBnuOXOrARWiF2ap3h/qhKs3F0dlaDvThsKuN2p7/bXwZcB1EDOqCJw/FW8AOlkpSC5oZwrPUo6hcmtlgZRjh1ISo1LTA5xRM68jDHgurYLp/Wwa6vpDCTyn+5gcvq+QmLhdYLkfhOgc1UX+Sq4r+4UWmyZ7FleVEampN6UVZyaCSsfIIpU5QYvvAAE8X8XSGZYoWJ8W6GKKdfiRQC52n1gG4UxRYlkqfVXSS3qFqYZLYTOedWu03hrD/p3OjMHq3RohKY14LaLy6MNgvu7aomlFgqhugV9U+p6Hu/5WNBFTZSPbIIK+/o3Nkm/6+N5XWbz2HoPY0uOrgOhnu7kcefnnQOXjTuboKHYBv0QASeggPwFhyCASDBveB58Dp4s/G71W7dbz2oWzeCZuYuWInW9h85Ahsf</latexit><latexit sha1_base64="xTX6u2XN05yTaRFRJqcoSnK6ZFs=">AAADaHicfVHrbtMwFHZWLiPcOpBAiD/WqkodQlMzIcEPJk3cxA8uQ6zdpDpUjuO01uw4sp3SYrxn4RF4HV6Bp8BpgrSuiCPF58v5zvmO5S8pONOm3/8VbLQuXb5ydfNaeP3GzVu321t3hlqWitABkVyqkwRryllOB4YZTk8KRbFIOD1OTl9W/PGMKs1kfmQWBY0FnuQsYwQbXxq3f3bRO9+dYrgPEaeZQRYVbEy+2F6y4x6jGUWiXPn9zCYC/60gxSZTg1zY9RT89r0650gzYRHBHH5wvUbBnuOXOrARWiF2ap3h/qhKs3F0dlaDvThsKuN2p7/bXwZcB1EDOqCJw/FW8AOlkpSC5oZwrPUo6hcmtlgZRjh1ISo1LTA5xRM68jDHgurYLp/Wwa6vpDCTyn+5gcvq+QmLhdYLkfhOgc1UX+Sq4r+4UWmyZ7FleVEampN6UVZyaCSsfIIpU5QYvvAAE8X8XSGZYoWJ8W6GKKdfiRQC52n1gG4UxRYlkqfVXSS3qFqYZLYTOedWu03hrD/p3OjMHq3RohKY14LaLy6MNgvu7aomlFgqhugV9U+p6Hu/5WNBFTZSPbIIK+/o3Nkm/6+N5XWbz2HoPY0uOrgOhnu7kcefnnQOXjTuboKHYBv0QASeggPwFhyCASDBveB58Dp4s/G71W7dbz2oWzeCZuYuWInW9h85Ahsf</latexit><latexit sha1_base64="xTX6u2XN05yTaRFRJqcoSnK6ZFs=">AAADaHicfVHrbtMwFHZWLiPcOpBAiD/WqkodQlMzIcEPJk3cxA8uQ6zdpDpUjuO01uw4sp3SYrxn4RF4HV6Bp8BpgrSuiCPF58v5zvmO5S8pONOm3/8VbLQuXb5ydfNaeP3GzVu321t3hlqWitABkVyqkwRryllOB4YZTk8KRbFIOD1OTl9W/PGMKs1kfmQWBY0FnuQsYwQbXxq3f3bRO9+dYrgPEaeZQRYVbEy+2F6y4x6jGUWiXPn9zCYC/60gxSZTg1zY9RT89r0650gzYRHBHH5wvUbBnuOXOrARWiF2ap3h/qhKs3F0dlaDvThsKuN2p7/bXwZcB1EDOqCJw/FW8AOlkpSC5oZwrPUo6hcmtlgZRjh1ISo1LTA5xRM68jDHgurYLp/Wwa6vpDCTyn+5gcvq+QmLhdYLkfhOgc1UX+Sq4r+4UWmyZ7FleVEampN6UVZyaCSsfIIpU5QYvvAAE8X8XSGZYoWJ8W6GKKdfiRQC52n1gG4UxRYlkqfVXSS3qFqYZLYTOedWu03hrD/p3OjMHq3RohKY14LaLy6MNgvu7aomlFgqhugV9U+p6Hu/5WNBFTZSPbIIK+/o3Nkm/6+N5XWbz2HoPY0uOrgOhnu7kcefnnQOXjTuboKHYBv0QASeggPwFhyCASDBveB58Dp4s/G71W7dbz2oWzeCZuYuWInW9h85Ahsf</latexit>

v2<latexit sha1_base64="3b9rdv5bDqbejxQDbrDOMR/SeeY=">AAADaHicfVHrbtMwFHZWLiPcOpBAiD/WqkodQlMzIcEPJk3cxA8uQ6zdpDpUjuO01uw4sp3SYrxn4RF4HV6Bp8BpgrSuiCPF58v5zvmO5S8pONOm3/8VbLQuXb5ydfNaeP3GzVu321t3hlqWitABkVyqkwRryllOB4YZTk8KRbFIOD1OTl9W/PGMKs1kfmQWBY0FnuQsYwQbXxq3f3bRO9+dYrgPEaeZQRYVbEy+2F6y4x6jGUWiXPn9zCYC/60gxSZTg1zY9RT89r0650gzYRHBHH5wvUbBnuOXOrARWiF2ap3h/qhKs3F0dlaDvThswLjd6e/2lwHXQdSADmjicLwV/ECpJKWguSEcaz2K+oWJLVaGEU5diEpNC0xO8YSOPMyxoDq2y6d1sOsrKcyk8l9u4LJ6fsJiofVCJL5TYDPVF7mq+C9uVJrsWWxZXpSG5qRelJUcGgkrn2DKFCWGLzzARDF/V0imWGFivJshyulXIoXAeVo9oBtFsUWJ5Gl1F8ktqhYmme1EzrnVblM46086NzqzR2u0qATmtaD2iwujzYJ7u6oJJZaKIXpF/VMq+t5v+VhQhY1UjyzCyjs6d7bJ/2tjed3mcxh6T6OLDq6D4d5u5PGnJ52DF427m+Ah2AY9EIGn4AC8BYdgAEhwL3gevA7ebPxutVv3Ww/q1o2gmbkLVqK1/Qc7VRsg</latexit><latexit sha1_base64="3b9rdv5bDqbejxQDbrDOMR/SeeY=">AAADaHicfVHrbtMwFHZWLiPcOpBAiD/WqkodQlMzIcEPJk3cxA8uQ6zdpDpUjuO01uw4sp3SYrxn4RF4HV6Bp8BpgrSuiCPF58v5zvmO5S8pONOm3/8VbLQuXb5ydfNaeP3GzVu321t3hlqWitABkVyqkwRryllOB4YZTk8KRbFIOD1OTl9W/PGMKs1kfmQWBY0FnuQsYwQbXxq3f3bRO9+dYrgPEaeZQRYVbEy+2F6y4x6jGUWiXPn9zCYC/60gxSZTg1zY9RT89r0650gzYRHBHH5wvUbBnuOXOrARWiF2ap3h/qhKs3F0dlaDvThswLjd6e/2lwHXQdSADmjicLwV/ECpJKWguSEcaz2K+oWJLVaGEU5diEpNC0xO8YSOPMyxoDq2y6d1sOsrKcyk8l9u4LJ6fsJiofVCJL5TYDPVF7mq+C9uVJrsWWxZXpSG5qRelJUcGgkrn2DKFCWGLzzARDF/V0imWGFivJshyulXIoXAeVo9oBtFsUWJ5Gl1F8ktqhYmme1EzrnVblM46086NzqzR2u0qATmtaD2iwujzYJ7u6oJJZaKIXpF/VMq+t5v+VhQhY1UjyzCyjs6d7bJ/2tjed3mcxh6T6OLDq6D4d5u5PGnJ52DF427m+Ah2AY9EIGn4AC8BYdgAEhwL3gevA7ebPxutVv3Ww/q1o2gmbkLVqK1/Qc7VRsg</latexit><latexit sha1_base64="3b9rdv5bDqbejxQDbrDOMR/SeeY=">AAADaHicfVHrbtMwFHZWLiPcOpBAiD/WqkodQlMzIcEPJk3cxA8uQ6zdpDpUjuO01uw4sp3SYrxn4RF4HV6Bp8BpgrSuiCPF58v5zvmO5S8pONOm3/8VbLQuXb5ydfNaeP3GzVu321t3hlqWitABkVyqkwRryllOB4YZTk8KRbFIOD1OTl9W/PGMKs1kfmQWBY0FnuQsYwQbXxq3f3bRO9+dYrgPEaeZQRYVbEy+2F6y4x6jGUWiXPn9zCYC/60gxSZTg1zY9RT89r0650gzYRHBHH5wvUbBnuOXOrARWiF2ap3h/qhKs3F0dlaDvThswLjd6e/2lwHXQdSADmjicLwV/ECpJKWguSEcaz2K+oWJLVaGEU5diEpNC0xO8YSOPMyxoDq2y6d1sOsrKcyk8l9u4LJ6fsJiofVCJL5TYDPVF7mq+C9uVJrsWWxZXpSG5qRelJUcGgkrn2DKFCWGLzzARDF/V0imWGFivJshyulXIoXAeVo9oBtFsUWJ5Gl1F8ktqhYmme1EzrnVblM46086NzqzR2u0qATmtaD2iwujzYJ7u6oJJZaKIXpF/VMq+t5v+VhQhY1UjyzCyjs6d7bJ/2tjed3mcxh6T6OLDq6D4d5u5PGnJ52DF427m+Ah2AY9EIGn4AC8BYdgAEhwL3gevA7ebPxutVv3Ww/q1o2gmbkLVqK1/Qc7VRsg</latexit><latexit sha1_base64="3b9rdv5bDqbejxQDbrDOMR/SeeY=">AAADaHicfVHrbtMwFHZWLiPcOpBAiD/WqkodQlMzIcEPJk3cxA8uQ6zdpDpUjuO01uw4sp3SYrxn4RF4HV6Bp8BpgrSuiCPF58v5zvmO5S8pONOm3/8VbLQuXb5ydfNaeP3GzVu321t3hlqWitABkVyqkwRryllOB4YZTk8KRbFIOD1OTl9W/PGMKs1kfmQWBY0FnuQsYwQbXxq3f3bRO9+dYrgPEaeZQRYVbEy+2F6y4x6jGUWiXPn9zCYC/60gxSZTg1zY9RT89r0650gzYRHBHH5wvUbBnuOXOrARWiF2ap3h/qhKs3F0dlaDvThswLjd6e/2lwHXQdSADmjicLwV/ECpJKWguSEcaz2K+oWJLVaGEU5diEpNC0xO8YSOPMyxoDq2y6d1sOsrKcyk8l9u4LJ6fsJiofVCJL5TYDPVF7mq+C9uVJrsWWxZXpSG5qRelJUcGgkrn2DKFCWGLzzARDF/V0imWGFivJshyulXIoXAeVo9oBtFsUWJ5Gl1F8ktqhYmme1EzrnVblM46086NzqzR2u0qATmtaD2iwujzYJ7u6oJJZaKIXpF/VMq+t5v+VhQhY1UjyzCyjs6d7bJ/2tjed3mcxh6T6OLDq6D4d5u5PGnJ52DF427m+Ah2AY9EIGn4AC8BYdgAEhwL3gevA7ebPxutVv3Ww/q1o2gmbkLVqK1/Qc7VRsg</latexit>

z1<latexit sha1_base64="hktfPEZtKLtq+gcpBVbOtequIgY=">AAADZHicfVHtbtMwFHVaPkYY0G3iFxKyqCp1CE3NhMT+TBqDH/zgY4i1m1SHyHGc1pqdRLZT2hnvWXgAXogX4DlwmiCtK+JK8T25595zLZ+44EzpweCX12rfun3n7sY9//7mg4ePOlvbI5WXktAhyXkuz2OsKGcZHWqmOT0vJMUi5vQsvnhT8WczKhXLs1O9KGgo8CRjKSNYu1LU+dlD7113guEhRJymGhlUsIh8Nf14175AM4pEufL7hU0E/ltBkk2mGlm/5yh4+b0650gxYRDBHH60/UbBXOOXOrARWiF2a53R4bhKsyi4uqrBfuhfRkHU6Q72BsuA6yBoQBc0cRJteT9QkpNS0EwTjpUaB4NChwZLzQin1kelogUmF3hCxw5mWFAVmuWzWthzlQSmuXRfpuGyen3CYKHUQsSuU2A9VTe5qvgvblzq9CA0LCtKTTNSL0pLDnUOK49gwiQlmi8cwEQyd1dIplhiop2TPsroN5ILgbOkejw7DkKD4pwn1V1yblC1ME5NN7DWrnbrwhp30rlWqTldo0UlMK8FlVtcaKUX3FlVTUixVPTRW+qeUtIPbsungkqsc/ncICydm3Nrmvy/NpbVbS77vvM0uOngOhjt7wUOf37ZPTpu3N0AT8Az0AcBeAWOwDtwAoaAeNvegffaO279bm+2d9qP69aW18zsgJVoP/0DkKQZpA==</latexit><latexit sha1_base64="hktfPEZtKLtq+gcpBVbOtequIgY=">AAADZHicfVHtbtMwFHVaPkYY0G3iFxKyqCp1CE3NhMT+TBqDH/zgY4i1m1SHyHGc1pqdRLZT2hnvWXgAXogX4DlwmiCtK+JK8T25595zLZ+44EzpweCX12rfun3n7sY9//7mg4ePOlvbI5WXktAhyXkuz2OsKGcZHWqmOT0vJMUi5vQsvnhT8WczKhXLs1O9KGgo8CRjKSNYu1LU+dlD7113guEhRJymGhlUsIh8Nf14175AM4pEufL7hU0E/ltBkk2mGlm/5yh4+b0650gxYRDBHH60/UbBXOOXOrARWiF2a53R4bhKsyi4uqrBfuhfRkHU6Q72BsuA6yBoQBc0cRJteT9QkpNS0EwTjpUaB4NChwZLzQin1kelogUmF3hCxw5mWFAVmuWzWthzlQSmuXRfpuGyen3CYKHUQsSuU2A9VTe5qvgvblzq9CA0LCtKTTNSL0pLDnUOK49gwiQlmi8cwEQyd1dIplhiop2TPsroN5ILgbOkejw7DkKD4pwn1V1yblC1ME5NN7DWrnbrwhp30rlWqTldo0UlMK8FlVtcaKUX3FlVTUixVPTRW+qeUtIPbsungkqsc/ncICydm3Nrmvy/NpbVbS77vvM0uOngOhjt7wUOf37ZPTpu3N0AT8Az0AcBeAWOwDtwAoaAeNvegffaO279bm+2d9qP69aW18zsgJVoP/0DkKQZpA==</latexit><latexit sha1_base64="hktfPEZtKLtq+gcpBVbOtequIgY=">AAADZHicfVHtbtMwFHVaPkYY0G3iFxKyqCp1CE3NhMT+TBqDH/zgY4i1m1SHyHGc1pqdRLZT2hnvWXgAXogX4DlwmiCtK+JK8T25595zLZ+44EzpweCX12rfun3n7sY9//7mg4ePOlvbI5WXktAhyXkuz2OsKGcZHWqmOT0vJMUi5vQsvnhT8WczKhXLs1O9KGgo8CRjKSNYu1LU+dlD7113guEhRJymGhlUsIh8Nf14175AM4pEufL7hU0E/ltBkk2mGlm/5yh4+b0650gxYRDBHH60/UbBXOOXOrARWiF2a53R4bhKsyi4uqrBfuhfRkHU6Q72BsuA6yBoQBc0cRJteT9QkpNS0EwTjpUaB4NChwZLzQin1kelogUmF3hCxw5mWFAVmuWzWthzlQSmuXRfpuGyen3CYKHUQsSuU2A9VTe5qvgvblzq9CA0LCtKTTNSL0pLDnUOK49gwiQlmi8cwEQyd1dIplhiop2TPsroN5ILgbOkejw7DkKD4pwn1V1yblC1ME5NN7DWrnbrwhp30rlWqTldo0UlMK8FlVtcaKUX3FlVTUixVPTRW+qeUtIPbsungkqsc/ncICydm3Nrmvy/NpbVbS77vvM0uOngOhjt7wUOf37ZPTpu3N0AT8Az0AcBeAWOwDtwAoaAeNvegffaO279bm+2d9qP69aW18zsgJVoP/0DkKQZpA==</latexit><latexit sha1_base64="hktfPEZtKLtq+gcpBVbOtequIgY=">AAADZHicfVHtbtMwFHVaPkYY0G3iFxKyqCp1CE3NhMT+TBqDH/zgY4i1m1SHyHGc1pqdRLZT2hnvWXgAXogX4DlwmiCtK+JK8T25595zLZ+44EzpweCX12rfun3n7sY9//7mg4ePOlvbI5WXktAhyXkuz2OsKGcZHWqmOT0vJMUi5vQsvnhT8WczKhXLs1O9KGgo8CRjKSNYu1LU+dlD7113guEhRJymGhlUsIh8Nf14175AM4pEufL7hU0E/ltBkk2mGlm/5yh4+b0650gxYRDBHH60/UbBXOOXOrARWiF2a53R4bhKsyi4uqrBfuhfRkHU6Q72BsuA6yBoQBc0cRJteT9QkpNS0EwTjpUaB4NChwZLzQin1kelogUmF3hCxw5mWFAVmuWzWthzlQSmuXRfpuGyen3CYKHUQsSuU2A9VTe5qvgvblzq9CA0LCtKTTNSL0pLDnUOK49gwiQlmi8cwEQyd1dIplhiop2TPsroN5ILgbOkejw7DkKD4pwn1V1yblC1ME5NN7DWrnbrwhp30rlWqTldo0UlMK8FlVtcaKUX3FlVTUixVPTRW+qeUtIPbsungkqsc/ncICydm3Nrmvy/NpbVbS77vvM0uOngOhjt7wUOf37ZPTpu3N0AT8Az0AcBeAWOwDtwAoaAeNvegffaO279bm+2d9qP69aW18zsgJVoP/0DkKQZpA==</latexit>

z2<latexit sha1_base64="ojUAhf9y5bj1vGNw/Rb5IZA9xrI=">AAADZHicfVHtbtMwFHVWPkYY0G3iFxKyqCp1CE1NhcT+TBqDH/zgY4i1m1SHyHGc1prtRLZT2pnsWXgAXogX4DlwmiCtK+JK8T25595zLZ8450ybfv+Xt9G6dfvO3c17/v2tBw8ftbd3RjorFKFDkvFMncdYU84kHRpmOD3PFcUi5vQsvnhT8WczqjTL5KlZ5DQUeCJZygg2rhS1f3bRe9edYHgIEaepQRblLCJfbS/eK1+gGUWiWPn9wiYC/60gxSZTg0q/6yh4+b0650gzYRHBHH4se42CvcYvdWAjtELs1Tqjw3GVZlFwdVWDQehfRoOo3env95cB10HQgA5o4iTa9n6gJCOFoNIQjrUeB/3chBYrwwinpY8KTXNMLvCEjh2UWFAd2uWzlrDrKglMM+U+aeCyen3CYqH1QsSuU2Az1Te5qvgvblyY9CC0TOaFoZLUi9KCQ5PByiOYMEWJ4QsHMFHM3RWSKVaYGOekjyT9RjIhsEyqxyvHQWhRnPGkukvGLaoWxqntBGVZrnabvLTupHOjU3u6RotKYF4Larc4N9osuLOqmlBiqeijt9Q9paIf3JZPOVXYZOq5RVg5N+elbfL/2pis21z2fedpcNPBdTAa7AcOf37ZOTpu3N0ET8Az0AMBeAWOwDtwAoaAeDvegffaO9743dpq7bYe160bXjOzC1ai9fQPkvcZpQ==</latexit><latexit sha1_base64="ojUAhf9y5bj1vGNw/Rb5IZA9xrI=">AAADZHicfVHtbtMwFHVWPkYY0G3iFxKyqCp1CE1NhcT+TBqDH/zgY4i1m1SHyHGc1prtRLZT2pnsWXgAXogX4DlwmiCtK+JK8T25595zLZ8450ybfv+Xt9G6dfvO3c17/v2tBw8ftbd3RjorFKFDkvFMncdYU84kHRpmOD3PFcUi5vQsvnhT8WczqjTL5KlZ5DQUeCJZygg2rhS1f3bRe9edYHgIEaepQRblLCJfbS/eK1+gGUWiWPn9wiYC/60gxSZTg0q/6yh4+b0650gzYRHBHH4se42CvcYvdWAjtELs1Tqjw3GVZlFwdVWDQehfRoOo3env95cB10HQgA5o4iTa9n6gJCOFoNIQjrUeB/3chBYrwwinpY8KTXNMLvCEjh2UWFAd2uWzlrDrKglMM+U+aeCyen3CYqH1QsSuU2Az1Te5qvgvblyY9CC0TOaFoZLUi9KCQ5PByiOYMEWJ4QsHMFHM3RWSKVaYGOekjyT9RjIhsEyqxyvHQWhRnPGkukvGLaoWxqntBGVZrnabvLTupHOjU3u6RotKYF4Larc4N9osuLOqmlBiqeijt9Q9paIf3JZPOVXYZOq5RVg5N+elbfL/2pis21z2fedpcNPBdTAa7AcOf37ZOTpu3N0ET8Az0AMBeAWOwDtwAoaAeDvegffaO9743dpq7bYe160bXjOzC1ai9fQPkvcZpQ==</latexit><latexit sha1_base64="ojUAhf9y5bj1vGNw/Rb5IZA9xrI=">AAADZHicfVHtbtMwFHVWPkYY0G3iFxKyqCp1CE1NhcT+TBqDH/zgY4i1m1SHyHGc1prtRLZT2pnsWXgAXogX4DlwmiCtK+JK8T25595zLZ8450ybfv+Xt9G6dfvO3c17/v2tBw8ftbd3RjorFKFDkvFMncdYU84kHRpmOD3PFcUi5vQsvnhT8WczqjTL5KlZ5DQUeCJZygg2rhS1f3bRe9edYHgIEaepQRblLCJfbS/eK1+gGUWiWPn9wiYC/60gxSZTg0q/6yh4+b0650gzYRHBHH4se42CvcYvdWAjtELs1Tqjw3GVZlFwdVWDQehfRoOo3env95cB10HQgA5o4iTa9n6gJCOFoNIQjrUeB/3chBYrwwinpY8KTXNMLvCEjh2UWFAd2uWzlrDrKglMM+U+aeCyen3CYqH1QsSuU2Az1Te5qvgvblyY9CC0TOaFoZLUi9KCQ5PByiOYMEWJ4QsHMFHM3RWSKVaYGOekjyT9RjIhsEyqxyvHQWhRnPGkukvGLaoWxqntBGVZrnabvLTupHOjU3u6RotKYF4Larc4N9osuLOqmlBiqeijt9Q9paIf3JZPOVXYZOq5RVg5N+elbfL/2pis21z2fedpcNPBdTAa7AcOf37ZOTpu3N0ET8Az0AMBeAWOwDtwAoaAeDvegffaO9743dpq7bYe160bXjOzC1ai9fQPkvcZpQ==</latexit><latexit sha1_base64="ojUAhf9y5bj1vGNw/Rb5IZA9xrI=">AAADZHicfVHtbtMwFHVWPkYY0G3iFxKyqCp1CE1NhcT+TBqDH/zgY4i1m1SHyHGc1prtRLZT2pnsWXgAXogX4DlwmiCtK+JK8T25595zLZ8450ybfv+Xt9G6dfvO3c17/v2tBw8ftbd3RjorFKFDkvFMncdYU84kHRpmOD3PFcUi5vQsvnhT8WczqjTL5KlZ5DQUeCJZygg2rhS1f3bRe9edYHgIEaepQRblLCJfbS/eK1+gGUWiWPn9wiYC/60gxSZTg0q/6yh4+b0650gzYRHBHH4se42CvcYvdWAjtELs1Tqjw3GVZlFwdVWDQehfRoOo3env95cB10HQgA5o4iTa9n6gJCOFoNIQjrUeB/3chBYrwwinpY8KTXNMLvCEjh2UWFAd2uWzlrDrKglMM+U+aeCyen3CYqH1QsSuU2Az1Te5qvgvblyY9CC0TOaFoZLUi9KCQ5PByiOYMEWJ4QsHMFHM3RWSKVaYGOekjyT9RjIhsEyqxyvHQWhRnPGkukvGLaoWxqntBGVZrnabvLTupHOjU3u6RotKYF4Larc4N9osuLOqmlBiqeijt9Q9paIf3JZPOVXYZOq5RVg5N+elbfL/2pis21z2fedpcNPBdTAa7AcOf37ZOTpu3N0ET8Az0AMBeAWOwDtwAoaAeDvegffaO9743dpq7bYe160bXjOzC1ai9fQPkvcZpQ==</latexit>

Global mean of x

Recall thatxi = m + Vzi + εi.

Therefore, each vector xi in the original space can be generated by aninfinite number of latent vector zi because εi is a random vector.


EM Formulation: M-Step

M-Step: Maximizing the Expectation of Complete Likelihood

Denote ω′ = {m′,V′,Σ′} as the new parameter sets. The E- andM-steps iteratively evaluate and maximize the expectation of thecomplete likelihood:

Q(ω′|ω) = Ez∼p(z|x){log p(X ,Z|ω′)|X ,ω}

= Ez∼p(z|x)

{∑ilog[p(xi|zi,ω′

)p(zi)

] ∣∣∣∣X ,ω}

= Ez∼p(z|x)

{∑ilog[N(xi|m′ + V′zi,Σ

′)N (zi|0, I)] ∣∣∣∣X ,ω

}.

(6)



Drop the symbol (′) in Eq. 6 and ignore the constant termsindependent on the model parameters:

Q(ω) = −∑

i

EZ{1

2log |Σ|+ 1

2(xi −m−Vzi)

TΣ−1(xi −m−Vzi)

}

=∑

i

[−1

2log |Σ| − 1

2(xi −m)TΣ−1(xi −m)

]

+∑

i

(xi −m)TΣ−1V 〈zi|xi〉 −1

2

[∑

i

⟨zTi VTΣ−1Vzi|xi

⟩].

(7)



Using the properties of matrix derivatives:

∂aTXb

∂X= abT

∂bTXTBXc

∂X= BTXbcT + BXcbT

∂

∂Alog |A−1| = −(A−1)T

We obtain

∂Q

∂V=∑

i

Σ−1(xi −m)〈zTi |xi〉 −∑

i

Σ−1V⟨ziz

Ti |xi

⟩. (8)



Setting ∂Q∂V = 0, we have

∑

i

V⟨ziz

Ti |xi

⟩=∑

i

(xi −m)〈zTi |xi〉 (9)

V =

[∑

i

(xi −m)〈zi|xi〉T][∑

i

⟨ziz

Ti |xi

⟩]−1. (10)

To find Σ, we evaluate

∂Q

∂Σ−1=

1

2

∑

i

[Σ− (xi −m)(xi −m)T

]+∑

i

(xi −m)〈zTi |xi〉VT

− 1

2

∑

i

V〈zizTi |xi〉VT.



Note that according to Eq. 9, we have

∂Q

∂Σ−1=

1

2

∑

i

[Σ− (xi −m)(xi −m)T

]+∑

i

(xi −m)〈zTi |xi〉VT

− 1

2

∑

i

(xi −m)〈zTi |xi〉VT.

Therefore, setting ∂Q∂Σ−1 = 0 we have

∑

i

Σ =∑

i

[(xi −m)(xi −m)T − (xi −m)〈zTi |xi〉VT

].

Rearranging, we have

Σ =1

N

∑

i

[(xi −m)(xi −m)T −V〈zi|xi〉(xi −m)T

].



To compute m, we evaluate

∂Q

∂m= −

∑

i

(Σ−1m−Σ−1xi

)+∑

i

Σ−1V〈zi|xi〉.

Setting ∂Q∂m = 0, we have

m =1

N

N∑

i=1

xi

where we have used the property∑N

i=1〈zi|xi〉 ≈ 0 when N issufficiently large. We have this property because of the assumptionthat the prior of z follows a Gaussian distribution, i.e., z ∼ N (z|0, I).


EM Formulation: E-Step

In the E-step, we compute the posterior means 〈zi|xi〉 and posteriormoments 〈zizTi |xi〉.Consider the posterior density:

p(zi|xi,ω)∝ p(xi|zi,ω)p(zi)

∝ exp

{−1

2(xi −m−Vzi)

TΣ−1(xi −m−Vzi)−1

2zTi zi

}

= exp

{zTi VTΣ−1(xi −m)− 1

2zTi

(I + VTΣ−1V

)zi

}.

(11)

A Gaussian distribution can be expressed as

N (z|µz,Cz) ∝ exp

{−1

2(z− µz)TC−1z (z− µz)

}

∝ exp

{zTC−1z µz −

1

2zTC−1z z

}.

(12)


EM Formulation: E-Step

Comparing Eq. 11 and Eq. 12, we obtain the posterior mean andmoment as follows:

〈zi|xi〉 = L−1VTΣ−1(xi −m)

〈zizTi |xi〉 = L−1 + 〈zi|X 〉〈zTi |xi〉

where L−1 = (I + VTΣ−1V)−1 is the posterior covariance matrix ofzi, ∀i.


EM Formulation

In summary, we have the following EM algorithm for factor analysis:

E-step:

〈zi|xi〉 = L−1VTΣ−1(xi −m)

〈zizTi |xi〉 = L−1 + 〈zi|xi〉〈zi|xi〉T

L = I + VTΣ−1V

M-step:

V′ =[∑

i(xi −m′)〈zi|xi〉T

] [∑i〈zizTi |xi〉

]−1

m′ =1

N

∑ixi

Σ′ =1

N

{∑N

i=1

[(xi −m′)(xi −m′)T −V′〈zi|xi〉(xi −m′)T

]}

(13)


Relation to PCA

Consider Σ = σ2I. Then, we have

L = I +1

σ2VTV

〈zi|xi〉 =1

σ2L−1VT(xi −m)

When σ2 → 0 and V is an orthogonal matrix such that V−1 = VT,then L→ 1

σ2 VTV and

〈zi|xi〉 →1

σ2σ2(VTV)−1VT(xi −m)

= V−1(VT)−1VT(xi −m)

= VT(xi −m)

(14)

Note that Eq. 14 is equivalent to PCA projection and that theposterior covariance (L−1) of z becomes 0.


Applications of FA to Facial Images

Source: S.J.D. Prince,Computer Vision: Models Learning and Inference,

Cambridge University Press, 2012

Different column vectors in V encode different types of variability inthe facial data, e.g., hue and pose.


Applications of Factor Analysis:I-Vectors


GMM I-Vectors

I-vector is an extension of the factor analysis in which the acousticvectors are generated by a Gaussian mixture model (GMM) instead ofa single Gaussian.

I-vector has been used in speaker, emotion, language, and ECGrecognition.

Given the i-th utterance, we denote Oi = {oi1, . . . ,oiTi} as a set ofF -dimensional observed vectors, which are assumed to be generatedby a GMM, i.e.,

p(oit) =

C∑

c=1

λcN (oit|µc,Σc), t = 1, . . . , Ti, (15)

where {λc,µc,Σc}Cc=1 are the parameters of the GMM, C is thenumber of mixtures, and Ti is the number of frames in the utterance.


GMM I-Vectors

Acoustic frames in speech are represented by low-dimensional acousticvectors.The density of acoustic vectors is modeled by a GMM.

GMM

Feature Space

Signal Space

Feature vectors from many utterances

GMM


I-Vectors: FA Model

GMM-supervector: Formed by stacking the mean vectors of a GMM.

GMM-supervector of the i-th utterance:

µi = µ(b) + Twi (16)

where µ(b) is obtained by stacking the mean vectors of a universalbackground model (UBM), T is a CF ×D low-rank total variabilitymatrix modeling the speaker and channel variability, and wi is thelatent factor of dimension D.


I-Vectors: FA Model

Eq. 16 can also be written in a component-wise form:

µic = µ(b)c + Tcwi, c = 1, . . . , C (17)

where µic ∈ <F is the c-th sub-vector of µi (similarly for µ(b)c ) and

Tc is an F ×D sub-matrix of T.

Given an utterance with acoustic vectors Oi, the i-vector xirepresenting the utterance is the posterior mean of wi, i.e.,

I-vector: xi = Ewi∼p(wi|Oi){wi|Oi}= 〈wi|Oi〉

This means that we need to determine the posterior density p(wi|Oi),which is a Gaussian with mean xi.


I-Vectors: FA Model

To determine xi, we consider the joint posterior distribution:

p(wi, yi,·,·|Oi) ∝ p(Oi|wi, yi,·,· = 1)p(yi,·,·)p(wi)

=

T∏

t=1

C∏

c=1

[λcp(oit|yi,t,c = 1,wi)]yi,t,c p(wi)

= p(wi)

T∏

t=1

C∏

c=1

[N(oit|µ(b)

c + Tcwi,Σ(b)c

)]yi,t,c

︸︷︷︸∝ p(wi|Oi)

λyi,t,cc ,

(18)

yi,.,. is an indicator variable of the GMM such that yi,t,c = 1 if thet-th frame is generated by the c-th Gaussian in the GMM.


I-Vectors: FA Model

Extracting terms depending on wi from Eq. 18, we obtain

log p(wi|Oi) ∝ −1

2

C∑

c=1

∑

t∈Hic

(oit − µ(b)c −Tcwi)

T(Σ(b)c )−1

(oit − µ(b)c −Tcwi)−

1

2wTi wi

= wTi

C∑

c=1

∑

t∈Hic

TTc (Σ

(b)c )−1(oit − µ(b)

c )

− 1

2wTi

I +

C∑

c=1

∑

t∈Hic

TTc (Σ

(b)c )−1Tc

wi,

(19)

where Hic comprises the frame indexes for which oit aligned tomixture c.


Frame Alignment

Oi = {oi1,oi2, . . . ,oiTi}

<latexit sha1_base64="QXnsBaEHlN2CKqzd1g8/WdaLXEg=">AAAC8nicfVHfixMxEE7XX+f6q6ePvgSLICKlWwVFEA71wRe5E9q7g6aUbHb2LlyyWZLZsyXkH/HpxFf/Dl/11f/GbFvFu4oDyXx8881MMpPXSjocDH52kkuXr1y9tnU9vXHz1u073e27+840VsBYGGXsYc4dKFnBGCUqOKwtcJ0rOMhP3rTxg1OwTppqhIsappofVbKUgmOkZt2Xngmu6G6YeRleMc9OgZqIs/DkNxxGqAqD7g8zmsnAwqzbG/QHS6ObIFuDHlnb3my7c8YKIxoNFQrFnZtkgxqnnluUQkFIWeOg5uKEH8EkwoprcFO//GSgDyNT0NLYeCqkS/bvDM+1cwudR6XmeOwuxlryX7FJg+WLqZdV3SBUYtWobBRFQ9uJ0UJaEKgWEXBhZXwrFcfccoFxrimr4KMwWvOqaIcXJtnUs9yoon2LUZ61DfPS97IQwnk11sHHG+boSj/aCOu2wHxV0MXGNTpcKKDLDKuXFVP2FuIoLbyPXXZrsByNfewZt0eaz4Nf+//JZLWSRZ+mcafZxQ1ugv1hP3vaH3541tt5vd7uFrlPHpBHJCPPyQ55R/bImAhyRr6R7+RHgsmn5HPyZSVNOuuce+ScJV9/ASCL9YM=</latexit>

N⇣µ(b)

c ,⌃(b)c

⌘<latexit sha1_base64="oSdBlAVN2b/rmulDRm+Aytd9Sk8=">AAADKXicfVFNbxMxEHUWaMvy0RSOXFZElVKEomxAggtSBRy4QItI2kpxWHm93sSqvV7s2ZLI8n/hzq/hxMeV34GEd5Mi2iBGsv305r0ZayYtBTfQ739vBVeuXtvY3Loe3rh56/Z2e+fOkVGVpmxElVD6JCWGCV6wEXAQ7KTUjMhUsOP09EWdPz5j2nBVDGFRsokk04LnnBLwVNL+sGsxJSI6cInl7hm2+IxFyuPYPTyHAw9FpsD8YYYJd9iFS+sbhwXLoeuTWFYJfW+76V7jxu/4VJJzBms+ncFe0u70e/0monUQr0AHreIw2Wl9wpmilWQFUEGMGcf9EiaWaOBUMBfiyrCS0FMyZWMPCyKZmdhmNi7a9UwW5Ur7U0DUsH87LJHGLGTqlZLAzFzO1eS/cuMK8qcTy4uyAlbQZaO8EhGoqB50lHHNKIiFB4Rq7v8a0RnRhIJfR4gL9pEqKUmR1SN343hicapEVv9FCYvrhmluO7Fz7qIaSmf9zeZgcjtcS8u6wHxZ0PjGJRhYCBY1Di2biiF+yfwoNXvtuxyUTBNQ+oHFRPt9zZ1dvf+T8WIp828Y+p3Glze4Do4GvfhRb/D2cWf/+Wq7W+geuo+6KEZP0D56hQ7RCFH0Ff1qbbQ2g8/Bl+Bb8GMpDVorz110IYKfvwF3Owlo</latexit>

Hic<latexit sha1_base64="LVB4UtU1aWxMvfLZFW+hIZp0YmI=">AAADOHicfVHLbhMxFPUMrzIUSEGwYTMiqpQiFGUCEmyQKmDRDbSIpK0Uh5HH40ms+jGyPSWRZYlP4Tv4E3asQGz5AjyTKaIN4kq2j869517rnqxkVJvB4GsQXrp85eq1jevRjc2bt253tu4calkpTMZYMqmOM6QJo4KMDTWMHJeKIJ4xcpSdvKrzR6dEaSrFyCxLMuVoJmhBMTKeSjufti3EiMX7LrXUvYAWnpJYepy4x2dw6CHLpdF/mFFKHXRRq33rICOF6fks5FWKP9hettPI4Xs64+iMgYrO5mYnWqn26onYpZ3uoD9oIl4HSQu6oI2DdCv4DHOJK06EwQxpPUkGpZlapAzFjLgIVpqUCJ+gGZl4KBAnemqbVbl42zN5XEjljzBxw/6tsIhrveSZr+TIzPXFXE3+KzepTPF8aqkoK0MEXg0qKhYbGdd7j3OqCDZs6QHCivq/xniOFMLGuxNBQT5iyTkSee2AmyRTCzPJ8vovkllYD8wK202cc+erTemsv8nC6MKO1tK8brBYNdR+cGm0WTISNwrFm44RfE38KhV546fsl0QhI9UjC5Hy7i2cbd//lVGxKvNvFHlPk4sOroPDYT950h++e9rdfdm6uwEegIegBxLwDOyCPXAAxgCD78FmcC+4H34Jv4U/wp+r0jBoNXfBuQh//QZA4Q49</latexit>


I-Vectors: FA Model

Comparing Eq. 19 with Eq. 12, we obtain the following posteriorexpectations:

〈wi|Oi〉 = L−1i

C∑

c=1

∑

t∈Hic

TTc

(Σ(b)c

)−1(oit − µ(b)

c )

= L−1i

C∑

c=1

TTc

(Σ(b)c

)−1 ∑

t∈Hic

(oit − µ(b)c ) (20)

〈wiwTi |Oi〉 = L−1i + 〈wi|Oi〉〈wT

i |Oi〉 (21)

where

Li = I +

C∑

c=1

∑

t∈Hic

TTc (Σ

(b)c )−1Tc. (22)


Hard Decisions for Frame Alignment

For each t, the posterior probabilities of yi,t,c, for c = 1, . . . , C, arecomputed. Then, oit is aligned to mixture c∗ when

c∗ = argmaxc

γc(oit)

where

γc(oit) ≡ Pr(Ct = c|oit)

=λ(b)c N (oit|µ(b)

c ,Σ(b)c )

∑Cj=1 λ

(b)j N (oit;µ

(b)j ,Σ

(b)j )

, c = 1, . . . , C(23)

are the posterior probabilities of the c-th mixture given oit.


Soft Decisions for Frame Alignment

Each frame is aligned to all of the mixtures with degree of alignmentaccording to the posterior probabilities γc(oit):

∑

t∈Hic

1 =

Ti∑

t=1

γc(oit)

∑

t∈Hic

(oit − µ(b)c ) =

Ti∑

t=1

γc(oit)(oit − µ(b)c ).

(24)

Baum-Welch (sufficient) statistics:

Nic ≡Ti∑

t=1

γc(oit) and f̃ ic ≡Ti∑

t=1

γc(oit)(oit − µ(b)c ). (25)


I-Vector Extraction

Substituting Eq. 25 into Eq. 20 and Eq. 22, we have the followingexpression for i-vectors:

xi = 〈wi|Oi〉

= L−1i

C∑

c=1

TTc

(Σ(b)c

)−1f̃ ic

= L−1i TT(Σ(b))−1f̃ i

(26)

where f̃ i = [f̃Ti1 · · · f̃

TiC ]

T and

Li = I +

C∑

c=1

NicTTc (Σ

(b)c )−1Tc = I + TT(Σ(b))−1NiT

where Ni is a CF × CF block diagonal matrix containing NicI,c = 1, . . . , C, as its block diagonal elements.


I-Vector Extraction

Align with UBM

Compute Sufficient Statistics

Compute Posterior

Mean

x<latexit sha1_base64="wvYNprTBdyC4QTI49WihFPc1uTs=">AAADfHicfVHbbtMwGHZaDiOcOrjkxqJM6jZ1ahACbiZNjAu4QBSxboO6jRzXaa3ZTmQ7o5Xld+FheAleBuGkAfWA+KXEn//v+w/yl+ScadPr/QwazRs3b93euRPevXf/wcPW7qNznRWK0AHJeKYuE6wpZ5IODDOcXuaKYpFwepFcnZb8xTVVmmXyzCxyOhJ4KlnKCDY+Fbd+7KFrCufwGCJOU9Mpb+8PIdKFiC05jtz4FEqPXEmcxWSMTF6K0Gc2FdjfbSfZd/tj243+SJBi05mpUyt9Krqs/ztpvclq3co2aUy60nf1clFsaMNq+TButXtHvSrgNohq0AZ19OPd4DuaZKQQVBrCsdbDqJebkcXKMMKpC1GhaY7JFZ7SoYcSC6pHtnpuB/d8ZgLTTPlPGlhlVyssFlovROKVApuZ3uTK5L+4YWHS1yPLZF4YKslyUFpwaDJYegcnTFFi+MIDTBTzu0IywwoT4x0OkaTfSCYElhPr38UNo5FFScYn5S4Zt6gcmKS2HTnn1tUmd9b/6dzo1J5t0aJsMF821H5wbrRZcAqrCiWqjiF6S/1TKvrBT/mYU4VNpg4swspbPHe2Pv8nY3Ip82dYehptOrgNzp8fRR5/etE+eVO7uwOegKegAyLwCpyAd6APBoAEB0E/+BJ8bfxqPmseNrtLaSOoax6DtWi+/A14GiI8</latexit><latexit sha1_base64="wvYNprTBdyC4QTI49WihFPc1uTs=">AAADfHicfVHbbtMwGHZaDiOcOrjkxqJM6jZ1ahACbiZNjAu4QBSxboO6jRzXaa3ZTmQ7o5Xld+FheAleBuGkAfWA+KXEn//v+w/yl+ScadPr/QwazRs3b93euRPevXf/wcPW7qNznRWK0AHJeKYuE6wpZ5IODDOcXuaKYpFwepFcnZb8xTVVmmXyzCxyOhJ4KlnKCDY+Fbd+7KFrCufwGCJOU9Mpb+8PIdKFiC05jtz4FEqPXEmcxWSMTF6K0Gc2FdjfbSfZd/tj243+SJBi05mpUyt9Krqs/ztpvclq3co2aUy60nf1clFsaMNq+TButXtHvSrgNohq0AZ19OPd4DuaZKQQVBrCsdbDqJebkcXKMMKpC1GhaY7JFZ7SoYcSC6pHtnpuB/d8ZgLTTPlPGlhlVyssFlovROKVApuZ3uTK5L+4YWHS1yPLZF4YKslyUFpwaDJYegcnTFFi+MIDTBTzu0IywwoT4x0OkaTfSCYElhPr38UNo5FFScYn5S4Zt6gcmKS2HTnn1tUmd9b/6dzo1J5t0aJsMF821H5wbrRZcAqrCiWqjiF6S/1TKvrBT/mYU4VNpg4swspbPHe2Pv8nY3Ip82dYehptOrgNzp8fRR5/etE+eVO7uwOegKegAyLwCpyAd6APBoAEB0E/+BJ8bfxqPmseNrtLaSOoax6DtWi+/A14GiI8</latexit><latexit sha1_base64="wvYNprTBdyC4QTI49WihFPc1uTs=">AAADfHicfVHbbtMwGHZaDiOcOrjkxqJM6jZ1ahACbiZNjAu4QBSxboO6jRzXaa3ZTmQ7o5Xld+FheAleBuGkAfWA+KXEn//v+w/yl+ScadPr/QwazRs3b93euRPevXf/wcPW7qNznRWK0AHJeKYuE6wpZ5IODDOcXuaKYpFwepFcnZb8xTVVmmXyzCxyOhJ4KlnKCDY+Fbd+7KFrCufwGCJOU9Mpb+8PIdKFiC05jtz4FEqPXEmcxWSMTF6K0Gc2FdjfbSfZd/tj243+SJBi05mpUyt9Krqs/ztpvclq3co2aUy60nf1clFsaMNq+TButXtHvSrgNohq0AZ19OPd4DuaZKQQVBrCsdbDqJebkcXKMMKpC1GhaY7JFZ7SoYcSC6pHtnpuB/d8ZgLTTPlPGlhlVyssFlovROKVApuZ3uTK5L+4YWHS1yPLZF4YKslyUFpwaDJYegcnTFFi+MIDTBTzu0IywwoT4x0OkaTfSCYElhPr38UNo5FFScYn5S4Zt6gcmKS2HTnn1tUmd9b/6dzo1J5t0aJsMF821H5wbrRZcAqrCiWqjiF6S/1TKvrBT/mYU4VNpg4swspbPHe2Pv8nY3Ip82dYehptOrgNzp8fRR5/etE+eVO7uwOegKegAyLwCpyAd6APBoAEB0E/+BJ8bfxqPmseNrtLaSOoax6DtWi+/A14GiI8</latexit><latexit sha1_base64="wvYNprTBdyC4QTI49WihFPc1uTs=">AAADfHicfVHbbtMwGHZaDiOcOrjkxqJM6jZ1ahACbiZNjAu4QBSxboO6jRzXaa3ZTmQ7o5Xld+FheAleBuGkAfWA+KXEn//v+w/yl+ScadPr/QwazRs3b93euRPevXf/wcPW7qNznRWK0AHJeKYuE6wpZ5IODDOcXuaKYpFwepFcnZb8xTVVmmXyzCxyOhJ4KlnKCDY+Fbd+7KFrCufwGCJOU9Mpb+8PIdKFiC05jtz4FEqPXEmcxWSMTF6K0Gc2FdjfbSfZd/tj243+SJBi05mpUyt9Krqs/ztpvclq3co2aUy60nf1clFsaMNq+TButXtHvSrgNohq0AZ19OPd4DuaZKQQVBrCsdbDqJebkcXKMMKpC1GhaY7JFZ7SoYcSC6pHtnpuB/d8ZgLTTPlPGlhlVyssFlovROKVApuZ3uTK5L+4YWHS1yPLZF4YKslyUFpwaDJYegcnTFFi+MIDTBTzu0IywwoT4x0OkaTfSCYElhPr38UNo5FFScYn5S4Zt6gcmKS2HTnn1tUmd9b/6dzo1J5t0aJsMF821H5wbrRZcAqrCiWqjiF6S/1TKvrBT/mYU4VNpg4swspbPHe2Pv8nY3Ip82dYehptOrgNzp8fRR5/etE+eVO7uwOegKegAyLwCpyAd6APBoAEB0E/+BJ8bfxqPmseNrtLaSOoax6DtWi+/A14GiI8</latexit>

�c(ot)<latexit sha1_base64="NxZt1OwG9putBEIlTzwmsy7qyqg=">AAADiHicfVFdb9MwFHUWPkb4WAePvFhUkzrQpgQhBg+Tpo0HeEAM0W6T6i5yXKe1ZjuR7YxWVv4LP4lX/g12GlA/EJYSH99z7rlX92YlZ9rE8a9gK7xz99797QfRw0ePn+x0dp9e6KJShA5IwQt1lWFNOZN0YJjh9KpUFIuM08vs5szzl7dUaVbIvpmXdCTwRLKcEWxcKO383EO3FM7gMUSc5qbnX59eQaQrkVpynNTXZ1A6VHuin5JrZEovQt/YRGD3tr1sv96/tgfJHwlSbDI1bWjJp6F9/t9KqybLeUvd5Ck5kM7VyUW1po3QBAvv0CiL1EXSTjc+jJsDN0HSgi5oz3m6G/xA44JUgkpDONZ6mMSlGVmsDCOc1hGqNC0xucETOnRQYkH1yDajr+Gei4xhXij3SQOb6HKGxULrucicUmAz1eucD/6LG1YmfzeyTJaVoZIsCuUVh6aAfo9wzBQlhs8dwEQx1yskU6wwMW7bEZL0OyncaOTYutHUw2RkUVbwse+l4Bb5glluu0ld16tqU9bW/enM6Nz2N2jhDWYLQ+0Kl0abOaewyVCicYzQB+pGqehnV+VLSRU2hXppEVZu3bPatvf/ZEwuZO6O/E6T9Q1ugovXh4nDX990T07b7W6D5+AF6IEEHIET8BGcgwEgwdsABTTIwyiMw6Pw/UK6FbQ5z8DKCU9/A1rTJfE=</latexit><latexit sha1_base64="NxZt1OwG9putBEIlTzwmsy7qyqg=">AAADiHicfVFdb9MwFHUWPkb4WAePvFhUkzrQpgQhBg+Tpo0HeEAM0W6T6i5yXKe1ZjuR7YxWVv4LP4lX/g12GlA/EJYSH99z7rlX92YlZ9rE8a9gK7xz99797QfRw0ePn+x0dp9e6KJShA5IwQt1lWFNOZN0YJjh9KpUFIuM08vs5szzl7dUaVbIvpmXdCTwRLKcEWxcKO383EO3FM7gMUSc5qbnX59eQaQrkVpynNTXZ1A6VHuin5JrZEovQt/YRGD3tr1sv96/tgfJHwlSbDI1bWjJp6F9/t9KqybLeUvd5Ck5kM7VyUW1po3QBAvv0CiL1EXSTjc+jJsDN0HSgi5oz3m6G/xA44JUgkpDONZ6mMSlGVmsDCOc1hGqNC0xucETOnRQYkH1yDajr+Gei4xhXij3SQOb6HKGxULrucicUmAz1eucD/6LG1YmfzeyTJaVoZIsCuUVh6aAfo9wzBQlhs8dwEQx1yskU6wwMW7bEZL0OyncaOTYutHUw2RkUVbwse+l4Bb5glluu0ld16tqU9bW/enM6Nz2N2jhDWYLQ+0Kl0abOaewyVCicYzQB+pGqehnV+VLSRU2hXppEVZu3bPatvf/ZEwuZO6O/E6T9Q1ugovXh4nDX990T07b7W6D5+AF6IEEHIET8BGcgwEgwdsABTTIwyiMw6Pw/UK6FbQ5z8DKCU9/A1rTJfE=</latexit><latexit sha1_base64="NxZt1OwG9putBEIlTzwmsy7qyqg=">AAADiHicfVFdb9MwFHUWPkb4WAePvFhUkzrQpgQhBg+Tpo0HeEAM0W6T6i5yXKe1ZjuR7YxWVv4LP4lX/g12GlA/EJYSH99z7rlX92YlZ9rE8a9gK7xz99797QfRw0ePn+x0dp9e6KJShA5IwQt1lWFNOZN0YJjh9KpUFIuM08vs5szzl7dUaVbIvpmXdCTwRLKcEWxcKO383EO3FM7gMUSc5qbnX59eQaQrkVpynNTXZ1A6VHuin5JrZEovQt/YRGD3tr1sv96/tgfJHwlSbDI1bWjJp6F9/t9KqybLeUvd5Ck5kM7VyUW1po3QBAvv0CiL1EXSTjc+jJsDN0HSgi5oz3m6G/xA44JUgkpDONZ6mMSlGVmsDCOc1hGqNC0xucETOnRQYkH1yDajr+Gei4xhXij3SQOb6HKGxULrucicUmAz1eucD/6LG1YmfzeyTJaVoZIsCuUVh6aAfo9wzBQlhs8dwEQx1yskU6wwMW7bEZL0OyncaOTYutHUw2RkUVbwse+l4Bb5glluu0ld16tqU9bW/enM6Nz2N2jhDWYLQ+0Kl0abOaewyVCicYzQB+pGqehnV+VLSRU2hXppEVZu3bPatvf/ZEwuZO6O/E6T9Q1ugovXh4nDX990T07b7W6D5+AF6IEEHIET8BGcgwEgwdsABTTIwyiMw6Pw/UK6FbQ5z8DKCU9/A1rTJfE=</latexit><latexit sha1_base64="NxZt1OwG9putBEIlTzwmsy7qyqg=">AAADiHicfVFdb9MwFHUWPkb4WAePvFhUkzrQpgQhBg+Tpo0HeEAM0W6T6i5yXKe1ZjuR7YxWVv4LP4lX/g12GlA/EJYSH99z7rlX92YlZ9rE8a9gK7xz99797QfRw0ePn+x0dp9e6KJShA5IwQt1lWFNOZN0YJjh9KpUFIuM08vs5szzl7dUaVbIvpmXdCTwRLKcEWxcKO383EO3FM7gMUSc5qbnX59eQaQrkVpynNTXZ1A6VHuin5JrZEovQt/YRGD3tr1sv96/tgfJHwlSbDI1bWjJp6F9/t9KqybLeUvd5Ck5kM7VyUW1po3QBAvv0CiL1EXSTjc+jJsDN0HSgi5oz3m6G/xA44JUgkpDONZ6mMSlGVmsDCOc1hGqNC0xucETOnRQYkH1yDajr+Gei4xhXij3SQOb6HKGxULrucicUmAz1eucD/6LG1YmfzeyTJaVoZIsCuUVh6aAfo9wzBQlhs8dwEQx1yskU6wwMW7bEZL0OyncaOTYutHUw2RkUVbwse+l4Bb5glluu0ld16tqU9bW/enM6Nz2N2jhDWYLQ+0Kl0abOaewyVCicYzQB+pGqehnV+VLSRU2hXppEVZu3bPatvf/ZEwuZO6O/E6T9Q1ugovXh4nDX990T07b7W6D5+AF6IEEHIET8BGcgwEgwdsABTTIwyiMw6Pw/UK6FbQ5z8DKCU9/A1rTJfE=</latexit>

nc, f c<latexit sha1_base64="Yaax5guaZtKhzQwrrk0SxOl5X/M=">AAADlXicfVFbb9MwFE5WLiPcOnjggReLqlIHbGoQErxUmuiE4AFtiHabVLeR4zqtNV8i2xmtrPwX/hYP/BfsNKBeEEdK/Pmc73zHOl+aM6pNt/sz3Gvcun3n7v696P6Dh48eNw+eXGhZKEyGWDKprlKkCaOCDA01jFzliiCeMnKZXvd9/fKGKE2lGJhlTsYczQTNKEbGpZLmrza8IWABegAykpmOv31+BaAueGJxLy4nfSAcKn1hkOAJNLknwW90xpG72056WB5O7FH8hwIVnc1NnVrTqcq+/++kTZH1vrXXZAk+Ek7V0XmxxY3acIa4l6ioMnEpx31d90VJs9U97lYBdkFcg1ZQx3lyEP6AU4kLToTBDGk9iru5GVukDMWMlBEsNMkRvkYzMnJQIE702FY+lKDtMlOQSeU+YUCVXe+wiGu95KljcmTmervmk/+qjQqTvR9bKvLCEIFXg7KCASOBNxVMqSLYsKUDCCvq3grwHCmEjbM+goJ8x9KtSUyt20w5iscWppJN/Vsks9APTDPbisuy3GSbvLTuTxZGZ3awU+ZeYLES1G5wbrRZMgKqDsUrxQieErdKRb64KWc5UchI9dJCpJz3i9LW5/9oVKxo7oy8p/G2g7vg4s1x7PDXt62TD7W7+8Hz4EXQCeLgXXASfArOg2GAw35IQxXqxrNGr3Ha+Lii7oV1z9NgIxpnvwGNjyp9</latexit><latexit sha1_base64="Yaax5guaZtKhzQwrrk0SxOl5X/M=">AAADlXicfVFbb9MwFE5WLiPcOnjggReLqlIHbGoQErxUmuiE4AFtiHabVLeR4zqtNV8i2xmtrPwX/hYP/BfsNKBeEEdK/Pmc73zHOl+aM6pNt/sz3Gvcun3n7v696P6Dh48eNw+eXGhZKEyGWDKprlKkCaOCDA01jFzliiCeMnKZXvd9/fKGKE2lGJhlTsYczQTNKEbGpZLmrza8IWABegAykpmOv31+BaAueGJxLy4nfSAcKn1hkOAJNLknwW90xpG72056WB5O7FH8hwIVnc1NnVrTqcq+/++kTZH1vrXXZAk+Ek7V0XmxxY3acIa4l6ioMnEpx31d90VJs9U97lYBdkFcg1ZQx3lyEP6AU4kLToTBDGk9iru5GVukDMWMlBEsNMkRvkYzMnJQIE702FY+lKDtMlOQSeU+YUCVXe+wiGu95KljcmTmervmk/+qjQqTvR9bKvLCEIFXg7KCASOBNxVMqSLYsKUDCCvq3grwHCmEjbM+goJ8x9KtSUyt20w5iscWppJN/Vsks9APTDPbisuy3GSbvLTuTxZGZ3awU+ZeYLES1G5wbrRZMgKqDsUrxQieErdKRb64KWc5UchI9dJCpJz3i9LW5/9oVKxo7oy8p/G2g7vg4s1x7PDXt62TD7W7+8Hz4EXQCeLgXXASfArOg2GAw35IQxXqxrNGr3Ha+Lii7oV1z9NgIxpnvwGNjyp9</latexit><latexit sha1_base64="Yaax5guaZtKhzQwrrk0SxOl5X/M=">AAADlXicfVFbb9MwFE5WLiPcOnjggReLqlIHbGoQErxUmuiE4AFtiHabVLeR4zqtNV8i2xmtrPwX/hYP/BfsNKBeEEdK/Pmc73zHOl+aM6pNt/sz3Gvcun3n7v696P6Dh48eNw+eXGhZKEyGWDKprlKkCaOCDA01jFzliiCeMnKZXvd9/fKGKE2lGJhlTsYczQTNKEbGpZLmrza8IWABegAykpmOv31+BaAueGJxLy4nfSAcKn1hkOAJNLknwW90xpG72056WB5O7FH8hwIVnc1NnVrTqcq+/++kTZH1vrXXZAk+Ek7V0XmxxY3acIa4l6ioMnEpx31d90VJs9U97lYBdkFcg1ZQx3lyEP6AU4kLToTBDGk9iru5GVukDMWMlBEsNMkRvkYzMnJQIE702FY+lKDtMlOQSeU+YUCVXe+wiGu95KljcmTmervmk/+qjQqTvR9bKvLCEIFXg7KCASOBNxVMqSLYsKUDCCvq3grwHCmEjbM+goJ8x9KtSUyt20w5iscWppJN/Vsks9APTDPbisuy3GSbvLTuTxZGZ3awU+ZeYLES1G5wbrRZMgKqDsUrxQieErdKRb64KWc5UchI9dJCpJz3i9LW5/9oVKxo7oy8p/G2g7vg4s1x7PDXt62TD7W7+8Hz4EXQCeLgXXASfArOg2GAw35IQxXqxrNGr3Ha+Lii7oV1z9NgIxpnvwGNjyp9</latexit><latexit sha1_base64="Yaax5guaZtKhzQwrrk0SxOl5X/M=">AAADlXicfVFbb9MwFE5WLiPcOnjggReLqlIHbGoQErxUmuiE4AFtiHabVLeR4zqtNV8i2xmtrPwX/hYP/BfsNKBeEEdK/Pmc73zHOl+aM6pNt/sz3Gvcun3n7v696P6Dh48eNw+eXGhZKEyGWDKprlKkCaOCDA01jFzliiCeMnKZXvd9/fKGKE2lGJhlTsYczQTNKEbGpZLmrza8IWABegAykpmOv31+BaAueGJxLy4nfSAcKn1hkOAJNLknwW90xpG72056WB5O7FH8hwIVnc1NnVrTqcq+/++kTZH1vrXXZAk+Ek7V0XmxxY3acIa4l6ioMnEpx31d90VJs9U97lYBdkFcg1ZQx3lyEP6AU4kLToTBDGk9iru5GVukDMWMlBEsNMkRvkYzMnJQIE702FY+lKDtMlOQSeU+YUCVXe+wiGu95KljcmTmervmk/+qjQqTvR9bKvLCEIFXg7KCASOBNxVMqSLYsKUDCCvq3grwHCmEjbM+goJ8x9KtSUyt20w5iscWppJN/Vsks9APTDPbisuy3GSbvLTuTxZGZ3awU+ZeYLES1G5wbrRZMgKqDsUrxQieErdKRb64KWc5UchI9dJCpJz3i9LW5/9oVKxo7oy8p/G2g7vg4s1x7PDXt62TD7W7+8Hz4EXQCeLgXXASfArOg2GAw35IQxXqxrNGr3Ha+Lii7oV1z9NgIxpnvwGNjyp9</latexit>

UBM

o1, . . . ,oT<latexit sha1_base64="uQDL38S7Oqz4wds5oeFdq/muSho=">AAADXXicfVFNbxMxEPUmFMq2tCkcOHCxiCKlKKqyCAkEQqqAAwc+ikjaSnGIvF5vYtVer+zZkmD2v3DhR3Hir+BNFqlpECPZM573ZsaaF+dSWOj3fwWN5o2tm7e2b4c7u3f29lsHd0+tLgzjQ6alNucxtVyKjA9BgOTnueFUxZKfxRevK/zskhsrdDaARc7Hik4zkQpGwacmrZ8d8s6zE4pfYiJ5CsSRXEzYF9eND8seueREFWvPz2Kq6N8MMWI6A1KGHQ/hb9+re06sUI4wKvGHslt3cFfwZR9cN1oDDsPK60nUIzLRYHur52DSaveP+kvDm0FUB21U28nkIPhBEs0KxTNgklo7ivo5jB01IJjkZUgKy3PKLuiUj3yYUcXt2C33WeKOzyQ41cafDPAye7XCUWXtQsWeqSjM7HWsSv4LGxWQPhs7keUF8IytBqWFxKBxJQ5OhOEM5MIHlBnh/4rZjBrKwEsYkox/ZVopmiXV1spRNHYk1jKp/qKlI9XAOHXtqCzLdTbkpfM3n4NN3WADVlWD+aqh9YNzsLCQXqOqwqhlx5C84X6Vhr/3Uz7m3FDQ5pEj1HgZ56Wr/f9oIlvRvA9Dr2l0XcHN4PTxUeTjT0/ax69qdbfRA/QQdVGEnqJj9BadoCFiwU4QBc+DF43fza3mbnNvRW0Edc09tGbN+38AbQQXdQ==</latexit><latexit sha1_base64="uQDL38S7Oqz4wds5oeFdq/muSho=">AAADXXicfVFNbxMxEPUmFMq2tCkcOHCxiCKlKKqyCAkEQqqAAwc+ikjaSnGIvF5vYtVer+zZkmD2v3DhR3Hir+BNFqlpECPZM573ZsaaF+dSWOj3fwWN5o2tm7e2b4c7u3f29lsHd0+tLgzjQ6alNucxtVyKjA9BgOTnueFUxZKfxRevK/zskhsrdDaARc7Hik4zkQpGwacmrZ8d8s6zE4pfYiJ5CsSRXEzYF9eND8seueREFWvPz2Kq6N8MMWI6A1KGHQ/hb9+re06sUI4wKvGHslt3cFfwZR9cN1oDDsPK60nUIzLRYHur52DSaveP+kvDm0FUB21U28nkIPhBEs0KxTNgklo7ivo5jB01IJjkZUgKy3PKLuiUj3yYUcXt2C33WeKOzyQ41cafDPAye7XCUWXtQsWeqSjM7HWsSv4LGxWQPhs7keUF8IytBqWFxKBxJQ5OhOEM5MIHlBnh/4rZjBrKwEsYkox/ZVopmiXV1spRNHYk1jKp/qKlI9XAOHXtqCzLdTbkpfM3n4NN3WADVlWD+aqh9YNzsLCQXqOqwqhlx5C84X6Vhr/3Uz7m3FDQ5pEj1HgZ56Wr/f9oIlvRvA9Dr2l0XcHN4PTxUeTjT0/ax69qdbfRA/QQdVGEnqJj9BadoCFiwU4QBc+DF43fza3mbnNvRW0Edc09tGbN+38AbQQXdQ==</latexit><latexit sha1_base64="uQDL38S7Oqz4wds5oeFdq/muSho=">AAADXXicfVFNbxMxEPUmFMq2tCkcOHCxiCKlKKqyCAkEQqqAAwc+ikjaSnGIvF5vYtVer+zZkmD2v3DhR3Hir+BNFqlpECPZM573ZsaaF+dSWOj3fwWN5o2tm7e2b4c7u3f29lsHd0+tLgzjQ6alNucxtVyKjA9BgOTnueFUxZKfxRevK/zskhsrdDaARc7Hik4zkQpGwacmrZ8d8s6zE4pfYiJ5CsSRXEzYF9eND8seueREFWvPz2Kq6N8MMWI6A1KGHQ/hb9+re06sUI4wKvGHslt3cFfwZR9cN1oDDsPK60nUIzLRYHur52DSaveP+kvDm0FUB21U28nkIPhBEs0KxTNgklo7ivo5jB01IJjkZUgKy3PKLuiUj3yYUcXt2C33WeKOzyQ41cafDPAye7XCUWXtQsWeqSjM7HWsSv4LGxWQPhs7keUF8IytBqWFxKBxJQ5OhOEM5MIHlBnh/4rZjBrKwEsYkox/ZVopmiXV1spRNHYk1jKp/qKlI9XAOHXtqCzLdTbkpfM3n4NN3WADVlWD+aqh9YNzsLCQXqOqwqhlx5C84X6Vhr/3Uz7m3FDQ5pEj1HgZ56Wr/f9oIlvRvA9Dr2l0XcHN4PTxUeTjT0/ax69qdbfRA/QQdVGEnqJj9BadoCFiwU4QBc+DF43fza3mbnNvRW0Edc09tGbN+38AbQQXdQ==</latexit><latexit sha1_base64="uQDL38S7Oqz4wds5oeFdq/muSho=">AAADXXicfVFNbxMxEPUmFMq2tCkcOHCxiCKlKKqyCAkEQqqAAwc+ikjaSnGIvF5vYtVer+zZkmD2v3DhR3Hir+BNFqlpECPZM573ZsaaF+dSWOj3fwWN5o2tm7e2b4c7u3f29lsHd0+tLgzjQ6alNucxtVyKjA9BgOTnueFUxZKfxRevK/zskhsrdDaARc7Hik4zkQpGwacmrZ8d8s6zE4pfYiJ5CsSRXEzYF9eND8seueREFWvPz2Kq6N8MMWI6A1KGHQ/hb9+re06sUI4wKvGHslt3cFfwZR9cN1oDDsPK60nUIzLRYHur52DSaveP+kvDm0FUB21U28nkIPhBEs0KxTNgklo7ivo5jB01IJjkZUgKy3PKLuiUj3yYUcXt2C33WeKOzyQ41cafDPAye7XCUWXtQsWeqSjM7HWsSv4LGxWQPhs7keUF8IytBqWFxKBxJQ5OhOEM5MIHlBnh/4rZjBrKwEsYkox/ZVopmiXV1spRNHYk1jKp/qKlI9XAOHXtqCzLdTbkpfM3n4NN3WADVlWD+aqh9YNzsLCQXqOqwqhlx5C84X6Vhr/3Uz7m3FDQ5pEj1HgZ56Wr/f9oIlvRvA9Dr2l0XcHN4PTxUeTjT0/ax69qdbfRA/QQdVGEnqJj9BadoCFiwU4QBc+DF43fza3mbnNvRW0Edc09tGbN+38AbQQXdQ==</latexit>

⇤ =n⇡(b)

c , µ(b)c ,⌃(b)

c

oC

c=1<latexit sha1_base64="0FasGiO3xkR6q79/5ovqleyDVYs=">AAAD53icfVLNbhMxEN5t+CnLT1s4crGIIqXQRlmEBJdKFeUAEogikrZSnERerze1antXtrcksvwMnEAc4Tm4wkPwNtibLUoawNKuP8988814xknBqNLd7q9wrXHl6rXr6zeim7du39nY3Lp7pPJSYtLHOcvlSYIUYVSQvqaakZNCEsQTRo6TswPvPz4nUtFc9PSsIEOOJoJmFCPtTOOtsNOC5wRMwR6AjGS67U+vHgGoSj42eC+2owMgHLLe0RvjEdSFJ8H3dMKRO5t2sm23R2Y3vqBASSenujYt6FRuH/8n07LIYtxCNdkY7wqn6ui8vMSNWnCCuJeoqPnYmxx5pw6M4GvXixRdXA8aWNALjZ1lyZ1/FARtfQFzYKPxZrPb6VYLrIK4Bs2gXoeuw19hmuOSE6ExQ0oN4m6hhwZJTTEjNoKlIgXCZ2hCBg4KxIkammqyFrScJQVZLt0nNKisixEGcaVmPHFMjvSpuuzzxr/5BqXOng0NFUWpicDzRFnJgM6BfyYgpZJgzWYOICypqxXgUyQR1u4xRVCQDzh3fRepcV2zg3hoYJKz1NeSMwN9wiQzzdhau8zWhTXuT6ZaZaa34uZeYDoXVC5xoZWeMQKqCMkrxQi+IK6VkrxxWd4WRCKdy4cGIulmN7Wm3v9Ho2JOc3vkZxpfnuAqOHrciR1+96S5/7ye7npwP3gQtIM4eBrsBy+Dw6Af4PBT+D38Ef5s0MbHxufGlzl1Laxj7gVLq/HtN3mlSdQ=</latexit><latexit sha1_base64="0FasGiO3xkR6q79/5ovqleyDVYs=">AAAD53icfVLNbhMxEN5t+CnLT1s4crGIIqXQRlmEBJdKFeUAEogikrZSnERerze1antXtrcksvwMnEAc4Tm4wkPwNtibLUoawNKuP8988814xknBqNLd7q9wrXHl6rXr6zeim7du39nY3Lp7pPJSYtLHOcvlSYIUYVSQvqaakZNCEsQTRo6TswPvPz4nUtFc9PSsIEOOJoJmFCPtTOOtsNOC5wRMwR6AjGS67U+vHgGoSj42eC+2owMgHLLe0RvjEdSFJ8H3dMKRO5t2sm23R2Y3vqBASSenujYt6FRuH/8n07LIYtxCNdkY7wqn6ui8vMSNWnCCuJeoqPnYmxx5pw6M4GvXixRdXA8aWNALjZ1lyZ1/FARtfQFzYKPxZrPb6VYLrIK4Bs2gXoeuw19hmuOSE6ExQ0oN4m6hhwZJTTEjNoKlIgXCZ2hCBg4KxIkammqyFrScJQVZLt0nNKisixEGcaVmPHFMjvSpuuzzxr/5BqXOng0NFUWpicDzRFnJgM6BfyYgpZJgzWYOICypqxXgUyQR1u4xRVCQDzh3fRepcV2zg3hoYJKz1NeSMwN9wiQzzdhau8zWhTXuT6ZaZaa34uZeYDoXVC5xoZWeMQKqCMkrxQi+IK6VkrxxWd4WRCKdy4cGIulmN7Wm3v9Ho2JOc3vkZxpfnuAqOHrciR1+96S5/7ye7npwP3gQtIM4eBrsBy+Dw6Af4PBT+D38Ef5s0MbHxufGlzl1Laxj7gVLq/HtN3mlSdQ=</latexit><latexit sha1_base64="0FasGiO3xkR6q79/5ovqleyDVYs=">AAAD53icfVLNbhMxEN5t+CnLT1s4crGIIqXQRlmEBJdKFeUAEogikrZSnERerze1antXtrcksvwMnEAc4Tm4wkPwNtibLUoawNKuP8988814xknBqNLd7q9wrXHl6rXr6zeim7du39nY3Lp7pPJSYtLHOcvlSYIUYVSQvqaakZNCEsQTRo6TswPvPz4nUtFc9PSsIEOOJoJmFCPtTOOtsNOC5wRMwR6AjGS67U+vHgGoSj42eC+2owMgHLLe0RvjEdSFJ8H3dMKRO5t2sm23R2Y3vqBASSenujYt6FRuH/8n07LIYtxCNdkY7wqn6ui8vMSNWnCCuJeoqPnYmxx5pw6M4GvXixRdXA8aWNALjZ1lyZ1/FARtfQFzYKPxZrPb6VYLrIK4Bs2gXoeuw19hmuOSE6ExQ0oN4m6hhwZJTTEjNoKlIgXCZ2hCBg4KxIkammqyFrScJQVZLt0nNKisixEGcaVmPHFMjvSpuuzzxr/5BqXOng0NFUWpicDzRFnJgM6BfyYgpZJgzWYOICypqxXgUyQR1u4xRVCQDzh3fRepcV2zg3hoYJKz1NeSMwN9wiQzzdhau8zWhTXuT6ZaZaa34uZeYDoXVC5xoZWeMQKqCMkrxQi+IK6VkrxxWd4WRCKdy4cGIulmN7Wm3v9Ho2JOc3vkZxpfnuAqOHrciR1+96S5/7ye7npwP3gQtIM4eBrsBy+Dw6Af4PBT+D38Ef5s0MbHxufGlzl1Laxj7gVLq/HtN3mlSdQ=</latexit><latexit sha1_base64="0FasGiO3xkR6q79/5ovqleyDVYs=">AAAD53icfVLNbhMxEN5t+CnLT1s4crGIIqXQRlmEBJdKFeUAEogikrZSnERerze1antXtrcksvwMnEAc4Tm4wkPwNtibLUoawNKuP8988814xknBqNLd7q9wrXHl6rXr6zeim7du39nY3Lp7pPJSYtLHOcvlSYIUYVSQvqaakZNCEsQTRo6TswPvPz4nUtFc9PSsIEOOJoJmFCPtTOOtsNOC5wRMwR6AjGS67U+vHgGoSj42eC+2owMgHLLe0RvjEdSFJ8H3dMKRO5t2sm23R2Y3vqBASSenujYt6FRuH/8n07LIYtxCNdkY7wqn6ui8vMSNWnCCuJeoqPnYmxx5pw6M4GvXixRdXA8aWNALjZ1lyZ1/FARtfQFzYKPxZrPb6VYLrIK4Bs2gXoeuw19hmuOSE6ExQ0oN4m6hhwZJTTEjNoKlIgXCZ2hCBg4KxIkammqyFrScJQVZLt0nNKisixEGcaVmPHFMjvSpuuzzxr/5BqXOng0NFUWpicDzRFnJgM6BfyYgpZJgzWYOICypqxXgUyQR1u4xRVCQDzh3fRepcV2zg3hoYJKz1NeSMwN9wiQzzdhau8zWhTXuT6ZaZaa34uZeYDoXVC5xoZWeMQKqCMkrxQi+IK6VkrxxWd4WRCKdy4cGIulmN7Wm3v9Ho2JOc3vkZxpfnuAqOHrciR1+96S5/7ye7npwP3gQtIM4eBrsBy+Dw6Af4PBT+D38Ef5s0MbHxufGlzl1Laxj7gVLq/HtN3mlSdQ=</latexit>

Tc<latexit sha1_base64="3FD/GgMnoEf/48ct3HtkO4LKDkY=">AAAD8nicfVJLbxMxEN5teJTl1cKRi0UUKYW0yiIk4FCpIhxAAlFE0laKk8jreBOrtndle0sia/8IpyKu/A6ucOXfMJtsUB6ApV1/nvnmm/GMo1RwY5vNX/5W5crVa9e3bwQ3b92+c3dn996JSTJNWYcmItFnETFMcMU6llvBzlLNiIwEO43OW4X/9IJpwxPVttOU9SQZKR5zSiyYBrv+ixq+YGiCDhEWLLb14vTmMcImkwNHD8O830IKUF442gPaxzYtSPgjH0kCZ1eP9vK9vtsPFxSs+WhsS9OSzpLCn1yrMsuRS/XEA7qvQBfoMlvjBjU8IrKQmFGTQWECcqMMBP9baMeQLG6IHU75QqSxqtn4R0U4L+/gWnlQXiMY7FSbB83ZQpsgLEHVK9cxNPsSDxOaSaYsFcSYbthMbc8RbTkVDJQzw1JCz8mIdQEqIpnpudmQc1QDyxDFiYZPWTSzLkc4Io2ZygiYktixWfcVxr/5upmNn/ccV2lmmaLzRHEmkE1Q8WLQkGtGrZgCIFRzqBXRMdGEWnhXAVbsE01gAGrooDN5N+w5HCViWNSSCIeLhFHsqmGe56tsm+YO/mxiTezaG25ZCEzmggYSp9bYqWBoFqHlTDHArxi0UrN3kOV9yjSxiX7kMNEww0nuyv1/NK7mNNiDYqbh+gQ3wcmTgxDwh6fVo5fldLe9B95Dr+6F3jPvyHvtHXsdj/qX/nf/h/+zYiufK18qX+fULb+Mue+trMq335l0TaA=</latexit><latexit sha1_base64="3FD/GgMnoEf/48ct3HtkO4LKDkY=">AAAD8nicfVJLbxMxEN5teJTl1cKRi0UUKYW0yiIk4FCpIhxAAlFE0laKk8jreBOrtndle0sia/8IpyKu/A6ucOXfMJtsUB6ApV1/nvnmm/GMo1RwY5vNX/5W5crVa9e3bwQ3b92+c3dn996JSTJNWYcmItFnETFMcMU6llvBzlLNiIwEO43OW4X/9IJpwxPVttOU9SQZKR5zSiyYBrv+ixq+YGiCDhEWLLb14vTmMcImkwNHD8O830IKUF442gPaxzYtSPgjH0kCZ1eP9vK9vtsPFxSs+WhsS9OSzpLCn1yrMsuRS/XEA7qvQBfoMlvjBjU8IrKQmFGTQWECcqMMBP9baMeQLG6IHU75QqSxqtn4R0U4L+/gWnlQXiMY7FSbB83ZQpsgLEHVK9cxNPsSDxOaSaYsFcSYbthMbc8RbTkVDJQzw1JCz8mIdQEqIpnpudmQc1QDyxDFiYZPWTSzLkc4Io2ZygiYktixWfcVxr/5upmNn/ccV2lmmaLzRHEmkE1Q8WLQkGtGrZgCIFRzqBXRMdGEWnhXAVbsE01gAGrooDN5N+w5HCViWNSSCIeLhFHsqmGe56tsm+YO/mxiTezaG25ZCEzmggYSp9bYqWBoFqHlTDHArxi0UrN3kOV9yjSxiX7kMNEww0nuyv1/NK7mNNiDYqbh+gQ3wcmTgxDwh6fVo5fldLe9B95Dr+6F3jPvyHvtHXsdj/qX/nf/h/+zYiufK18qX+fULb+Mue+trMq335l0TaA=</latexit><latexit sha1_base64="3FD/GgMnoEf/48ct3HtkO4LKDkY=">AAAD8nicfVJLbxMxEN5teJTl1cKRi0UUKYW0yiIk4FCpIhxAAlFE0laKk8jreBOrtndle0sia/8IpyKu/A6ucOXfMJtsUB6ApV1/nvnmm/GMo1RwY5vNX/5W5crVa9e3bwQ3b92+c3dn996JSTJNWYcmItFnETFMcMU6llvBzlLNiIwEO43OW4X/9IJpwxPVttOU9SQZKR5zSiyYBrv+ixq+YGiCDhEWLLb14vTmMcImkwNHD8O830IKUF442gPaxzYtSPgjH0kCZ1eP9vK9vtsPFxSs+WhsS9OSzpLCn1yrMsuRS/XEA7qvQBfoMlvjBjU8IrKQmFGTQWECcqMMBP9baMeQLG6IHU75QqSxqtn4R0U4L+/gWnlQXiMY7FSbB83ZQpsgLEHVK9cxNPsSDxOaSaYsFcSYbthMbc8RbTkVDJQzw1JCz8mIdQEqIpnpudmQc1QDyxDFiYZPWTSzLkc4Io2ZygiYktixWfcVxr/5upmNn/ccV2lmmaLzRHEmkE1Q8WLQkGtGrZgCIFRzqBXRMdGEWnhXAVbsE01gAGrooDN5N+w5HCViWNSSCIeLhFHsqmGe56tsm+YO/mxiTezaG25ZCEzmggYSp9bYqWBoFqHlTDHArxi0UrN3kOV9yjSxiX7kMNEww0nuyv1/NK7mNNiDYqbh+gQ3wcmTgxDwh6fVo5fldLe9B95Dr+6F3jPvyHvtHXsdj/qX/nf/h/+zYiufK18qX+fULb+Mue+trMq335l0TaA=</latexit><latexit sha1_base64="3FD/GgMnoEf/48ct3HtkO4LKDkY=">AAAD8nicfVJLbxMxEN5teJTl1cKRi0UUKYW0yiIk4FCpIhxAAlFE0laKk8jreBOrtndle0sia/8IpyKu/A6ucOXfMJtsUB6ApV1/nvnmm/GMo1RwY5vNX/5W5crVa9e3bwQ3b92+c3dn996JSTJNWYcmItFnETFMcMU6llvBzlLNiIwEO43OW4X/9IJpwxPVttOU9SQZKR5zSiyYBrv+ixq+YGiCDhEWLLb14vTmMcImkwNHD8O830IKUF442gPaxzYtSPgjH0kCZ1eP9vK9vtsPFxSs+WhsS9OSzpLCn1yrMsuRS/XEA7qvQBfoMlvjBjU8IrKQmFGTQWECcqMMBP9baMeQLG6IHU75QqSxqtn4R0U4L+/gWnlQXiMY7FSbB83ZQpsgLEHVK9cxNPsSDxOaSaYsFcSYbthMbc8RbTkVDJQzw1JCz8mIdQEqIpnpudmQc1QDyxDFiYZPWTSzLkc4Io2ZygiYktixWfcVxr/5upmNn/ccV2lmmaLzRHEmkE1Q8WLQkGtGrZgCIFRzqBXRMdGEWnhXAVbsE01gAGrooDN5N+w5HCViWNSSCIeLhFHsqmGe56tsm+YO/mxiTezaG25ZCEzmggYSp9bYqWBoFqHlTDHArxi0UrN3kOV9yjSxiX7kMNEww0nuyv1/NK7mNNiDYqbh+gQ3wcmTgxDwh6fVo5fldLe9B95Dr+6F3jPvyHvtHXsdj/qX/nf/h/+zYiufK18qX+fULb+Mue+trMq335l0TaA=</latexit>


I-Vectors: Model Training

Model training involves estimating the total variability matrix T usingthe EM algorithm.

Using Eq. 13 and Eq. 19, the M-step for estimating T is

Tc =[∑

if̃ic〈wi|Oi〉T

] [∑iNic〈wiw

Ti |Oi〉

]−1, c = 1, . . . , C.

(27)


Applications of I-Vectors

The most popular application of i-vectors is speaker verification


Applications of I-Vectors

Adapting DNNs for patient-dependent ECG classification

Source: S. Xu, M.W. Mak and C.C. Cheung, ”I-Vector Based Patient Adaptation

of Deep Neural Networks for Automatic Heartbeat Classification”, IEEE Journal of

Biomedical and Health Informatics, May 2019


Software Tools

Factor Analysis

Python scikit-learn: sklearn.decomposition.FactorAnalysisMatlab factoran

I-Vectors

SIDEKIT:https://projets-lium.univ-lemans.fr/sidekit/overview/index.htmlmPLDA: http://bioinfo.eie.polyu.edu.hk/mPLDAKaldi: https://kaldi-asr.orgMSR Identity Toolbox


factor analysis and i-vectorsmwmak/papers/fa-beamer.pdf · wiki: \factor analysis is a statistical...

Documents